 All right, so without further ado, I'm just gonna follow the order that was listed on the webpage. So I'm just going to introduce Ezra Wischograd, am I saying your name correct? Wow, on the first try, very impressive. I'm a linguist, hey, not bad. Really good. Ezra, I'm just gonna ask you to give a quick brief summary of who you are, what your current job is and how you got there. Go ahead. Sure, so I would classify myself as an early career, a early career career linguist. I had a background academically in linguistics, went to Columbia undergrad, majored in it, went to Georgetown and got a master's. I entered in the PhD program and I left shortly after receiving a master's degree. My research interests were in phonetics and phonology. I did field work in West Africa and the Ivory Coast. I did field work for the Kurdish languages, part of the Endangered Language Alliance in New York City when I was at Columbia. I did a little bit of dialectology stuff as well. I did some Boston accent research back home where I'm from. And since grad school, I worked at Expert System USA, which is definitely a smaller company that does defense subcontracting, which is Washington, D.C.E.'s for helping the government out with linguistic type problems, document categorization, information extraction, things like that. I'm now working at a much larger company, of course, Amazon working with Alexa. And I would say caution everyone that hopefully you don't have an Alexa around you turned on, because I might be activating a bunch of people's Alexa's right now by saying that. But basically in the past few years, I've gone from being a grad student very academically minded to a smaller company to a larger company. So I think I've seen definitely a bunch of different scenes in the past two or three years. Great. So give us a sense of when you started looking towards industry as a career path. Sure. In grad school? So in grad school, and actually it comes back to actually one particular experience I had in grad school where at Georgetown, the phonetics, phonology people had a reading group every Friday. And we had a particularly, for lack of a better term, gifted guest speaker that was coming and spoke to us about to geek out a little bit about epithetic vowels in Georgian, which phonetics, phonology people love. And one of the most brilliant talks I've ever heard in my life, linguistics or otherwise. And afterwards, we came to talk about her struggles in finding a job in academia. And it was incredibly disheartening because this was one of the smartest people I've ever met. She had applied all over the world without exaggerating to all these different departments in the English and French speaking world to try to find an academic position. And she was telling us, you know, there's maybe three, four, five jobs that might come up in a particular season that attend your track to next phonology. And it was so disheartening for me to hear that you have to travel so far, work so hard, you know, with very questionable returns that, you know, I don't know if I was willing to put myself through that, butting family through that. So that was kind of the day where I figured it just might not be worth it, which might not be worth it for me, which could be worth it for others, but it just, I didn't really see it fitting into my life. Let me ask you this, Ezra, give us an example of a task you do at work on a daily or weekly basis. Just give us a sampling. Sure, well, at least at Amazon, for instance, one of the things that I might do is, you know, try to look at the current, the utterances, current sentences that Alexa might be able to understand at this point and try to find what kind of utterances are we not giving our customers? What utterances that the data might suggest might be necessary, do we not have? And I'm going through a lot of linguistic data to do that. And, you know, I'm in charge basically from start to finish on an entire design of I have an idea of something that Alexa should be able to understand. I'm gonna execute it from testing it with the current machinery that Alexa has all the way to the release into Alexa's software. So that's, you know, something I'm doing every day. I'm going through a bunch of linguistic data and, you know, seeing what utterances could make us better. All right, so I'm gonna get super detailed here. What is the format of that data? Is it a CSV file? Is it a special tooling? So this is where I begin to dance with my NDA. Oh, fair enough, fair enough. Sure, so for those who might not know, there's a non-disclosure agreement at many tech firms, both large and small. And I think the one thing they might tell you during training is to be careful there, but I would say that the format that I'm looking at it in is definitely something approachable if you've had linguistic training at a bachelor's degree level, master's degree level and onwards, that's for sure. Excellent, okay. Do you miss academics? It's a very fair question. And to me, it's a very easy answer. I don't. And that's not to cast aspersions onto anyone who might be listening who does miss academia, really enjoys it. I think that's wonderful that you do. And it's obviously a really, really noble thing to be in academia. I really believe that in my heart. I would say that the thing that I really don't miss is the incentive structures in academia. There's no limiting factor. You should always be publishing more. A lot of your major career approvals have to be of your peers and not the customer. So getting tenure has a lot to do with impressing the academy, i.e. other academics, versus trying to make the customer experience best. And every evaluation scheme is fraught, but I'd much rather work on impressing customers than other academics, just a personal predilection. Great. Can you think of a skill you've learned since you've entered industry? Oh boy. Yes, I- I don't think anyone. Yeah, I think for, even though I'm certainly not a project manager, I would say a lot of those type of skills of scheduling meetings, making sure that everyone's on task, timekeeping, things like that. Anything that involves keeping people up to date with information on an evolving project, that's not something that I think grad school really prepares you for, particularly in linguistics where research teams tend to be small. I can, if I would say that my, my first boss out of grad school could definitely tell you that, this whole timekeeping thing took me several months to figure out, and it's a little silly thing like that, but it's really something that is kind of alien when you're a grad student, because a grad student, your time isn't viewed the same way. There's not really working hours, you're just working. That's a great point. Side note, several years ago, we hired a guy straight out of the army. He went out the army as a high school student. He was about 25, 26. He left, he'd never had a real job, a real job, he'd only been in the army. We hired him into IBM, and he shows up the first day and kind of looks around and goes, when do I start? He was so used to that rigid army structure of somebody telling him exactly what to do every second, every minute. But in the industry, you show up at 9 a.m., show up at 9.05 a.m., no one's keeping track of exactly where you are at any moment, but you have a job to do, you have tasks to complete. How did you find the Amazon job? Sure, basically, I was given some really, really good advice very early on in my career, i.e. a year and a half ago, which is basically, even if you have a wonderful job and you're working in that, you're in the middle of that, you don't have to necessarily have immediate plans to be leaving, still look for a job. Never stop looking. Which was something very odd to me because I got an amazing job out of grad school and I was like, why should I keep doing this? But really, those opportunities will show up when you're not necessarily ready for them and it doesn't necessarily make sense, but you should still know about those opportunities. So, I was looking at different job boards constantly throughout my first job and as it happens, on Amazon.jobs, which is the Amazon job engine, the most up to date place for Amazon jobs, something came up in the city that I knew I'd be moving to relatively soon and I jumped on it. It was just a matter of being very pesky at looking at all the resources that you have in front of you. That brings up a good point is relocation for your first job, you were already in DC and you stayed in DC, correct? Correct. So, but it sounds like you will be relocating soon. That's right. I'm actually gonna be going back to where I grew up in Boston. I'm actually in DC right now speaking to you from the bottom, but Amazon is letting me work from the DC area until early next year. Great. But of course it's the COVID thing, but it's the relocation thing is really, really tough and I don't think that's necessarily something that's unique to linguistics. And I would say that one of the challenges that I certainly had coming out of grad school is that while DC is a pretty good market for linguistics jobs, it's certainly not the best one, particularly for linguistics and tech. It's no Seattle, it's no Boston, it's not even a New York. And that can be really tough and that was honestly a challenge that I had when looking through all these different, LinkedIn listings and they're just not your city and that can be really tough. Right, right. So since you've recently been in the job search, you have a good insight into this, better than say me or some of the other folks who've been at the same job for a while. You know, how, I don't wanna say easy, but like how did you go searching for linguist jobs? Now strictly keyword searching or what were your, what were your tips? Oh, it's terrible. It's really terrible because we're a very misunderstood people, us linguists. To give you an example, when I searched linguist in LinkedIn, as grad school is ending, you get translator jobs. Nothing wrong with being a translator. That's not what we think linguists are. So what it really was, keyword searches kind of drew me, got me a little bit crazy, but ultimately what ended up being great was reaching out to other linguists. And I know it's something that I think people must have heard in the past few weeks, but it sounds tired, but it really is true. And I was very lucky to be in a department that had Alex Johnston, who was a really incredible, incredible, incredible resource for linguistics. In academia, I was really lucky to have a hiring manager slash first boss, Emily Pace as well, was a linguist herself and really understood linguists. I think it's really about being in depth in the linguist community. And I think that actually helps most directly. Great, all right. So I wanna make sure we have plenty of time for everybody. So what I'm gonna do is I ask you one last question and I stole this question from one of my favorite podcasts, so it's not mine. This is, I feel petty. I want you to think about something you feel petty about. And for example, I work at IBM and I am completely petty about the career framework. They're kind of, we have to do so many kind of onerous, silly things to progress in our career. We have to fill out all those paperwork. We have to do these fake classes. They're horrible. I'm very petty about the career framework at IBM. Can you think of something you feel petty about in your career? Oh, for sure. And I'm glad you tapped into my petty side. This is great. I am definitely most petty about job skill lists on job sites because I took it as a bare minimum. Here's all the lists you have. Really, it's like a Christmas wish list, more likely. I remember seeing, some of them aren't even nonsensical. I remember seeing that some job wanted something like 10 or 12 years of Python 3 experience. I don't even know what Python 3 came out then. Like a lot of things that are just really like nonsensical. They're not gonna, I don't know what candidate would fill all this. So really just like, I think taking the list of qualifications with a little bit of a grain of salt, I think was something that I really wish I knew at the beginning. I was very petty about that. Great, great. Actually, I think maybe I will ask you one last question because you brought up something that I'm going to ask everybody. And that is, what is the role of coding in your day to day job? You mentioned Python. Do you know Python? Do you work with that? I, it's a great question. I know Python the way a linguist would know Python. I don't know Python the way a NLP person would know Python. And there's definitely, I think definitely a distinction there I've had about, I would say three courses in Python academically and have done it in practice in my last job and a little bit in this job. I would say that even in roles where you don't do coding directly, the thought processes that coding gives you is tremendously invaluable in whatever NLP related job you're doing, even if you're not directly coding. So like for those who are listening who are early enough in their career, well, actually it's never too late to teach yourself anything. But I would say for those particularly who are still in grad school, even taking one or two courses, even if it's not for the content, it's for the style of thinking. So I would say. Perfect. Okay, and we will still be able to come back to Ezra later on, but I am going to move on now. I see there are some questions coming up. I'm using some general questions in the chat. Let's circle back to those at the end but there's really important questions there. Now I want to move on to Esme. Esme Manam dies. Did I pronounce that correctly? No. Nope. Manam dies. Manam dies. Beautiful. Okay, but that's okay. People call me so often times merchant dies, so. Okay, so I'm Esme Manam dies. I was born and raised in Brussels, Belgium. And I came to the States to study linguistics and graduated from the University of Arizona. And my thesis advisor was Dr. Susan Steele. I believe she's in the audience today. And when I finished, so my interest in graduate school was formal syntax and Montague semantics and any type of formal grammars. And early on I knew that I also wanted to go into coding so I took already courses in computer sciences. So I feel a little of a cheater here. But I did major in linguistics with Manam in computer science. When I finished my PhD, I actually got three jobs in academia, two in the state ones in Flagstaff, at university there when it's up to say and then one at home back in Belgium. And I'm a French speaker from Belgium. So jobs at universities are very few and rare. And after, by Christmas, I knew that I didn't know, I didn't want to teach about language and linguistics. I actually wanted to do a code development and create a product that was based on natural language. And so I was in touch with Susan Steele when I was back there and she kept sending me job notification that she came across. So I ended up finding a job in the area that I wanted was to develop a work on natural language technologies. So my main tenure professionally is being at IBM at E.J. Watson's at the research center. My last manager was David Ferrucci when he was still big on the so-called Jeopardy. But when he left, we still went working on the, we work on the health industry. But another manager of mine was Dr. Michael McCord who was a mathematician actually, but he developed all this language technology and parsing technology that was used at, which is and was at the core of the Watson project of Jeopardy. And while I was there, living the life of, since you are IBM, of the IBMmer, people do a whole career there, somebody contacted me through LinkedIn, my current manager, I think to it. And I usually get a lot of inquiries through LinkedIn, monthly people from various office wanted to know if they want, I'm interested in a new job. But this one, rather than being long and tedious, was funny and it picked my curiosity. And I agreed to a job, to a first contact by phone. And I thought the problem that they wanted to solve was really fascinating. And I went for the job interview, the former manager grew with homework because in this space of natural language technology, typically you receive a natural language problem for which you have to code a solution that you deliver to the team that's gonna do the hiring and the interview. And then you go to the interview and you discuss why you implemented it this way, what are the possible outcomes and the pitfalls and all of that. No, I have to confess, it took me three months to accept the offer. Yeah, so I've been with Intuit for four years and a half and I work on, my data is always unstructured and then updated to both texts and between scenes. Fantastic, so I am very familiar with the slot grammar as a matter of fact. Yeah, sure. The grammar that Michael McCord. Yes. Not for the rest of the crowd, but we actually are trying to get that open sourced. There's still a group of IBMers who are trying to get that released. It would make so much sense if it were. I would then work on Intuit to use it. Oh, fantastic. Can you spacey another type of prices up there? Yeah. So that leads to the question related to the Python question, which is what kind of tools do you normally use, NLP tools in this new Intuit work? Okay, all day long, probably half of my time is actual hands-on development coding directly in Python. Then this company has a very special, the development mentality because there is one that's very different from the one at IBM. At IBM, my whole career is that I was developing tools with IBM technology for IBM usage. Here, they like to accelerate things and so we use a lot of open source third parties of code libraries. And so I use what's out there. I poke around with behind the scene and then I tailor and see what it would take to adapt it to my end goals. So if it's open source and you can modify directly the code, then I'll just do it and that's it, yeah. So you've done a lot of research stuff in your career. So this may, I think the answer may be obvious, but I'll ask it anyways, how important is the PhD to what you do? Or could you do this job with, say, just a master's? Well, I don't think so. One thing, the two things, I think that by the time you go to the PhD, you're ready. So you have an ID, that's the purpose of your PhD. You want to prove it, right? And so you're not only right about it, but you investigate it and you develop the proof of concept and you test it. And I think these are the type of analytical skills that are often when you work on a project that is a little more high level where you actually have to take the initiative to solve a problem rather than be just an executioner. And I think the PhD develops all these analytical skills in the space of natural language. So you learn to analyze the data, to interpret it, to create various tools, to have even more insight on massive data. You learn to validate the evidence and you learn all of that while you do your PhD. I was very fortunate to take a lot of individual studies with faculty members and especially my test advisor. And she was a tough cookie. She was always asking me, why? Why? You know, like a child, why, why? And so that forced me to always think about why am I saying this? What is to prove? What is to come to prove? And I think this is very useful. I don't think that at the master level, the programs demands that you push the envelope this way. You're still learning a lot of basic skills. Really? Yeah. So here's potentially a touchy question for political reasons. Which or have you worked on any non-English NLP problems? Yes, but that was at IBM. So initially when I got the job at IBM, it was to take the natural language technology that Michael McCord had developed for English. And I was asked to take that technology and do it for the Romance languages. And the Romance languages at the time for IBM were defined as French, Spanish, Brazilian Portuguese and Italian. But so I devised a way of doing more of a universal grammar and use switches or flags. So write a common grammar and the code behind that covered those four Romance languages. And then since you knew Adele said the language you were crossing, then if the flag would tell you what, given this context, this pattern, then you need to apply what applies to the source language. Yeah. And so peripheral to that, and this is kind of where I'm going with this question is, how did you find non-English NLP resources? And what I mean is, were they much worse than the NLP resources for English? Were they harder to use? Were they harder, et cetera? Yeah, as I said, actually, because it was within the IBM frame, everything is IBM. So at the time, I didn't look outside the IBM entity. And so all the, whatever texts were available, they were already in-house, everything was in-house. So there were no looking outside IBM, which is something I wouldn't do now. I did because after four years, I've been to it before I implement anything or looking to implement it, something I go see what's out there. For context for the folks, Zoom, generally speaking, the NLP world revolves around languages from rich countries. So English has tons of NLP resources. Western European languages spoken in Western European countries have reasonably good NLP resources. You can find that region, the availability and workability of a lot of NLP simply drops off, just a little. Yes, good, no resource languages, yeah. So let's go way back to when you left academia, how did you find your first job? I'm not sure you mentioned that. No, I mentioned it through, I was always in touch with my thesis advisor and she was sending me whenever she came across job offers. I mean, just listening, she would just communicate them to me and that's how it happened. So for me, again, it is networking, but it's a type of passive networking. It's word of mouth if you want to in that. And I think it works. If people know you, you spread the word and you say, oh, I know such and such person. So I didn't have to look for one you know, anything in linking for that matter. They just contacted me. I was more fortunate. So how have you found LinkedIn in general? How long have you been on LinkedIn? Oh, for a long time, but I think I follow some people in the space of NLP. Typically the people are more controversial because you really, in the end, I do NLP, right? But I find that people who do the real partition of NLP often have been training computer science and there are some spaces like natural language understanding. But I think they lack the understanding of the granularity of semantics that I needed to put into natural language technology. And so there are some people who are arguing that there's some limits to how much we can do with NLP. And I like the very controversial and I typically follow them on LinkedIn because they post things where they are open about their beliefs that they are limits to data-driven natural language approaches. Great. I think we have, there's a shift actually going on right now. I think we're gonna go away from this blind faith in massive data to do natural language analysis. Interesting, okay, great. You're giving me the thumbs up. Thank you, Ezra. Yeah, traditionally data scientists have loved big data, right? That's, they always say, no more data's the problem than that solves it. But of course we know that that's actually not true. Yeah. Can you think of some advice you've gotten in your career, particularly mid-career, once you were already in it that you thought was really, really useful? Okay. I'm gonna say this as a woman here, okay? And I don't know if it's appropriate, but I have to say it. My whole career I have been the uptoken in the teams. Often times the only woman there. And what my career advice is that, let's say for the other female linguists, right? Stand your ground, don't be intimidated. If you have something to offer, just discuss it in a cool, scientific way and never think that you can't do the job, that's all. Fantastic. And let me ask you, have you been on hiring committees in your time in industry? Well, I've been internally usually, not outside, but within the companies I've held, I do what is called a, for the assess job candidates, assess within the company, people who want to go up for promotion. And typically they never in the field of natural language, it's pure engineering work, but I have to develop that side to be successful. And if you are able to do a PhD in linguistics and develop all these analytical skills, you can learn anything, thank you. If you have the passion and the willingness to learn it, really go ahead and do it. And so what I'm getting at is, so when you look at a resume, how do you see that passion? Like what's the kind of things that jump out at you in a good resume? Well, this is the question about the hardness of skills, right? I mean, it's a little abstract. I would see how the linguist presented himself, herself in terms of their hard skills, right? I think it's important to emphasize the analytical skills, because I think as I mentioned that, it still irritates me when people ask me how many languages do you speak? I just can't stand it because people still assume that linguistics is all about speaking languages. That has nothing to do with what I know and the lots in the audience only speak English, right? And they're still the professional linguists. So the important thing is precisely make clear all these analytical skills that are developed doing a linguistic as a scientific systematic approach of language, right? Data and languages. And make that prominent in the resumes because these analytical skills are skills that you can use in software engineering, in design, in taxonomy development, in content architecture. And I think that a linguist that are well-prepared can succeed in something that may be outside of linguistics, but it's possible. So that's what I would say. Oh, yeah. And then the soft skills have to be assessed. I would also see what type of soft skills could indicate the passion and the disponibility to learn more, right? Yeah. And in general over your career, do you attend things like meetups or conferences? Do you still stay connected to things like that? Yeah, I try to give one paper or two papers a year. But they typically, nowadays they were in NMT and engineering, but they always have to do with natural language. So can you name a couple of conferences that you would generally like to talk about? So I did the one, the latest one was at the Florida Artificial Intelligence Society. The paper was published on one of the things we did into it. There was another one hosted also by the ACL in Europe, ESL, it was about financial narrative. So I tailored the paper to the type of work I do and I leverage what I've learned by doing and then I abstract away and share. But in practice, typically the companies are more interested in patterns. They would rather have pattern submissions than paper published. Yeah. Very true. Yeah, so I do at least one pattern a year. Let's also talk for a moment. And then the important thing, sorry. An important thing is that don't be a loner if my experience to be best, I always work in team and I do everything, the papers is collaborative, the patterns are collaborative. And never present yourself as the one that always knows best and goes your heart, your heart, ride your horse along. Yeah, team is better. I guess, yeah, totally agree. So I wanna ask real quick about work-life balance. Do you feel you've been pressured to work, and I mean just in general, to work more than 40, 50 hours a week? I haven't been pressured to do it. I'm a workaholic. I was already a workaholic when I was in grad school, which means that the one there was socially done in the department, I was one of the ones who never showed up. But this is a preference, it's my personality, but I haven't been pressured. I've just learned that I have a family and you know, but I've learned to just work of time when things are quiet or whatever around the house, but depression not by the companies, you know. Good, that's very good, okay. So along those lines, just quickly, is there something non-linguistic and non-professional that you do in your personal life? Are you a knitter? Are you a horse rider? I'm a avid walker. Beautiful, fantastic. I walk late at night when it's quiet, with a dog by the way, a big dog, a 60 pounder, so I can feel protected. Okay, I am gonna save some time at the end for the questions that are in the chat, but let's go ahead and move on. We're gonna move on to Rich Campbell. Rich, you've had quite a career, I was looking at your LinkedIn page, so why don't you give us the quick two-minute summary of where you started and where you are now? All right, I got a PhD in linguistics from UCLA and specializing in syntax, and I originally went into an academic career and I spent 10 years in academic positions, two one-year positions, and then I went and got a position at a small state university in Michigan and got tenure there and was there for eight years total. About that time, I had some dissatisfaction with various aspects of my career and felt like I wanted to move back to the West Coast, which is where I'm from, and an opportunity fell in my lap. I knew someone who worked at Microsoft Research, a guy I went to graduate school with, and he arranged for me to come out for an interview, and I really didn't think much of it. I thought this is quite a long shot, but they ended up offering me the job and found it kind of hard to refuse. So my wife and family and I moved out here to Seattle at about that time. I worked at Microsoft Research for five years, then I left and went to work for another company called Cataphora for nine years, a very different line of work. And then after that, Cataphora started to wind down. I found another job working for a company called Interactive Intelligence, which the title of this just encouraged all of my family members who didn't understand what I was doing to think that I was actually a spy, but of course it has nothing to do with that. Interactive Intelligence was swallowed up by Genesis, so I've been at that job under one name or another for about seven years, actually almost eight years now. So it has changed a lot. I guess that my experience in companies, working in industry is small company, big company, medium-sized company. So I guess I've done it all in that sense. And I work in a small group of linguists where we've sort of carved out a niche for what we intend to do. Fantastic. So it's great that you've had a variety of experiences. How has being a linguist been perceived or received by the companies you've worked at? Have you had to sort of fight to be recognized? Actually, no. I was lucky that in all of these places I was hired specifically as a linguist and I was expected to do linguistic work, at least at the beginning. So Microsoft Research, there was an NLP team in which we were developing various applications in a variety of languages. And I think at the time it was the grammar checker was the main product when I joined that was being worked on. When I went to Cataphora they were looking specifically for linguists. There was a team of linguists under Dick Early was the chief linguist. And it's interesting in that case we were working directly with customers. So it was a very different kind of work. And all of us having PhDs in linguistics or advanced degrees in linguistics was kind of a selling point for the customers. So the company actually charged for our time. We had an hourly rate and we had to bill by the hour. And they could charge higher for a PhD in linguistics. So it was all part of a sales job. I mean, there was linguistics to do also but it was also part of a sales job. And when I started out at Interactive Intelligence we were doing text-to-speech and automatic speech recognition systems for different languages. They were specifically looking for linguists to compile the resources, work out phonology problems, that sort of thing. And that job has evolved quite a bit over the years especially since the merger. But in every case, I was hired specifically as a linguist. So I didn't really have that fight. At least I didn't have to do it individually. And did you ever write code or was that not really ever part of your career? So I never wrote code that's used in a product but I use Python quite a bit to write, to create tools that I or my teammates will use for the various kind of analyses that we want to do. It doesn't go into a product, it's just for use among myself and my teammates. Example of something's created? Most of them are the sort of thing where I create it and I'll use it for a few days until we go on to the next project. But for example, we're working on chat bots for example in voice bots and like everybody I guess and we have an engine that gives us results in a JSON format that we need to analyze those results. So oftentimes I'll write scripts to take in those results, perform some kind of analysis on them, sort them in some way so that we can look at them and do our sort of qualitative analysis on it more fully. That's a very common kind of script that I would write. Fantastic. And you would write that in Python? Python, yes. So I'm just gonna, I'm going to anticipate a question that you're gonna ask me about what is a great piece of advice? Years ago when I joined Cataphora, Dick Hurley said, learn Python and I did and it's been a great help. Do you work with other languages or people who write in other languages? Well, definitely I will work with a lot of engineers who also write in Python or they might write in other languages if they see fit. I don't generally have to deal much with those other languages in terms of looking at their code. Python is even among the machine learning engineers in our company, in our division. It's Python is still the tool that's most favored, I think. Yeah, I think that's generally true across NLP, my experience. Now, have you worked with any other non-English natural languages? I have, but again, mostly the rich ones. So we are, the things we work on are driven by what the company sees as its needs. And these tend to be English, Western European languages, maybe Japanese and Mandarin. It's really been limited to that, but I have worked a lot on those, both at my current job and the previous job and that we're still doing a lot of that work, trying to expand our abilities into other languages, that the company sees as in its sort of near-term future. So we're doing that a lot, actually. Fantastic, great. And do you, like in your current roles, since you've been, you're in a senior role, so do you get input into the kinds of products or services the company is seeking to produce or move into? Yes, but I get some input. I can't swear that it's all paid close attention to, but yes, we do get consulted on, somebody is developing a new idea for things we want bots to do in the next generation of bots. For example, we might get consulted on that. And so we have a project that might last several months where we go back and forth with them about how feasible we think this is, whether it's really well-conceived, that sort of thing. In terms of directly proposing what kind of products or we should do this or that, I'm not that senior. In general, as you progress through your career, do you feel like you have the opportunities that other people in the company that are maybe engineers and such have had in terms of career advancement? So has being a linguist hurt you, helped you or been irrelevant? I think on the whole, I think it's helped me in that I think that because myself and the team that I work with, our linguists, we sort of have this and we're a group and we refer to ourselves as the linguists and other groups that are within the AI division refer to us as the linguist team, right? So we've sort of carved out this identity as being the experts in these certain areas. And I think that has helped us, right? That we've been able to, like I said, carve out this area of expertise and people come to us with problems and propose new problems that they think we might deal with. And I think it's helped that we're kind of different, right? I think that if there's one thing that I've kind of noticed in industry for linguists is that linguists are kind of treated as special in both a good way and bad way, and that we can benefit from that. And one of the ways we're special is that we know about things that they wish they knew about. And the other way we're special is that we don't know about things that they wish we did know about. So we get special treatment that way too. Right, right, right, fantastic. So I asked this question of Esme, I'm gonna ask it of you. Do you sit on hiring committees? And if yes, what do you look for in new applicants? Not committees per se, but I do sometimes review resumes if it's something within our group or in a closely related group. And I think what I'm looking for, again, of course it's gonna vary a little bit depending on specifically what the company is looking for in this position, what's been approved. But I think what I'm looking for is someone who has experience doing linguistics and sort of repeating what Esme said here. It's somebody who has shows that they've engaged with linguistics a little bit or a medium amount as opposed to somebody who's just a... This is gonna sound like I'm putting them down and it's not what I mean, but somebody who's a computer scientist, an LP person who has worked on language related problems, that doesn't show me that they're necessarily engaged in linguistics. And so I think for our group, we sort of have this identity as bringing this different kind of qualitative research to the table and so I'm looking for that. Great, I've passed that. And it's good that you agree with Esme, it means there's a consensus. So I'm gonna fill this last one out at you. What do you feel petty about in industry? I feel petty about a lot of things. I think when you mentioned that before to Ezra, I think what crossed my mind, something that I feel petty about is the way that something that came up recently, I won't give any specifics, but the way that higher executives and the company come up with these ideas for various things that they wanna do without really thinking about how it's gonna work and what it's gonna take to do that. And this is just my personal attitude and this is a blanket statement that probably isn't true, but I sometimes feel like executives are looking for something to justify their job. I was gonna say, I hope it's not being recorded, but I'm sure it is. So that's my pettyness, that's my main pettyness lately. Fair enough, fair enough, good, good. Azna, I forgot to ask you that question. Do you have something in industry that you feel petty about that you'd like to share? If not, that's okay. Probably a lot, but having been born in Europe and raised in Europe, if I feel petty about something, I usually or immediately think, oh well, if I were in Europe, I would have a union rep and that's where I would go. So I usually don't express anything about that. All right. Okay, I'm gonna get to some of the questions in the chat now. Thank you panelists for your one-on-ones. These questions I'm gonna ask to the panelists as a whole. And also there are folks sitting in who have quite good experience in industry as well. You're welcome to pitch in as well. I'll basically open this up to all the experts on the call. One of the questions is about resources. So one of the things that I'll just speak to myself, I entered industry in the mid-2000s. And for me, I would have been a much better entry-level candidate if I had started now, if only because the resources for learning about NLP are so much better now than they were 10 or 15 years ago. Tutorials, cable competitions, just books, even just the packages themselves, something like Spacey, that just didn't exist 15 years ago. You had to really, really learn. Like learning Python 15 years ago was way harder than learning now. So what I wanna go and Ezra, I'm gonna start with you since you're the earliest career linguist. What kind of resources do you reach out to when you need to teach yourself something, et cetera? Sure, I think for general networking things, the special interest group, which we're, I think all aware of here, linguistics beyond academia was really incredible. I think particularly reaching out to the people that were the head of it, I think it's been shown in my experience, I think that none of those linguists are above talking to a new career person. And it's really been a beautiful thing. I remember early on when I was really wondering what I was gonna be doing with my life after a linguistics degree, reaching out to Laurel Sutton who has a bunch of experience in the professional linguistics world, very willing to talk to me. And I think that's kind of been the feeling that I've had in reaching out to people. So I think definitely that special interest group was a really, really big thing for me. And I would say beyond that, I think LinkedIn, I think with caution has been really, really helpful to me, particularly because it allows you to see your trajectory of different linguists that maybe are five or six years ahead of you. And I found that to be incredibly helpful. It's incredibly helpful internally, even within Amazon where you can see what kind of roles people are going to and what kind of companies people are transferring to and what people's jobs were coming out of grad school. I think those two resources, I think were incredibly helpful. And I'm sure there's more, but that's definitely what worked for me. Great. Esme, you work with a lot of, it sounds like you do a lot of just core coding. Do you find yourself reaching out to online resources, et cetera, to teach yourself how something works, like say, space here, et cetera? Or do you try to figure it out yourself? What's your method? That's a mix of both. But of course, since I basically, I have to find solution on the go quickly when I'm programming something. So I use Stack Overflow a lot to check a few things. Yeah. And so you learn to do it and you learn to develop intuitions about what to trust, you know, quickly skim through it. But that's one of my main resource for on-the-spot clarifying coding something. But one of the things that I've had to learn is to be very self-sufficient in this space. All the toolings that I have to use, the platforms and you just do it. But I have to say this is an anecdote, but I have a summer intern. She's a rising junior from Caltech in computer science and she's working on a problem for me and with me on CUI Conversational User Interface. And one of the problems is intent classification, right? For customer care. And she had absolutely no background in NFP, nothing. Yet it's an NFP related problem, a very important one for Intuit. And one of the resources, she took the initiative, she signed up on Coursera and some other online code and she completed some NFP courses over the weekends early on. And these are resources that I would recommend to people early in their career as major immunistics to look at what's available out there. And they were very well done and they're given by specialists. I've checked a few of them to see, I don't know. Fantastic. Rich, do you have any other resources you reach out to when you need to teach yourself something? Well, I just wanna second the idea about Stack Overflow, which I use practically every day. And also Coursera is a great resource, if you can find a course or something you're interested in. Other than that, I think that you can find a lot of things online, I mean, everything's online now, right? So if it's a problem about some specific, Python specific problem, you could just look into the Python documentation, which is all online for looking up a particular module and figure out how to do it. And I use that a lot. It's just really, I think just being able to use the search engine and finding that stuff and knowing what to look for is very helpful. And then co-workers, I have co-workers that have a different background and it's their job to know a lot of stuff that I don't know. So I don't hesitate to ask them. And actually, and just to jump in and one more resource that you guys are causing me to think of is O'Reilly Books. And basically they've been really incredible for those that don't know, it's a publisher who publishes different educational books about tech and they tend to do it really, really quickly, bare bones, illustrations, kind of stuff. I'm reading one right now about VUI's voice user interfaces and they can be really, really cutting edge. They don't wait for publication like a lot of other more corporate textbooks do. And that's been tremendously helpful early in the career. And I think many O'Reilly Books are published both on a physical for cost basis and given out as a free PDF. That's right, absolutely. It doesn't cost that much, yeah. This is not so much a question but a lot of us are working on chatbots now. It's just a big thing. And I just wanted to point out to everyone and I would like to kind of start a discussion here with the professionals is I just saw a blog post and I can try to find and put it in the chat about a chatbot designer who had discovered Grycean Maxims. And for those who don't know, Grycean Maxims are these conversational principles about how people cooperate in a conversation, make things relevant, make them short, et cetera. And she'd written up this really beautiful blog post about how she was using Grycean Maxims to make her chatbot better. And the conversation starters is what a brilliantly non-engineering expertise that is, right? But that's not an engineer's expertise but it's going to make a chatbot incredibly better if you have an understanding of that. For those of you working on chatbots kind of what kind of non-engineering linguistically things are you bringing to those developments? Like understanding terms and things like that. Can I go? One of the things that we are working, me and my intern is the three classifiers that we have created and trained after some other. We want to refine the distinction. It's a notion that people training and then being computer scientists are not familiar with. The difference between form and function is that you can express a speech act, what they call the dialogue, in many ways. And so for instance, let me give you an example. The customer care there is this request for a life help and it is, give me a life person now, and on and on and on. And the intent is feedback hate, right? But it's not different from saying, give me life help. Is that the speech act is still the same and request for something, it just happens to be in the abusive language, right? So this is one of the things that needs to be improved in all of the CUI. This is a linguistic notion of the distinction between form and function. And a lot of the strictly ML-based approaches to classification, intent classification is just schemes the surface of the language result having really a deep understanding of this distinction. So because you can add the same request said in many, many, many work ways with lots of F, F, F, F and other things, that's it. That's brilliant, yeah, that's fantastic. Yeah, so there's lots of, I think with chatbots, I think we're going to see an explosion in the need for pure linguistics, non-engineering pure linguistics. He's designing chatbots. Really, you have to have an understanding of conversation and human language. So let me point out everyone, if you're not following the chat, people are putting in other resources in the chat. So please do follow that. The chat, I don't think is preserved after we quit the Zoom call. So if you want those links, get them now. There is as much, not resistance, but there seems to be as much anxiety in the linguistics world now as there was when I entered 15 some years ago about whether or not linguists really fit in engineering companies. I think we've got a pretty good sense of our free panelists, that that's not true, that they do. But I want to go to the panelists and say, have you encountered anything that you would call resistance to being a linguist on the teams you were on that you had to kind of overcome? Rich, you've already kind of addressed this. I think you had a pretty positive experience. But let me ask Ezra and Esme, do you feel at any point, you kind of had some resistance to, you're just a linguist? Have you ever encountered that? I haven't, thankfully. And I think that that might have to do with the fact that the two roles that I've been at are teams of linguists. Right now I'm on a team of I believe seven. Beforehand, I was on a team of a similar number. So there is a significant amount of, I would say, insulation if that culture exists within the company. I would say that, I think part of that might come from if that culture did exist in the past of people being maybe a little bit confused about exactly why linguists are useful. And I think that that is actually becoming clearer and clearer as why linguists are particularly useful. And just to give a recent example, one of the things that Alexa is working on, of course, is multi-term conversation. There's gonna be a lot of give and take within Alexa. And one of the questions that you have to ask is are people treating Alexa like a human? And one of the things that that kind of manifests as is thanking Alexa and saying, please and thank you. And that's something even as Alexa used that I find. And that has been discussed recently. And that's something that linguists are bringing up to engineers. And that's something that if you're really on the engineering end of things, it's something that might not necessarily occur to you, but it's something that is being discussed by linguists. These kinds of conversations that engineers that I work with find really, really valuable are really, really coming from linguists. And I felt that our role has been, I think, clearly useful in my still very short career. Although I imagine diachronically, it's not always necessarily been that way. And I can't necessarily speak about 10 years in the past. Rich, I forgot to ask you, do you engage in any kind of meetups or conferences or any kind of connections to a wider community? Not on a regular basis. When I was at Microsoft, I did. That was a research-oriented position and went to conferences frequently and published papers. Since then, my work has been much more product-oriented and it's really more haphazard. I go to things now and then, but not by any means on a regular basis. I know. Do you do any kind of reach back? Do you, like, I know people reach out to me all the time, just random grad students saying, hey, I'm interested in industry. Do you find people reaching out to you that way? Occasionally, yeah. Like, you know, my professors or something might reach out to me and say, we'd like you to come and talk about your experience, you know, kind of like this, right? Your experiences in industry, going from academics into industry and because students are interested in that. So that does happen occasionally. And Esme, you've got, you said you've got this intern. Have you had, have you worked with a lot of interns in the past? Is that generally part of your job? Yeah, every year, I have an intern. And Intuit is really actually an engineering company. So all the interns that I work with are people in computer that are in computer science programs and they typically work on an engineering problem that I'm working on, but that uses natural language somewhere, right? As input or whatever. I never stray far from that type of data. Don't worry. But yeah, every year I work with somebody and they always end up learning something about natural language. Oh, fantastic. Oh yeah, yeah. Esme, let me flip it on you. Did you do any internships? I didn't. Oddly enough, maybe I should have. I had done sort of, you know, research. In the back when I was doing academia, I had done sort of, you know, research internships, but in terms of in the professional linguist world, I didn't. And it could be because, you know, I just wasn't aware of, you know, more pure linguistics internships that might have existed. Actually, I would say this is maybe somewhat related. Between grad school and my first job, I did actually do an internship that I think was more political in nature. It was with the Joint National Committee on Languages, which is an umbrella group that does lobbying for language and linguistics-related causes for Congress, which, unsurprisingly, their center is here in DC. So I've worked a little bit there, but I would say that it was a very ancillarily linguistics. It was mostly congressional lobbying, which is a whole other world. But in terms of, you know, internships that are more, I would say the linguistics end of NLP, you know, I wasn't really aware of it. Fair enough. Yeah. So I'll ask this question very carefully and you are all, of course, allowed to answer or not answer as carefully as you want, but I want to talk about salaries, getting paid. First question, have you ever asked for a raise? Yes. As you specifically asked for a raise. So you start off at something, you were at the job, how long were you at the job before you asked for a raise? I mean, I asked for a raise at the job offer. So the raise offer was X and you said, I need it to go up. Correct. And why do you need it to go up? I'm sorry? Why did you believe it was low? You know, I, it's a negotiation mindset thing. I would always assume that an initial job offer, they are as they should be trying to low-value. And I think that's just a general job offer game that you have to play. I think that, you know, they've identified skills in you if they want a job offer, they are financially incentivized to employ you for as little as possible. That's just a good business sense on their part. So of course, counter-offer is my mindset there. I would say that I think it's important when you've worked there for a certain amount of time, I'm not saying it's six months, not saying it's a year, but it's some amount of time to reevaluate how much you make, especially if you're giving a lot to the team and you feel that you're not being required financially by the role that you have. I mean, I think that's a certainly a separate question. And where did you learn that? Like, is this just kind of a part of who you are or did somebody teach you to be that way? So I think sort of a negotiation mindset is a little bit about who I am, but I would say specifically to talk about that like upfront at the job offer was something that had to be taught to me. Cause again, coming out of academia, I had no idea what the heck to do regarding job offers. That was actually came from a lot of my friends in college who oddly enough worked in things like finance, worked in consulting where, you know, job offers and negotiation is really everything. And that's not really a linguistics stereotype, I would suppose. And I think it's somewhat based in truth. It was really kind of friends teaching me that. I think that that is not something I've seen fellow linguistics friends do. And I think just general salary negotiation is something that is good for linguists and good for everybody. Great. That's my same question. Have you ever asked for a raise? Only once in my life, actually, when I was at IBM, when I switched from micromanagement to David Ferrucci, since I had got my hands really dirty at working on all the coding and CC++ from English parsing technology to the romance language, I thought, well, I know enough about the insights. And I did. And you know what? It was so easy to say, oh, no problem. I was surprised. So it was very easy. So I have no recommendation because Intuit gave me such an amazing package that I didn't have to negotiate. So I don't have recommendations there. And again, being European, we grow up within the brackets for salaries given a job description. I was never told to negotiate these things. We take salaries given the job description for granted. But anyway. Yeah. Richard, have you ever asked for a raise? Yes, once. When I was teaching at a university and was offered the job at Microsoft, I asked the university to reconsider my salary. And they came back with an offer and I said that wasn't good enough and they came back with another offer. And that was much better, but there are other factors that they couldn't address. So I ended up not accepting anyway. But there was something that Esme said that I wanted to follow up on. But now I think I've lost the train of my thought a little bit. Ma'am. Can't think of it. Well, Esme, you mentioned something that I think maybe I'll speak to briefly. Obviously I'm at IBM, but I've been at other large corporations which have the same idea of bands. And not every corporation does this, but it's not uncommon for positions to be in a band. What a band is is X number of dollars to X number of dollars. So if you're a band nine, that goes from 120,000 to 160,000. As long as you're in band nine, you can only get a raise up to that. If you wanna go above that, you need to do the career move of moving up to the next band. And that involves all this career machinations and things like that. And typically involves different roles. Each band has a different role. And that's certainly the way it is in IBM. That's the way it was at some other companies I've been at. Yeah, I am. Yes, sorry. I would say that there's a couple of good questions, I think, opposed by Alex in the chat that I think it was something I was passionate about mentioning anyway. And it's particularly about doing individual research on companies that you're applying to and also salary ranges for entry level positions. I certainly agree theoretically with that point. I think one of the challenges that I've had is that linguist jobs are rare enough, which makes sense. I mean, not everyone's linguist, very rare choice, educationally and career-wise. That a lot of those resources, like for instance, Classdoor, don't have a lot of salaries on there. There just aren't a lot of data points. As a matter of fact, there are so little data points that when I looked at the salary range from my current position, the low side of the range was 60% lower than the high side of the range. That's a really big range. This is not like looking for first law firm job out of college, out of law school rather, where it's a very set range. It can really be large just because there just aren't that many linguists out there. And that's frankly a challenge. Absolutely. And there's a good point to also be made that salary is of course dependent on region, you know, cities, et cetera. But it's also dependent on small, medium and large companies. Large companies tend to be much more rigid in their salaries because they're dealing with so many people, they can't have that much negotiation. This can allow a 60% range, whereas a small company can often really go back and forth. It kind of depends on how ambitious and how comfortable you are negotiating that. Yeah, I don't know that I can speak to entry level salaries, like what is an entry level salary? I don't know. I was never entry level because I jumped from being a grad student to being essentially a product manager. And that's a bit rare. So my jump was a bit unusual. But I'm really, I think sometimes it's nice just to mention numbers. Not everyone is comfortable talking about their salary, but I'm comfortable talking about it. So I will say, when I was a grad student in 2004, and I jumped to a consulting company in Washington, BC, I asked for $90,000 at the starting salary. And they gave it to me and I'm mad because I'm certain they would have given me more. I did opposite of what Ezra did. I did not negotiate. But at the time I was making $14,000 a year living in Buffalo. So I was pretty happy with that. But that was more for a product lead role. So again, it wasn't entry level. But I do just want to kind of put some numbers out there to say, tech companies make a lot of money. You might as well make it too. That's my position. Somebody at that company is getting rich. Sure. And definitely in the spirit of numbers, I will say this is particularly from my first job search right after grad school. I was in a space where I wanted a job that was linguistics related and there was not many more requirements after that. So I was looking at a very wide variety of things, everything from vaguely linguistically related consulting things, looking at tech, looking politics, government, linguistics things which is more relevant here around DC. I was seeing starting salaries anywhere between $35,000 and $150,000, which if that sounds crazy it is. There's a lot of different salary points. It depends on which kind of linguistics role you want to get into. And again, I think that this is a problem that ultimately gets solved by time where you're beginning to get more data points, you have more linguist networking with each other where those numbers become a little bit clearer. We're a little bit, I feel, in a period of kind of wild west of employment of linguists. Just to give you an example, the job of language engineer at Amazon which is my current title that didn't exist five or six years ago. A lot of these roles are very, very new. So the salary numbers and salary cultural benchmarks I think are not, they're a little bit fluid. So they're not quite ossified. Wayne is posing a couple of interesting questions in the chat. So I just want to address them real quick. How much does the salary difference between large tech companies and small companies? Again, there are a lot of dependent factors but I will say that my experience has been small tech companies pay better entry level than large tech companies. Large tech companies, again, they have their bands are kind of fixed. Small tech companies tend to be a lot more open-minded and you can negotiate better. I love small tech companies as an environment. I think it's just a fantastic place to be. The problem with small tech companies is you don't know if they're gonna exist 30 days from now. They go out of business easily. So you're taking a risk but I do love the environment of a small tech company. The next question is regional differences. I actually can speak to this and of course the panelists can speak to this as well but I've moved across the country four times in my career. So I've gotten a sense of the difference between costs. I will say that again, especially at a big company they will literally have a number attached to the city you live in which says whatever your baseline salary is if you live in San Francisco, you get 12% more. If you move from San Francisco to Boston you get 6% less than you were making. They literally have this written back. They know or at least they think they know what the cost of living difference is and they will adjust it for you. And a lot of times you have no control. Again, it depends on where you are in your band there's some other things there, et cetera. But I will tell you right now salary San Francisco is as far as I can tell the most expensive place in the country right now. And so you do get a bump, but that bump doesn't afford. This may be a bit of an exaggeration but honestly I would not move to San Francisco for less than $200,000 a year. And it's just insanely expensive right now. There are, because tech, I'm sorry I don't mean to dominate but let me fill this out to the team. In my opinion, because tech is fairly diffused you have tech centers. As where you mentioned DC is not the best but DC is not bad. There's Austin, Dallas, Boston, Seattle, San Francisco. You've got some choice there but all of those are pretty expensive cities. Even Austin, Texas is getting expensive now. Tech companies bring gentrification with them and prices go up. It's just a fact. So let me fill that out there. We're running low on time. I just wanna take a quick, oh some of the questions that are coming in are not so much about being a linguist in industry but just being in industry. In general, sort of Ezra, I'll pose this as the final question and I'll just go down the line and ask all three of you. Linguistics aside, salary aside. What kind of field do you want from a job in terms of commute, in terms of office life and what's around the office? What do you look for in a job? Sure. I would say that as far as commute to handle that first, as close as possible, commuting is terrible and a car costs a lot of money. I think that's understandable. I've been able to avoid having a car for my entire 20s and I hope to keep it that way. I would say like from an office, I would say that I want an office that not only do people feel comfortable communicating with each other but there's really, really good cross-team communication. There is a book that I read in my first job thanks to Emily Pace that's called Team of Teams. It's written by General McChrystal and I think it's a really, really good example of a place that I want to work in which is having teams that interact really, really well with each other and not being siloed with each other. And one of the things that I liked about expert system, one of the things I liked about Amazon and I hope to have in future roles as well is teams that are comfortable interacting with each other and not just through choke points. I think that's something that from a corporate structural point of view is really, really important. As far as other things, just general flexibility, you know, being able to get off to religious holidays relevant to fellow Jews and definitely to Muslims as well. And I would say also making it really, really clear how one moves up. I think that kind of clarity is really important. We do have that at Amazon, which has been great. It doesn't always necessarily happen. So a place where if your career progresses with that particular corporation knowing how to do it, Lajom. Great. Azmeh, what do you look for? Okay. I think every second what Ezra said about the team, I would mention that earlier for me, it's who I work with and how they engage with me in a non-conflictive fashion is extremely important. I like to be able to discuss whatever is going on implementation coding, problem solving in a non-conflictive manner. And that's the most important thing. And I want always the group to be accountable as a group for whatever happens. Okay. Above all in the field of engineering when there are problems it's collective not single out anybody. That's very important for me. And also respect for my expertise. If somebody did not respect my expertise and deleted it in any way, well, I would not go for that. I would be very unspoken about it. And so basically no discrimination of any type. Fantastic. Rich, what do you look for? I would definitely second the idea about teams being able to work together. I think that's very important. Being able to work with your teammates, that these are people you can get along, not just work with them, but you have to have water cooler time, right? And so there has to be some kind of chemistry fit there as well. And I have worked remotely for 17 years. So I don't commute. And so the fact that my company is located 2,000 miles away from where I live. I don't, I've been there a couple of times but it's not a big deal. If you don't have that opportunity, I would look for what life is around, what life is like around where you're working. That's why I came to Seattle. I mean, that's why I left my academic career to come here. And so personally, if I'm looking for a company, I'm looking for a company that's gonna let me stay here. Absolutely. And that's a great point. We're living in a world where work from home is very popular. I've been working from home for five years. So yeah, it's becoming a more of a thing. But basically at the end of our time, and so I just wanna end with my kind of impression, which is all three of our panelists today have had a pretty positive experience in the industry. And I think that's a really good lesson for everyone who sat in on this to take away is there's a role for you. And, you know, academia is suffering a lot right now. I don't know how many of you listened to academic Twitter, but oh my God, it's like a nightmare, health scape right now. I hope it's not really that bad, but it sounds like industry is a heck of a good choice for a lot of linguists right now.