 and welcome to the Linguistics Career Launch. My name is Alex Johnston. I'm one of the co-organizers of the LCL 2021 and director of the MA in Language and Communication Program at Georgetown University in the Department of Linguistics. And today I'm so pleased to host a panel titled Applied Research in Non-Profit, Poor Profit, and Federal Agencies. We have a representative of each type of organization on our panel today. And the motivation behind this panel is because I've heard from so many students who are thinking of following a career path outside of tenure track academia that they will miss doing research or they don't know about the opportunities to do research or what it looks like outside of higher education. And so we have three panelists here to talk today about how they perform research in their organizations, how it's similar to and different from doing research in a university setting. So I'm very pleased to welcome our three speakers today. We have Marcus Berger, who is a social science analyst in the Language and Cross-Cultural Research Group at the Center for Behavioral Science Methods in the U.S. Census Bureau. So did anybody know that there is social science research happening in the U.S. Census Bureau? Yes, Marcus, it's one of those people. We also have Jill Burstein, who is the Principal Assessment Scientist at Duolingo, an educational technology company that many of you may have the app for on your phone, Learning Languages. And we also have Francesca DeSoviet, who is Director of World Languages at the Center for Applied Linguistics in Washington, D.C. So I very much appreciate we have three linguists who are going to talk to us today about their research programs in their organizations and I'll turn to them for a more in-depth introduction. So if you don't mind just letting us all know where you did your linguistics training, what did you study in your linguistics program and what is the current scope of responsibility for your job. So if I could start with Marcus, then go to Jill and then Francesca, please. Marcus, would you mind telling us just a bit about your linguistics background and what you currently do at work? Sure, absolutely. So I did my PhD in linguistics at the University of Michigan. I focused on syntax and documenting endangered languages. So I spent most of my research time in Amazonian Peru working with speakers of a couple different languages down there and work primarily on generative syntax analysis of those languages. Pretty different from where I ended up. I now work for the US Census Bureau. Most of what our work involves is a lot of pretesting of materials that are produced by the Census Bureau. We go through in languages other than English. We do pretesting. We do expert reviews. We do focus groups. We do cognitive interviews and really make sure that the materials that we're producing are understandable, that people are understanding the questions in the same way across languages, and that we try to avoid any ambiguities. But we also try to make sure that the materials we produce are culturally appropriate for each of those languages. Great. Thank you so much. May I turn to you, Jill, which you mind letting us know about your linguistics background and what you do now at Duolingo. Sure. So I did my bachelors in linguistics in Spanish at NYU, and then I went and did my PhD at the Graduate Center at the City University of New York in linguistics. I did a theoretical dissertation in morphology, but at the same time, I would just want to take a step back. This was in the middle to late 80s, so I was doing a concentration in computational linguistics at the time, which there weren't that many programs that were doing it now. There's a lot of programs that were doing it. And while I was a graduate student, I worked at AT&T Bell Laboratories, which was a huge research institution as far as there really isn't much of the labs left. But I got a lot. I got some of my practical training, sort of like an internship sale, though it was a very well-paid internship for the time. And so that's sort of me in a nutshell in terms of my educational background. And I just joined Duolingo in May. I was previously an educational testing service. So I work on the Duolingo English test. The Duolingo, most people probably know the app, but there's also the Duolingo English test. And the Duolingo English test is what we call a digital first English language proficiency test. And it's really for academic English language proficiency. And what I do, and I wrote this down to make sure that it was clear because it's a little esoteric, is that I support sort of the, right now, I support a lot of things, but my key responsibility right now is to support continuous development of a theoretical framework for the test. So remember, I worked at ETS for many years. So I have now a pretty strong assessment background, which sort of got through, you know, working there through osmosis. Anyway, the framework is especially important. Since everything on the test is digitally mediated, specifically all the items are automatically generated and scored. And so another part of my contribution is research related to AI and NLP methods that can be used for test development and associated evaluation. And so a lot of my, all my work at ETS was, I was basically, I was an NLP scientist. So I was sort of in the linguistics and computational linguistics space. And then because of where I was living at ETS, I became very familiar with assessment and I learned about assessment. So I think that that's sort of one of the things that's important that wherever you end up being situated, you're going to learn about that, you know, how you're going to learn about that area. Chances are you won't be working for a linguistics company, but you need to sort of understand the content of where you are. So that's me in a nutshell. Jill, I think that's such an important point to note is, you know, I can imagine that many people would think that if you go to work at educational testing service, you know, you should have had formal classes in language assessment, perhaps. But as you say, you are, you are learning about that content, you're learning about that special content area on the job. That's right. So in a way, this gets back to what I often say is to not be your own gatekeeper when you're applying to positions, you are able to apply to positions, for example, at a testing service without having had a lot of formal background in assessment because of all the on-the-job training that you'll then go through. Is that fair? It wasn't my case. Yeah. Yeah. Great. Thank you. And I see the question in the chat that I will look forward to getting back to in just a minute. Francesca, would you mind letting us know your training in linguistics and what's the scope of your responsibilities now at the Center for Applied Linguistics? Sure. So I did my undergraduate study. I studied both Spanish and Italian, and I minored in linguistics kind of at the end of my undergraduate years because I found it really interesting. And so then when I went back to grad school, I actually did a master's in language and communication through the program that Alex works in at Georgetown University. It's a fairly broad program. I personally focused a bit on sociolinguistics. But then following completion of that degree, I started working at the Center for Applied Linguistics, or Cal, as a research assistant in the language assessment division. And that's where I still am. I've just moved up. So my current title is Director of World Languages. I work a lot on language assessments, development, and research related to that, but not only. We also do research on kind of language learning outcomes. We do a lot of development of training materials related to administering and rating assessments, but also kind of classroom use of assessments. So that's like teacher training. And we do a lot of qualitative research with users of these materials to figure out what they need, what works, what gaps there are. Great. Thank you so much, Francesca. So now I think I'd like to go back, put yourself back in the place where you were when you were finishing up your final degree in linguistics and see how you made that leap from your degree program to your first job post-degree. And, you know, Marcus, I'd like to turn back to you, having, you know, the focus that you did in your degree, working on endangered languages. How did you learn that you could apply to a job at the Census Bureau, which I believe was your first job post-doctorate, right? How did you learn about it? How did you position yourself for that job? It is my first job post-grad school. So I initially, while I was in my graduate program at Michigan, we were funded through our first summer. And starting later in our grad program, we had to sort of find our own summer funding, whether it be through grants and fellowships or other things. And I was talking to a committee member of mine who had a former student who works for the Census Bureau. And there was an opportunity for an internship that summer. And I thought, you know, it's, I didn't know that they needed people linguists working at the Census, but I was working at that time. In 2020, we were going to be doing our first Census where you could answer online. And they wanted to do research on different language groups and what technology they would be most likely to use. So would Spanish speakers be most likely to use a mobile device or a library computer or something like that. And so we were doing that for the most spoken languages across the country. And that was my first internship there where I was able to find out more about, you know, the jobs, linguistic-related work that was being done at the Census. And so after that internship, I made sure to stay in contact with some of the people I had met during that time and was able to look forward to other positions later. My final year in grad school, I did another summer internship there in a different part of the Census and was very lucky in that I was able to basically continue in that position until I graduated and I applied to a postdoc position, which is where I am now. That's so interesting. I mean, it shows the importance of faculty advisors keeping track of their students and where they work because your advisor, your faculty member was able to tell you, oh, I have a student who works at the Census Bureau and then kind of connect you that way. How important do you think networking is and, you know, keeping track of people, making contact with people? Incredibly important. And, you know, a lot for that reason. I was very lucky that this member had kept in contact with her former students. I think, you know, there's that and then also keeping in contact, you know, not only the faculty members keeping in contact with their former students, but myself as a former student keeping in contact with my former advisors and faculty members just to really stay in contact with the linguistics community. Part of my job, we still present at conferences. There are different conferences that I'm used to. We do, for example, the American Association on Public Opinion Research. But I still am interested in continuing some of the work I was doing in grad school and so I'm still applying to conferences related to Indigenous languages sort of outside the scope of my Census work on my own free time. Oh, that's so interesting. First of all, I'm so glad that you mentioned the other types of professional associations that are out there that are venues for this type of work because that's really important to know that there are different venues for different types of research outside of the academic conferences that we may be used to, like the Linguistic Society of America annual meeting, the American Association of Apropology, or the American Association of Applied Linguistics, the kind of the big, big three in linguistics. So American Association of Public Opinion Research is where you and other members of your units present your research, right? Yes, that's sort of our LSA. That's the big conference that happens every year. And in fact, this most recent one, we did a lot to recognize Rob Santos, who is now going to become, if all goes well, in his congressional hearings, the director of the Census. So we were able to kind of interact with him at that conference and it was sort of neat being able to talk to someone who's going to be in that position. Yeah, exactly. And so again, this is the type of conference that if, you know, I always recommend that people find out what are the conferences associated with the field that you're interested in. And this could be something that people could research and consider attending to get a sense of what is presented at these conferences. Thanks so much for putting that in the chat, Marcus Robinson. Thank you. Great. So Jill, could I turn to you and find out your story of how you found your job at ETS in particular? Was that your first post-degree job at ETS? It was. So I mentioned earlier that I was working at, you know, as a graduate student in a graduate student capacity at AT&T Bell Labs. And so I was learning, I was learning a bit more, I was learning in school about computational linguistics and taking courses, but I was learning in a more applied environment there. And a friend of mine, and this goes back to net working, it's a bit old school, but she saw a little handwritten job post that was posted on the bullet board. I don't think that really happens anymore, but it would be the equivalent of, I don't know, I'm going to actually post a job from Duolingo into chat before I forget that just came up yesterday. That would be the equivalent. And she said here, there's something at ETS, there's a phone number, they're starting an NLP group, and it was literally a ripped piece of paper that somebody had posted there. So there weren't a lot of jobs in the field of NLP at the time. There was sort of a bit of a drought from a lot of the research had been funded by Department of Defense and that money had dried up for a short time. It came back again, but it was kind of dry. There weren't a lot of opportunities. So I did apply for this job. I called them, I guess it was the call phone number at that point. And I applied and I interviewed and it took several months. It took about eight months to get the job offer because at that time it was still, do we want to bring somebody in? Do we not? Because they were really starting something new. It turned out now they have a huge NLP presence in the educational, in education. But yeah, that was how I started. And networking, it's definitely, it was just like an old school form of networking where my friend, you know, I was looking and she saw something, a bulletin board and I totally agree with everything that was just said and I just want to offer if anyone on this call wants to link in with me, you know, at the end of the session, that's definitely something to do. You want to, you want to really build your networks. And I will post this job in the chat, but I didn't want to forget before the end. Yeah, I just noticed yesterday on LinkedIn that your colleague Cindy Blanco had posted several open positions at Duolingo. So everybody should check out. Yeah, there are. I happen to have one. I just posted it. Thank you. And the chat, people want to take a look. Yeah. And you know, that also, your story shows the importance of peer networks too. It's just you let your friend know. You told your friend, I'm looking. Your friend knew. So your friend was kind of activated to notice things in the environment that everyone needs to know that you're looking and all your friends need to be looking out for you and you need to have a lot of them. So yeah. Tell everybody in your peer network. Well, LinkedIn is a good resource. It's a really good resource for sort of connecting. I find it useful. And I'm happy to connect with people and help people out. Thank you for that. Yes, we've been pushing LinkedIn throughout the whole thing with six career lunch. That's for sure. So Francesca, could I turn to you and ask you about how you located your first job post grad school? Sure. Yeah. So I was at Georgetown in Washington, D.C. and Cal is also in Washington, D.C. I was pretty familiar with their work. There's a lot of people at Cal who've gone through Georgetown or had connections there. So I think I knew of it as an organization. I was interested in what opportunities there might be there. So I set up an informational interview, an in-person, probably 30 minute interview with someone who works in the team I ultimately joined just to ask about what kind of work they did at Cal and what kind of opportunities they had. And then a job opening was posted pretty soon thereafter, a couple of months after. And with the person I interviewed with, she was on the hiring team. So it was a pretty direct connection. But then I fully applied and got the job. So I guess that's a little bit of networking word of mouth, but a plug for informational interviews. Absolutely. Yeah. And that was a really good way to make yourself known to that person who then would be on the hiring committee. I don't think you could have predicted that the person that you interviewed with would then be on the hiring committee per se, but it's great to make yourself known to organizations in that way. So I'd like to turn to all of you. And I said at the beginning that this panel was motivated because not a lot of students currently in their undergraduate or graduate programs know that know the range of research that is done outside of universities, because in so many cases, the research that's done is either proprietary. It's kept within the organization and it's not published outside, or it's presented at conferences that academic linguists who are in higher education may not go to, like the conference that Marcus Berger mentioned. So I would love for you to give us an idea of how is the research you do similar to and different from research processes at institutions of higher education. Would anybody like to address that and just talk about ways that your research is similar to or different from? I guess I can kind of start on that. I had no idea coming into my current position how that was going to transfer. And it ended up transferring a little more than I thought it would. A lot of what we do in our research is cognitive interviewing with respondents. We give them a survey, we have them answer the survey, we ask how they interpreted different parts of it, why they answered the way they did answer them questions. And it's incredibly similar to a socio-linguistic interview or to the types of interviews I was doing with syntactical limitations in the field. And I think having that training from my graduate research really transferred over to the sort of research that I'm doing now. There is new stuff that I wish I had done more in graduate school. There's a lot of statistics that goes on and I sort of, I took a stats class in grad school and said, oh, I'll never use this. So I am using it now. And I'm having to go back and relearn a lot of things. But I'm happy to sort of have the background that I did. And like Jill was saying earlier, there is a lot of on-the-job training. And as an academic at heart, I'm sort of excited by that opportunity for lifelong learning. And so I would say there is definitely some stuff that did cross over and definitely other things I am learning new. Yeah, you know, Marcus, a couple of things that I love about what you said. First of all, the use of the term cognitive interviewing isn't often used in socio-linguistic research. We tend to call it a socio-linguistic interview or a semi-structured interview. In applied linguistics, you might call it a stimulated recall protocol. There are so many different terms for leading somebody through a talk-aloud protocol of how they thought about questions and answers on a survey, right? So it's knowing that there are different terms used in different sectors for the same thing, which is really important. So you can make sure that you're using the language that your interviewer or your networking partner may understand. So I just want to make note of that term cognitive interviewing. And also, I just really appreciate that you mentioned that you have opportunities for lifelong learning and for a stimulating research environment outside of higher education. That's not the domain only of universities. You can continue to keep learning and keep stretching and growing in jobs outside of academia. So thanks for plugging that lifelong learning in federal agencies. So Jill, would you like to address the question of how research is similar to or different from the research that you've done in higher education? Yeah, I don't have a whole lot to say about this because I mean, I've never worked in higher education. I've worked through grants. I've worked with higher education institutions. As a PhD student, I've done research within a university, but that was a very long time ago. And I think there's a lot of differences between how research is maybe done now and how it was done then. Like, for instance, I mean, I know in universities now, they let faculty have startups and things because they've been leaving. So that was kind of a new thing. So I don't have a lot to offer in that context. What I can offer, and I probably think is true, and this is a bit of an assumption, is that I've always really liked applied research. I never was drawn to academia. And so what I really like about what I do when I did also an ETS was research that goes into products, things that go out into the world that people use, like things that have like an immediate social impact. And so I found, and I guess my grant work, you know, with the Department of Education, that's sort of the kind of thing where that's more broadening knowledge and that's maybe more like academic research, what you would do in an academic institution. It's very long term. It takes a while to get anything done, whereas in industry, even though some things do take a while to get done, you do sometimes have shorter term satisfaction of building something, getting it into a product, seeing it used, being able to see how people are using it. You log data, you can do research with it. And so you get some very, you might get some very immediate feedback about what you're doing in application that I enjoy. Yeah. So there's this difference in timeline and difference in the immediacy of feedback of just knowing how the results of your research are being applied. Great. How about you, Francesca? What do you think are some similarities and differences in research? Yeah, piggybacking off the other answers. I feel like at the Center for Plain Linguistics, we're a nonprofit organization, so we exist fully off of soft money, which means the projects that we do are grants and contracts that we win. So a lot of times our research is very applied, but it depends on the funding source, you know, what they want. So we do research, especially in the area of world languages that's funded by, for example, the Department of Education. And that's some of the more longer term, you know, really looking at the data. And, you know, it takes a long time, you need like a theory of change, you need a framework. So in that way, it's actually very academic, but then we do some more applied research in our language testing that is for products that are going to be available for sale. So it's a little bit of both. But I would say it is similar to what I experienced in higher education in terms of training, in terms of setting up the research, fully executing it, you know, from methodology to really writing up reports for different stakeholders and different audiences. That's something I definitely think I learned from education. And similar to Marcus, I did not focus at all on statistics and wish I had, although Cal has a department for that, you know, we have PhD psychometricians. But I do a lot of work that touches that area and have learned a bit on the job. But I would advise if people have the opportunity to take those kind of courses, it can be really helpful because no matter what kind of research you're doing, qualitative or quantitative, you need that ability to analyze. And it's, it could look really nice when you're applying for a job as well. Okay, so a plug for having a well rounded set of methodological tools, both qualitative and quantitative. Great. I'm wondering about your about the necessity of having a PhD to do applied research in your organization. So what does that PhD degree allow you to do in your organization that a master's or bachelor's holder would not get to do? I can speak for Cal. We definitely have a mix. We have, we have people, you know, we have research assistants certainly coming in just with bachelor's degrees. We have some people with master's degrees and plenty of PhDs. And the difference is somewhat the level, the position you can attain at Cal, although what we do here, which I really appreciate having only a master's myself is, you know, most of the jobs that we post and then the classifications we have, they have a combination of education and experience. So if you have a bachelor's degree or a master's degree, you need the equivalent of more experience than someone coming in with a higher degree. But in terms of actual application of the PhD, it is very helpful when we are applying for these grants. Some, you know, you need a PhD as the principal investigator. And so if your team doesn't have that, or we don't have capacity for that, we might partner with other groups, we might bring in someone with a PhD as a senior advisor. And then for those very specialized fields, particularly within language testing, you know, like psychometricians, they really need that degree. They're bringing in a wealth of background knowledge that they have learned in their PhD programs. Great. I'm so glad that you're bringing up, you know, the mix of degree levels in your organization and how people can work together on teams. Because I think a lot of students who are who have the model of, you know, the academic model of people who have PhDs going into assistant professorships, like those are the people who do research. However, in many organizations, people of any degree level can do applied research and research within that organization and work in teams together to engage in research. Can I address that question to either Jill or Marcus about the role of PhD holders versus other degree levels in your organization? Yeah, I can say a little about the Census Bureau, at least. The team that I'm on, I think we would be, we're fairly small, but we're a little over half of us, I think, our PhDs. And most of the rest, I think, have masters. We have one who is a bachelor's. The advantage, I think, of coming in with a PhD is an easier way to onboard. A lot of people on our team have come on to the team through the Census Postdoc program, where you basically submit an idea that you have for postdoc research and you address how it would be beneficial for the Census Bureau to have that research and how you would benefit carrying out that research in a position at the Census Bureau. So multiple people have come on to our team with that program. I'm currently a postdoc in that program and I'm looking forward to applying for a more permanent position down the road. So that's definitely been helpful there. And the other main difference is the pay level that you would come in at. There's a certain level with a master's you would come in at what we call a GS9, with a PhD you would come in at a GS11. But those are the primary differences. And here Marcus is referencing these pay scales and seniority scales in the government, which are very precise as part of the salaries associated with them. So those are publicly available. You can look up the GS scales and find salaries and associated with each of those levels. Jill, can I ask you about the opportunities for people who hold different degrees at Duolingo to do research? Yeah. So I can talk about the Duolingo English test. I don't know all of what's going on on the other side in the app world, but to do the kind of research that I do, you need a PhD. The kinds of work that you can do at Duolingo without a PhD with a master's typically, or maybe a bachelor's is more technical. So Duolingo is a technology company. So you can be a machine learning engineer, you can be a software engineer, you can be a data scientist, you can be a product manager type thing. Sometimes these people come in with more technical skills like they have an engineering degree or they have a computer science degree because they have to understand the technical aspects of the work being done. And so I think because it's a technology company, it's the more technical roles that where you tend to be able to have not a PhD. So if that helps, and I would advise, I mean, going back to the statistics that people do get some skills in Python or R and data visualization skills and things that are sort of aligned with data science and they'll really come in handy. Thank you. And as I'm continuing to ask questions, the floor is open to anybody in the audience who would like to raise their digital hand or wave a nonverbal hand or ask your question in the chat and we'll be happy to read it out. So at any point, please feel free to ask questions of our panelists. What I'd like to find out now is, you know, when graduate students in doctoral programs are going through their research, it's very much motivated by what they're interested in. As an assistant professor, you have a research program, which you develop on your own. But when you're working outside of academia, you may not have as much individual choice in what you choose to research. So I'm wondering if each of you could tell us, you know, who are your your clients or your stakeholders or the beneficiaries of your research? Like who are you kind of beholden to as far as what you research? Would anybody like to address that? I can start. Since I'm in industry, they're definitely stakeholders. Pretty strong influence. So so our clients are institutional programs who accept the test obviously, and then the test takers who purchase the test. So the test, you know, we have to make sure that from the test that the stakeholders who are the institutions who are using the score to for academic admissions are really confident about, you know, the score that they have, and that it really represents the test taker and the test taker, we have to make sure that they have a really good experience taking the test so that when they the test is a reasonable price, when they sit down to take the test, it goes really smoothly. So all of our research to make the test better and to make the experience better for test takers who are our stakeholders and beneficiaries of the test, that really drives the research. The research is really about making the test really solid, really good. And also as good an experience as any test could be, so making sure that it's a really smooth experience. So that's my answer. Yes, very clear about who the stakeholders and beneficiaries are. Thanks so much. Marcus or Francesca, would you like to address that? Marcus? Sure. So for the Census Bureau, it's we have some things that are just sort of our job, right? So the decennial census that happens every 10 years, we have to do research on that. And that's sort of, you know, within our normal job responsibilities, the same with the American Community Survey that goes out every year to the sample of the population. We're currently in the middle of our content test for that that we do every five years. So those are sort of unsponsored, I guess you would say, jobs. We also have work that comes from other areas. So we work with the National Health Interview Survey, which is from the National Institute of Health NIH. We work with the National Household Education Survey through the Department of Education. So we sort of have stakeholders in those areas. We help out the New York City Housing and Vacancy Survey. We work with a lot of different stakeholders on a survey. We're currently putting out the Household Pulse Survey, which started as sort of a rapid response to the COVID pandemic, where we wanted to find out how people's lives were affected, how people were getting their food if they didn't feel comfortable going to the grocery store. It was sort of an early on thing. And then now we're testing how comfortable people are being vaccinated, whether or not they have been vaccinated. If they are not being vaccinated, why not? And so it's very rapidly changing. And there are questions on that survey from lots of different organizations. We want to find out how this pandemic is affecting people's lives. So most of our stakeholders are within the government or a few are outside of it, but we do have several of them. Yeah, thank you for that survey. Francesca, in your area, who would you say are the stakeholders and the beneficiaries? Yeah, so at the Center for Applied Linguistics, I guess I'm sort of in two areas. So in world languages, I would say broadly it's language learners and language teachers. So depending on the audience are always both. You know, we're trying to improve the teaching and learning of world languages and English language and also just mutual understanding of linguistically and culturally diverse people. So we have a lot of policy work in that area as well. And then in language testing, we actually work on develop content for and deliver a variety of tests. And those are K-12 and adult. And on both ends of that, there's actually, you know, there's federal accountability requirements. So a big stakeholder is actually the federal government and what the users of these tests, so school districts and students, what they have to report, you know, it needs to be meaningful. There needs to be validity studies behind it showing that the test is measuring what it's intended to measure, you know, showing the gains that we are trying to measure. And so I think those are major stakeholders. I'm thinking of a particular project I'm working on right now where every seven years we have to submit for approval to the federal government. It's a subset of the Department of Education to allow this test not to be sold, but it is primarily sold to adult education programs that report to the federal government and they receive funding related to that recording. So we actually have to do research to support the tests in order to be approved by the federal government as a test that can be used for those purposes. Great. Thank you. And I noticed that Bellina has had her hand up for quite a while. Bellina, would you like to ask your question, please? Yeah. Hi, everyone. I have two questions. The first one, I would like to know when you're doing career in your positions and you get higher position, that does mean that you get kind of farther from the linguistic fields. It's like more into like manager or more related into like kind of business, depending on if it's nonprofit or not. Or you still keep having a very related job in that sense. And the second question is related to like the requirements related to computational linguistics. Like I just searched the dual-lingual position, the geo share. And this is something that I kind of heard recently. So I don't really know much and I would like to know when talking about computational linguistics, mostly we speak about Python as the main programming language. But what about Prologue? I just heard that Prologue is another language really related to. And I don't know, do you ever use it? Or is it just in like a research field of academia? Because I found it related to computational linguistics in Spain. So this is a kind of far question. But I just recently heard about it. And I would like to know if it's a skill that it can be. If I really learn a well, I can use it in a job market in the future. Thank you. I'm not aware of anyone who uses Prologue anymore. I learned Prologue when I was a graduate student. But it's, I don't know who's using it. I mean, Python and R and Java and these are the languages that people are using regularly in computational linguistics now. I forget your other questions that I sort of focused on. Oh, the manager one. Yeah, the manager one. So the manager one in my previous role at ETS, it was not, I mean, when people sort of started rising up, they definitely became much further away from the research because they're spending a lot of time on budgets. Personally, I maintained my research role, but I don't know that that's that can always happen in every organization. I haven't worked at Duolingo long enough to know what will happen as I rise up. But I do know that my manager, I mean, you do end up spending less time in research, but it just depends who you are and how much bandwidth you have and how much you can sort of insert yourself and make sure you're watching things. But you make a distinction between individual contributor, which would be a researcher and sort of a management physician. So individual researchers or contributors, individual contributors typically are going to be more hands on research. So they'll be right in there. It's a good question. Francesca, since you've spent your career at Cal, rising in the managerial ranks, what what might you add to this question? Yeah, I think it's a good question at Cal because it's relatively small. We have about 60 employees and that's actually everyone. So on the program side, not only research is probably only 30 people. So I think it's a bit more you stay closer to the research or you have the ability to assuming you want to. But moving up in the organization, Jill mentioned, yeah, you have a lot more responsibility for budgets for managing projects overall and managing the people who work on projects. And sometimes with the research, again, depending on the project and the funding source, you might you do a lot of work or you can do a lot of work on research design. But then you have other individuals who are like implementing the research. And then it comes back to you when you're doing more kind of like final reviews. So I still do a lot of research. I appreciate that about Cal. That's why I am happy to be here and doing it. But it definitely depends on the project. And I think there is a track, especially if you have a PhD, where you can be like maintaining research only. But if you're trying to move up more based on like a level of education and experience, bringing that project management into it does become a bigger role. But I don't think you have to completely lose touch with the research, at least in my experience. Okay, I'd like to go back to a question in the chat that is asking about salary ranges. Because I think this is a really good way to, you know, kind of compare with salaries in academia, especially if you're starting out as a researcher as an assistant professor in the higher education. So Marcus, you raised the issue of salary ranges by referring to the GS scale, which is publicly available, searchable. Would either of you, Francesca and Jill, be able to share kind of imagine somebody coming in with a doctorate? What might they be able, what's a range of salary that they might be able to find in your organization? I'm trying to think if I can give a precise answer. But I will say that Center for Applied Linguistics has two jobs posted right now. And I think they're more, they're slightly more programmatic than research. But that'll give an example of the range, because they both at least require a master's. And there's definitely all of our jobs have a range based on education, and it's higher for PhDs. So I think, I don't know, I'm just going to totally make up a number that if there were a job that were being posted that was really looking for a PhD, you know, aiming at that. I think probably the lower end of the range would be maybe like 70,000. But don't like hold me to that. Yeah, absolutely. Understand that this is very general. Jill, would you have a general range that you could point to? I really, I don't. In fact, I don't even know what the pay range is for the job I just posted. And I'm not even sure I could say. I don't know if I'm at liberty to really say. I can tell you it's for a PhD is probably much higher than 70,000. But it's definitely higher than 70,000 for PhD or probably even a master's student, because we're an industry. So I think if you just look out there for, you know, in industry for what the typical ranges are, like, for particular positions, if you don't last door or something, I think Duolingo is competitive with, you know, maybe not quite as competitive with Google and Facebook. But it's pretty competitive. I think it's probably competitive with Microsoft, because we seem to be lowering a lot of people for Microsoft, at least for engineering goals. So. And anything that you could say about ETS, if you recall, just very generally, would this be kind of on par with Duolingo or? No, it's not quite known. I don't know if they don't pay as high as Duolingo in my experience. Definitely a little bit. I would, think it's a little bit lower, but it's still good. It's still a pretty decent salary. And they may be getting better because they're, you know, they have to compete. So I really don't know. Yeah, and thank you, Marcus Berger, for posting the general schedule pay scale in the chat so people can get an idea of the GS levels in government. And, you know, for comparison, if you're looking at an assistant professorship at a higher education institution, well, there's quite a range there, depending on public and private and depending on if you're not a 10-year line or a 10-year line, but you could be looking at, you know, $70,000, $80,000 a year-ish. And for some non-10-year line, faculty or full-time who teach, you know, three courses, fall semester, three courses, spring, they could get anywhere from 40 to 60 to 70. So just something to think about. You really have to do the research when you're looking at salary ranges. I'm actually curious to find out just a bit about your day-to-day life in your workplace and, you know, what you tend to work on day-to-day. So, you know, Marcus, would you care to tell us, you know, how you manage your schedule? Is it pretty routine? Is there always something new to work on? So there's often something new to work on, which is kind of a nice thing about the job, I think. We have these different projects that come in at different times, and our work is sort of determined by those projects. We talk, as our team, I know early at the beginning of this, he mentioned that I'm part of the language and cross-cultural research group within the Center for Behavioral Science Methods, and our language team is seven people right now, and we sort of meet once a week. We talk about the different projects that are going on, who's on which projects, what projects are coming up, who's going to have free time coming up, and sort of determine based on that who will work on which projects. We are sort of constrained within our team, you know, although we are seven people, we are four Spanish speakers, and so any project that involves Spanish review or Spanish work, we're sort of limited on who can work on those projects as well. But it often is a mix of doing interviews, doing analysis, writing up outlines for new projects, finishing up reports on projects where we already have data collection, and so it's a pretty constant cycle between all of those different tasks. Great. Jill, what's your day-to-day like at Duolingo? There's always a new question every day, but I would say just from a day-to-day, I mean, part of my day is usually, you know, I'm on different teams, so I'm on a team that talks about, you know, that talks about new item development types, and then, you know, somebody from another team might ask me a question because I have expertise in something. So I have these meetings on different topics, typically day-to-day, and then I have time for my reading and writing and research, and I usually kind of block off. I usually like to do that in blocks, so I find blocks of time that I can do that. And then at Duolingo, one day a week is deliberately set aside just for focus for whatever you want to do, and people are not supposed to schedule meetings on those days, and so that's when I typically do more focused research. So that's kind of, without going into, you know, I get up, I have a cup of coffee, the details, that's kind of what my day looks like, and yeah. Well, I think that's really neat that the company has this policy of having a focus day of like a no meeting day. That's great. It used to be thinking Thursdays now it's focus Fridays. They moved it to Friday, which I think makes more sense. I think it's great, and people are really respectful of that. So yeah, I love that. I wish more organizations took that one. I think it's brilliant. Yeah. That's very special. And we're getting some some really good questions in the chat. So Francesca, if you want to take that question about kind of your day to day, sort of the pace of work, and maybe also, you know, work in what skills you really make most use of day to day? Sure. Yeah. So my job at Cal, I work on a number of different projects, and everyone on my team does like at least four or five at the same time. So I do a lot of balancing and switching between projects. Some other people at Cal work on, for example, the WEDA access for ELLs, that's a large scale test for English learners in K-12 schools, they work exclusively on that. But I jump around. So some of the like activities I do, I do, I have people I supervise. So I do some meetings, I manage those people and I manage project budgets and project plans. I do a lot of writing and editing of reports, research, presentations. I also sometimes, and for particular projects, actually write items. And I do a lot of item reviews. And then reviews of like radar training materials and executing research on those radar training materials. So it's a wide variety of things. But I guess I would say skills, definitely writing, strong writing skills are really critical and writing for different audiences. So sometimes we produce very technical reports, and sometimes we're producing things that are for educators to use daily in the classroom. And so they need to be really actionable and bite-sized. So being able to tailor to your audience. And then I guess project management is a big skill that I have. And that helps to move up at Cal and some other organizations depending on your goals. So general organizational skills and ability to kind of track different deadlines for yourself and others if you're managing people is a really important skill. And there's a question from Alfonso, I see that's just specific to Cal. There are not, oh, I can't say for the census, but you are not required to be a citizen to work for Cal. Each of our job postings probably has details on that. But we definitely do have people apply who have visas that are arranged in different ways. It would depend on the job, whether that person could be hired and then make those arrangements. Thank you. And you're correct. There is a citizenship requirement for the census, yes. And just to keep going with that question about visa requirements or citizenship, Jill, do you know anything about Duolingo and what's required there as far as citizenship or visa status? I really don't know. I mean, I really don't know. To be honest, I have, I've been there only a short period of time. So I haven't, but I think it's if you look in any of the job postings, it would say if they sponsor visas or not, usually they'll, I think job postings will say that kind of thing. Okay. And going to another question in the chat, which is mainly for Marcus, would a co-op position be a good way to get into the government or is a research assistant position seen as a more high ranking position? And maybe you could explain what a co-op position is for those of us who don't know too, like me. I'm also in that very familiar with a co-op position. If it's one that involves any sort of academic sort of research or one that involves research methodology, as long as you can explain to the hiring process that you're going through what your responsibilities were and the sorts of skills that you gained from it, I think either is probably an acceptable way to enter. Really the sort of job that we do is, in my branch, is very research oriented. And so we're just interested in knowing that you have those skills. So as long as you're able to express that. Okay. And going back to the skills question, I'd love to hear from you Marcus and also from Jill, what are the skills that you make use of day to day? And how might some potential applicant talk about the skills that they have that would be required in your position? Yeah, definitely. It's very similar, I guess, to the sort of research that I was doing in grad school. One skill that has come very much in handy is grant writing. So we do have different sponsors and we do rely on those sponsors to get the money to fund our projects. Some of the projects come to us one way or the other, but some we have to do that for. We also contract out some of our work in non-English languages where we don't have the skills for those. For example, right now we're doing research with American Sign Language and we don't have anyone who does ASL on our team. And so writing out this sort of necessary paperwork for contracting out different jobs is necessary as well. That's not something that was part of grad schools for me, but it's something that came with the sort of grant writing experience that I got. I mentioned earlier the sociolinguistic or cognitive interviews that skill has transferred over and really just academic writing because we are still academics in these positions and so it's that sort of writing skill that has really transferred over. You know, and Mark is just before I turn to Jill, I really appreciate how you say, you keep mentioning, you know, we're academics and you have this identity that you've retained as an academic, which I think some people who are, you know, at that choice point, at the end of their degree program, especially their doctorate, might be thinking, oh, I'll lose this part of my identity if I don't get work as an assistant professor, if I don't stay within higher education. But there are these opportunities to do research and to retain that part of your identity in many other types of organizations. Yes, absolutely. And, you know, even some of the research that I'm still doing on my own time has sort of helped me keep in touch with that sort of indigenous language sort of research that I was doing. It's stuff that I could continue to do and associate with my census identity, but then it has to go through a very long review process. And I don't think the Census Bureau can say this is our position unlogic for pronouns in Amazonian languages. So some of that is stuff that I still do on my own time as well. And I'm really impressed that you have your own time to do this with your busy job. How do you do that, Marcus? It's very slow. And right now, I recently, last week, submitted an abstract to a small conference. And it was really just the outcomes of my dissertation. So I'm still, a lot of that hasn't been published or presented anywhere. And so I'm trying to get that out there still. But I'm trying to keep in contact the language I worked on. There's one other researcher in Germany who does research on that language. So I stay in contact with him and try to keep up with what's going on with those languages as far as research goes. I think that's so important to, you know, this goes back to networking, to always staying in contact with people who are doing things of interest to you. And to kind of keep your, keep a foot in different worlds. And just keep those contacts alive, because you really never know what the future may hold and where you may move your career path. So that's another way that you can keep some options open and be on the lookout for other opportunities. So now I'd like to turn to Jill to ask you about that skills question about what you make use of day to day. And I also know that you have a hard stop coming up at a quarter after. So if you have also some last words of advice about anything I haven't asked about, please, you know, feel free, because I know you have to go soon. Yeah, so the academic writing for sure that and writing for different audiences, that's really, really important. And I can't even write for all the different audiences I need to write for. But that's really important. I mean, I have I bring a lot of different skills to the table, you know, given my background. So I've done a lot of work. So obviously, I'm wearing my linguistics hat more than I have in the past at ETS, because, you know, you, when you're thinking about what to put on tests, and you're thinking about the language and, you know, what, what kinds of language you want to elicit from test takers either spoken or written. So thinking about the construct of the different modalities in which you deliver a test and how you're going to capture it and how you're going to capture digitally because the test is all digital. And so if I want a writing task, you know, how am I going to grab the what NLP methods am I going to use to grab the information that I'm interested in about their writing, what NLP or speech features and we're going to use to, to understand, you know, to be able to evaluate automatically the speech that is produced from a test taker, that kind of thing. So it's sort of like I'm in, I'm bringing to the table my background and assessment, my background and writing studies, like a lot of work with it, you know, collaborating with writing studies people at ETS because my work was focused on automated writing evaluation. And also thinking about the construct, you know, we, you know, if we develop an item, what parts of the construct are we testing. So I'm also going into sort of wearing a bit my, my second language acquisition hat. I work with a lot of language assessment researchers. And that's not my, my focus, but obviously in a linguistics program, you're going to learn about sort of second language acquisition and socio linguistics. And so I'm kind of pulling that stuff back out of the hat. So I think you just never know what you're going to use, and you need to be flexible. And you need to really be open to learning new things. I think you just need to really open to learning new things and skill building. Yeah, so again, you know, there's, there are these opportunities for lifelong learning outside of academia and all manner of organizations. Anything else you'd like to leave us with, Jill, before you need to go to your next obligation? I think I've said all the things that I wanted to say. And I posted, I've actually posted the jobs if anybody is interested, you know, feel free to apply. And it's a new grad job, actually. So it's for people who are, I think it's, I think it's like people who are graduating anytime from August this year through the fall of 2022 that the range is on there. It's kind of, it's a bit of a unicorn. You don't see that that often. So there's that. And just, yeah, just keep an open mind and be flexible about, you know, what's possible and keep your networks really, really large and don't be ashamed to reach out to people and ask them to help you because everybody, you know, you may, if you're not doing it, someone else is. So I think that's that's sort of a really important thing to learn. And I'll, I'm going to look through the chat also, because I think they're, they're, I got a couple of direct messages here. So I'll sit here and answer them while you finish up here. Thank you. Oh, thank you so much for being here, Jill. If you have to hop off while we're still talking. Yeah, I'm good for, I'm good. I think I'm good. Okay. Great. Thank you. So I'll, I'll pose the same question to, to Marcus and Francesca just, are there any kind of last words to leave our audience with as they're considering opportunities to do applied research outside of higher education? Anything you want to hit home on with respect to networking, respect to skill building? I think I'll go back to, yeah, if you do have a chance to bolster your statistical skills, do that. And then also, I imagine, based on this panel and this, this larger event, there are a lot of people who are coming to that point or thinking ahead to that point of, you know, do I want to stay in academia or not? Or what are my options? And I would say, there's a lot of options outside of academia, if that's of interest to you, don't feel like, you know, you're restricted in that and look at organizations, you know, both government organizations, nonprofit organizations and for profit companies see what's out there. And you'll do others have said you'll do a ton of learning on the job. And I think Alex said too, so like, I think this is a advice for anyone, regardless of your background, not just linguistics, PhD students, but, you know, look at the qualifications listed in a job posting and obviously write your cover letter for those. But if you're not quite there, that's not necessarily a problem. A lot of people think you have to have 100% of all of the things. And that's absolutely not true. So don't short sell yourself in that way. Thank you so much for highlighting that that list of qualifications is a wish list. And we don't have to achieve 100% or even 80%. Right. In most, in most cases, the answer is just go for it. Go through that process. Marcus, what would you like to leave us with? Sort of piggybacking up of what Francesca just said, there are definitely jobs out there outside of academia, if that's what you're interested in. For the qualifications, I will say for if you go through USA Jobs, which is where most you'll find most government jobs, there are a list of questions that they use as sort of screener questions. So if you are able to answer yes to those questions without lying, that's a good way to get through to the next portion of the, I guess, get through the computer screener into a human screener. And I'll close also the link for the census postdoc positions as well in case anyone's looking into something like that. It is a very well laid out page that tells you exactly what is being looked for for those sort of positions. So basically once you submit that in someone, the census sees your application and says oh, I would really like someone to do research on our team for that, but it will scoop you up onto their team. So definitely if you have questions, I posted my email earlier in the chat. I'll be out of the office the rest of this week, but I'll be happy to get back to you once and back next week. Great. Well, I so appreciate all of you. Francesca, work is Jill, your time spent here has been so helpful in giving some hope, I think to those who are watching now and who may watch later that there are ways to do research outside of higher education that can be very fulfilling and can be in any structure of organization, whether it's not profit for profit or in the federal agency. So I'm going to stop recording now and just thank you so much for your participation here today at the linguistics career launch.