 Today I put some objectives together of things that we're going to talk about. We're going to talk about some regulatory and sub-regulatory, as I have learned, what the new terminology is around laboratories, the differentiate between the various areas of health and human services area of oversight, talk about dean status, and talk about various... talk about some stuff about analytical validation, especially related to reagents, kits, instruments, and laboratory-developed assays, tests, or procedures. I'll use assays, tests, and procedures somewhat interchangeably for laboratory-developed tests. So let's first talk about regulatory versus non-regulatory entities. So to make it sort of in my terms, regulatory is there's a law around it. And if you don't follow the law, you can go to jail. And I put jail in parentheses. It's not most labs that don't follow the law don't necessarily go to jail, but they do have huge fines that they pay. The law is in the Federal Register. It's subject to notice of proposed rulemaking. And it allows comment. Usually, when you put a notice of proposed rulemaking out there, you can make a comment, or the public can make comments before it becomes a final law. If there's a non-regulatory oversight of laboratories, I haven't listed them all, but the Association for Molecular Pathology, the American College of Medical Genetics and Genomics, the College of American Pathologists, National Comprehensive Cancer Networks, they develop guidelines that determine practice of medicine and how laboratories should be, what they should be doing and how they should be doing it. They have no oversight capabilities, but they do influence on how laboratories are regulated. So I have to say this slide, this is a little bit of my simplified version of how to explain the Secretary of Health and Human Services, which is HHS, who is appointed by the President, oversees the FDA, clinical CMS, Centers for Medicaid and Medicare Services, and the National Institutes of Health. So I've sort of called the FDA has oversight over manufacturers. CMS is more clinical, and NIH has researched. These are big buckets. There's some overlap around in this area, but just to sort of give you an idea. So CMS, well, let's start with the FDA actually. FDA, we'll talk about it in more detail, but has put out guidelines, but they're actually going to oversee clinical laboratories, and that's why I have a dotted line. FDA does oversee clinical laboratories that are involved in transplant and transplant testing. I haven't put that on there, but because I've made it sort of more as a big bucket type thing. CMS oversees payment for Medicare, the CLIA program, which we'll talk about in more detail for clinical laboratories, and oversees the CDC and the Public Health Laboratory, and CDC has oversight over the Public Health Laboratories. Again, these are broad buckets. There are lots of details that I haven't gone into, but just to sort of give you a sort of background to areas of health and human services. So first we're going to talk about CLIA. So the Center for Medicaid and Medicare Services or CMS under there has CLIA 88, which was responsible for clinical laboratory testing. Under CLIA 88, the Clinical Laboratory Improvement Amendment, they talk about quality assurance standards, proficiency standards, record maintenance, personnel qualifications, and quality control. This is all what is in law. CMS also oversees Medicare reimbursement, and one thing to think about if you're not doing medical testing, you don't necessarily need a CLIA license. So some examples I have are ancestry, paternity, and recreational genetics, and things I think about in recreational genetics is if you want to find out what your dog, you have a nice little fluffy dog that you got from the pound and you want to try to determine what the background of that dog is, that sort of recreational genetics, or I'm trying to think of some other things that sort of fit in there. If you want to know if you have blue eyes and want to get a test around that that doesn't necessarily relate to medical care, that is recreational genetics. Under CLIA 88, or the Clinical Laboratory Improvement Amendment of 1988, the statute is an amendment to the Public Health Service Act in which Congress provides the federal program for certification and oversight of clinical laboratory testing, and 42 CFR 493 is the law. To have a dean status, to be dean status, you have to have higher standards than what's in the law in CLIA, and some examples include New York State Department of Health and the College of American Pathologist. So if we look at CLIA CAP in New York State, CLIA is really overseas, any medical test, and that is built to CMS. The College of American Pathologist, or CAP, most laboratories obtain CAP status or accreditation, and this suggests a higher quality than CLIA, and CAP also administers a proficiency testing program. New York State Department of Health is only required for those labs patients who reside in the state of New York. It also suggests a higher quality than CLIA, and they administer a proficiency testing program. One thing that I've had to learn in the laboratory world is that the terminology, or CLIA speak, is a lot different than FDA speak, which is maybe different than normal person speak. So in CLIA terminology, or CLIA speak, the laboratory director is the CLIA license holder. There is only one laboratory director per CLIA license. There is a technical supervisor under CLIA, and that's often referred to as a director. So when you talk to a laboratory and you say, I want to talk to the director, a lot of times you're going to get the technical supervisor for genetics or the technical supervisor for cytogenetics or somebody else. And they're really a laboratory director, and usually a PhD or an MD or an MD PhD, but under CLIA they're called a technical supervisor. A general supervisor is a supervisor, whether it's in CLIA or normally. A technical consultant can be an MD, PhD, or a DO, but it cannot be a genetic counselor. So often in genetic labs, genetic counselors really serve to guide laboratory testing and help physicians and people order correct genetic tests, but under CLIA they can't be a technical consultant to the laboratory. And then there's the technologies that perform the work and the testing in the laboratory. CLIA terminology or CLIA speak, too, is the laboratories have different classifications. So there's called physician office laboratories that do a lot of either low complexity or point of care testing. Some examples are the strep test, the pregnancy test, that there's qualifications for the person who administers that there's education qualifications for people who can administer those point of care or the low complexity test. Then there's medium complexity laboratories. There are that are medium complexity. One that I can think of is I think there's one for factor five light in from a company, but most all genetic tests fall in a high complexity laboratory, and especially if you think the newer technology, such as next generation sequencing, really fall into this high complexity, that the level of education for the people that perform the test day to day must be much higher than for a glucose test or a strep test. The next thing that I'm going to talk about is proficiency testing, PT, or alternative assessment. It is required two times per year for every analyte performed in the laboratory. Both the College of American Pathologists and the New York State Department of Health have proficiency testing programs. There are other programs available, because if you look at all the genetic tests that are currently out there, there's not a proficiency testing program for every genetic test that is out there. But laboratories often may supplement what's available at CAP in New York State from other countries, such as Europe has a program EMQN. There's one out of Germany called INSTAD, and then if there's no laboratory, must do alternative assessment. Most commonly, this is done by exchanging samples with another laboratory, and if you're the only laboratory in the country that performs that test, the method to do is you retest samples in a blinded manner in your own laboratory. There have been some entities that says that genetic testing is not required, and that's not quite true. The number of tests found in subpart one in the CLIA regulations. But also in CLIA, CLIA requires laboratories to take steps to assure the accuracy of testing in lieu of proficiency testing samples. And CLIA requires that at least twice annually that you verify the accuracy of a test or procedure that you perform that is not listed there. So lawyers are people playing with words saying that there's not proficiency testing required twice a year. It is required twice a year for every analyte at a laboratory test. I'm not going to go in too much detail about the College of American Pathologists, about this is a slide from them, just to familiarize yourself with them. But they have been offering laboratory accreditation since the mid-60s, and they really helped laboratories achieve a high standard of excellence. There's more than 7,000 cath-accredited labs in more than 50 countries. So there's a lot of laboratories that participate in the CAP and the CAP Proficiency Program. Right in too much detail, I'm just going to hit on a few highlights to go over. One is CAP actually comes into the laboratories every other year. They do by enial inspection. And the year that you're not inspected, you do a self-inspection or a self-assessment. How laboratories gain excellence is they have checklists, and these checklists are updated every year. And for the laboratories, they're really open for tests. Have you done your instrument-to-instrument comparison two times each year? Yes or no? And if the answer is no, you get a deficiency, and you have to correct that. Have you checked all your reagents that they're used within their expiration date? Yes or no? No, there's a deficiency. Have you recorded the temperatures on everything that requires? So these checklists are actually like 100, there's like 100 or more questions usually for each checklist, around 100 questions for each checklist. So it's very detailed that the labs must do. The inspectors for CAP are actually other laboratory directors and personnel that if your lab gets inspected, then part of the CAP process is that CAP will then recruit you to co-inspect another laboratory. So I think that's about it on the highlight of that slide. This is just to say, again, it's a two-year cycle where one year they come in and inspect and then the following year that you do a self-assessment. The other thing about these inspections when the inspectors come from CAP is actually it's an unannounced inspection. So in a three-month window, they can show up any day in that three-month window. And not that labs aren't always ready to be inspected, but you never know when your CAP inspector is going to come within that three-month window. For laboratory accreditation, it includes the director and the personnel. It also involves the physical resources. Do you have, is the lab safe? Do you have adequate space? Do you have a quality manager? The third standard is around quality management. So to make sure that there is quality testing and there's patient safety. And then the fourth standard is around administrative requirements with checklist, inspection, self-assessments, and record and documentation. This is just a list of some of their checklists. What I have highlighted in red, the three in red is all laboratories. Every laboratory has three checklists that every laboratory is in common to all the laboratories. And then depending on your laboratory, whether it's a genetics laboratory that does pharmacogenetic testing, which follows under molecular pathology or cytogenetics, and even physician office labs or labs that do point-of-care testing, there is a checklist around that. So there's this very detailed checklist around the type that are tailored to the type of testing that is done in that laboratory. So the value that you actually have a peer come in and do it is a peer is familiar with the testing that's performed in a laboratory. So if I went to a urinalysis laboratory, I know nothing about urinalysis. So I have other geneticists that come into my laboratory and inspect me, and then I go inspect other laboratories that perform molecular pathology or genetic testing. So you have somebody who's familiar with the type of testing that's performed, and make sure that lab is performing at a high quality. We're going to switch and talk some about the FDA. I have never worked for the FDA, but I'm going to talk about what's going on with the FDA from attending meetings. So under the FDA, their mandate under law is to ensure that a device is safe and effective. They have different routes to do that. They have pre-market approval or a PMA, which is any new device that goes through the FDA. And if the device gets pre-market approval, that device is called FDA-approved. Then they have what's called 510K, and that's equivalent to a predicate device. So if a DNA sequencer, I'm going to give an example such as the Illumina DNA sequencer, that's for cystic fibrosis, that went through as a new device that got FDA-approved. If there's another DNA sequencer that then goes through the FDA, that device is compared to the original device and then goes through under an FDA clearance and gets called FDA-cleared. The FDA has what they call the NOVO 510K, which is a lower-risk new device, also gets called FDA-cleared once it's approved. So an example is that a new assay may be on an approved platform. So if cystic fibrosis went through on this Illumina sequencer, another CF assay that goes through or another new assay on that sequencer, then that's the NOVO 510K. And then there's a humanitarian device exemption. So these are four devices that are for really rare stuff or new and emerging infectious disease. One example is like the Ebola. When the Ebola crisis was happening, there was a test that was approved in Nigeria for testing for Ebola. There is, actually, I think just recently for Zika virus, there was a test that got approved under the humanitarian device exception for Zika virus. The device that goes through the FDA requires QSR or Quality System Regulations. These regulations are very similar to ISO, the International Organization for Standardization Regulations, but they've got a U.S. spin on them a little bit. So the only thing to remember is any, and the FDA will tell you this too, is any device that is FDA-approved or FDA-cleared does not determine reimbursement. So there may be devices, there may be tests out there that are FDA-cleared or approved, but there may be no reimbursement. One such example was the UGT-1A1 assay that Hologic had provided at one time. It was an FDA-cleared, I believe it was an FDA-cleared device because it was based on a predicate device, a new analyte on a predicate device, but there was never any reimbursement for that test. And eventually, the Hologic pulled it from the market due to lack of utilization. So I did mention briefly that the FDA has issued a draft guidance for the oversight of laboratory-developed tests or laboratory-developed procedures. It was issued, they gave on July 31st in 2014 a 60-day notice to Congress as required by law. And then it was placed in the federal register in October 2014. The goal of the FDA was to ensure analytical and clinical validity. The FDA has said in 2016 that their intent is to finalize the guidance. I don't know the status of this and can't comment on that. So where we are in the FDA oversight of LDTs is we're in this sort of Neverland right here before year one. For their guidance, they plan to over, once the guidance is finalized, then the clock starts ticking. And the FDA plans to start with high-risk laboratory-developed tests to do pre-market review and then take nine years to complete the full implementation. We're minus one area. We're still waiting for the final FDA guidance of laboratories. The FDA restricts off-label promotion and tests. You can only talk about what's in the product label. Under CLEA, we are allowed to provide a clinical consultation. This may or may not be allowed once there's FDA oversight that are quite different from QFR regulations under FDA and how that's going to be resolved as you have to be determined or decided. The laboratory doesn't have a package insert for its tests. They have SOPs, they have service directories, but they don't really have a package insert and how that will be regulated will be determined. One of the bigger differences is laboratories are part of the practice of medicine and have malpractice insurance in laboratories. A laboratory is then required to go under FDA. Laboratories will have to have product liability insurance. When I look at a lot of this stuff, there's a lot of duplication, but I also see a lot of expense going to be added on the laboratory test that will then be passed on to the patient because this is going to all cost a lot of money. Professional organizations or societies really have no regulatory oversight. As I called it before, it's more sub-regulatory oversight. They determine standards of care around guidelines, but they really have no oversight of laboratories. We're going to talk a little bit about tests which includes all or most of the test components and a written procedure. In step one, you do this. In step two, you incubate. In step three, you do this. In step four, you do that. That is an FDA-approved kit. I gave an example here such as the aphymetric cytoscan, DX assay that's used for cytogenomic arrays, which is FDA-cleared, and it went through as a DeNovo 510K. The package insert for that test, the laboratory has to validate those changes. The Illumina FDA-cleared MySeq instrument, patients are also FDA-cleared or approved. If I use it for my favorite gene, we'll call it the Vicki gene. If I use it for the Vicki gene, then even though I'm using an FDA-cleared instrument and FDA-cleared reagent, because that gene is not indicated in the package insert, I have to validate any changes that I make. So I have to validate that from beginning to end to make sure that works for that test. There are, and there's very few of them that are called Investigational Use Only or IUO. It includes all of them. There's very few of these on the market. Usually these type of tests are undergoing initial development and validation with clinical studies. So when a company is developing a new test and they're getting ready to submit to the FDA and they're doing side-by-side trials in laboratories, they may have an investigational use only. And they're not to be used as a diagnostic procedure without confirmation of the diagnosis by a second medically established diagnostic device or procedure. If you modify anything in that kit insert or package insert that comes with that test, again, you have to validate any changes that are not in the package insert. I will concede that laboratories have to consult with their legal advisors. Some people, under the eyes of the FDA, this is still considered an investigational use only device. Though some legal, some entities, if your legal advice may say it would be a laboratory-developed procedure. Then there's research use only test out there. It includes all or most of the test components and a written procedure. It's intended for performing basic science or animal research and not intended for diagnostic testing. One thing to research use only test is good manufacturing practices are not required. So there may be some quality control issues around research use only test. And the FDA does not intend that to be used as a building block for laboratory-developed tests or assays. You can modify a research use only. It is still considered a research use only test. And again, the institution where you work may consider these laboratory-developed procedures. I'm telling you what the FDA considers it's still to be a research use only. There are test kits that you can get from Europe that are CE-marked. And it's okay to use those in Europe, but in the U.S., if the FDA has not reviewed them, the FDA considers these to be research use only. It's better for forensics use only that are used in clinical laboratories. One such thing is identity testing, such as paternity testing. That is okay for forensic use, but may be considered research use only in clinical tests because the FDA does not oversee... As far as I'm aware, the FDA does not oversee forensic test and test kits. Laboratory-developed procedures or tests or LDTs or LDPs, the components of the assay are determined by the laboratory and may use kits for part of the assay. And I'll explain how they may use kits in more detail. So we're going to talk about non-kits reagents. So there are ASRs. Usually this is a single reagent, such as an antibody or a nucleic acid probe that can be used by laboratories to validate a functional clinical assay. One such example that I could think of are fish assays that are not FDA-cleared or approved, that they use analyte-specific reagents for that intervalidated end test. Under the ASR rule, the ASRs must register their establishment and list their reagents with the FDA. The reagents must be manufactured under quality system regulations and the manufacturer can provide information about the reagent to the user, such as identity or purity, but cannot provide any specific performance characteristics. There are research use-only reagents. And one such example, what I think of are, it may include a component of a test with a written procedure, such as a DNA extraction, but it may not be specific to an analyte. In a previous laboratory I worked in, actually all our DNA extractions were FDA. We used a kit for methylation studies that was not FDA-cleared, but it was not specific to the test. It was only specific to helping with methylation. And we used it as a component of our test. There are general-purpose reagents, and these are reagents with general laboratory application, and they're not labeled or intended for specific diagnostic applications, such as some examples are buffers, restriction enzymes, alcohols, laboratory equipment, laboratory software, and automated power systems cannot be considered as a general purpose or considered as general purpose. So laboratory-developed tests or procedures are completely developed in the laboratory. They may include the following kits or reagents, kits without a specific analyte, such as DNA extraction kits. They may include RUO reagents, may or may not. I'm just giving examples, ASRs and general-purpose reagents. So when you have a laboratory-developed procedure or test, you have to undergo analytical validation, as well as if you make any modifications to an FDA-cleared or approved test. So we're going to talk a little bit about the steps and things that you must do in analytical validation. In validation or verification, it's really proving you can detect what you say you can't detect in the most simplest terms. Under verification that the specified requirements have been fulfilled, it's a one-time process to confirm the test performance or characteristics. In the U.S., we use this for all FDA-cleared or approved assays where we're not changing what's in the package. We're not changing any steps from what's in the package insert. In Europe, they actually use this term for if there's an existing test or technology in the lab to add in additional analytes. So if you're doing a sequencing testing for cystic fibrosis and you want to add in a sequencing test for something else, when you add in that next gene, that's verification in Europe, but that is not the case for the U.S. It produces an unmodified FDA-cleared or approved test system must verify the following performance characteristics before reporting patient test results. You must confirm the decision reference interval and reportable range. Under validation, the definition is that requirements for specific intended use or application have been fulfilled. In the U.S., we use this for all laboratory-developed tests or procedures, or modifications to any FDA-cleared or approved assay, as well as any modifications to any assay that has a really... We talked about RUO, but you still have to validate those performance characteristics. In Europe, that's used for any novel test or technology. So under clear law in the Federal Register, each laboratory that modifies an FDA-cleared or approved test system that introduces a test system not subject to FDA clearance or approval must establish, that's the words I have in, must validate, establish, and I'm going to substitute the word invalidate, the following test performance characteristics before reporting patient test results. Accuracy, precision, reportable range, reference interval, analytical sensitivity, analytical specificity, and any other characteristics necessary to perform the test. This paper about analytical validation, it must go through validation. We have... Really the point of this slide is to point out that there's usually ongoing validation in the laboratory. Very rarely is any test, whether it's FDA-cleared or approved or lab-developed tests, is completely set in stone. There is an SOP. The SOP must be followed in the laboratory, but often there's continual improvement of maybe a physician add the new... what would like a new specimen type accepted by the laboratory. So before the laboratory can accept that specimen type, they have to do... add in, do a validation before they can accept a new specimen type. So usually there's some sort of ongoing validation. We've had a vendor that just continued a specific reagent or a specific part of the test, and so then you have to revalidate that replacement before you can perform clinical tests. So that's what I mean about ongoing validation. I've been doing analytical validation for a long time. I think the words are very confusing. What does sensitivity mean? What does specificity mean? People talk about it and refer to it as different things. And I'm going to give you my definitions of it. So accuracy is trueness of a measurement. It's really a formula. Accuracy is equal to the true result over the true result plus the false result. And I really like the bullseye analogy here that I've provided that an accurate and precise result is actually hit that bullseye. But not quite precise is near the bullseye, is around the bullseye, but not dead square in the center of the bullseye. You can have an assay that's not accurate, but very precise, which is the dots are clustered together, but they're not in the bullseye, so it's not accurate. You can have a not accurate and not precise where there's lots of scatter and one or very few samples hit the bullseye. In the previous slide, it's really the repeatability and the reproducibility of a test result. This can vary some within the technologist, so just setting up the test day in, day out, a different day, they can have slight variability in what they do and how they do things. You can have variability between technologists within a run and between runs. So during analytical validation, you test for all this to ensure that the test is accurate and precise. Bias of measurement is an estimate of systemic measurement errors. So there's three types of bias. There's constant bias. So if I look at an example, I don't do chemistry, but I won't give that as an answer, but I'll talk about Fragilex. So if we do Fragilex testing, it's a triplet repeat disorder, and you have to give the number of repeats. If there's a constant bias, if the term that means that the lab is always systematically off from what the true number is, so that's a constant bias. A proportional bias is where it may be accurate at one end of the spectrum, but at the other end of the spectrum, it's off, and that's a proportional bias. And then a constant and proportional bias, it means that more commonly we see this as where most of the patient samples are. It's very accurate there, but when you get to the far extremes, it's not accurate. The ability of a test to detect a mutation or when the mutation is present. This is really... it's an equation. It's the true... sensitivity is equal to the true positive or the true positive plus the false negative. In my mind, when I've always done analytical... when I've done these calculations, it's very important to document confidence intervals, and I'll show you an example. Some people refer to analytical sensitivity as the lower limit of detection, and we'll use that, especially if you're doing infectious disease testing. Talking about lower limit of detection, I'm talking about lower limit of detection. Analytical sensitivity is different in my... is something a little... slightly different in my mind. Analytical specificity is the ability of a test to give a normal negative result in specimens without a mutation being tested. So it's the true negatives over the true negatives plus the false positives. Some people refer to... especially immunology is a good example where analytical specificity talks about the ability of the test to detect the analyte without cross-reacting to other substances. And if I talk about cross-reactivity, I talk about cross-reactivity when I'm doing analytical validation. I don't talk to it... don't refer to it as analytical specificity. I'll use an example. If I test in my analytical validation, if I obtain 25 positives and 75 negatives, and I do my test, and I get 25 positives in those positive samples and 75 negatives in those negative samples and no false negatives and no false positives, my analytical sensitivity is 86% and my analytical specificity is 95%. And it's a function of these numbers. But I mean, the analytical sensitivity is 100% and specificity is 100%. But the confidence interval is 86% to 100% and 95% to 100%. Troy, that only tests 15 samples and it may be a really rare assay. It's not... people say my analytical sensitivity is 100% and specificity is 100% or greater than 99%, but they've only tested 15 samples. But if you look at their confidence intervals around that, if they only test five positives and those are positive, that their sensitivity is 56% to 100% and their specificity is 72% to 100%. So there's differences if you don't test a lot... larger numbers of samples, so there may be instances where that doesn't make sense because the test is so rare. To go over these very quickly is how to verify an unmodified FDA assay as you go through... you perform according to the specifications, use positive and negative samples, and then you verify that the test is working as it is expected to. When you validate a laboratory-developed test or procedure, you first have to develop a procedure that is fit for purpose. And then you use multiple positive and negative samples to confirm that you detect what you say you're going to detect. So to finalize it, it's important to understand the regulations of clinical testing and performing the laboratory services. Thank you.