 Hi, I'm Daniel Weisberg, Search Advocate at Google. And today, I'll provide a short introduction to how Google crawls pages and provide definitions for terms such as crawl rate, crawl demand, and crawl budget. Then, I'll dive into the Search Console CrawlStats report, which provides data on crawl requests, average response time, and more. The report contains information for several Google crawlers, but I'll focus on the search aspect of it. As a disclaimer, this presentation will be more relevant if you work with a large website. If you have a site with fewer than a few thousand pages, you do not need to worry about it. But if you ask me, never hurts to learn something new. Who knows? Your site may become the next big thing. The crawling process begins with a list of web addresses from past crawls and site maps provided by site owners. Google uses web crawlers to visit these addresses, read the information in them, and follow links on those pages. The crawlers will revisit pages already in the list to check if they have changed, and also crawl new pages they discover. During this process, the crawlers have to make important decisions, such as prioritizing when and what to crawl, while making sure the website can handle the server requests made by Google. Crawl demand is how much the content is desired by Google. It is affected by URLs that haven't been crawled by Google before, and by Google's estimation on how often content changes on the non-urls. The successfully crawled pages are processed and passed to Google indexing to prepare the content for serving on Google search results. Google wants to make sure that it doesn't overload your servers, as a good citizen would do. With that in mind, Google computes your site's crawl rate, which represents the maximum number of concurrent connections a crawler may use to crawl your site. This value is calculated by Google periodically, based on your site's responsiveness, or in other words, how much crawling traffic it can handle. If the site is quick and consistent to respond to crawlers, the rate goes up if there is demand from indexing. If the site slows down or responds with server errors, the rate goes down, and Google crawls less. In rare cases where Google crawlers overload your servers, you can set a crawl rate limit using the crawl rate settings report in Search Console. Taking crawl rate and crawl demand together, we can define crawl budget as the number of URLs Google can and wants to crawl. My colleague, the Gary, wrote a detailed blog post about crawl budget in the search central blog. Check it out. To find out how often Google crawls your site and what the responses were, you should use the Search Console crawl stats report, which provides statistics about Google's crawling behavior. Here are some questions you can answer with the data provided. What's your site's general availability? What's the average page response for a crawl request? And how many requests were made by Google to your site in the last 90 days? Let's dive in to learn more about the report. Log in to Search Console and find the settings page where you can open the crawl stats report. Know that this report is only available for properties at the domain level. It is not available for properties that include a path. In the summary page, you'll find a lot of information. The main elements are the crawling trends chart, the whole status details, and the crawl request breakdown. The chart shows the transfer three metrics. The total crawl request for URLs on your site, whether successful or not. Request resources hosted outside of your site are not counted. So if your images are served on another domain, such as a content delivery network or CDN, they will not appear here. The total download size from your site during crawling. If Google cache the page resource that is used by multiple pages, the resource is only requested the first time. And the average page response time for a crawl request to retrieve the page content. This metric does not include retrieving page resources such as scripts, images, and other linked or embedded content. The response time does not account for page rendering time. Look for major spikes, drops, and trends over time in your data. For example, if you see a significant drop in total crawl requests, make sure no one added a new robot's text rule to your site. Or maybe your site is responding slowly to Googlebot. This might be understood as a sign your server cannot handle all the requests. Another example would be a consistent increase in average response time. This might not affect your crawl rate immediately, but it's a good indicator that your servers might not be handling all the load. Ultimately, this may affect user experience, too. The host status is an easy way for you to check your site's general availability in the last 90 days. If you verify your website as a domain property, you will see all child hosts separately here. This can be very handy to evaluate all your host performance in one place. When you click to get the host status details, you'll find three categories. Errors in this section mean Google cannot crawl your site for a technical reason. Robot's text fetch tells you the failure rate when crawling your robot's text file. Your site is not required to have a robot's text file, but it must return a successful response 200 or 404 when asked for this file. If Googlebot has a connection issue, for example, a 503, it will stop crawling your site. DNS resolution tells you when your DNS server didn't recognize your host name or didn't respond during crawling. If you see an issue here, check with your registrar to make sure your site is correctly set up and that your server is properly connected to the internet. Server connectivity tells you when your server was unresponsive or did not provide a full response for your rel during a crawl. If you're seeing spikes or consistent connectivity issues here, you might need to talk to your provider about increasing your capacity or fixing availability issues. If Google found at least one of those errors on your site in the last week, you will find a warning alert in you of problems last week. Dig deeper into the error and fix it. If Google found at least one error on your site in the last 90 days, but it occurred more than a week ago, you will see a warning that you had problems in the past. You should check your server logs or contact a developer to review what the problems were and decide whether you need to take any action. If Google didn't find any crawl availability issues on your site in the past 90 days, you're all green. Good job. The crawl request cards show several breakdowns to help you understand what Google Crawlers found in your website. There are four available breakdowns. Crawl response shows the responses that Google received when crawling your site. Common response types would be 200, 301, 404, or server errors. Crawl file type shows file types returned by the request. Common file types would be HTML, image, video, or JavaScript. Crawl purples could be discovery when a URL is new to Google or refresh for a re-crawl of a non-page. And Googlebot type shows the user agent used by Google to make the crawl request. For example, smartphone, desktop, image, and others. Click any row to drill down to that value and review its trends and specific sample URLs. Check out the Help Center article about the crawl stats report to learn more details about it. To recap, crawl budget is the number of URLs Google can and wants to crawl websites every day. It is relevant to large websites where Google needs to prioritize what to crawl first, how much to crawl, and how frequently to re-crawl. To help you understand and optimize Google's crawling, you should use the search console crawl stats report. Use the summary page chart to analyze crawling volume and trends. Use the whole status details to check your site's general availability. And use the crawl request breakdown to understand what Googlebot is finding when crawling your website. And make sure to check my search console training YouTube series to learn more about how to use the product to succeed on search. That's all the crawling I'm doing today. Stay tuned.