> ## Knowledge Base Index > Fetch the complete knowledge base index at: https://help.pandectes.io/sitemap.xml > Use this file to discover available pages before exploring further. > Pure-Markdown content can be obtained by appending a '.md' suffix to the content URLs listed in the sitemap (without the trailing slash). # Bots and crawlers: control banner behavior and consent records ### Overview The **Bots & crawlers** settings let you control how the Pandectes consent banner behaves for automated visitors such as search crawlers, SEO tools, scanners, monitoring services, and other bots. You can let an automated visitor follow the normal consent flow, or hide the banner and apply either a granted or denied consent state without creating a consent record. This feature is available on the **Enterprise** plan. ### How it works By default, automated visitors follow the regular banner flow, just like other storefront visitors. If a bot sees the banner and interacts with it, its choice can be recorded in the consent log. Repeated automated interactions can therefore affect statistics calculated from those consent records. For example, if automated visits consistently accept: | Traffic | Visits | Accepted | Acceptance rate | | ---- | | Human visitors | 100 | 70 | 70% | | Automated visitors | 100 | 100 | 100% | | Combined | 200 | 170 | 85% | This example is illustrative. The effect depends on how automated visitors interact with your storefront. Pandectes provides three possible behaviors: | Behavior | Banner | Consent-controlled scripts | Consent record | | ---- | | **Show the banner** | Shown normally | Follow the regular consent flow | Recorded if the bot makes a choice | | **Hide and treat as consent given** | Hidden | Run as if consent was granted | Not recorded | | **Hide and treat as consent denied** | Hidden | Remain in the denied state | Not recorded | ### Key points to understand * **Showing the banner uses the regular flow.** If the automated visitor makes a consent choice, that interaction is handled like another banner interaction. * **Both hidden modes keep the automated visit out of the consent records.** * **Treat as consent given** is useful when an automated tool needs to inspect the storefront with consent-controlled resources enabled. * **Treat as consent denied** is useful when a tool needs to inspect the storefront in its pre-consent or denied state. * The general **Known bots & crawlers** setting applies to bots recognized by Pandectes. * **Specific bots** can override the general behavior for matching user agents. * Existing consent records are not retroactively changed when you update these settings. * These settings control Pandectes banner and consent behavior. They do not block a crawler from requesting your website. ### Configure known bots and crawlers Open: **Settings > Banner > Advanced > Bots & crawlers** Under **Known bots & crawlers**, enable: **Hide the banner and treat as consent given** When enabled, Pandectes applies this behavior to automated visitors it recognizes: * The banner is not displayed. * The visit is treated as having granted consent for Pandectes-controlled behavior. * No consent record is created for that automated visit. Use **See the list** in the app to review the bots and crawlers currently recognized by Pandectes. Keeping the list in the application avoids relying on a static documentation list that may become outdated. If this option is disabled and no specific rule applies, the automated visitor follows the regular banner flow. ### Configure specific bots The **Specific bots** section lets you define behavior for individual user-agent patterns. Add one value per line under one of these groups: * **Show the banner** * **Hide and treat as consent given** * **Hide and treat as consent denied** Specific entries override the general known-bot behavior when they match. For example: * Add a testing crawler to **Show the banner** when you want to inspect the normal consent experience. * Add a cookie-scanning tool to **Hide and treat as consent given** when it needs consent-controlled services to run. * Add an auditing tool to **Hide and treat as consent denied** when you want it to inspect the storefront before consent. ### How user-agent matching works Specific entries are matched against the visitor's user-agent string. * Enter one pattern per line. * Matching can occur anywhere in the user-agent string. * Matching is case-insensitive. * Use lowercase entries for consistency. For example, an entry such as `googlebot` can match a user agent containing that text without requiring you to enter the complete browser-style user-agent value. Keep patterns as specific as possible. Generic browser terms such as `mozilla`, `chrome`, `safari`, or `webkit` can also appear in normal shoppers' user agents and may therefore match real visitors. If Pandectes flags an entry as unusually broad, review the pattern before using it. ### Using the feature with cookie scanners A cookie or storage scanner can only observe resources that actually load during its visit. If consent-controlled scripts remain disabled, the scanner may not reach everything that would normally load after consent. Assigning that scanner to **Hide and treat as consent given** lets the storefront run in the granted-consent state without creating a consent receipt for the scanner. This does not guarantee that every scanner will find every cookie or storage item. Results can also depend on the scanner's JavaScript support, page coverage, authentication, storefront behavior, and other technical conditions. ### Using the feature for blocked-state audits To inspect how the storefront behaves before optional consent is granted, add the auditing user agent under: **Hide and treat as consent denied** Pandectes then keeps its consent-controlled behavior in the denied state while hiding the banner from that automated visitor. This can help when checking whether integrations and scripts managed by your consent configuration behave as expected before consent. It is a technical testing state and should not be treated as a determination that the store meets any particular legal requirement. ### Does this block crawlers from the store? No. The **Bots & crawlers** feature controls what consent state Pandectes applies to an automated visit. It does not prevent the crawler from requesting your pages and does not edit your `robots.txt` configuration. Crawler access, indexing instructions, firewall rules, and CDN bot controls are separate configurations. Google, for example, documents `robots.txt` as a mechanism for providing crawling instructions to supported crawlers. ([Google for Developers][1]) ### Is user-agent matching proof that a visitor is really that bot? No. A user-agent value identifies what the requester claims to be, but it is not authentication. Google explicitly notes that crawler user-agent strings can be spoofed and provides IP-address and reverse-DNS verification methods when genuine Google crawler identity needs to be confirmed. ([Google for Developers][2]) Use the Pandectes setting for consent-banner behavior. If verified crawler identity is important for security or access control, handle that separately at the server, CDN, or infrastructure level. ### Does treating a bot as consent given determine whether tracking is legally permitted? No. This setting defines the technical consent state that Pandectes applies to an automated request. It does not determine whether a particular request, script, or third-party service is subject to privacy or other legal requirements. Review your storefront configuration and applicable requirements separately.