Do AI Crawlers Read Your IR Site's JavaScript? | AIO & LLMO for IR (Technical)

You ask a generative AI about your own company, and the earnings figures come back right, yet the briefing materials and the timely disclosure list that you know are on your IR site are not referenced at all. When you meet an answer like that, what should you suspect?

Before you question whether the content is correct, there is something to establish. Is that page in a state where the AI side can retrieve it at all?

This article deals with that single point. Among corporate websites, IR sites are structurally unusual, and that unusualness is exactly where the question of fetchability arises. The field also carries two simplifications: "AI cannot read JavaScript" and "Google renders it, so we are fine." On the basis of observation, this article rejects both.

• Which structures on an IR site become points that need checking when seen from the AI side
• What a 41-day controlled experiment and large-scale server log observation show, and what they do not show
• Why "Google renders it, so we are fine" does not square with the observations
• What it means for robots.txt design that AI crawlers are split by purpose
• Where the line falls between what can and cannot be asserted about embeds, third-party components and PDFs
• Where the timely disclosure regime and XBRL are separate from the technical structure of an IR site
• A procedure for checking the fetchability of your own IR site, in a form you can take to your vendor tomorrow

Written for IR staff at listed companies and for anyone whose job includes talking to IR site vendors and IR support vendors. No engineering background is assumed. Technical terms are explained where they first appear.

• In the 41-day controlled experiment published by Search Engine Land on August 19, 2026, none of GPTBot, ClaudeBot, OAI-SearchBot, ChatGPT-User, Meta-ExternalAgent, Amazonbot or PerplexityBot reached a single hierarchy page behind a JavaScript-inserted link (Bingbot reached three, which the author interprets as discovery by another route)
• In the same experiment, Googlebot, the crawler that builds the search index, reached 2% of the 293 pages in the JavaScript link group and 5% of the 748 pages in the HTML link group
• The 33rd "Fact-Finding Survey on IR Activities" by the Japan Investor Relations Association (published May 14, 2026; 948 responding companies) reports use of IR support vendors at 81.9%

The subject of AI crawlers and JavaScript is not peculiar to IR. The underlying mechanics are covered in What Is LLMO (Large Language Model Optimization). There is still a reason to treat IR sites on their own. Among corporate websites, IR sites are distinctly unusual in structure.

1-1. Five structures peculiar to IR sites

1. They are sometimes placed on a domain separate from the corporate site. IR information sits on a subdomain such as ir.example.com, or on an entirely separate domain. That is not wrong in itself, but robots.txt (explained below) and the server settings exist separately for each domain, so what you confirmed on the corporate site does not necessarily hold for the IR site.

2. An IR support vendor's system is sometimes built into the site. In the 33rd "Fact-Finding Survey on IR Activities" by the Japan Investor Relations Association (published May 14, 2026; 948 responding companies, with a different base for each question), 81.9% of companies conducting IR use an IR support vendor, and among services currently in use, "video production and delivery" is reported at 41.0% and "English translation of disclosure materials" at 42.7%. Some builds also use components loaded from outside, such as share price information or an IR calendar, and in that case the display mechanism is subject to the vendor's specifications.

3. Key materials are sometimes provided as PDFs. A build in which the body of the web page is a summary while the exact figures sit inside a PDF is common in IR. §7 discusses how to handle it. To say it in advance: this article asserts nothing here.

4. Share price information, IR calendars and video are sometimes embedded from third-party providers. The representative method is the iframe, a mechanism that loads another page as a frame inside your page (§6).

5. The timely disclosure list is sometimes built to load dynamically. The list is fetched by JavaScript after the page opens and appears afterwards. For a human reader, nothing goes wrong. The problem is what happens when the reader is not human.

1-2. Why these five points place this article in the IR lane

For the items above described as "sometimes," this article gives no percentages. We have not confirmed quantitative support for them.

2-1. Four words first

A crawler is a program that automatically goes around retrieving web pages. A user agent is the string by which a crawler identifies itself, and it appears in access logs under names such as GPTBot or Googlebot. Rather than treating "AI crawlers" as one block, it matters to look at them name by name.

The initial HTML response is the page the server returns first, before JavaScript runs. A browser receives it, then executes JavaScript and paints in what is missing. That process is rendering, and what a person sees on screen is the state after it. robots.txt is a notice for crawlers placed at the entrance to a site.

2-2. A 41-day controlled experiment (published August 19, 2026)

In an article published in Search Engine Land on August 19, 2026, Vinicius Stanula reports a 41-day controlled experiment.

Working with a business directory of roughly 2,400 pages, he split 21 top-level categories into two groups: in 11 categories the links to the pages below were written directly into the initial HTML response, and in 10 they were inserted afterwards by JavaScript. He also disabled the sitemap and removed breadcrumbs and hierarchy links, closing off routes of discovery other than JavaScript. Reaching one level below a category page was treated as evidence of JavaScript execution.

The results for period 1 (27 days). The figures are the number of hierarchy pages reached.

Source: Search Engine Land, published August 19, 2026. On the three pages in the JavaScript group reached by Bingbot, the author points to the possibility of discovery by another route rather than JavaScript execution.

2-3. What was observed

GPTBot swept all 759 pages of the HTML group on the day of publication, but in the JavaScript group it stopped at the 10 category pages and went no further down. ClaudeBot traced the same shape the following day. Deep progress, halting at the JavaScript boundary.

Bingbot was active on 20 of the 27 days of period 1 and re-fetched the JavaScript group's category pages across 18 days without following a single inserted link. It reached three pages on days 26 and 27, but did not spread to other pages in those categories, so the author reads this as discovery by another route.

ChatGPT-User is retrieval that runs when an actual user asks ChatGPT a question. During the experiment it reached the HTML side through real use, but did not reach anything behind the JavaScript links. A page a user was trying to fetch could not be fetched. Behavior from period 2 onward is covered in §4-5.

2-4. The limits of this experiment (stated as a condition of writing it up)

• It is an experiment on one site, one industry, 41 days, a new domain. The site had just launched and had no external links, and the author himself states that results could differ on an established site
• The experimental site is owned and operated by the author
• ChatGPT-User retrieval amounted to 46 pages, so the base behind that zero is small
• PerplexityBot fetched nothing beyond the homepage and robots.txt in period 1, and its arrival at a hierarchy page was recorded on day 40. This experiment alone does not establish that it does not execute JavaScript
• The author reads the fact that GoogleOther exceeded Googlebot on HTML pages by roughly 14 to 1 as a by-product of crawl budget on a new domain
• A crawl record does not indicate indexing, that is, registration for search. The author notes this himself

On that basis, this article does not assert in the present tense that major AI crawlers do not execute JavaScript. The condition "as observed in the 41-day experiment published on August 19, 2026" is attached throughout.

A controlled experiment can narrow the conditions at the cost of scale. On the other side sits observation of real logs at scale.

3-1. The observation by Vercel and MERJ (published December 17, 2024)

The joint study published by Vercel and MERJ on December 17, 2024 analyzes actual access records that passed through Vercel's network. It uses nextjs.org as its main data and also examines two job sites with different technical builds. Retrieval counts over one month were as follows.

The combined figure for GPTBot, Claude, AppleBot and PerplexityBot is about 1.3 billion, put at a little over 28% of Googlebot. Source: Vercel, published December 17, 2024.

3-2. Whether rendering happens

• Reported as not executing JavaScript: OpenAI (OAI-SearchBot / ChatGPT-User / GPTBot), Anthropic (ClaudeBot), Meta (Meta-ExternalAgent), ByteDance (Bytespider), Perplexity (PerplexityBot), and Common Crawl (CCBot), which is used as a training dataset
• Reported as executing it: Gemini (which uses Googlebot's infrastructure) and AppleBot (which uses a browser-based method)

ChatGPT's crawler spends 11.50% of its retrieval on fetching JavaScript files and Claude 23.84%, yet they do not execute them: that is what the same study reports. Fetching a file and running it are separate things.

The same study also records an important exception. JSON data contained inside the initial HTML response, and React Server Components that arrive later, can be picked up — AI models can interpret formats other than HTML. The question is not "are you using JavaScript" but "is the information you need, and the route to it, inside what comes back first."

3-3. Types of content retrieved, and URLs that are not present

In the observation on nextjs.org, ChatGPT's crawler devoted 57.70% of its retrieval to HTML and Claude 35.17% to images. These are retrieval ratios on a single site and do not represent the population of IR sites or corporate sites.

The same study also reports that ChatGPT's crawler spent 34.82% of its retrieval, and Claude 34.16%, on pages that are not present (404), against 8.22% for Googlebot.

Do not draw causation from this. The claim that reducing 404s makes you more likely to be cited by AI has not been demonstrated. What can be said is that there is ordinary operational sense in reducing retrieval of invalid URLs and keeping current URLs reachable.

3-4. The limits of this study

• It was published on December 17, 2024. Crawler behavior can change. Read it with the publication date attached, not as a present-tense assertion
• It is not a population of IR sites or corporate sites
• Interested-party status: Vercel is a platform vendor providing server-side rendering among other things, and MERJ is a technical SEO vendor. The article concludes in favor of server-side rendering and also introduces Vercel's own crawler-blocking feature. It has to be read as a study by an interested party whose commercial position points the same way as its conclusion
• Microsoft Copilot is out of scope because it does not have a user agent of its own
• The study also states that asking Claude or ChatGPT about recent documentation does not always produce an immediate retrieval record on the server side, and points to possible reliance on caches or trained data. This connects directly to the measurement design in §10

This is the crux of the article. When the technical structure of an IR site comes up, internally or in a vendor meeting, two claims appear. Neither squares with the published observations.

4-1. Claim A: "Google renders it, so we are fine"

The basis for this claim is that Google states officially that it renders JavaScript. That statement is itself correct. The problem is that "Google" is not one crawler.

In the 41-day experiment, what reached the far side of the JavaScript links were two Google user agents. So far, this matches claim A. The breakdown changes the picture. Of the 293 pages in the JavaScript group, GoogleOther reached 142 (48%), while Googlebot, the crawler that builds the search index, reached 7 (2%).

The author writes it plainly: the 66% and 48% figures for GoogleOther say nothing about whether those pages appear in Google search. What bears on visibility in search is the Googlebot side, and that figure is 2%.

Being rendered and being on the path into the search index are separate events. Counting everything that calls itself Google as one number risks a false read of how far retrieval reached. If a vendor supplies your logs, whether Googlebot and GoogleOther are counted separately is worth checking before anything else.

4-2. But do not let the 2% travel on its own

In the same experiment, Googlebot reached 35 of the 748 pages in the HTML link group (5%). The 2% in the JavaScript group is a figure to be read alongside the 5% in the HTML group. The author himself reads GoogleOther's roughly 14-to-1 lead over Googlebot on HTML pages as a by-product of crawl budget on a new domain, and states that the Googlebot share would be higher on an established site.

So what this experiment supports is not "Googlebot cannot execute JavaScript." What it supports is that under these conditions the reach of the crawler that builds the search index was thin to begin with, and thinner still on the JavaScript side. Capability is one question; whether your own pages are actually receiving that processing is another.

There is supporting material as well. In the same experiment, GoogleOther's discoveries on the JavaScript side lagged the HTML side by 18% to 27%, because a rendering queue and an execution step sit in between. Reach was uneven too: one category got to 52 pages while two others sat at a single page for weeks. Some branches never come up in the queue, the author writes. Brought back to IR, the question becomes whether the pages you cannot afford to have retrieved late are sitting on the side that tends to be late.

4-3. Claim B: "AI cannot read JavaScript"

The assertion in the opposite direction is just as precarious.

First, some do execute it. In the observation published on December 17, 2024, Gemini and AppleBot are reported to execute JavaScript.

Second, a limited counter-example is reported. In an experiment report published by EdgeComet on January 15, 2026 and updated on February 17, 2026, log observation on a new domain found that ChatGPT-User retrieved raw HTML documents alone, while for GPTBot the report observed one AJAX request among several hundred requests. The report summarizes this as showing that it can execute but rarely does. Interested-party status and limits: EdgeComet is a vendor providing rendering, and the article closes with a route into its own service. It is an in-house experiment on a single new domain, and the author states explicitly that it is not put forward as a final conclusion. It is properly treated as a limited counter-example to the proposition that AI never executes JavaScript, and it does not support "it executes, so you can relax."

Third, the zero for PerplexityBot cannot be given a meaning from this experiment alone. In period 1 it fetched nothing beyond the homepage and robots.txt, and its first arrival at a hierarchy page was on day 40. You cannot say that something which did not come to fetch is unable to execute.

Fourth, the question was never "are you using JavaScript." Formats other than HTML, such as JSON inside the initial HTML response, can also be picked up. What is being asked is whether the information you need, and the route to it, are inside what comes back first.

4-4. A crawl record does not show that something was used in an answer

The author of the 41-day experiment states explicitly that crawl logs show whether something could be reached, not whether it was indexed. The December 2024 observation adds that asking about recent documentation does not always produce an immediate retrieval record, pointing to possible reliance on caches or trained data. Retrieved is not indexed, and indexed is not used in an answer. That is why the measurement in §10 looks at both the retrieval side (logs) and the answer side (questions put to AI).

4-5. An asymmetry that shows up after a redesign

Period 2 contains an observation that bears directly on running an IR site. After the links were rewritten into HTML, GPTBot retrieved 250 pages within two days, Bingbot added 212, and Meta-ExternalAgent also swept the site by day 41. For some crawlers, rediscovery was observed within a short period after the rewrite to HTML. Not every crawler came back the same way, however.

ClaudeBot, meanwhile, kept re-fetching after the rewrite without picking up a single newly linked page, and Amazonbot had left six days before the rewrite. Google was slowest: as of day 41, Googlebot had visited one page among the rewritten categories and GoogleOther none.

This is an observation of one site over 41 days, not a general law. It does give you a way to separate a retrieval lag from other causes when an AI answer feels stale right after a redesign.

4-6. So what can be written?

Placing your key information, and the links that lead to it, in an initial HTML response that can be retrieved without assuming JavaScript execution is consistent with the observations currently available.

Nothing beyond that. In particular, as to what percentage this improves the accuracy of AI answers or the way they cite you, within the scope of this review, we could not confirm a figure. This article also does not specify an implementation method. There are several, and the choice depends on the site's build and on your contract with your vendor. What has to be decided is not the method; it is that you measure the current state first.

When the question "should we block AI crawlers" comes up internally, the question can be too coarse as posed, because crawlers are split by purpose.

5-1. OpenAI's official split

Source: OpenAI, "Overview of OpenAI Crawlers." Four notes matter in practice.

1. The settings are independent of one another. It is stated that you can permit OAI-SearchBot while refusing GPTBot
2. It can take about 24 hours for an update to robots.txt to be reflected on the search side. Which means: do not judge the result immediately after a change
3. It is noted that a site that has opted out of OAI-SearchBot may still be shown as a navigation link
4. It is stated that ChatGPT-User, being a user-initiated action, may not have robots.txt rules applied to it. You cannot say that writing it into robots.txt means no access at all from ChatGPT

5-2. Do not equate the 2024 GPTBot with today's ChatGPT search

The GPTBot reported as crawling heavily in the December 2024 observation is retrieval for training. What runs when a user asks a question is ChatGPT-User, and what bears on display in the search feature is OAI-SearchBot. Talking about them as one "ChatGPT bot" leads to mistakes in robots.txt design.

Elsewhere there are ClaudeBot (Anthropic), PerplexityBot, Meta-ExternalAgent, Bytespider, Amazonbot, CCBot (Common Crawl), Google-Extended and others. Each provider divides its crawlers differently, and the divisions are updated. Check each provider's official documentation.

5-3. What this means in practice for IR sites

If the IR site sits on a separate domain, robots.txt is specific to that domain. Checking the corporate site's settings is not enough. Blocking can also be applied at the CDN or WAF layer (the mechanism that shuts out unauthorized access), separately from robots.txt. A build in which you believe you are permitting access while it is being stopped earlier is entirely possible. Whether blanket blocking is advisable is covered in What Is LLMO (Large Language Model Optimization).

Some IR sites load share price information, IR calendars, video and timely disclosure lists from outside services. Here, this article does not adopt an assertion you often see.

6-1. You cannot write that "the contents of an iframe are invisible"

"If what is inside the iframe runs on JavaScript, a crawler that does not execute JavaScript cannot see it." It sounds plausible, but within the scope of this review, we could not confirm a published study that tests this directly.

There is a leap in the logic as well, because a crawler may discover the URL that the iframe loads and retrieve it as a separate request. Whether the frame can be read and whether the URL inside it can be retrieved are different questions. What can be written goes one step:

Using third-party components increases the number of retrieval paths. That is why checking is needed.

Not "invisible," not "visible." There is more to check, and that is all.

6-2. The shape of what to check

For third-party parts, this is what you can ask a vendor or an IR support vendor.

• Is the component included in the initial HTML response on your own domain, or loaded from another domain
• If it is loaded from another domain, what does that domain's robots.txt say
• Does the same information also exist as text on your own domain, separately from the embed
• Is the cookie consent screen built so that it covers the body of the page on first display

On the last point too, as to what percentage it lowers the accuracy of AI answers, within the scope of this review, we could not confirm a figure.

6-3. A workable landing point

You do not have to stop using third-party components. The realistic line is to check whether the same information also exists as text in the initial HTML response on your own domain. You do not need to hold raw share price data twice, but you can design so that the date of an earnings announcement, the date of a briefing, and the title and date of a disclosure do not sit inside the embedded component alone.

In IR, key materials are sometimes provided as PDFs. Not as PDFs alone, however: for the earnings report (kessan tanshin), XBRL data is also provided through TDnet (§8-3). On a PDF-centered IR site, "does AI read PDFs" is a pressing question. This article does not answer it with an assertion.

Google states in its official documentation that PDFs composed of text can be indexed by Google Search, and notes that OCR may be applied to image-based PDFs. The generalization "PDFs are not read by search engines" is wrong, at least for Google Search.

On the other hand, as to a published experiment directly testing how AI crawlers handle the body of a PDF — how far they interpret the tables and notes inside a retrieved PDF, and how they use them in generating an answer — within the scope of this review, we could not confirm one. The December 2024 observation goes as far as ratios of retrieved content types.

This article therefore writes neither "PDFs are not read" nor "PDFs are read too." The question is placed in §11 (what could not be confirmed), and §10-6 treats it as something to check for yourself.

IR documents come in formats where even a human reader can go wrong: tables that span pages, footnotes, two-column layouts, financial tables rendered as images. Checking against the way your own materials are built is more practical than a general argument. Whether you provide an HTML version or a PDF version, and how that bears on AI answers, is covered in the parent article How AI Describes Your Financials.

Technical discussion tends to blur into regulatory discussion. Without sorting this out, internal debate spins.

8-1. TDnet is an exchange system, not your own site

TDnet is the Tokyo Stock Exchange's mechanism for publishing timely disclosure in real time. The periods set for it are 31 days for public viewing and 10 years for searching by company. TDnet is a system operated by the exchange, not a statute in itself. And having disclosed on TDnet, and having the pages of your own IR site in a state where AI can retrieve them, are entirely separate events. Meeting a disclosure obligation under the rules does not guarantee technical fetchability.

8-2. XBRL and schema.org are separate things

On TDnet, earnings reports, revisions to earnings and dividend forecasts, corporate governance reports and others are produced in XBRL.

This is where the confusion starts. XBRL is a disclosure data format used by the exchange; schema.org is markup you add to a web page. Reading this as "we file XBRL, so structured data is handled" is wrong: the purpose, the reader and the route all differ. This article does not go into schema.org design.

8-3. The fact that "state" is an independent disclosure category

The Tokyo Stock Exchange defines revisions to earnings forecasts, differences between forecast and actual results, dividend forecasts and their revisions, corporate reorganizations, and tender offers as independent categories of timely disclosure. The same "revenue of 100 billion yen" is an entirely different piece of information for an investment decision depending on whether it is an initial forecast, a revised forecast, an actual result or a corrected figure. The fact that the regime distinguishes these determines how you read an AI answer when you check it (§10-3).

8-4. The duty to maintain an IR framework and AI-space perception are different matters

Listed companies are required, under the Code of Corporate Conduct, to maintain a framework for providing the information needed to build relationships with shareholders and investors, an IR framework, and the state of that framework has to be disclosed in the corporate governance report. The specific content of the framework is left to each company's judgment.

There is a line here that must not be crossed. This article does not state that the Tokyo Stock Exchange requires management of perception in AI space. Maintaining an IR framework is an obligation under exchange rules, but measuring and managing perception of your own company inside AI is not an obligation. What can be written goes one step: how far, within the existing IR framework, you treat disclosure inconsistencies in AI as something to manage. That is for each company to judge, and it is not something the regime directs.

9-1. Where this sits in the regime

Companies listed on the Prime Market have been required, since April 1, 2025, to disclose earnings information and timely disclosure information in English at the same time as in Japanese. It is an obligation under exchange rules, with separate provisions for grace treatment.

On the scope of English disclosure for timely disclosure information, the Tokyo Stock Exchange's FAQ sets out specific questions and answers noting that a part or a summary of the Japanese disclosure may be acceptable in some cases. That simultaneous disclosure of the full text is not required for every timely disclosure in every case is a premise worth holding. Check the same FAQ for details.

In the 33rd "Fact-Finding Survey on IR Activities" by the Japan Investor Relations Association (published May 14, 2026), among the questions put to companies that have an IR site, the earnings report was the most common English disclosure material at 74.1%. Because the base differs by question, this figure cannot be added to figures from other questions.

9-2. What to check technically

What this article deals with in this section is one point. When two pieces of official information exist, a Japanese version and an English version, are you confirming the fetchability of each separately?

• Is the English IR page on the same domain as the Japanese page, on a different path, or on a different domain
• Does the language switch run on JavaScript. Does the content after switching exist in the initial HTML response
• Can the English list and archive be traversed with the same structure as the Japanese side
• Do the English and Japanese versions diverge in scope, point in time, figures or summarized content

The last point is also a regulatory question. Where the English is a part or a summary, you are the one who can judge whether the difference is intended or an omission in updating.

Note that this article does not go into whether AI answers change between Japanese and English. That belongs to perception across languages and regions, and is the territory of When the Language Changes, So Does the AI's Answer. What this article covers is whether the Japanese page and the English page are each in a state where they can be retrieved.

This is the core of the article. The aim is not to end at reading, but to leave you able to talk to your vendor tomorrow.

10-0. How to think about measurement

Perception inside AI cannot be measured with a single question. The same question produces answers that vary from run to run, so a one-off result cannot be treated as a standing state.

Vaipm measures AI-space perception through a total of 25 stateless queries across multiple AI engines.

This is not an optimum derived from research; it is Vaipm's operational design.

Stateless means carrying no context over from a previous conversation. Asking a follow-up inside a running conversation pulls the answer toward what came before, and you lose sight of how that AI would answer on its own. The procedures below are built to look at both the technical side (is it being retrieved) and the answer side (how are you actually being described).

10-1. Step 1: Turn off JavaScript and look at your own pages

The cheapest, and the most effective. Disable JavaScript in your browser and open your IR site. In Chrome, go to site settings from "Privacy and security" in settings and change JavaScript to "Don't allow sites to use JavaScript." Be sure to change it back when you are done.

1. Can you get from the top page to the IR top page
2. From the IR top page, can you get to the timely disclosure list, the earnings report list and the briefing materials list
3. Does each list carry actual items, titles and dates. Are they blank
4. Can you reach individual disclosure pages and material pages by following links
5. Do the pages that carry figures, such as financial highlights, show the figures

What you are looking at is not broken presentation. The layout will break, naturally. What you are checking is whether the information, and the route to it, are there.

There is a more precise method. Right-click on the page and choose "View page source" to see the initial HTML response as it is. Use the search function to see whether the title or the figure of the disclosure you want to check is in there. If it is not, it is absent from the initial HTML response.

10-2. Step 2: Put the same question repeatedly, and independently

Alongside the technical side, look at the answer side. Limit the questions to ones where a correct answer exists in your own disclosure. That a correct answer can be defined is IR's advantage.

• "What is [company name]'s most recent full-year earnings forecast"
• "What is [company name]'s most recent dividend forecast"
• "When was [company name]'s most recent earnings briefing held"
• "What are the three most recent timely disclosures by [company name]"
• "Where is the earnings report on [company name]'s IR site"

Put these as a new conversation each time. Do not repeat them inside the same conversation. Record the date and time of each run and the name of the service used.

10-3. Step 3: Check figures, point in time and state against your own disclosure

When an answer comes back, look at three things separately.

Checking "state" is specific to IR, and it is the most often missed. As set out in §8-3, the Tokyo Stock Exchange treats revisions to earnings forecasts, and differences between forecasts and results, as independent disclosure categories. Even when the figures agree, getting the state wrong is a serious error.

The concrete design of that checking — how to build a reference table, at what frequency, in what record format — is covered in the sister article How to Verify What AI Says About Your Financials. This article stays on the technical side.

10-4. Step 4: Take inventory of the URLs cited

When an answer shows a URL, check which of your pages it is. Your own IR site or an external information site. If it is yours, is it a current page or one on an old domain or an old structure. Does that URL open and display now. Then look from the other direction. List your main pages and check whether any of them have never been cited.

There can be several reasons for not being cited. It was not retrieved; it was retrieved but not used; the question did not match; a better source exists — this alone will not tell you. But if step 1 has already shown you that the page is absent from the initial HTML response, the possibility of a retrieval failure is worth investigating before the others.

10-5. Step 5: Look at server logs by user agent

This is the check closest to what is actually being retrieved. User agent names can be spoofed, so authenticity has to be established, and IR staff cannot always see server logs directly. So it is set out here in the form of a request.

[Request: crawler access to our IR site]
Hello. This is [name] from the IR department.
As part of an effort to understand how our company's information is handled
in generative AI, could you share the state of crawler access to our IR site?
■ Scope
    Our IR site (target domain: ____________________)
    * If it is on a different domain from the corporate site, the IR site alone is fine
■ Period
    The most recent three months (six if possible)
■ What we would like shared
    1. Access counts by user agent
         (Googlebot / GoogleOther / GPTBot / OAI-SearchBot / ChatGPT-User /
          ClaudeBot / PerplexityBot / Bingbot / Amazonbot / Meta-ExternalAgent)
         * Please count Googlebot and GoogleOther separately
         * Please judge authenticity as far as the official IP ranges published by
           each provider allow, and count unverified user agent names separately
    2. For each of the above, the range of URLs reached
         (the IR top page alone, as far as the list pages, or as far as individual
          disclosure pages)
    3. Counts by HTTP status (2xx / 3xx / 401 / 403 / 404 / 429 / 5xx)
         * Please include 401, 403, 429 and 5xx, as these directly indicate
           obstruction of retrieval
    4. The current contents of robots.txt
    5. If access control is applied at a layer other than robots.txt, such as a CDN
         or WAF, those settings
    6. Whether an XML sitemap exists, and the range of URLs it contains
■ Please also check
    - For the timely disclosure list, the earnings report list and the briefing
      materials list, whether the list items and the links to individual pages can
      be retrieved with JavaScript not executed
    - If share price information, an IR calendar, video and the like are embedded
      from third parties, the domains they are loaded from, and whether the same
      information exists on our own domain
This request is not premised on any particular measure; the purpose is to
understand the current state.

State that the purpose of the request is to understand the current state. Naming a measure first brings back a proposal rather than a check.

The experiment in §2 disabled the XML sitemap in order to create a condition in which URLs could not be discovered other than through JavaScript. A live IR site can have other routes, a sitemap among them. "The link is absent from the initial HTML response" and "a crawler cannot discover the URL" are not the same thing.

10-6. Step 6: Check PDFs for yourself

As set out in §7, this article does not assert how AI crawlers handle the body of a PDF. Instead, here is a way to check for yourself.

1. Pick one figure that appears nowhere except inside your earnings report PDF (not in the body of a web page, not in a summary)
2. Write a question that asks for that figure
3. Put it several times as a new conversation each time, in the same way as step 2
4. Record whether it is answered correctly, and what it gives as the basis when it is

What you get is an observation about your own materials, not an answer to the general proposition that AI reads PDFs or does not. Keep that distinction in internal reporting too. The same approach works for tables that span pages and for financial tables rendered as images.

10-7. A record format

The point is holding the technical state in the same table. It lets you separate afterwards whether an error in an answer was a content problem or a retrieval problem.

10-8. Readings to avoid

• Do not judge from a single answer. They vary from run to run
• Do not promise an improvement rate. Do not explain internally that putting something in the initial HTML response increases citations by some percentage. That causation has not been demonstrated. What you can explain goes as far as "we confirmed it is in a state where it can be retrieved"
• Do not read log figures as impact on answers (§4-4). Do not count everything that calls itself Google as one number (§4-1). Do not draw a conclusion immediately after a settings change (§5-1)
• Do not correct an AI's error with information that has not been published. This is a disclosure rule, not a technical matter. Keep corrections to re-presenting already published materials

As a definitive article, we state what could not be confirmed. For each of the items below, within the scope of this review, we could not confirm one. That is not an assertion of absence.

This blank space means you have to measure it yourself. That is why §10 is the core of this article.

Q1. Can AI not read JavaScript?

It cannot be put in a word. In the observation published on December 17, 2024, Gemini (which uses Googlebot's infrastructure) and AppleBot were reported to execute it, and the crawlers of OpenAI, Anthropic, Meta, ByteDance and Perplexity were reported not to. It is a report carrying an observation date, not a present-tense assertion. What matters is whether the information you need, and the route to it, are in the initial HTML response.

Q2. I was told "Google renders it, so we are fine." Is that right?

It cannot be taken as it stands. In the 41-day experiment, most of the Google access that reached the far side of the JavaScript links came from GoogleOther, and Googlebot's reach was 2%. In the same experiment, though, Googlebot also stopped at 5% in the HTML link group, and the author reads that as the effect of crawl budget on a new domain.

Q3. Our IR site is on a separate domain. What should we check?

robots.txt exists separately for each domain, so what you confirmed on the corporate site does not apply to the IR site. For the IR site's domain, check robots.txt, whether blocking is applied at the CDN or WAF layer, and the range reached by each user agent, separately (the request in §10-5).

Q4. Our timely disclosure list is loaded by JavaScript. Should we fix it right away?

Measure before you fix. Disable JavaScript, open the list page, and check whether the items and the links to individual pages are there (§10-1). If they are not, it becomes a candidate for rework. This article does not specify an implementation method, however. What can be written is that placing your key information, and the links that lead to it, in the initial HTML response is consistent with the observations currently available.

Q5. We embed share price information in an iframe. Does that mean AI cannot see it?

We cannot write "cannot see." Within the scope of this review, we could not confirm a study that tests that directly. A crawler may also discover the URL the iframe loads and retrieve it as a separate request. What can be said is that retrieval paths increase, so checking is needed.

Q6. Are PDFs read by AI?

This article does not assert either way. Google states in its official documentation that PDFs composed of text can be indexed by Google Search. But as to a published experiment directly testing how AI crawlers handle the body of a PDF, within the scope of this review, we could not confirm one. The procedure for checking with your own materials is in §10-6.

Q7. Should we block all AI crawlers?

The question is too coarse as posed. OpenAI's official documentation lists GPTBot for training, OAI-SearchBot for search and ChatGPT-User for user-initiated retrieval, and explains that the settings are independent of one another. Blanket blocking can also stop the retrieval that exists for display in search.

Q8. If we write it into robots.txt, will access from ChatGPT stop?

It may not. OpenAI's official documentation states that ChatGPT-User, being a user-initiated action, may not have robots.txt rules applied to it. It also states explicitly that an update can take about 24 hours to be reflected.

Q9. We file XBRL, so is it right that structured data is handled?

No. XBRL is a disclosure data format used by the exchange and schema.org is markup you add to a web page; the purpose, the reader and the route differ. Disclosure on TDnet, and having the pages of your own IR site in a state where AI can retrieve them, are also separate.

Q10. Does the Tokyo Stock Exchange require management of perception in AI space?

It does not. Listed companies are under an obligation in the Code of Corporate Conduct to maintain an IR framework, but measuring and managing perception of your own company inside AI is not an obligation. How much you treat as something to manage is each company's judgment.

Q11. We redesigned our IR site. When will AI answers update?

It cannot be stated uniformly. In period 2 of the 41-day experiment, GPTBot retrieved 250 pages within two days of the rewrite, while ClaudeBot picked up no newly linked page and Google had barely come back as of day 41. It is an observation of one site, but the fact that update speed can differ greatly by crawler is a premise for internal explanation.

Q12. By what percentage will AI answers improve if we rebuild?

That figure cannot be given. As to what percentage moving to the initial HTML response improves the accuracy of AI answers or the way they cite you, within the scope of this review, we could not confirm a figure. What you can explain internally goes as far as "we confirmed it is in a state where it can be retrieved."

IR sites are structurally unusual. They are placed on separate domains, support vendors' systems are built into them, key materials are centered on PDFs, and share prices and timely disclosure lists are sometimes loaded from outside. None of that is wrong in itself. But whether the AI side is able to retrieve them has to be checked separately, structure by structure.

This article rejected two simplifications in circulation. Some execute JavaScript and some are reported not to, and within Google, the reach of the crawler that builds the search index was thin under the conditions observed. Retrieved, indexed and used in an answer are separate events.

There is one conclusion that can be written. Placing your key information, and the links that lead to it, in an initial HTML response that can be retrieved without assuming JavaScript execution is consistent with the observations currently available. By what percentage that improves AI answers could not be confirmed.

So what to do first is not a rebuild; it is a measurement. Turn off JavaScript and look at your pages. Put the same question several times, independently. Take inventory of the URLs cited. Look at server logs by user agent. All four can start today.

Treating corporate perception inside AI as a continuing object of management is covered in AI Perception Management (AIPM), what an error means in IR and the wider picture in How AI Describes Your Financials, and the procedure for verification testing in How to Verify What AI Says About Your Financials.

[Layer 1] Primary sources and official documentation

1. OpenAI, "Overview of OpenAI Crawlers" — https://developers.openai.com/api/docs/bots (verified for this article: August 31, 2026)
2. Google Search Central, "PDFs in Google search results" — https://developers.google.com/search/blog/2011/09/pdfs-in-google-search-results
3. Google Search Central, "AI features and your website" — https://developers.google.com/search/docs/appearance/ai-features
4. Japan Exchange Group, "Investor Relations" — https://www.jpx.co.jp/equities/listing/investor-relations/index.html
5. Japan Exchange Group, "Corporate Information Requiring Timely Disclosure" — https://www.jpx.co.jp/equities/listing/disclosure/info/
6. Japan Exchange Group, "TDnet (Timely Disclosure network)" — https://www.jpx.co.jp/english/equities/listing/disclosure/tdnet/
7. Japan Exchange Group, "XBRL (TDnet)" — https://www.jpx.co.jp/english/equities/listing/disclosure/xbrl/03.html
8. Japan Exchange Group, "List of Companies Granted Grace for Mandatory English Disclosure" — https://www.jpx.co.jp/listing/others/en-disclosures/index.html (includes the description of mandatory English disclosure on the Prime Market. Verified for this article: August 31, 2026)
9. Japan Exchange Group, "FAQ (English disclosure on the Prime Market, from April 2025)" — https://faq.jpx.co.jp/disclo/tse/web/category2511.html
10. Japan Investor Relations Association, 33rd "Fact-Finding Survey on IR Activities" (published May 14, 2026; 948 responding companies, with a different base for each question) — https://www.jira.or.jp/activity/research.html

[Layer 2] Non-peer-reviewed practitioner observation and controlled experiments

1. Vinicius Stanula, "JavaScript links can make your pages invisible to AI search," Search Engine Land, published August 19, 2026 — https://searchengineland.com/javascript-links-pages-invisible-ai-search-485228

A controlled experiment on a single site owned and operated by the author, one industry, 41 days, a new domain. The author is with LOCOMOTIVE. Search Engine Land is owned by Semrush.

1. Giacomo Zecchini, Alice Alexandra Moore, Malte Ubl, Ryan Siddle, "The rise of the AI crawler," Vercel + MERJ, published December 17, 2024 — https://vercel.com/blog/the-rise-of-the-ai-crawler

Vercel is a platform vendor providing server-side rendering among other things; MERJ is a technical SEO vendor. A study by an interested party, and the article recommends server-side rendering and its own crawler-blocking feature.

1. Helen Durant, "OpenSeoTest: How GPTBot and ChatGPT-User Handle JavaScript," EdgeComet, published January 15, 2026, updated February 17, 2026 — https://edgecomet.com/blog/openseotest-how-gptbot-and-chatgpt-user-handle-javascript/

EdgeComet is a vendor providing rendering. An in-house experiment by an interested party, observed on a single new domain. The author states explicitly that it is not a final conclusion. Confirmed at the primary source when this article was written (August 31, 2026).

Sources verified: August 31, 2026 (the contents of every Layer 1 and Layer 2 URL were confirmed at the time of writing)