Once the reference table is ready, the next stage is obtaining AI answers and matching them against it. This is the core of measurement design.
3-1. Why one measurement is not enough
A single answer is one observation. But from an answer obtained by sending a single prompt once to a single model, you cannot estimate the stable tendency of what that AI returns about your company. There are four reasons.
Reason 1: the same question does not always return the same answer. Generative AI output is probabilistic. Temperature settings, sampling settings, and internal routing all mean that the same question can return different answers. One observation is one sample from a distribution. With a sample of one, you cannot distinguish a tendency from an outlier.
Reason 2: consistent does not mean correct. As the hub article showed, on the FinQA benchmark 15–23% of the "high-confidence" answers in which 8 resamplings all agreed were wrong. The stability of an answer is not evidence of its correctness. Put the other way, if you cannot observe the variation, you will mistake a stable error for a correct answer.
Reason 3: answers differ by engine. Major AI services each have different models, different search indexes, and different citation policies. You cannot look at one service alone and conclude "this is how AI talks about our company."
Reason 4: session history contaminates the result. When you ask successive questions inside the same chat, the earlier answer remains as context. An answer returned after you say "please correct the figure you just gave" is not the answer that AI returns about that company under normal conditions. Measurement has to be carried out from a clean state (stateless) every time.
3-2. Vaipm's measurement design
Vaipm measures AI-space perception through a total of 25 stateless queries across multiple AI engines.
Three points about this design should be made explicit.
- This is not an optimum derived from research; it is Vaipm's operational design. It is set as a point that captures the variation in answers to a degree that is workable in practice, while staying within a running cost that can be sustained over time.
- Being stateless is the essential part. Each measurement is independent and does not carry a previous answer forward as context. What is observed is therefore the distribution of answers that AI returns to that question under normal conditions.
- Multiple AI engines are covered. The result from a single service is not treated as the whole picture.
If you build this in-house, we recommend keeping these three — repeating the measurement, not carrying history forward, and looking at multiple services — as design premises. The number of repetitions can be set to match your operating capacity, but do not design it so that a single measurement produces a conclusion.
3-3. What to record
One record of a verification test carries the following items.
| Record item | Contents | Purpose |
| run_id | A distinct ID for the execution | Identifying the same batch |
| question_id | Reference to the reference table | Matching against the correct value |
| prompt_text | The wording submitted (verbatim) | Ensuring reproducibility |
| executed_at | Execution timestamp (with time zone) | Judging the order relative to the disclosure date |
| engine | Name of the AI service | Analyzing differences between services |
| model_label | The model name and version displayed | Comparison before and after an update |
| locale | Region and language settings | Detecting Japanese/English and regional differences |
| session_state | Login state and whether history is present | Making the conditions explicit |
| response_text | Full text of the answer | Re-judging at a later point |
| extracted_value | The figure extracted | The object of mechanical matching |
| cited_urls | Source URLs presented in the answer | Calculating the primary-source citation rate |
| verdict | Judgment (match / mismatch / partial match / no answer) | The denominator for the KPIs |
| mismatch_type | Type of mismatch (fiscal period / standard / currency and unit / scope / other) | Root-cause analysis |
| delta | Difference from the correct value | Judging severity |
Always carry `mismatch_type`. Mapping it to the four error types (fiscal period, accounting standard, currency/unit, scope) tells you which error is the most frequent. If you look only at the overall consistency rate, you cannot identify what needs to be addressed.
An example of the record schema follows.
run_id,question_id,prompt_text,executed_at,engine,model_label,locale,session_state,extracted_value,cited_urls,verdict,mismatch_type,delta
R2026-0816-001,Q001,2026年3月期の連結売上高は,2026-08-16T10:00:00+09:00,EngineA,modelX,ja-JP,stateless,1234567,"https://example.co.jp/ir/tanshin_2026q4.pdf",match,,0
R2026-0816-002,Q001,2026年3月期の連結売上高は,2026-08-16T10:00:30+09:00,EngineA,modelX,ja-JP,stateless,1210000,"https://example.com/news/article",mismatch,fiscal_period,-24567
3-4. Conditions that must be stated
Even for the same question, the result can change when the following conditions change. You have to choose one of two: fix the conditions, or record them separately for each condition.
- Variation in the query: "net sales," "revenue," and "operating revenue" have different formal names depending on the company. Prepare several prompts with different wording.
- Region: the sources referred to can change depending on the region the access comes from.
- Language: Japanese and English answers do not necessarily agree. The more a company discloses in English, the more it is worth measuring both.
- Login state: whether you are logged in, and differences in paid plans, can change which model or search function is used.
- Model and version differences: behavior changes across a model update even within the same service. Record `model_label` and annotate any comparison that spans an update.
3-5. Designing the frequency
The following is a practical design example presented in this article, not an optimal configuration derived from research. It is offered as one example of a frequency; set it according to your risk, disclosure events, and running cost.
| Timing | Scope | Purpose |
| Immediately after a quarterly earnings announcement (announcement day to 3 business days) | All questions on the most recent figures | To measure how quickly the update is reflected |
| Two weeks after the announcement | Same as above | To identify items that were not reflected |
| Monthly | Basic attributes and shareholder returns | To detect persistence of stale information |
| After the annual securities report is filed | All questions | To confirm consistency with statutory disclosure |
| After a serious wrong answer is detected (14 and 30 days after remediation) | The relevant questions | To confirm the effect of the remediation |
| After a reorganization or a change of officers is announced | The relevant questions | To detect persistence of stale information |
The measurement immediately after an earnings announcement carries the most information. It lets you observe when and how AI answers change from the moment the disclosure is updated.
3-6. Combining with other measurement methods
The verification test measures the content of AI answers. Combine it with methods that measure reach through AI.
- The Google Search Console performance report: impressions and clicks that arrive through AI Overviews and AI Mode are counted within the overall data for the "Web" search type. Filter by the IR site directory and compare before and after a disclosure update.
- Server logs: which bots crawl which pages of the IR site, and how often. As the log study in GenAI as a Reader showed, this information itself can be obtained. However, a user agent does not prove that the content was used in the final answer. Do not confuse being crawled with being cited.
- Referrers: whether there is traffic from AI services. Note, though, that a path in which someone reads an AI answer and then arrives via search cannot be captured.
These three measure different layers. Accuracy of content (the verification test), exposure on the search surface (Search Console), and machine retrieval (logs) are not substitutes for one another.