Chatbot ROI for Executives: The AI Analysis Report

The chatbot has been running for a few months. Renewal is coming up, and the person who signs off on the budget asks a plain question in a plain meeting: is it actually working?

Someone opens the admin dashboard and shares their screen. A chart goes up, then a list of conversations. Everybody nods politely. Nobody can say, in one sentence, what the company got for the money. The approver leaves the room no more convinced than when they walked in, and that is how a chatbot that works fine gets quietly cancelled.

The problem is rarely the bot. It is that a dashboard is built for the person who operates it, and a budget decision is made by someone who will never log in. This article is about the document that closes that gap: the AI Analysis Report (AI分析レポート) in OneBot. It is a PDF meant to be read in a management meeting, and it is designed so that you can trust what is printed on it.

What a decision-maker actually wants to know

Strip the meeting down and the approver has four questions.

• Are customers really using it, or did we just switch it on?
• Did it save anybody any time?
• Where does it fail?
• What should we do next month that we are not doing now?

A screenshot of a conversation log answers none of these. A raw count of messages answers the first, badly. What is needed is a short document that puts the numbers next to the assumptions behind them, and ends with a recommendation someone can act on.

What is in the report

The report is a PDF with a fixed structure. Roughly in reading order:

• Volume. How many questions were asked and how many conversations they belonged to, for a period you choose.
• Hours saved. An estimate of staff time the bot took off the team's plate.
• Give-up rate. The share of questions where the bot could not help and the person gave up or was handed off.
• Answer satisfaction. Based on the ratings end users actually gave.
• Category breakdown. What people ask about, grouped, so you can see whether it is opening hours or returns or something you had not expected.
• Up to three improvement proposals. Concrete suggestions for the knowledge base: this question keeps failing, this answer is outdated, this topic has no source document at all.

Three proposals, not thirty. That cap is deliberate. A list of thirty items gets filed; three items get done.

The hours-saved number is an assumption, and the PDF says so

Everyone who has been shown "hours saved" on a slide has wondered where it came from. In this report the calculation is not hidden. By default each handled inquiry is counted as 3 minutes of staff time avoided, and the formula is printed on the PDF itself, so a reader can see: number of inquiries multiplied by minutes per inquiry.

Three minutes is a starting assumption, not a measurement. It will be too generous for a one-line opening-hours question and too stingy for a question that used to take a phone call and a lookup. So the figure is editable when the report is prepared. Use your own estimate if you have one, for example from watching how long your team really takes.

We would rather you argue with the assumption than trust a number nobody can trace. If your finance lead thinks two minutes is more honest, change it to two, and the PDF will show two.

Designed so the numbers can be trusted

Five design choices decide whether a number on the page can be trusted.

You generate it yourself. Choose the period, generate the report and export the PDF.

Assumptions are visible and editable. The minutes-per-inquiry figure behind hours saved can be checked and adjusted, and the formula is printed on the PDF.

Personal information is masked before analysis. Customer messages are masked for personal data before the AI looks at them for categories and proposals: masking happens first, and analysis works on the masked text. For the wider picture on where data is stored and processed, see our APPI and data residency guide. Storage sits on servers located in Japan; AI processing uses external AI APIs, and we do not claim that data never leaves Japan.

The period is fixed in Japan time. Reporting periods are calculated in JST, so a month means the month your business experienced, with no drifting midnight boundaries.

Metrics without enough evidence are left out. If there are too few ratings to say anything about satisfaction, the satisfaction block is omitted. If the give-up rate cannot be supported, it is omitted. The report does not fill the gap with a guess. A shorter report with a missing block is more useful than a complete-looking one that is partly made up, and it is the design choice we would defend hardest.

What it does not do

Being straight about limits is better than a surprise later.

• It does not guarantee accuracy of the analysis. Counts come from your conversation data; the categories and proposals come from AI analysis. Read the numbers together with their assumptions and check anything doubtful against the original conversations.
• The three-minute saving is an assumption, as above.
• It reports on what the chatbot handled. It will not tell you how much revenue a conversation led to unless that data exists elsewhere.

How it differs from the report you build by hand

If you are an agency you may already send clients a monthly report assembled from the dashboard. That workflow, with its template and the time it costs across 20 clients, is covered in monthly report automation for agencies. The manual report is flexible and yours; you decide the story.

The AI Analysis Report is a different thing: a standardized document with the assumptions printed on it. Some agencies will want both, using the manual report for the relationship and the analysis report as the evidence behind it. It also helps when a client's approver, not your day-to-day contact, is the one deciding.

What a good review meeting does with the three proposals

The proposals are the part most readers skip and the part that pays back. A useful meeting treats the report as an agenda, not a verdict.

Start with the give-up rate and the category breakdown together. A high give-up rate in a category people ask about constantly is the first place to look. Then take the proposals one at a time and ask three things: is this a missing document, an outdated one, or a question the bot should hand to a person? Who owns the fix? By when?

Then close the loop. Each accepted proposal becomes a change to the training data: a new FAQ entry, a corrected source, a rewritten answer. Next period's report shows whether that category improved. Without this step the report is a nicer screenshot. With it, the report becomes the way the chatbot gets better each month. Teams that struggle after launch often lack exactly this loop; our piece on why chatbot projects stall goes into the pattern.

Who it suits

It suits two situations. A company where the person funding the chatbot is not the person running it. And an agency that has to justify a monthly fee to a client who does not look at dashboards. If you are a two-person team who checks the admin screen every morning anyway, you probably do not need a formal PDF yet.

If you want to see what a report looks like for your own setup, start with a free trial or talk to us. We will be candid about whether this level of reporting fits what you are doing.

FAQ

FAQ

Can we generate the AI Analysis Report ourselves?

Yes. You choose the period, generate the report yourself and export the PDF.

Where does the "hours saved" figure come from?

It is an estimate: handled inquiries multiplied by minutes per inquiry. The default is 3 minutes, editable, and the formula is printed on the PDF. Treat it as an assumption, not a time study.

What if there is not enough data for a metric?

That block is omitted. For example, satisfaction is left out when there are too few ratings. The report does not estimate values it cannot support.

Is customer personal information sent to the AI?

Personal information is masked before analysis, so the analysis works on masked text. Data is stored on servers located in Japan, while AI processing uses external AI APIs. We do not claim data never leaves Japan.

Can we change the assumptions behind the numbers?

The assumptions, such as minutes per inquiry, are shown and editable, and the formula is printed on the PDF so readers can see what the numbers rest on.

Which time zone defines the reporting period?

Japan Standard Time (JST).

How is this different from a monthly report we write ourselves?

A self-written report is flexible and carries your own narrative. The AI Analysis Report has a standardized structure and shows its assumptions. Agencies can use both.