Propose an AI experiment reviewers can assess
For employees with an AI idea and the governance reviewers who weigh it.
Fictional example: at Harbor & Pine Group, a shared-services team proposes a small, human-checked test. Follow it to see what a clear proposal says about the problem, the users, the data, the risks and the measures of success.
The example in brief
The team proposes a four-week test in which an AI tool suggests a category and a draft routing note for copied, historical internal IT requests. A coordinator checks every suggestion in a separate test workspace. No test decision enters the live queue. The decision to assess this limited prototype was recorded on 10 October 2026.
Problem and baseline
Say what is slow today, and how you know
Shared-services coordinators read each incoming internal IT request and choose a category and a resource group. In this fictional example, that is about 120 routine requests a week, such as access resets, software installs and peripheral orders.
Operational context (illustrative figures)
- Review takes about 4 minutes per request, or roughly 8 coordinator-hours a week.
- About 1 in 10 requests is re-categorized after first routing. This describes the operational problem; it is not the accuracy baseline for the offline test.
A comparable offline baseline
Before testing, the work owner establishes reviewed reference categories for an approved historical sample. Compare manual-only and AI-assisted review on comparable sample groups using the same timing method and reference categories. Record the group sizes and request mix so the result can be interpreted.
The narrow question is whether suggestions reduce review effort while preserving the accuracy of the coordinator’s final test decision. Downstream re-categorization in live work would require a separate approved live evaluation; this offline test cannot measure it.
Intended users and boundaries
A proposal should name who uses the tool and what stays with people
Four shared-services coordinators take part. They already review these requests and can tell a good suggestion from a poor one.
- Test users see suggestions inside a separate test workspace, not in the live queue.
- Each coordinator records whether they accepted, changed or rejected every suggestion.
For each routine request in the test sample, the tool proposes a category and a one-line draft routing note.
- The coordinator verifies both the category and the note.
- No test output is sent to employees or routed into the live queue.
The tool does not reply to employees, close requests or change priorities.
- Urgent, sensitive or ambiguous requests are excluded from the sample and handled as they are today.
- Any request a coordinator is unsure about goes to the usual escalation route, with no AI suggestion attached.
Options compared
Illustrative judgments for this fictional example, not measured results. The chosen option is in the first option column.
| Criterion | Human-checked suggestions | Rules-only update | Automated routing |
|---|---|---|---|
| Effort to try | Small test on copied sample requests | Moderate: keyword rules rewritten by hand | High: connects to the live queue |
| Who decides routing | A coordinator, every time | A coordinator for exceptions | The system, reviewed afterwards |
| Unusual wording | Suggestion may miss it; coordinator catches it | Missed unless a rule exists | Can be misrouted without anyone noticing |
| Reversibility | Stop the test; nothing to unwind | Restore the previous rule list | Hard to undo once requests are routed |
| Main risk | Coordinators over-trust suggestions | Rules go stale | Wrong routing reaches employees |
Data and risks
Reviewers need the data scope and what could go wrong
Data scope
- A sample of about 300 closed, routine requests from the previous quarter, chosen by the data owner for the request system.
- Names, attachments and free-text fields that could hold personal or confidential details are removed or masked before use.
- Access is limited to the four test coordinators and the proposing team lead, and the copied sample is deleted when the test ends.
- The test begins only after the data owner approves this scope in writing.
Risks and how the test limits them
- Sensitive content exposure: a masked sample and a restricted workspace, with no live requests.
- Incorrect categorization: every suggestion is checked by a coordinator. AI edits and final test-decision errors are recorded separately against the reviewed reference categories.
- Uneven performance: results are broken down by request type and by short, vague or non-standard wording, so weak spots are visible and not averaged away.
- Drift toward live use: testing stays away from live decisions. Any move to the live queue needs a new proposal.
Decision record
A short record that lets a reviewer see what was decided, when, why and by whom
| Item | Entry |
|---|---|
| Decision date | 10 October 2026 |
| Decision | Assess a limited, human-checked prototype before any live use. |
| Options considered | Human-checked suggestions, a rules-only update, and automated routing (see the comparison above). |
| Rationale | The prototype can stop without unwinding live automated routing, and a named person remains accountable for every test decision. Its effort and cost are still recorded. Any live use needs a separate reviewed proposal. |
| Responsible owner | The proposing shared-services team lead, who owns the test plan and reports the results. |
| Review before testing | AI governance review of the data scope, boundaries and success criteria, and data owner approval of the sample. |
| Revisit when | The test sample has been reviewed, or if the scope, data or users change. |
Success criteria and readiness
Agree the measures before the test, not after
- Review time per request for manual-only and AI-assisted reviews, measured the same way on comparable approved historical sample groups
- Final coordinator-selected categories against the same reviewed reference categories in both groups, with sample sizes and request mix reported
- AI suggestions accepted, changed or rejected as a separate diagnostic—not a substitute for final-decision accuracy
- Results by request type and by vague or non-standard wording
- Every excluded or escalated case, with the reason it was excluded; no claim about live downstream re-categorization
- A coordinator checks every suggestion before any routing decision
- No live requests, employee replies or priority changes
- Sample data stays masked, access-controlled and deleted at the end of the test
- The problem and baseline are stated with how they were measured
- Intended users and exclusions are named
- The data scope has a named data owner
- Success measures and the test sample size are agreed
- An accountable owner is named