What we learned from the scheduler sign-in incident
A de-identified review for service teams and employees, covering what happened, why it reached staff, and who owns each fix.
Northstar Services is a fictional organization used for this worked example. The incident, times and figures are illustrative.
Impact in brief
Who was affected, and for how long
On the morning of 4 March, about 310 of Northstar's roughly 1,900 staff could not sign in to the internal scheduling tool. All of them had changed team or role in the previous week. Other systems and all other staff were unaffected.
The scheduling tool itself was healthy. The cause was in the identity directory: an overnight update of group memberships had not finished, so the scheduler rejected sign-ins for people whose access had just changed.
Supervisors posted shifts by message while the fix was applied, so no shifts were missed. No data was lost or exposed, and this review contains no individual employee or customer records.
The incident in numbers
Illustrative figures for the sample incident on 4 March. Counts are of the fictional Northstar workforce on that day.
Timeline: Tuesday 4 March
-
08:52
First sign-in reports reach the Service Desk
Three staff on the same team report an "access not authorised" message when opening the scheduler within ten minutes. -
09:25
Incident raised and the Scheduling Service team joins
The Service Desk links the reports, sees they share a recent team change, and raises an incident. The on-call scheduler engineer joins. -
09:40
Scheduler ruled out as the cause
The scheduling team confirms the application is healthy. Sign-in tokens are rejected only for accounts with group changes since the previous Tuesday. Identity Operations is asked to join. -
10:05
Stalled directory update found
Identity Operations finds that the 02:00 group-membership update started but never completed. It stopped at a validation stage, and no alert fired. -
10:25
Update restarted
Identity Operations restarts the update. Group memberships refresh for the affected accounts. -
10:40
Sign-ins recover
Test sign-ins from three affected teams succeed. Supervisors stop using message-based shift posting. -
11:15
Incident closed with a Service Desk update
The Service Desk tells affected teams the issue is resolved and asks them to report any sign-in problems that persist. -
Tue 18 Mar
Blameless review held
Identity Operations, the Scheduling Service team, the Service Desk and two supervisors review the timeline and agree the corrective actions below.
Contributing factors
The review separates the technical condition from the process gaps that let it reach staff. Neither is attributed to an individual.
What went wrong in the systems.
- The overnight group-membership update stopped at a validation stage and did not finish.
- The scheduler trusts directory group membership at sign-in, so accounts changed in the last week were rejected.
- The update job reported no error, so nothing alerted anyone.
Why a stalled job became a morning of failed sign-ins.
- The morning check on the directory update confirmed the job had started, not that it had finished.
- The scheduler's sign-in was not part of that check, so no one tested a recently changed account.
- Reports from one team took 33 minutes to be linked into one incident, because each was handled as a single-user request.
Corrective actions and owners
Owners are roles in this fictional example. Each action has one accountable owner and a check date counted from the review.
Alert when the directory update does not finish
- Alert the on-call identity engineer if the update has not completed by 03:30.
- Check date: 30 days after the review, by running a test update and confirming the alert fires.
Add a sign-in test for a recently changed account
- Include one account with a group change in the last seven days in the morning health check.
- Check date: 30 days after the review, by confirming the check fails when group data is stale.
Link related reports into one incident sooner
- When three reports from one team arrive within 15 minutes, raise an incident and ask what changed for those accounts.
- Check date: 60 days after the review, by sampling that month's multi-user reports.
Reading this review
Terms and routes for service teams and employees.
What does "blameless review" mean here?
The review asks how the systems and processes allowed the problem, not who caused it. Role names appear only to show who owns a follow-up.
Who owns this page and keeps it accurate?
IT and Security own this page. If you spot a factual error or a missing detail, send a correction through the Service Desk and name this review. The owner updates the page when an action's status or check date changes.
Is this page the place to check whether something is down now?
No. This page is written after recovery and is not a live status view. For a current problem, contact the Service Desk, who will direct you to the active incident updates.
When will a review like this be published?
A review follows an incident once the service has recovered and the blameless review meeting has taken place. In this example, that was 14 days after the incident.