Monitoring deployed AI: what to watch after launch

Deployment is the beginning of the operating lifecycle, not the end of evaluation.

Performance

Track task outcomes that matter to users and the business. Where ground truth arrives later, create delayed review samples rather than assuming stable quality.

Input and usage drift

Watch for changes in who is using the system, what they ask it to do, and which data or tools it touches. A stable model can behave differently when its operating environment changes.

Permission and action logs

For agents, monitor tool calls, failed actions, privilege changes, unusual sequences and human overrides.

Incidents and near misses

Near misses are valuable signals. Record patterns before they become expensive failures.

Change management

Model updates, prompt changes, new integrations and policy changes should trigger proportionate re-evaluation.