mirror of
https://github.com/wshobson/agents.git
synced 2026-03-18 17:47:16 +00:00
47a5dbc3f9
Remove references to non-existent resource files (references/, assets/, scripts/, examples/) from 115 skill SKILL.md files. These sections pointed to directories and files that were never created, causing confusion when users install skills. Also fix broken Code of Conduct links in issue templates to use absolute GitHub URLs instead of relative paths that 404.
310 lines
7.5 KiB
Markdown
310 lines
7.5 KiB
Markdown
---
|
|
name: on-call-handoff-patterns
|
|
description: Master on-call shift handoffs with context transfer, escalation procedures, and documentation. Use when transitioning on-call responsibilities, documenting shift summaries, or improving on-call processes.
|
|
---
|
|
|
|
# On-Call Handoff Patterns
|
|
|
|
Effective patterns for on-call shift transitions, ensuring continuity, context transfer, and reliable incident response across shifts.
|
|
|
|
## When to Use This Skill
|
|
|
|
- Transitioning on-call responsibilities
|
|
- Writing shift handoff summaries
|
|
- Documenting ongoing investigations
|
|
- Establishing on-call rotation procedures
|
|
- Improving handoff quality
|
|
- Onboarding new on-call engineers
|
|
|
|
## Core Concepts
|
|
|
|
### 1. Handoff Components
|
|
|
|
| Component | Purpose |
|
|
| -------------------------- | ----------------------- |
|
|
| **Active Incidents** | What's currently broken |
|
|
| **Ongoing Investigations** | Issues being debugged |
|
|
| **Recent Changes** | Deployments, configs |
|
|
| **Known Issues** | Workarounds in place |
|
|
| **Upcoming Events** | Maintenance, releases |
|
|
|
|
### 2. Handoff Timing
|
|
|
|
```
|
|
Recommended: 30 min overlap between shifts
|
|
|
|
Outgoing:
|
|
├── 15 min: Write handoff document
|
|
└── 15 min: Sync call with incoming
|
|
|
|
Incoming:
|
|
├── 15 min: Review handoff document
|
|
├── 15 min: Sync call with outgoing
|
|
└── 5 min: Verify alerting setup
|
|
```
|
|
|
|
## Templates
|
|
|
|
### Template 1: Shift Handoff Document
|
|
|
|
````markdown
|
|
# On-Call Handoff: Platform Team
|
|
|
|
**Outgoing**: @alice (2024-01-15 to 2024-01-22)
|
|
**Incoming**: @bob (2024-01-22 to 2024-01-29)
|
|
**Handoff Time**: 2024-01-22 09:00 UTC
|
|
|
|
---
|
|
|
|
## 🔴 Active Incidents
|
|
|
|
### None currently active
|
|
|
|
No active incidents at handoff time.
|
|
|
|
---
|
|
|
|
## 🟡 Ongoing Investigations
|
|
|
|
### 1. Intermittent API Timeouts (ENG-1234)
|
|
|
|
**Status**: Investigating
|
|
**Started**: 2024-01-20
|
|
**Impact**: ~0.1% of requests timing out
|
|
|
|
**Context**:
|
|
|
|
- Timeouts correlate with database backup window (02:00-03:00 UTC)
|
|
- Suspect backup process causing lock contention
|
|
- Added extra logging in PR #567 (deployed 01/21)
|
|
|
|
**Next Steps**:
|
|
|
|
- [ ] Review new logs after tonight's backup
|
|
- [ ] Consider moving backup window if confirmed
|
|
|
|
**Resources**:
|
|
|
|
- Dashboard: [API Latency](https://grafana/d/api-latency)
|
|
- Thread: #platform-eng (01/20, 14:32)
|
|
|
|
---
|
|
|
|
### 2. Memory Growth in Auth Service (ENG-1235)
|
|
|
|
**Status**: Monitoring
|
|
**Started**: 2024-01-18
|
|
**Impact**: None yet (proactive)
|
|
|
|
**Context**:
|
|
|
|
- Memory usage growing ~5% per day
|
|
- No memory leak found in profiling
|
|
- Suspect connection pool not releasing properly
|
|
|
|
**Next Steps**:
|
|
|
|
- [ ] Review heap dump from 01/21
|
|
- [ ] Consider restart if usage > 80%
|
|
|
|
**Resources**:
|
|
|
|
- Dashboard: [Auth Service Memory](https://grafana/d/auth-memory)
|
|
- Analysis doc: [Memory Investigation](https://docs/eng-1235)
|
|
|
|
---
|
|
|
|
## 🟢 Resolved This Shift
|
|
|
|
### Payment Service Outage (2024-01-19)
|
|
|
|
- **Duration**: 23 minutes
|
|
- **Root Cause**: Database connection exhaustion
|
|
- **Resolution**: Rolled back v2.3.4, increased pool size
|
|
- **Postmortem**: [POSTMORTEM-89](https://docs/postmortem-89)
|
|
- **Follow-up tickets**: ENG-1230, ENG-1231
|
|
|
|
---
|
|
|
|
## 📋 Recent Changes
|
|
|
|
### Deployments
|
|
|
|
| Service | Version | Time | Notes |
|
|
| ------------ | ------- | ----------- | -------------------------- |
|
|
| api-gateway | v3.2.1 | 01/21 14:00 | Bug fix for header parsing |
|
|
| user-service | v2.8.0 | 01/20 10:00 | New profile features |
|
|
| auth-service | v4.1.2 | 01/19 16:00 | Security patch |
|
|
|
|
### Configuration Changes
|
|
|
|
- 01/21: Increased API rate limit from 1000 to 1500 RPS
|
|
- 01/20: Updated database connection pool max from 50 to 75
|
|
|
|
### Infrastructure
|
|
|
|
- 01/20: Added 2 nodes to Kubernetes cluster
|
|
- 01/19: Upgraded Redis from 6.2 to 7.0
|
|
|
|
---
|
|
|
|
## ⚠️ Known Issues & Workarounds
|
|
|
|
### 1. Slow Dashboard Loading
|
|
|
|
**Issue**: Grafana dashboards slow on Monday mornings
|
|
**Workaround**: Wait 5 min after 08:00 UTC for cache warm-up
|
|
**Ticket**: OPS-456 (P3)
|
|
|
|
### 2. Flaky Integration Test
|
|
|
|
**Issue**: `test_payment_flow` fails intermittently in CI
|
|
**Workaround**: Re-run failed job (usually passes on retry)
|
|
**Ticket**: ENG-1200 (P2)
|
|
|
|
---
|
|
|
|
## 📅 Upcoming Events
|
|
|
|
| Date | Event | Impact | Contact |
|
|
| ----------- | -------------------- | ------------------- | ------------- |
|
|
| 01/23 02:00 | Database maintenance | 5 min read-only | @dba-team |
|
|
| 01/24 14:00 | Major release v5.0 | Monitor closely | @release-team |
|
|
| 01/25 | Marketing campaign | 2x traffic expected | @platform |
|
|
|
|
---
|
|
|
|
## 📞 Escalation Reminders
|
|
|
|
| Issue Type | First Escalation | Second Escalation |
|
|
| --------------- | -------------------- | ----------------- |
|
|
| Payment issues | @payments-oncall | @payments-manager |
|
|
| Auth issues | @auth-oncall | @security-team |
|
|
| Database issues | @dba-team | @infra-manager |
|
|
| Unknown/severe | @engineering-manager | @vp-engineering |
|
|
|
|
---
|
|
|
|
## 🔧 Quick Reference
|
|
|
|
### Common Commands
|
|
|
|
```bash
|
|
# Check service health
|
|
kubectl get pods -A | grep -v Running
|
|
|
|
# Recent deployments
|
|
kubectl get events --sort-by='.lastTimestamp' | tail -20
|
|
|
|
# Database connections
|
|
psql -c "SELECT count(*) FROM pg_stat_activity;"
|
|
|
|
# Clear cache (emergency only)
|
|
redis-cli FLUSHDB
|
|
```
|
|
````
|
|
|
|
### Important Links
|
|
|
|
- [Runbooks](https://wiki/runbooks)
|
|
- [Service Catalog](https://wiki/services)
|
|
- [Incident Slack](https://slack.com/incidents)
|
|
- [PagerDuty](https://pagerduty.com/schedules)
|
|
|
|
---
|
|
|
|
## Handoff Checklist
|
|
|
|
### Outgoing Engineer
|
|
|
|
- [x] Document active incidents
|
|
- [x] Document ongoing investigations
|
|
- [x] List recent changes
|
|
- [x] Note known issues
|
|
- [x] Add upcoming events
|
|
- [x] Sync with incoming engineer
|
|
|
|
### Incoming Engineer
|
|
|
|
- [ ] Read this document
|
|
- [ ] Join sync call
|
|
- [ ] Verify PagerDuty is routing to you
|
|
- [ ] Verify Slack notifications working
|
|
- [ ] Check VPN/access working
|
|
- [ ] Review critical dashboards
|
|
|
|
````
|
|
|
|
### Template 2: Quick Handoff (Async)
|
|
|
|
```markdown
|
|
# Quick Handoff: @alice → @bob
|
|
|
|
## TL;DR
|
|
- No active incidents
|
|
- 1 investigation ongoing (API timeouts, see ENG-1234)
|
|
- Major release tomorrow (01/24) - be ready for issues
|
|
|
|
## Watch List
|
|
1. API latency around 02:00-03:00 UTC (backup window)
|
|
2. Auth service memory (restart if > 80%)
|
|
|
|
## Recent
|
|
- Deployed api-gateway v3.2.1 yesterday (stable)
|
|
- Increased rate limits to 1500 RPS
|
|
|
|
## Coming Up
|
|
- 01/23 02:00 - DB maintenance (5 min read-only)
|
|
- 01/24 14:00 - v5.0 release
|
|
|
|
## Questions?
|
|
I'll be available on Slack until 17:00 today.
|
|
````
|
|
|
|
### Template 3: Incident Handoff (Mid-Incident)
|
|
|
|
```markdown
|
|
# INCIDENT HANDOFF: Payment Service Degradation
|
|
|
|
**Incident Start**: 2024-01-22 08:15 UTC
|
|
**Current Status**: Mitigating
|
|
**Severity**: SEV2
|
|
|
|
---
|
|
|
|
## Current State
|
|
|
|
- Error rate: 15% (down from 40%)
|
|
- Mitigation in progress: scaling up pods
|
|
- ETA to resolution: ~30 min
|
|
|
|
## What We Know
|
|
|
|
1. Root cause: Memory pressure on payment-service pods
|
|
2. Triggered by: Unusual traffic spike (3x normal)
|
|
3. Contributing: Inefficient query in checkout flow
|
|
|
|
## What We've Done
|
|
|
|
- Scaled payment-service from 5 → 15 pods
|
|
- Enabled rate limiting on checkout endpoint
|
|
- Disabled non-critical features
|
|
|
|
## What Needs to Happen
|
|
|
|
1. Monitor error rate - should reach <1% in ~15 min
|
|
2. If not improving, escalate to @payments-manager
|
|
3. Once stable, begin root cause investigation
|
|
|
|
## Key People
|
|
|
|
- Incident Commander: @alice (handing off)
|
|
- Comms Lead: @charlie
|
|
- Technical Lead: @bob (incoming)
|
|
|
|
## Communication
|
|
|
|
- Status page: Updated at 08:45
|
|
- Customer support: Notified
|
|
- Exec team: Aware
|