dpnmw/agents
1
0
Fork 0
mirror of https://github.com/wshobson/agents.git synced 2026-03-18 09:37:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add Agent Skills and restructure documentation

Browse Source 
- Add 47 Agent Skills across 14 plugins following Anthropic's specification
  - Python (5): async patterns, testing, packaging, performance, UV package manager
  - JavaScript/TypeScript (4): advanced types, Node.js patterns, testing, modern JS
  - Kubernetes (4): manifests, Helm charts, GitOps, security policies
  - Cloud Infrastructure (4): Terraform, multi-cloud, hybrid networking, cost optimization
  - CI/CD (4): pipeline design, GitHub Actions, GitLab CI, secrets management
  - Backend (3): API design, architecture patterns, microservices
  - LLM Applications (4): LangChain, prompt engineering, RAG, evaluation
  - Blockchain/Web3 (4): DeFi protocols, NFT standards, Solidity security, Web3 testing
  - Framework Migration (4): React, Angular, database, dependency upgrades
  - Observability (4): Prometheus, Grafana, distributed tracing, SLO
  - Payment Processing (4): Stripe, PayPal, PCI compliance, billing
  - API Scaffolding (1): FastAPI templates
  - ML Operations (1): ML pipeline workflow
  - Security (1): SAST configuration

- Restructure documentation into /docs directory
  - agent-skills.md: Complete guide to all 47 skills
  - agents.md: All 85 agents with model configuration
  - plugins.md: Complete catalog of 63 plugins
  - usage.md: Commands, workflows, and best practices
  - architecture.md: Design principles and patterns

- Update README.md
  - Add Agent Skills banner announcement
  - Reduce length by ~75% with links to detailed docs
  - Add What's New section showcasing Agent Skills
  - Add Popular Use Cases with real examples
  - Improve navigation with Core Guides and Quick Links

- Update marketplace.json with skills arrays for 14 plugins

All 47 skills follow Agent Skills Specification:
- Required YAML frontmatter (name, description)
- Use when activation clauses
- Progressive disclosure architecture
- Under 1024 character descriptions
This commit is contained in:
Seth Hobson
2025-10-16 20:33:27 -04:00
parent 1962091501
commit 65e5cb093a
80 changed files with 32158 additions and 1025 deletions
Download Patch File Download Diff File Expand all files Collapse all files

75
.claude-plugin/marketplace.json
View File

@@ -126,6 +126,11 @@
"./agents/backend-architect.md",
"./agents/graphql-architect.md",
"./agents/tdd-orchestrator.md"
],
"skills": [
"./skills/api-design-principles",
"./skills/architecture-patterns",
"./skills/microservices-patterns"
]
},
{
@@ -421,6 +426,12 @@
"agents": [
"./agents/ai-engineer.md",
"./agents/prompt-engineer.md"
],
"skills": [
"./skills/langchain-architecture",
"./skills/llm-evaluation",
"./skills/prompt-engineering-patterns",
"./skills/rag-implementation"
]
},
{
@@ -508,6 +519,9 @@
"./agents/data-scientist.md",
"./agents/ml-engineer.md",
"./agents/mlops-engineer.md"
],
"skills": [
"./skills/ml-pipeline-workflow"
]
},
{
@@ -660,6 +674,12 @@
"./agents/performance-engineer.md",
"./agents/database-optimizer.md",
"./agents/network-engineer.md"
],
"skills": [
"./skills/distributed-tracing",
"./skills/grafana-dashboards",
"./skills/prometheus-configuration",
"./skills/slo-implementation"
]
},
{
@@ -741,6 +761,12 @@
"commands": [],
"agents": [
"./agents/kubernetes-architect.md"
],
"skills": [
"./skills/gitops-workflow",
"./skills/helm-chart-scaffolding",
"./skills/k8s-manifest-generator",
"./skills/k8s-security-policies"
]
},
{
@@ -774,6 +800,12 @@
"./agents/terraform-specialist.md",
"./agents/network-engineer.md",
"./agents/deployment-engineer.md"
],
"skills": [
"./skills/cost-optimization",
"./skills/hybrid-cloud-networking",
"./skills/multi-cloud-architecture",
"./skills/terraform-module-library"
]
},
{
@@ -806,6 +838,12 @@
"./agents/kubernetes-architect.md",
"./agents/cloud-architect.md",
"./agents/terraform-specialist.md"
],
"skills": [
"./skills/deployment-pipeline-design",
"./skills/github-actions-templates",
"./skills/gitlab-ci-patterns",
"./skills/secrets-management"
]
},
{
@@ -955,6 +993,12 @@
"agents": [
"./agents/legacy-modernizer.md",
"./agents/architect-review.md"
],
"skills": [
"./skills/angular-migration",
"./skills/database-migration",
"./skills/dependency-upgrade",
"./skills/react-modernization"
]
},
{
@@ -1071,6 +1115,9 @@
],
"agents": [
"./agents/security-auditor.md"
],
"skills": [
"./skills/sast-configuration"
]
},
{
@@ -1213,6 +1260,9 @@
"./agents/graphql-architect.md",
"./agents/fastapi-pro.md",
"./agents/django-pro.md"
],
"skills": [
"./skills/fastapi-templates"
]
},
{
@@ -1529,6 +1579,12 @@
"commands": [],
"agents": [
"./agents/blockchain-developer.md"
],
"skills": [
"./skills/defi-protocol-templates",
"./skills/nft-standards",
"./skills/solidity-security",
"./skills/web3-testing"
]
},
{
@@ -1584,6 +1640,12 @@
"commands": [],
"agents": [
"./agents/payment-integration.md"
],
"skills": [
"./skills/billing-automation",
"./skills/paypal-integration",
"./skills/pci-compliance",
"./skills/stripe-integration"
]
},
{
@@ -1670,6 +1732,13 @@
"./agents/python-pro.md",
"./agents/django-pro.md",
"./agents/fastapi-pro.md"
],
"skills": [
"./skills/async-python-patterns",
"./skills/python-testing-patterns",
"./skills/python-packaging",
"./skills/python-performance-optimization",
"./skills/uv-package-manager"
]
},
{
@@ -1699,6 +1768,12 @@
"agents": [
"./agents/javascript-pro.md",
"./agents/typescript-pro.md"
],
"skills": [
"./skills/typescript-advanced-types",
"./skills/nodejs-backend-patterns",
"./skills/javascript-testing-patterns",
"./skills/modern-javascript-patterns"
]
},
{

1207
README.md
View File

File diff suppressed because it is too large Load Diff

224
docs/agent-skills.md Normal file
View File

@@ -0,0 +1,224 @@
# Agent Skills
Agent Skills are modular packages that extend Claude's capabilities with specialized domain knowledge, following Anthropic's [Agent Skills Specification](https://github.com/anthropics/skills/blob/main/agent_skills_spec.md). This plugin ecosystem includes **47 specialized skills** across 14 plugins, enabling progressive disclosure and efficient token usage.
## Overview
Skills provide Claude with deep expertise in specific domains without loading everything into context upfront. Each skill includes:
- **YAML Frontmatter**: Name and activation criteria
- **Progressive Disclosure**: Metadata → Instructions → Resources
- **Activation Triggers**: Clear "Use when" clauses for automatic invocation
## Skills by Plugin
### Kubernetes Operations (4 skills)
| Skill | Description |
|-------|-------------|
| **k8s-manifest-generator** | Create production-ready Kubernetes manifests for Deployments, Services, ConfigMaps, and Secrets following best practices |
| **helm-chart-scaffolding** | Design, organize, and manage Helm charts for templating and packaging Kubernetes applications |
| **gitops-workflow** | Implement GitOps workflows with ArgoCD and Flux for automated, declarative deployments |
| **k8s-security-policies** | Implement Kubernetes security policies including NetworkPolicy, PodSecurityPolicy, and RBAC |
### LLM Application Development (4 skills)
| Skill | Description |
|-------|-------------|
| **langchain-architecture** | Design LLM applications using LangChain framework with agents, memory, and tool integration |
| **prompt-engineering-patterns** | Master advanced prompt engineering techniques for LLM performance and reliability |
| **rag-implementation** | Build Retrieval-Augmented Generation systems with vector databases and semantic search |
| **llm-evaluation** | Implement comprehensive evaluation strategies with automated metrics and benchmarking |
### Backend Development (3 skills)
| Skill | Description |
|-------|-------------|
| **api-design-principles** | Master REST and GraphQL API design for intuitive, scalable, and maintainable APIs |
| **architecture-patterns** | Implement Clean Architecture, Hexagonal Architecture, and Domain-Driven Design |
| **microservices-patterns** | Design microservices with service boundaries, event-driven communication, and resilience |
### Blockchain & Web3 (4 skills)
| Skill | Description |
|-------|-------------|
| **defi-protocol-templates** | Implement DeFi protocols with templates for staking, AMMs, governance, and lending |
| **nft-standards** | Implement NFT standards (ERC-721, ERC-1155) with metadata and marketplace integration |
| **solidity-security** | Master smart contract security to prevent vulnerabilities and implement secure patterns |
| **web3-testing** | Test smart contracts using Hardhat and Foundry with unit tests and mainnet forking |
### CI/CD Automation (4 skills)
| Skill | Description |
|-------|-------------|
| **deployment-pipeline-design** | Design multi-stage CI/CD pipelines with approval gates and security checks |
| **github-actions-templates** | Create production-ready GitHub Actions workflows for testing, building, and deploying |
| **gitlab-ci-patterns** | Build GitLab CI/CD pipelines with multi-stage workflows and distributed runners |
| **secrets-management** | Implement secure secrets management using Vault, AWS Secrets Manager, or native solutions |
### Cloud Infrastructure (4 skills)
| Skill | Description |
|-------|-------------|
| **terraform-module-library** | Build reusable Terraform modules for AWS, Azure, and GCP infrastructure |
| **multi-cloud-architecture** | Design multi-cloud architectures avoiding vendor lock-in |
| **hybrid-cloud-networking** | Configure secure connectivity between on-premises and cloud platforms |
| **cost-optimization** | Optimize cloud costs through rightsizing, tagging, and reserved instances |
### Framework Migration (4 skills)
| Skill | Description |
|-------|-------------|
| **react-modernization** | Upgrade React apps, migrate to hooks, and adopt concurrent features |
| **angular-migration** | Migrate from AngularJS to Angular using hybrid mode and incremental rewriting |
| **database-migration** | Execute database migrations with zero-downtime strategies and transformations |
| **dependency-upgrade** | Manage major dependency upgrades with compatibility analysis and testing |
### Observability & Monitoring (4 skills)
| Skill | Description |
|-------|-------------|
| **prometheus-configuration** | Set up Prometheus for comprehensive metric collection and monitoring |
| **grafana-dashboards** | Create production Grafana dashboards for real-time system visualization |
| **distributed-tracing** | Implement distributed tracing with Jaeger and Tempo to track requests |
| **slo-implementation** | Define SLIs and SLOs with error budgets and alerting |
### Payment Processing (4 skills)
| Skill | Description |
|-------|-------------|
| **stripe-integration** | Implement Stripe payment processing for checkout, subscriptions, and webhooks |
| **paypal-integration** | Integrate PayPal payment processing with express checkout and subscriptions |
| **pci-compliance** | Implement PCI DSS compliance for secure payment card data handling |
| **billing-automation** | Build automated billing systems for recurring payments and invoicing |
### Python Development (5 skills)
| Skill | Description |
|-------|-------------|
| **async-python-patterns** | Master Python asyncio, concurrent programming, and async/await patterns |
| **python-testing-patterns** | Implement comprehensive testing with pytest, fixtures, and mocking |
| **python-packaging** | Create distributable Python packages with proper structure and PyPI publishing |
| **python-performance-optimization** | Profile and optimize Python code using cProfile and performance best practices |
| **uv-package-manager** | Master the uv package manager for fast dependency management and virtual environments |
### JavaScript/TypeScript (4 skills)
| Skill | Description |
|-------|-------------|
| **typescript-advanced-types** | Master TypeScript's advanced type system including generics and conditional types |
| **nodejs-backend-patterns** | Build production-ready Node.js services with Express/Fastify and best practices |
| **javascript-testing-patterns** | Implement comprehensive testing with Jest, Vitest, and Testing Library |
| **modern-javascript-patterns** | Master ES6+ features including async/await, destructuring, and functional programming |
### API Scaffolding (1 skill)
| Skill | Description |
|-------|-------------|
| **fastapi-templates** | Create production-ready FastAPI projects with async patterns and error handling |
### Machine Learning Operations (1 skill)
| Skill | Description |
|-------|-------------|
| **ml-pipeline-workflow** | Build end-to-end MLOps pipelines from data preparation through deployment |
### Security Scanning (1 skill)
| Skill | Description |
|-------|-------------|
| **sast-configuration** | Configure Static Application Security Testing tools for vulnerability detection |
## How Skills Work
### Activation
Skills are automatically activated when Claude detects matching patterns in your request:
```
User: "Set up Kubernetes deployment with Helm chart"
→ Activates: helm-chart-scaffolding, k8s-manifest-generator
User: "Build a RAG system for document Q&A"
→ Activates: rag-implementation, prompt-engineering-patterns
User: "Optimize Python async performance"
→ Activates: async-python-patterns, python-performance-optimization
```
### Progressive Disclosure
Skills use a three-tier architecture for token efficiency:
1. **Metadata** (Frontmatter): Name and activation criteria (always loaded)
2. **Instructions**: Core guidance and patterns (loaded when activated)
3. **Resources**: Examples and templates (loaded on demand)
### Integration with Agents
Skills work alongside agents to provide deep domain expertise:
- **Agents**: High-level reasoning and orchestration
- **Skills**: Specialized knowledge and implementation patterns
Example workflow:
```
backend-architect agent → Plans API architecture
↓
api-design-principles skill → Provides REST/GraphQL best practices
↓
fastapi-templates skill → Supplies production-ready templates
```
## Specification Compliance
All 47 skills follow the [Agent Skills Specification](https://github.com/anthropics/skills/blob/main/agent_skills_spec.md):
- ✓ Required `name` field (hyphen-case)
- ✓ Required `description` field with "Use when" clause
- ✓ Descriptions under 1024 characters
- ✓ Complete, non-truncated descriptions
- ✓ Proper YAML frontmatter formatting
## Creating New Skills
To add a skill to a plugin:
1. Create `plugins/{plugin-name}/skills/{skill-name}/SKILL.md`
2. Add YAML frontmatter:
```yaml
---
name: skill-name
description: What the skill does. Use when [activation trigger].
---
```
3. Write comprehensive skill content using progressive disclosure
4. Add skill path to `marketplace.json`:
```json
{
"name": "plugin-name",
"skills": ["./skills/skill-name"]
}
```
### Skill Structure
```
plugins/{plugin-name}/
└── skills/
└── {skill-name}/
└── SKILL.md # Frontmatter + content
```
## Benefits
- **Token Efficiency**: Load only relevant knowledge when needed
- **Specialized Expertise**: Deep domain knowledge without bloat
- **Clear Activation**: Explicit triggers prevent unwanted invocation
- **Composability**: Mix and match skills across workflows
- **Maintainability**: Isolated updates don't affect other skills
## Resources
- [Anthropic Skills Repository](https://github.com/anthropics/skills)
- [Agent Skills Documentation](https://docs.claude.com/en/docs/claude-code/skills)

325
docs/agents.md Normal file
View File

@@ -0,0 +1,325 @@
# Agent Reference
Complete reference for all **85 specialized AI agents** organized by category with model assignments.
## Agent Categories
### Architecture & System Design
#### Core Architecture
| Agent | Model | Description |
|-------|-------|-------------|
| [backend-architect](../plugins/backend-development/agents/backend-architect.md) | opus | RESTful API design, microservice boundaries, database schemas |
| [frontend-developer](../plugins/multi-platform-apps/agents/frontend-developer.md) | sonnet | React components, responsive layouts, client-side state management |
| [graphql-architect](../plugins/backend-development/agents/graphql-architect.md) | opus | GraphQL schemas, resolvers, federation architecture |
| [architect-reviewer](../plugins/comprehensive-review/agents/architect-review.md) | opus | Architectural consistency analysis and pattern validation |
| [cloud-architect](../plugins/cloud-infrastructure/agents/cloud-architect.md) | opus | AWS/Azure/GCP infrastructure design and cost optimization |
| [hybrid-cloud-architect](../plugins/cloud-infrastructure/agents/hybrid-cloud-architect.md) | opus | Multi-cloud strategies across cloud and on-premises environments |
| [kubernetes-architect](../plugins/kubernetes-operations/agents/kubernetes-architect.md) | opus | Cloud-native infrastructure with Kubernetes and GitOps |
#### UI/UX & Mobile
| Agent | Model | Description |
|-------|-------|-------------|
| [ui-ux-designer](../plugins/multi-platform-apps/agents/ui-ux-designer.md) | sonnet | Interface design, wireframes, design systems |
| [ui-visual-validator](../plugins/accessibility-compliance/agents/ui-visual-validator.md) | sonnet | Visual regression testing and UI verification |
| [mobile-developer](../plugins/multi-platform-apps/agents/mobile-developer.md) | sonnet | React Native and Flutter application development |
| [ios-developer](../plugins/multi-platform-apps/agents/ios-developer.md) | sonnet | Native iOS development with Swift/SwiftUI |
| [flutter-expert](../plugins/multi-platform-apps/agents/flutter-expert.md) | sonnet | Advanced Flutter development with state management |
### Programming Languages
#### Systems & Low-Level
| Agent | Model | Description |
|-------|-------|-------------|
| [c-pro](../plugins/systems-programming/agents/c-pro.md) | sonnet | System programming with memory management and OS interfaces |
| [cpp-pro](../plugins/systems-programming/agents/cpp-pro.md) | sonnet | Modern C++ with RAII, smart pointers, STL algorithms |
| [rust-pro](../plugins/systems-programming/agents/rust-pro.md) | sonnet | Memory-safe systems programming with ownership patterns |
| [golang-pro](../plugins/systems-programming/agents/golang-pro.md) | sonnet | Concurrent programming with goroutines and channels |
#### Web & Application
| Agent | Model | Description |
|-------|-------|-------------|
| [javascript-pro](../plugins/javascript-typescript/agents/javascript-pro.md) | sonnet | Modern JavaScript with ES6+, async patterns, Node.js |
| [typescript-pro](../plugins/javascript-typescript/agents/typescript-pro.md) | sonnet | Advanced TypeScript with type systems and generics |
| [python-pro](../plugins/python-development/agents/python-pro.md) | sonnet | Python development with advanced features and optimization |
| [ruby-pro](../plugins/web-scripting/agents/ruby-pro.md) | sonnet | Ruby with metaprogramming, Rails patterns, gem development |
| [php-pro](../plugins/web-scripting/agents/php-pro.md) | sonnet | Modern PHP with frameworks and performance optimization |
#### Enterprise & JVM
| Agent | Model | Description |
|-------|-------|-------------|
| [java-pro](../plugins/jvm-languages/agents/java-pro.md) | sonnet | Modern Java with streams, concurrency, JVM optimization |
| [scala-pro](../plugins/jvm-languages/agents/scala-pro.md) | sonnet | Enterprise Scala with functional programming and distributed systems |
| [csharp-pro](../plugins/jvm-languages/agents/csharp-pro.md) | sonnet | C# development with .NET frameworks and patterns |
#### Specialized Platforms
| Agent | Model | Description |
|-------|-------|-------------|
| [elixir-pro](../plugins/functional-programming/agents/elixir-pro.md) | sonnet | Elixir with OTP patterns and Phoenix frameworks |
| [django-pro](../plugins/api-scaffolding/agents/django-pro.md) | sonnet | Django development with ORM and async views |
| [fastapi-pro](../plugins/api-scaffolding/agents/fastapi-pro.md) | sonnet | FastAPI with async patterns and Pydantic |
| [unity-developer](../plugins/game-development/agents/unity-developer.md) | sonnet | Unity game development and optimization |
| [minecraft-bukkit-pro](../plugins/game-development/agents/minecraft-bukkit-pro.md) | sonnet | Minecraft server plugin development |
| [sql-pro](../plugins/database-design/agents/sql-pro.md) | sonnet | Complex SQL queries and database optimization |
### Infrastructure & Operations
#### DevOps & Deployment
| Agent | Model | Description |
|-------|-------|-------------|
| [devops-troubleshooter](../plugins/incident-response/agents/devops-troubleshooter.md) | sonnet | Production debugging, log analysis, deployment troubleshooting |
| [deployment-engineer](../plugins/cloud-infrastructure/agents/deployment-engineer.md) | sonnet | CI/CD pipelines, containerization, cloud deployments |
| [terraform-specialist](../plugins/cloud-infrastructure/agents/terraform-specialist.md) | sonnet | Infrastructure as Code with Terraform modules and state management |
| [dx-optimizer](../plugins/team-collaboration/agents/dx-optimizer.md) | sonnet | Developer experience optimization and tooling improvements |
#### Database Management
| Agent | Model | Description |
|-------|-------|-------------|
| [database-optimizer](../plugins/observability-monitoring/agents/database-optimizer.md) | sonnet | Query optimization, index design, migration strategies |
| [database-admin](../plugins/database-migrations/agents/database-admin.md) | sonnet | Database operations, backup, replication, monitoring |
| [database-architect](../plugins/database-design/agents/database-architect.md) | opus | Database design from scratch, technology selection, schema modeling |
#### Incident Response & Network
| Agent | Model | Description |
|-------|-------|-------------|
| [incident-responder](../plugins/incident-response/agents/incident-responder.md) | opus | Production incident management and resolution |
| [network-engineer](../plugins/observability-monitoring/agents/network-engineer.md) | sonnet | Network debugging, load balancing, traffic analysis |
### Quality Assurance & Security
#### Code Quality & Review
| Agent | Model | Description |
|-------|-------|-------------|
| [code-reviewer](../plugins/comprehensive-review/agents/code-reviewer.md) | opus | Code review with security focus and production reliability |
| [security-auditor](../plugins/comprehensive-review/agents/security-auditor.md) | opus | Vulnerability assessment and OWASP compliance |
| [backend-security-coder](../plugins/data-validation-suite/agents/backend-security-coder.md) | opus | Secure backend coding practices, API security implementation |
| [frontend-security-coder](../plugins/frontend-mobile-security/agents/frontend-security-coder.md) | opus | XSS prevention, CSP implementation, client-side security |
| [mobile-security-coder](../plugins/frontend-mobile-security/agents/mobile-security-coder.md) | opus | Mobile security patterns, WebView security, biometric auth |
#### Testing & Debugging
| Agent | Model | Description |
|-------|-------|-------------|
| [test-automator](../plugins/codebase-cleanup/agents/test-automator.md) | sonnet | Comprehensive test suite creation (unit, integration, e2e) |
| [tdd-orchestrator](../plugins/backend-development/agents/tdd-orchestrator.md) | sonnet | Test-Driven Development methodology guidance |
| [debugger](../plugins/error-debugging/agents/debugger.md) | sonnet | Error resolution and test failure analysis |
| [error-detective](../plugins/error-debugging/agents/error-detective.md) | sonnet | Log analysis and error pattern recognition |
#### Performance & Observability
| Agent | Model | Description |
|-------|-------|-------------|
| [performance-engineer](../plugins/observability-monitoring/agents/performance-engineer.md) | opus | Application profiling and optimization |
| [observability-engineer](../plugins/observability-monitoring/agents/observability-engineer.md) | opus | Production monitoring, distributed tracing, SLI/SLO management |
| [search-specialist](../plugins/content-marketing/agents/search-specialist.md) | haiku | Advanced web research and information synthesis |
### Data & AI
#### Data Engineering & Analytics
| Agent | Model | Description |
|-------|-------|-------------|
| [data-scientist](../plugins/machine-learning-ops/agents/data-scientist.md) | opus | Data analysis, SQL queries, BigQuery operations |
| [data-engineer](../plugins/data-engineering/agents/data-engineer.md) | sonnet | ETL pipelines, data warehouses, streaming architectures |
#### Machine Learning & AI
| Agent | Model | Description |
|-------|-------|-------------|
| [ai-engineer](../plugins/llm-application-dev/agents/ai-engineer.md) | opus | LLM applications, RAG systems, prompt pipelines |
| [ml-engineer](../plugins/machine-learning-ops/agents/ml-engineer.md) | opus | ML pipelines, model serving, feature engineering |
| [mlops-engineer](../plugins/machine-learning-ops/agents/mlops-engineer.md) | opus | ML infrastructure, experiment tracking, model registries |
| [prompt-engineer](../plugins/llm-application-dev/agents/prompt-engineer.md) | opus | LLM prompt optimization and engineering |
### Documentation & Technical Writing
| Agent | Model | Description |
|-------|-------|-------------|
| [docs-architect](../plugins/code-documentation/agents/docs-architect.md) | opus | Comprehensive technical documentation generation |
| [api-documenter](../plugins/api-testing-observability/agents/api-documenter.md) | sonnet | OpenAPI/Swagger specifications and developer docs |
| [reference-builder](../plugins/documentation-generation/agents/reference-builder.md) | haiku | Technical references and API documentation |
| [tutorial-engineer](../plugins/code-documentation/agents/tutorial-engineer.md) | sonnet | Step-by-step tutorials and educational content |
| [mermaid-expert](../plugins/documentation-generation/agents/mermaid-expert.md) | sonnet | Diagram creation (flowcharts, sequences, ERDs) |
### Business & Operations
#### Business Analysis & Finance
| Agent | Model | Description |
|-------|-------|-------------|
| [business-analyst](../plugins/business-analytics/agents/business-analyst.md) | sonnet | Metrics analysis, reporting, KPI tracking |
| [quant-analyst](../plugins/quantitative-trading/agents/quant-analyst.md) | opus | Financial modeling, trading strategies, market analysis |
| [risk-manager](../plugins/quantitative-trading/agents/risk-manager.md) | sonnet | Portfolio risk monitoring and management |
#### Marketing & Sales
| Agent | Model | Description |
|-------|-------|-------------|
| [content-marketer](../plugins/content-marketing/agents/content-marketer.md) | sonnet | Blog posts, social media, email campaigns |
| [sales-automator](../plugins/customer-sales-automation/agents/sales-automator.md) | haiku | Cold emails, follow-ups, proposal generation |
#### Support & Legal
| Agent | Model | Description |
|-------|-------|-------------|
| [customer-support](../plugins/customer-sales-automation/agents/customer-support.md) | sonnet | Support tickets, FAQ responses, customer communication |
| [hr-pro](../plugins/hr-legal-compliance/agents/hr-pro.md) | opus | HR operations, policies, employee relations |
| [legal-advisor](../plugins/hr-legal-compliance/agents/legal-advisor.md) | opus | Privacy policies, terms of service, legal documentation |
### SEO & Content Optimization
| Agent | Model | Description |
|-------|-------|-------------|
| [seo-content-auditor](../plugins/seo-content-creation/agents/seo-content-auditor.md) | sonnet | Content quality analysis, E-E-A-T signals assessment |
| [seo-meta-optimizer](../plugins/seo-technical-optimization/agents/seo-meta-optimizer.md) | haiku | Meta title and description optimization |
| [seo-keyword-strategist](../plugins/seo-technical-optimization/agents/seo-keyword-strategist.md) | haiku | Keyword analysis and semantic variations |
| [seo-structure-architect](../plugins/seo-technical-optimization/agents/seo-structure-architect.md) | haiku | Content structure and schema markup |
| [seo-snippet-hunter](../plugins/seo-technical-optimization/agents/seo-snippet-hunter.md) | haiku | Featured snippet formatting |
| [seo-content-refresher](../plugins/seo-analysis-monitoring/agents/seo-content-refresher.md) | haiku | Content freshness analysis |
| [seo-cannibalization-detector](../plugins/seo-analysis-monitoring/agents/seo-cannibalization-detector.md) | haiku | Keyword overlap detection |
| [seo-authority-builder](../plugins/seo-analysis-monitoring/agents/seo-authority-builder.md) | sonnet | E-E-A-T signal analysis |
| [seo-content-writer](../plugins/seo-content-creation/agents/seo-content-writer.md) | sonnet | SEO-optimized content creation |
| [seo-content-planner](../plugins/seo-content-creation/agents/seo-content-planner.md) | haiku | Content planning and topic clusters |
### Specialized Domains
| Agent | Model | Description |
|-------|-------|-------------|
| [arm-cortex-expert](../plugins/arm-cortex-microcontrollers/agents/arm-cortex-expert.md) | sonnet | ARM Cortex-M firmware and peripheral driver development |
| [blockchain-developer](../plugins/blockchain-web3/agents/blockchain-developer.md) | sonnet | Web3 apps, smart contracts, DeFi protocols |
| [payment-integration](../plugins/payment-processing/agents/payment-integration.md) | sonnet | Payment processor integration (Stripe, PayPal) |
| [legacy-modernizer](../plugins/framework-migration/agents/legacy-modernizer.md) | sonnet | Legacy code refactoring and modernization |
| [context-manager](../plugins/agent-orchestration/agents/context-manager.md) | haiku | Multi-agent context management |
## Model Configuration
Agents are assigned to specific Claude models based on task complexity and computational requirements.
### Model Distribution Summary
| Model | Agent Count | Use Case |
|-------|-------------|----------|
| Haiku | 47 | Fast execution tasks: testing, documentation, ops, database optimization, business |
| Sonnet | 97 | Complex reasoning, architecture, language expertise, orchestration, security |
### Model Selection Criteria
#### Haiku - Fast Execution & Deterministic Tasks
**Use when:**
- Generating code from well-defined specifications
- Creating tests following established patterns
- Writing documentation with clear templates
- Executing infrastructure operations
- Performing database query optimization
- Handling customer support responses
- Processing SEO optimization tasks
- Managing deployment pipelines
#### Sonnet - Complex Reasoning & Architecture
**Use when:**
- Designing system architecture
- Making technology selection decisions
- Performing security audits
- Reviewing code for architectural patterns
- Creating complex AI/ML pipelines
- Providing language-specific expertise
- Orchestrating multi-agent workflows
- Handling business-critical legal/HR matters
### Hybrid Orchestration Patterns
The plugin ecosystem leverages Sonnet + Haiku orchestration for optimal performance and cost efficiency:
#### Pattern 1: Planning → Execution
```
Sonnet: backend-architect (design API architecture)
↓
Haiku: Generate API endpoints following spec
↓
Haiku: test-automator (generate comprehensive tests)
↓
Sonnet: code-reviewer (architectural review)
```
#### Pattern 2: Reasoning → Action (Incident Response)
```
Sonnet: incident-responder (diagnose issue, create strategy)
↓
Haiku: devops-troubleshooter (execute fixes)
↓
Haiku: deployment-engineer (deploy hotfix)
↓
Haiku: Implement monitoring alerts
```
#### Pattern 3: Complex → Simple (Database Design)
```
Sonnet: database-architect (schema design, technology selection)
↓
Haiku: sql-pro (generate migration scripts)
↓
Haiku: database-admin (execute migrations)
↓
Haiku: database-optimizer (tune query performance)
```
#### Pattern 4: Multi-Agent Workflows
```
Full-Stack Feature Development:
Sonnet: backend-architect + frontend-developer (design components)
↓
Haiku: Generate code following designs
↓
Haiku: test-automator (unit + integration tests)
↓
Sonnet: security-auditor (security review)
↓
Haiku: deployment-engineer (CI/CD setup)
↓
Haiku: Setup observability stack
```
## Agent Invocation
### Natural Language
Agents can be invoked through natural language when you need Claude to reason about which specialist to use:
```
"Use backend-architect to design the authentication API"
"Have security-auditor scan for OWASP vulnerabilities"
"Get performance-engineer to optimize this database query"
```
### Slash Commands
Many agents are accessible through plugin slash commands for direct invocation:
```bash
/backend-development:feature-development user authentication
/security-scanning:security-sast
/incident-response:smart-fix "memory leak in payment service"
```
## Contributing
To add a new agent:
1. Create `plugins/{plugin-name}/agents/{agent-name}.md`
2. Add frontmatter with name, description, and model assignment
3. Write comprehensive system prompt
4. Update plugin definition in `.claude-plugin/marketplace.json`
See [Contributing Guide](../CONTRIBUTING.md) for details.

379
docs/architecture.md Normal file
View File

@@ -0,0 +1,379 @@
# Architecture & Design Principles
This marketplace follows industry best practices with a focus on granularity, composability, and minimal token usage.
## Core Philosophy
### Single Responsibility Principle
- Each plugin does **one thing well** (Unix philosophy)
- Clear, focused purposes (describable in 5-10 words)
- Average plugin size: **3.4 components** (follows Anthropic's 2-8 pattern)
- **Zero bloated plugins** - all plugins focused and purposeful
### Composability Over Bundling
- Mix and match plugins based on needs
- Workflow orchestrators compose focused plugins
- No forced feature bundling
- Clear boundaries between plugins
### Context Efficiency
- Smaller tools = faster processing
- Better fit in LLM context windows
- More accurate, focused responses
- Install only what you need
### Maintainability
- Single-purpose = easier updates
- Clear boundaries = isolated changes
- Less duplication = simpler maintenance
- Isolated dependencies
## Granular Plugin Architecture
### Plugin Distribution
- **63 focused plugins** optimized for specific use cases
- **23 clear categories** with 1-6 plugins each for easy discovery
- Organized by domain:
- **Development**: 4 plugins (debugging, backend, frontend, multi-platform)
- **Security**: 4 plugins (scanning, compliance, backend-api, frontend-mobile)
- **Operations**: 4 plugins (incident, diagnostics, distributed, observability)
- **Languages**: 7 plugins (Python, JS/TS, systems, JVM, scripting, functional, embedded)
- **Infrastructure**: 5 plugins (deployment, validation, K8s, cloud, CI/CD)
- And 18 more specialized categories
### Component Breakdown
**85 Specialized Agents**
- Domain experts with deep knowledge
- Organized across architecture, languages, infrastructure, quality, data/AI, documentation, business, and SEO
- Model-optimized (47 Haiku, 97 Sonnet) for performance and cost
**15 Workflow Orchestrators**
- Multi-agent coordination systems
- Complex operations like full-stack development, security hardening, ML pipelines, incident response
- Pre-configured agent workflows
**44 Development Tools**
- Optimized utilities including:
- Project scaffolding (Python, TypeScript, Rust)
- Security scanning (SAST, dependency audit, XSS)
- Test generation (pytest, Jest)
- Component scaffolding (React, React Native)
- Infrastructure setup (Terraform, Kubernetes)
**47 Agent Skills**
- Modular knowledge packages
- Progressive disclosure architecture
- Domain-specific expertise across 14 plugins
- Spec-compliant (Anthropic Agent Skills Specification)
## Repository Structure
```
claude-agents/
├── .claude-plugin/
│ └── marketplace.json # Marketplace catalog (63 plugins)
├── plugins/ # Isolated plugin directories
│ ├── python-development/
│ │ ├── agents/ # Python language agents
│ │ │ ├── python-pro.md
│ │ │ ├── django-pro.md
│ │ │ └── fastapi-pro.md
│ │ ├── commands/ # Python tooling
│ │ │ └── python-scaffold.md
│ │ └── skills/ # Python skills (5 total)
│ │ ├── async-python-patterns/
│ │ ├── python-testing-patterns/
│ │ ├── python-packaging/
│ │ ├── python-performance-optimization/
│ │ └── uv-package-manager/
│ ├── backend-development/
│ │ ├── agents/
│ │ │ ├── backend-architect.md
│ │ │ ├── graphql-architect.md
│ │ │ └── tdd-orchestrator.md
│ │ ├── commands/
│ │ │ └── feature-development.md
│ │ └── skills/ # Backend skills (3 total)
│ │ ├── api-design-principles/
│ │ ├── architecture-patterns/
│ │ └── microservices-patterns/
│ ├── security-scanning/
│ │ ├── agents/
│ │ │ └── security-auditor.md
│ │ ├── commands/
│ │ │ ├── security-hardening.md
│ │ │ ├── security-sast.md
│ │ │ └── security-dependencies.md
│ │ └── skills/ # Security skills (1 total)
│ │ └── sast-configuration/
│ └── ... (60 more isolated plugins)
├── docs/ # Documentation
│ ├── agent-skills.md # Agent Skills guide
│ ├── agents.md # Agent reference
│ ├── plugins.md # Plugin catalog
│ ├── usage.md # Usage guide
│ └── architecture.md # This file
└── README.md # Quick start
```
## Plugin Structure
Each plugin contains:
- **agents/** - Specialized agents for that domain (optional)
- **commands/** - Tools and workflows specific to that plugin (optional)
- **skills/** - Modular knowledge packages with progressive disclosure (optional)
### Minimum Requirements
- At least one agent OR one command
- Clear, focused purpose
- Proper frontmatter in all files
- Entry in marketplace.json
### Example Plugin
```
plugins/kubernetes-operations/
├── agents/
│ └── kubernetes-architect.md # K8s architecture and design
├── commands/
│ └── k8s-deploy.md # Deployment automation
└── skills/
├── k8s-manifest-generator/ # Manifest creation skill
├── helm-chart-scaffolding/ # Helm chart skill
├── gitops-workflow/ # GitOps automation skill
└── k8s-security-policies/ # Security policy skill
```
## Agent Skills Architecture
### Progressive Disclosure
Skills use a three-tier architecture for token efficiency:
1. **Metadata** (Frontmatter): Name and activation criteria (always loaded)
2. **Instructions**: Core guidance and patterns (loaded when activated)
3. **Resources**: Examples and templates (loaded on demand)
### Specification Compliance
All skills follow the [Agent Skills Specification](https://github.com/anthropics/skills/blob/main/agent_skills_spec.md):
```yaml
---
name: skill-name # Required: hyphen-case
description: What the skill does. Use when [trigger]. # Required: < 1024 chars
---
# Skill content with progressive disclosure
```
### Benefits
- **Token Efficiency**: Load only relevant knowledge when needed
- **Specialized Expertise**: Deep domain knowledge without bloat
- **Clear Activation**: Explicit triggers prevent unwanted invocation
- **Composability**: Mix and match skills across workflows
- **Maintainability**: Isolated updates don't affect other skills
See [Agent Skills](./agent-skills.md) for complete details on the 47 skills.
## Model Configuration Strategy
### Two-Tier Architecture
The system uses Claude Opus and Sonnet models strategically:
| Model | Count | Use Case |
|-------|-------|----------|
| Haiku | 47 agents | Fast execution, deterministic tasks |
| Sonnet | 97 agents | Complex reasoning, architecture decisions |
### Selection Criteria
**Haiku - Fast Execution & Deterministic Tasks**
- Generating code from well-defined specifications
- Creating tests following established patterns
- Writing documentation with clear templates
- Executing infrastructure operations
- Performing database query optimization
- Handling customer support responses
- Processing SEO optimization tasks
- Managing deployment pipelines
**Sonnet - Complex Reasoning & Architecture**
- Designing system architecture
- Making technology selection decisions
- Performing security audits
- Reviewing code for architectural patterns
- Creating complex AI/ML pipelines
- Providing language-specific expertise
- Orchestrating multi-agent workflows
- Handling business-critical legal/HR matters
### Hybrid Orchestration
Combine models for optimal performance and cost:
```
Planning Phase (Sonnet) → Execution Phase (Haiku) → Review Phase (Sonnet)
Example:
backend-architect (Sonnet) designs API
↓
Generate endpoints (Haiku) implements spec
↓
test-automator (Haiku) creates tests
↓
code-reviewer (Sonnet) validates architecture
```
## Performance & Quality
### Optimized Token Usage
- **Isolated plugins** load only what you need
- **Granular architecture** reduces unnecessary context
- **Progressive disclosure** (skills) loads knowledge on demand
- **Clear boundaries** prevent context pollution
### Component Coverage
- **100% agent coverage** - all plugins include at least one agent
- **100% component availability** - all 85 agents accessible across plugins
- **Efficient distribution** - 3.4 components per plugin average
### Discoverability
- **Clear plugin names** convey purpose immediately
- **Logical categorization** with 23 well-defined categories
- **Searchable documentation** with cross-references
- **Easy to find** the right tool for the job
## Design Patterns
### Pattern 1: Single-Purpose Plugin
Each plugin focuses on one domain:
```
python-development/
├── agents/ # Python language experts
├── commands/ # Python project scaffolding
└── skills/ # Python-specific knowledge
```
**Benefits:**
- Clear responsibility
- Easy to maintain
- Minimal token usage
- Composable with other plugins
### Pattern 2: Workflow Orchestration
Orchestrator plugins coordinate multiple agents:
```
full-stack-orchestration/
└── commands/
└── full-stack-feature.md # Coordinates 7+ agents
```
**Orchestration:**
1. backend-architect (design API)
2. database-architect (design schema)
3. frontend-developer (build UI)
4. test-automator (create tests)
5. security-auditor (security review)
6. deployment-engineer (CI/CD)
7. observability-engineer (monitoring)
### Pattern 3: Agent + Skill Integration
Agents provide reasoning, skills provide knowledge:
```
User: "Build FastAPI project with async patterns"
↓
fastapi-pro agent (orchestrates)
↓
fastapi-templates skill (provides patterns)
↓
python-scaffold command (generates project)
```
### Pattern 4: Multi-Plugin Composition
Complex workflows use multiple plugins:
```
Feature Development Workflow:
1. backend-development:feature-development
2. security-scanning:security-hardening
3. unit-testing:test-generate
4. code-review-ai:ai-review
5. cicd-automation:workflow-automate
6. observability-monitoring:monitor-setup
```
## Versioning & Updates
### Marketplace Updates
- Marketplace catalog in `.claude-plugin/marketplace.json`
- Semantic versioning for plugins
- Backward compatibility maintained
- Clear migration guides for breaking changes
### Plugin Updates
- Individual plugin updates don't affect others
- Skills can be updated independently
- Agents can be added/removed without breaking workflows
- Commands maintain stable interfaces
## Contributing Guidelines
### Adding a Plugin
1. Create plugin directory: `plugins/{plugin-name}/`
2. Add agents and/or commands
3. Optionally add skills
4. Update marketplace.json
5. Document in appropriate category
### Adding an Agent
1. Create `plugins/{plugin-name}/agents/{agent-name}.md`
2. Add frontmatter (name, description, model)
3. Write comprehensive system prompt
4. Update plugin definition
### Adding a Skill
1. Create `plugins/{plugin-name}/skills/{skill-name}/SKILL.md`
2. Add YAML frontmatter (name, description with "Use when")
3. Write skill content with progressive disclosure
4. Add to plugin's skills array in marketplace.json
### Quality Standards
- **Clear naming** - Hyphen-case, descriptive
- **Focused scope** - Single responsibility
- **Complete documentation** - What, when, how
- **Tested functionality** - Verify before committing
- **Spec compliance** - Follow Anthropic guidelines
## See Also
- [Agent Skills](./agent-skills.md) - Modular knowledge packages
- [Agent Reference](./agents.md) - Complete agent catalog
- [Plugin Reference](./plugins.md) - All 63 plugins
- [Usage Guide](./usage.md) - Commands and workflows

374
docs/plugins.md Normal file
View File

@@ -0,0 +1,374 @@
# Complete Plugin Reference
Browse all **63 focused, single-purpose plugins** organized by category.
## Quick Start - Essential Plugins
> 💡 **Getting Started?** Install these popular plugins for immediate productivity gains.
### Development Essentials
**code-documentation** - Documentation and technical writing
```bash
/plugin install code-documentation
```
Automated doc generation, code explanation, and tutorial creation for comprehensive technical documentation.
**debugging-toolkit** - Smart debugging and developer experience
```bash
/plugin install debugging-toolkit
```
Interactive debugging, error analysis, and DX optimization for faster problem resolution.
**git-pr-workflows** - Git automation and PR enhancement
```bash
/plugin install git-pr-workflows
```
Git workflow automation, pull request enhancement, and team onboarding processes.
### Full-Stack Development
**backend-development** - Backend API design and architecture
```bash
/plugin install backend-development
```
RESTful and GraphQL API design with test-driven development and modern backend architecture patterns.
**frontend-mobile-development** - UI and mobile development
```bash
/plugin install frontend-mobile-development
```
React/React Native component development with automated scaffolding and cross-platform implementation.
**full-stack-orchestration** - End-to-end feature development
```bash
/plugin install full-stack-orchestration
```
Multi-agent coordination from backend → frontend → testing → security → deployment.
### Testing & Quality
**unit-testing** - Automated test generation
```bash
/plugin install unit-testing
```
Generate pytest (Python) and Jest (JavaScript) unit tests automatically with comprehensive edge case coverage.
**code-review-ai** - AI-powered code review
```bash
/plugin install code-review-ai
```
Architectural analysis, security assessment, and code quality review with actionable feedback.
### Infrastructure & Operations
**cloud-infrastructure** - Cloud architecture design
```bash
/plugin install cloud-infrastructure
```
AWS/Azure/GCP architecture, Kubernetes setup, Terraform IaC, and multi-cloud cost optimization.
**incident-response** - Production incident management
```bash
/plugin install incident-response
```
Rapid incident triage, root cause analysis, and automated resolution workflows for production systems.
### Language Support
**python-development** - Python project scaffolding
```bash
/plugin install python-development
```
FastAPI/Django project initialization with modern tooling (uv, ruff) and production-ready architecture.
**javascript-typescript** - JavaScript/TypeScript scaffolding
```bash
/plugin install javascript-typescript
```
Next.js, React + Vite, and Node.js project setup with pnpm and TypeScript best practices.
---
## Complete Plugin Catalog
### 🎨 Development (4 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **debugging-toolkit** | Interactive debugging and DX optimization | `/plugin install debugging-toolkit` |
| **backend-development** | Backend API design with GraphQL and TDD | `/plugin install backend-development` |
| **frontend-mobile-development** | Frontend UI and mobile development | `/plugin install frontend-mobile-development` |
| **multi-platform-apps** | Cross-platform app coordination (web/iOS/Android) | `/plugin install multi-platform-apps` |
### 📚 Documentation (2 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **code-documentation** | Documentation generation and code explanation | `/plugin install code-documentation` |
| **documentation-generation** | OpenAPI specs, Mermaid diagrams, tutorials | `/plugin install documentation-generation` |
### 🔄 Workflows (3 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **git-pr-workflows** | Git automation and PR enhancement | `/plugin install git-pr-workflows` |
| **full-stack-orchestration** | End-to-end feature orchestration | `/plugin install full-stack-orchestration` |
| **tdd-workflows** | Test-driven development methodology | `/plugin install tdd-workflows` |
### ✅ Testing (2 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **unit-testing** | Automated unit test generation (Python/JavaScript) | `/plugin install unit-testing` |
| **tdd-workflows** | Test-driven development methodology | `/plugin install tdd-workflows` |
### 🔍 Quality (3 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **code-review-ai** | AI-powered architectural review | `/plugin install code-review-ai` |
| **comprehensive-review** | Multi-perspective code analysis | `/plugin install comprehensive-review` |
| **performance-testing-review** | Performance analysis and test coverage review | `/plugin install performance-testing-review` |
### 🛠️ Utilities (4 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **code-refactoring** | Code cleanup and technical debt management | `/plugin install code-refactoring` |
| **dependency-management** | Dependency auditing and version management | `/plugin install dependency-management` |
| **error-debugging** | Error analysis and trace debugging | `/plugin install error-debugging` |
| **team-collaboration** | Team workflows and standup automation | `/plugin install team-collaboration` |
### 🤖 AI & ML (4 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **llm-application-dev** | LLM apps and prompt engineering | `/plugin install llm-application-dev` |
| **agent-orchestration** | Multi-agent system optimization | `/plugin install agent-orchestration` |
| **context-management** | Context persistence and restoration | `/plugin install context-management` |
| **machine-learning-ops** | ML training pipelines and MLOps | `/plugin install machine-learning-ops` |
### 📊 Data (2 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **data-engineering** | ETL pipelines and data warehouses | `/plugin install data-engineering` |
| **data-validation-suite** | Schema validation and data quality | `/plugin install data-validation-suite` |
### 🗄️ Database (2 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **database-design** | Database architecture and schema design | `/plugin install database-design` |
| **database-migrations** | Database migration automation | `/plugin install database-migrations` |
### 🚨 Operations (4 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **incident-response** | Production incident management | `/plugin install incident-response` |
| **error-diagnostics** | Error tracing and root cause analysis | `/plugin install error-diagnostics` |
| **distributed-debugging** | Distributed system tracing | `/plugin install distributed-debugging` |
| **observability-monitoring** | Metrics, logging, tracing, and SLO | `/plugin install observability-monitoring` |
### ⚡ Performance (2 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **application-performance** | Application profiling and optimization | `/plugin install application-performance` |
| **database-cloud-optimization** | Database query and cloud cost optimization | `/plugin install database-cloud-optimization` |
### ☁️ Infrastructure (5 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **deployment-strategies** | Deployment patterns and rollback automation | `/plugin install deployment-strategies` |
| **deployment-validation** | Pre-deployment checks and validation | `/plugin install deployment-validation` |
| **kubernetes-operations** | K8s manifests and GitOps workflows | `/plugin install kubernetes-operations` |
| **cloud-infrastructure** | AWS/Azure/GCP cloud architecture | `/plugin install cloud-infrastructure` |
| **cicd-automation** | CI/CD pipeline configuration | `/plugin install cicd-automation` |
### 🔒 Security (4 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **security-scanning** | SAST analysis and vulnerability scanning | `/plugin install security-scanning` |
| **security-compliance** | SOC2/HIPAA/GDPR compliance | `/plugin install security-compliance` |
| **backend-api-security** | API security and authentication | `/plugin install backend-api-security` |
| **frontend-mobile-security** | XSS/CSRF prevention and mobile security | `/plugin install frontend-mobile-security` |
### 🔄 Modernization (2 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **framework-migration** | Framework upgrades and migration planning | `/plugin install framework-migration` |
| **codebase-cleanup** | Technical debt reduction and cleanup | `/plugin install codebase-cleanup` |
### 🌐 API (2 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **api-scaffolding** | REST/GraphQL API generation | `/plugin install api-scaffolding` |
| **api-testing-observability** | API testing and monitoring | `/plugin install api-testing-observability` |
### 📢 Marketing (4 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **seo-content-creation** | SEO content writing and planning | `/plugin install seo-content-creation` |
| **seo-technical-optimization** | Meta tags, keywords, and schema markup | `/plugin install seo-technical-optimization` |
| **seo-analysis-monitoring** | Content analysis and authority building | `/plugin install seo-analysis-monitoring` |
| **content-marketing** | Content strategy and web research | `/plugin install content-marketing` |
### 💼 Business (3 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **business-analytics** | KPI tracking and financial reporting | `/plugin install business-analytics` |
| **hr-legal-compliance** | HR policies and legal templates | `/plugin install hr-legal-compliance` |
| **customer-sales-automation** | Support and sales automation | `/plugin install customer-sales-automation` |
### 💻 Languages (7 plugins)
| Plugin | Description | Install |
|--------|-------------|---------|
| **python-development** | Python 3.12+ with Django/FastAPI | `/plugin install python-development` |
| **javascript-typescript** | JavaScript/TypeScript with Node.js | `/plugin install javascript-typescript` |
| **systems-programming** | Rust, Go, C, C++ for systems development | `/plugin install systems-programming` |
| **jvm-languages** | Java, Scala, C# with enterprise patterns | `/plugin install jvm-languages` |
| **web-scripting** | PHP and Ruby for web applications | `/plugin install web-scripting` |
| **functional-programming** | Elixir with OTP and Phoenix | `/plugin install functional-programming` |
| **arm-cortex-microcontrollers** | ARM Cortex-M firmware and drivers | `/plugin install arm-cortex-microcontrollers` |
### 🔗 Blockchain (1 plugin)
| Plugin | Description | Install |
|--------|-------------|---------|
| **blockchain-web3** | Smart contracts and DeFi protocols | `/plugin install blockchain-web3` |
### 💰 Finance (1 plugin)
| Plugin | Description | Install |
|--------|-------------|---------|
| **quantitative-trading** | Algorithmic trading and risk management | `/plugin install quantitative-trading` |
### 💳 Payments (1 plugin)
| Plugin | Description | Install |
|--------|-------------|---------|
| **payment-processing** | Stripe/PayPal integration and billing | `/plugin install payment-processing` |
### 🎮 Gaming (1 plugin)
| Plugin | Description | Install |
|--------|-------------|---------|
| **game-development** | Unity and Minecraft plugin development | `/plugin install game-development` |
### ♿ Accessibility (1 plugin)
| Plugin | Description | Install |
|--------|-------------|---------|
| **accessibility-compliance** | WCAG auditing and inclusive design | `/plugin install accessibility-compliance` |
## Plugin Structure
Each plugin contains:
- **agents/** - Specialized agents for that domain
- **commands/** - Tools and workflows specific to that plugin
- **skills/** - Optional modular knowledge packages (progressive disclosure)
Example:
```
plugins/python-development/
├── agents/
│ ├── python-pro.md
│ ├── django-pro.md
│ └── fastapi-pro.md
├── commands/
│ └── python-scaffold.md
└── skills/
├── async-python-patterns/
├── python-testing-patterns/
├── python-packaging/
├── python-performance-optimization/
└── uv-package-manager/
```
## Installation
### Step 1: Add the Marketplace
```bash
/plugin marketplace add wshobson/agents
```
This makes all 63 plugins available for installation, but **does not load any agents or tools** into your context.
### Step 2: Install Specific Plugins
Browse available plugins:
```bash
/plugin
```
Install only the plugins you need:
```bash
/plugin install python-development
/plugin install backend-development
```
Each installed plugin loads **only its specific agents and commands** into Claude's context.
## Plugin Design Principles
### Single Responsibility
- Each plugin does **one thing well** (Unix philosophy)
- Clear, focused purposes (describable in 5-10 words)
- Average plugin size: **3.4 components** (follows Anthropic's 2-8 pattern)
### Minimal Token Usage
- Install only what you need
- Each plugin loads only its specific agents and tools
- No unnecessary resources loaded into context
- Better context efficiency with granular plugins
### Composability
- Mix and match plugins for complex workflows
- Workflow orchestrators compose focused plugins
- Clear boundaries between plugins
- No forced feature bundling
## See Also
- [Agent Skills](./agent-skills.md) - 47 specialized skills across plugins
- [Agent Reference](./agents.md) - Complete agent catalog
- [Usage Guide](./usage.md) - Commands and workflows
- [Architecture](./architecture.md) - Design principles

371
docs/usage.md Normal file
View File

@@ -0,0 +1,371 @@
# Usage Guide
Complete guide to using agents, slash commands, and multi-agent workflows.
## Overview
The plugin ecosystem provides two primary interfaces:
1. **Slash Commands** - Direct invocation of tools and workflows
2. **Natural Language** - Claude reasons about which agents to use
## Slash Commands
Slash commands are the primary interface for working with agents and workflows. Each plugin provides namespaced commands that you can run directly.
### Command Format
```bash
/plugin-name:command-name [arguments]
```
### Discovering Commands
List all available slash commands from installed plugins:
```bash
/plugin
```
### Benefits of Slash Commands
- **Direct invocation** - No need to describe what you want in natural language
- **Structured arguments** - Pass parameters explicitly for precise control
- **Composability** - Chain commands together for complex workflows
- **Discoverability** - Use `/plugin` to see all available commands
## Natural Language
Agents can also be invoked through natural language when you need Claude to reason about which specialist to use:
```
"Use backend-architect to design the authentication API"
"Have security-auditor scan for OWASP vulnerabilities"
"Get performance-engineer to optimize this database query"
```
Claude Code automatically selects and coordinates the appropriate agents based on your request.
## Command Reference by Category
### Development & Features
| Command | Description |
|---------|-------------|
| `/backend-development:feature-development` | End-to-end backend feature development |
| `/full-stack-orchestration:full-stack-feature` | Complete full-stack feature implementation |
| `/multi-platform-apps:multi-platform` | Cross-platform app development coordination |
### Testing & Quality
| Command | Description |
|---------|-------------|
| `/unit-testing:test-generate` | Generate comprehensive unit tests |
| `/tdd-workflows:tdd-cycle` | Complete TDD red-green-refactor cycle |
| `/tdd-workflows:tdd-red` | Write failing tests first |
| `/tdd-workflows:tdd-green` | Implement code to pass tests |
| `/tdd-workflows:tdd-refactor` | Refactor with passing tests |
### Code Quality & Review
| Command | Description |
|---------|-------------|
| `/code-review-ai:ai-review` | AI-powered code review |
| `/comprehensive-review:full-review` | Multi-perspective analysis |
| `/comprehensive-review:pr-enhance` | Enhance pull requests |
### Debugging & Troubleshooting
| Command | Description |
|---------|-------------|
| `/debugging-toolkit:smart-debug` | Interactive smart debugging |
| `/incident-response:incident-response` | Production incident management |
| `/incident-response:smart-fix` | Automated incident resolution |
| `/error-debugging:error-analysis` | Deep error analysis |
| `/error-debugging:error-trace` | Stack trace debugging |
| `/error-diagnostics:smart-debug` | Smart diagnostic debugging |
| `/distributed-debugging:debug-trace` | Distributed system tracing |
### Security
| Command | Description |
|---------|-------------|
| `/security-scanning:security-hardening` | Comprehensive security hardening |
| `/security-scanning:security-sast` | Static application security testing |
| `/security-scanning:security-dependencies` | Dependency vulnerability scanning |
| `/security-compliance:compliance-check` | SOC2/HIPAA/GDPR compliance |
| `/frontend-mobile-security:xss-scan` | XSS vulnerability scanning |
### Infrastructure & Deployment
| Command | Description |
|---------|-------------|
| `/observability-monitoring:monitor-setup` | Setup monitoring infrastructure |
| `/observability-monitoring:slo-implement` | Implement SLO/SLI metrics |
| `/deployment-validation:config-validate` | Pre-deployment validation |
| `/cicd-automation:workflow-automate` | CI/CD pipeline automation |
### Data & ML
| Command | Description |
|---------|-------------|
| `/machine-learning-ops:ml-pipeline` | ML training pipeline orchestration |
| `/data-engineering:data-pipeline` | ETL/ELT pipeline construction |
| `/data-engineering:data-driven-feature` | Data-driven feature development |
### Documentation
| Command | Description |
|---------|-------------|
| `/code-documentation:doc-generate` | Generate comprehensive documentation |
| `/code-documentation:code-explain` | Explain code functionality |
| `/documentation-generation:doc-generate` | OpenAPI specs, diagrams, tutorials |
### Refactoring & Maintenance
| Command | Description |
|---------|-------------|
| `/code-refactoring:refactor-clean` | Code cleanup and refactoring |
| `/code-refactoring:tech-debt` | Technical debt management |
| `/codebase-cleanup:deps-audit` | Dependency auditing |
| `/codebase-cleanup:tech-debt` | Technical debt reduction |
| `/framework-migration:legacy-modernize` | Legacy code modernization |
| `/framework-migration:code-migrate` | Framework migration |
| `/framework-migration:deps-upgrade` | Dependency upgrades |
### Database
| Command | Description |
|---------|-------------|
| `/database-migrations:sql-migrations` | SQL migration automation |
| `/database-migrations:migration-observability` | Migration monitoring |
| `/database-cloud-optimization:cost-optimize` | Database and cloud optimization |
### Git & PR Workflows
| Command | Description |
|---------|-------------|
| `/git-pr-workflows:pr-enhance` | Enhance pull request quality |
| `/git-pr-workflows:onboard` | Team onboarding automation |
| `/git-pr-workflows:git-workflow` | Git workflow automation |
### Project Scaffolding
| Command | Description |
|---------|-------------|
| `/python-development:python-scaffold` | FastAPI/Django project setup |
| `/javascript-typescript:typescript-scaffold` | Next.js/React + Vite setup |
| `/systems-programming:rust-project` | Rust project scaffolding |
### AI & LLM Development
| Command | Description |
|---------|-------------|
| `/llm-application-dev:langchain-agent` | LangChain agent development |
| `/llm-application-dev:ai-assistant` | AI assistant implementation |
| `/llm-application-dev:prompt-optimize` | Prompt engineering optimization |
| `/agent-orchestration:multi-agent-optimize` | Multi-agent optimization |
| `/agent-orchestration:improve-agent` | Agent improvement workflows |
### Testing & Performance
| Command | Description |
|---------|-------------|
| `/performance-testing-review:ai-review` | Performance analysis |
| `/application-performance:performance-optimization` | App optimization |
### Team Collaboration
| Command | Description |
|---------|-------------|
| `/team-collaboration:issue` | Issue management automation |
| `/team-collaboration:standup-notes` | Standup notes generation |
### Accessibility
| Command | Description |
|---------|-------------|
| `/accessibility-compliance:accessibility-audit` | WCAG compliance auditing |
### API Development
| Command | Description |
|---------|-------------|
| `/api-testing-observability:api-mock` | API mocking and testing |
### Context Management
| Command | Description |
|---------|-------------|
| `/context-management:context-save` | Save conversation context |
| `/context-management:context-restore` | Restore previous context |
## Multi-Agent Workflow Examples
Plugins provide pre-configured multi-agent workflows accessible via slash commands.
### Full-Stack Development
```bash
# Command-based workflow invocation
/full-stack-orchestration:full-stack-feature "user dashboard with real-time analytics"
# Natural language alternative
"Implement user dashboard with real-time analytics"
```
**Orchestration:** backend-architect → database-architect → frontend-developer → test-automator → security-auditor → deployment-engineer → observability-engineer
**What happens:**
1. Database schema design with migrations
2. Backend API implementation (REST/GraphQL)
3. Frontend components with state management
4. Comprehensive test suite (unit/integration/E2E)
5. Security audit and hardening
6. CI/CD pipeline setup with feature flags
7. Observability and monitoring configuration
### Security Hardening
```bash
# Comprehensive security assessment and remediation
/security-scanning:security-hardening --level comprehensive
# Natural language alternative
"Perform security audit and implement OWASP best practices"
```
**Orchestration:** security-auditor → backend-security-coder → frontend-security-coder → mobile-security-coder → test-automator
### Data/ML Pipeline
```bash
# ML feature development with production deployment
/machine-learning-ops:ml-pipeline "customer churn prediction model"
# Natural language alternative
"Build customer churn prediction model with deployment"
```
**Orchestration:** data-scientist → data-engineer → ml-engineer → mlops-engineer → performance-engineer
### Incident Response
```bash
# Smart debugging with root cause analysis
/incident-response:smart-fix "production memory leak in payment service"
# Natural language alternative
"Debug production memory leak and create runbook"
```
**Orchestration:** incident-responder → devops-troubleshooter → debugger → error-detective → observability-engineer
## Command Arguments and Options
Many slash commands support arguments for precise control:
```bash
# Test generation for specific files
/unit-testing:test-generate src/api/users.py
# Feature development with methodology specification
/backend-development:feature-development OAuth2 integration with social login
# Security dependency scanning
/security-scanning:security-dependencies
# Component scaffolding
/frontend-mobile-development:component-scaffold UserProfile component with hooks
# TDD workflow cycle
/tdd-workflows:tdd-red User can reset password
/tdd-workflows:tdd-green
/tdd-workflows:tdd-refactor
# Smart debugging
/debugging-toolkit:smart-debug memory leak in checkout flow
# Python project scaffolding
/python-development:python-scaffold fastapi-microservice
```
## Combining Natural Language and Commands
You can mix both approaches for optimal flexibility:
```
# Start with a command for structured workflow
/full-stack-orchestration:full-stack-feature "payment processing"
# Then provide natural language guidance
"Ensure PCI-DSS compliance and integrate with Stripe"
"Add retry logic for failed transactions"
"Set up fraud detection rules"
```
## Best Practices
### When to Use Slash Commands
- **Structured workflows** - Multi-step processes with clear phases
- **Repetitive tasks** - Operations you perform frequently
- **Precise control** - When you need specific parameters
- **Discovery** - Exploring available functionality
### When to Use Natural Language
- **Exploratory work** - When you're not sure which tool to use
- **Complex reasoning** - When Claude needs to coordinate multiple agents
- **Contextual decisions** - When the right approach depends on the situation
- **Ad-hoc tasks** - One-off operations that don't fit a command
### Workflow Composition
Compose multiple plugins for complex scenarios:
```bash
# 1. Start with feature development
/backend-development:feature-development payment processing API
# 2. Add security hardening
/security-scanning:security-hardening
# 3. Generate comprehensive tests
/unit-testing:test-generate
# 4. Review the implementation
/code-review-ai:ai-review
# 5. Set up CI/CD
/cicd-automation:workflow-automate
# 6. Add monitoring
/observability-monitoring:monitor-setup
```
## Agent Skills Integration
Agent Skills work alongside commands to provide deep expertise:
```
User: "Set up FastAPI project with async patterns"
→ Activates: fastapi-templates skill
→ Invokes: /python-development:python-scaffold
→ Result: Production-ready FastAPI project with best practices
User: "Implement Kubernetes deployment with Helm"
→ Activates: helm-chart-scaffolding, k8s-manifest-generator skills
→ Guides: kubernetes-architect agent
→ Result: Production-grade K8s manifests with Helm charts
```
See [Agent Skills](./agent-skills.md) for details on the 47 specialized skills.
## See Also
- [Agent Skills](./agent-skills.md) - Specialized knowledge packages
- [Agent Reference](./agents.md) - Complete agent catalog
- [Plugin Reference](./plugins.md) - All 63 plugins
- [Architecture](./architecture.md) - Design principles

564
plugins/api-scaffolding/skills/fastapi-templates/SKILL.md Normal file
View File

@@ -0,0 +1,564 @@
---
name: fastapi-templates
description: Create production-ready FastAPI projects with async patterns, dependency injection, and comprehensive error handling. Use when building new FastAPI applications or setting up backend API projects.
---
# FastAPI Project Templates
Production-ready FastAPI project structures with async patterns, dependency injection, middleware, and best practices for building high-performance APIs.
## When to Use This Skill
- Starting new FastAPI projects from scratch
- Implementing async REST APIs with Python
- Building high-performance web services and microservices
- Creating async applications with PostgreSQL, MongoDB
- Setting up API projects with proper structure and testing
## Core Concepts
### 1. Project Structure
**Recommended Layout:**
```
app/
├── api/ # API routes
│ ├── v1/
│ │ ├── endpoints/
│ │ │ ├── users.py
│ │ │ ├── auth.py
│ │ │ └── items.py
│ │ └── router.py
│ └── dependencies.py # Shared dependencies
├── core/ # Core configuration
│ ├── config.py
│ ├── security.py
│ └── database.py
├── models/ # Database models
│ ├── user.py
│ └── item.py
├── schemas/ # Pydantic schemas
│ ├── user.py
│ └── item.py
├── services/ # Business logic
│ ├── user_service.py
│ └── auth_service.py
├── repositories/ # Data access
│ ├── user_repository.py
│ └── item_repository.py
└── main.py # Application entry
```
### 2. Dependency Injection
FastAPI's built-in DI system using `Depends`:
- Database session management
- Authentication/authorization
- Shared business logic
- Configuration injection
### 3. Async Patterns
Proper async/await usage:
- Async route handlers
- Async database operations
- Async background tasks
- Async middleware
## Implementation Patterns
### Pattern 1: Complete FastAPI Application
```python
# main.py
from fastapi import FastAPI, Depends
from fastapi.middleware.cors import CORSMiddleware
from contextlib import asynccontextmanager
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Application lifespan events."""
# Startup
await database.connect()
yield
# Shutdown
await database.disconnect()
app = FastAPI(
title="API Template",
version="1.0.0",
lifespan=lifespan
)
# CORS middleware
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Include routers
from app.api.v1.router import api_router
app.include_router(api_router, prefix="/api/v1")
# core/config.py
from pydantic_settings import BaseSettings
from functools import lru_cache
class Settings(BaseSettings):
"""Application settings."""
DATABASE_URL: str
SECRET_KEY: str
ACCESS_TOKEN_EXPIRE_MINUTES: int = 30
API_V1_STR: str = "/api/v1"
class Config:
env_file = ".env"
@lru_cache()
def get_settings() -> Settings:
return Settings()
# core/database.py
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from app.core.config import get_settings
settings = get_settings()
engine = create_async_engine(
settings.DATABASE_URL,
echo=True,
future=True
)
AsyncSessionLocal = sessionmaker(
engine,
class_=AsyncSession,
expire_on_commit=False
)
Base = declarative_base()
async def get_db() -> AsyncSession:
"""Dependency for database session."""
async with AsyncSessionLocal() as session:
try:
yield session
await session.commit()
except Exception:
await session.rollback()
raise
finally:
await session.close()
```
### Pattern 2: CRUD Repository Pattern
```python
# repositories/base_repository.py
from typing import Generic, TypeVar, Type, Optional, List
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy import select
from pydantic import BaseModel
ModelType = TypeVar("ModelType")
CreateSchemaType = TypeVar("CreateSchemaType", bound=BaseModel)
UpdateSchemaType = TypeVar("UpdateSchemaType", bound=BaseModel)
class BaseRepository(Generic[ModelType, CreateSchemaType, UpdateSchemaType]):
"""Base repository for CRUD operations."""
def __init__(self, model: Type[ModelType]):
self.model = model
async def get(self, db: AsyncSession, id: int) -> Optional[ModelType]:
"""Get by ID."""
result = await db.execute(
select(self.model).where(self.model.id == id)
)
return result.scalars().first()
async def get_multi(
self,
db: AsyncSession,
skip: int = 0,
limit: int = 100
) -> List[ModelType]:
"""Get multiple records."""
result = await db.execute(
select(self.model).offset(skip).limit(limit)
)
return result.scalars().all()
async def create(
self,
db: AsyncSession,
obj_in: CreateSchemaType
) -> ModelType:
"""Create new record."""
db_obj = self.model(**obj_in.dict())
db.add(db_obj)
await db.flush()
await db.refresh(db_obj)
return db_obj
async def update(
self,
db: AsyncSession,
db_obj: ModelType,
obj_in: UpdateSchemaType
) -> ModelType:
"""Update record."""
update_data = obj_in.dict(exclude_unset=True)
for field, value in update_data.items():
setattr(db_obj, field, value)
await db.flush()
await db.refresh(db_obj)
return db_obj
async def delete(self, db: AsyncSession, id: int) -> bool:
"""Delete record."""
obj = await self.get(db, id)
if obj:
await db.delete(obj)
return True
return False
# repositories/user_repository.py
from app.repositories.base_repository import BaseRepository
from app.models.user import User
from app.schemas.user import UserCreate, UserUpdate
class UserRepository(BaseRepository[User, UserCreate, UserUpdate]):
"""User-specific repository."""
async def get_by_email(self, db: AsyncSession, email: str) -> Optional[User]:
"""Get user by email."""
result = await db.execute(
select(User).where(User.email == email)
)
return result.scalars().first()
async def is_active(self, db: AsyncSession, user_id: int) -> bool:
"""Check if user is active."""
user = await self.get(db, user_id)
return user.is_active if user else False
user_repository = UserRepository(User)
```
### Pattern 3: Service Layer
```python
# services/user_service.py
from typing import Optional
from sqlalchemy.ext.asyncio import AsyncSession
from app.repositories.user_repository import user_repository
from app.schemas.user import UserCreate, UserUpdate, User
from app.core.security import get_password_hash, verify_password
class UserService:
"""Business logic for users."""
def __init__(self):
self.repository = user_repository
async def create_user(
self,
db: AsyncSession,
user_in: UserCreate
) -> User:
"""Create new user with hashed password."""
# Check if email exists
existing = await self.repository.get_by_email(db, user_in.email)
if existing:
raise ValueError("Email already registered")
# Hash password
user_in_dict = user_in.dict()
user_in_dict["hashed_password"] = get_password_hash(user_in_dict.pop("password"))
# Create user
user = await self.repository.create(db, UserCreate(**user_in_dict))
return user
async def authenticate(
self,
db: AsyncSession,
email: str,
password: str
) -> Optional[User]:
"""Authenticate user."""
user = await self.repository.get_by_email(db, email)
if not user:
return None
if not verify_password(password, user.hashed_password):
return None
return user
async def update_user(
self,
db: AsyncSession,
user_id: int,
user_in: UserUpdate
) -> Optional[User]:
"""Update user."""
user = await self.repository.get(db, user_id)
if not user:
return None
if user_in.password:
user_in_dict = user_in.dict(exclude_unset=True)
user_in_dict["hashed_password"] = get_password_hash(
user_in_dict.pop("password")
)
user_in = UserUpdate(**user_in_dict)
return await self.repository.update(db, user, user_in)
user_service = UserService()
```
### Pattern 4: API Endpoints with Dependencies
```python
# api/v1/endpoints/users.py
from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy.ext.asyncio import AsyncSession
from typing import List
from app.core.database import get_db
from app.schemas.user import User, UserCreate, UserUpdate
from app.services.user_service import user_service
from app.api.dependencies import get_current_user
router = APIRouter()
@router.post("/", response_model=User, status_code=status.HTTP_201_CREATED)
async def create_user(
user_in: UserCreate,
db: AsyncSession = Depends(get_db)
):
"""Create new user."""
try:
user = await user_service.create_user(db, user_in)
return user
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@router.get("/me", response_model=User)
async def read_current_user(
current_user: User = Depends(get_current_user)
):
"""Get current user."""
return current_user
@router.get("/{user_id}", response_model=User)
async def read_user(
user_id: int,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(get_current_user)
):
"""Get user by ID."""
user = await user_service.repository.get(db, user_id)
if not user:
raise HTTPException(status_code=404, detail="User not found")
return user
@router.patch("/{user_id}", response_model=User)
async def update_user(
user_id: int,
user_in: UserUpdate,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(get_current_user)
):
"""Update user."""
if current_user.id != user_id:
raise HTTPException(status_code=403, detail="Not authorized")
user = await user_service.update_user(db, user_id, user_in)
if not user:
raise HTTPException(status_code=404, detail="User not found")
return user
@router.delete("/{user_id}", status_code=status.HTTP_204_NO_CONTENT)
async def delete_user(
user_id: int,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(get_current_user)
):
"""Delete user."""
if current_user.id != user_id:
raise HTTPException(status_code=403, detail="Not authorized")
deleted = await user_service.repository.delete(db, user_id)
if not deleted:
raise HTTPException(status_code=404, detail="User not found")
```
### Pattern 5: Authentication & Authorization
```python
# core/security.py
from datetime import datetime, timedelta
from typing import Optional
from jose import JWTError, jwt
from passlib.context import CryptContext
from app.core.config import get_settings
settings = get_settings()
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
ALGORITHM = "HS256"
def create_access_token(data: dict, expires_delta: Optional[timedelta] = None):
"""Create JWT access token."""
to_encode = data.copy()
if expires_delta:
expire = datetime.utcnow() + expires_delta
else:
expire = datetime.utcnow() + timedelta(minutes=15)
to_encode.update({"exp": expire})
encoded_jwt = jwt.encode(to_encode, settings.SECRET_KEY, algorithm=ALGORITHM)
return encoded_jwt
def verify_password(plain_password: str, hashed_password: str) -> bool:
"""Verify password against hash."""
return pwd_context.verify(plain_password, hashed_password)
def get_password_hash(password: str) -> str:
"""Hash password."""
return pwd_context.hash(password)
# api/dependencies.py
from fastapi import Depends, HTTPException, status
from fastapi.security import OAuth2PasswordBearer
from jose import JWTError, jwt
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.database import get_db
from app.core.security import ALGORITHM
from app.core.config import get_settings
from app.repositories.user_repository import user_repository
oauth2_scheme = OAuth2PasswordBearer(tokenUrl=f"{settings.API_V1_STR}/auth/login")
async def get_current_user(
db: AsyncSession = Depends(get_db),
token: str = Depends(oauth2_scheme)
):
"""Get current authenticated user."""
credentials_exception = HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Could not validate credentials",
headers={"WWW-Authenticate": "Bearer"},
)
try:
payload = jwt.decode(token, settings.SECRET_KEY, algorithms=[ALGORITHM])
user_id: int = payload.get("sub")
if user_id is None:
raise credentials_exception
except JWTError:
raise credentials_exception
user = await user_repository.get(db, user_id)
if user is None:
raise credentials_exception
return user
```
## Testing
```python
# tests/conftest.py
import pytest
import asyncio
from httpx import AsyncClient
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from sqlalchemy.orm import sessionmaker
from app.main import app
from app.core.database import get_db, Base
TEST_DATABASE_URL = "sqlite+aiosqlite:///:memory:"
@pytest.fixture(scope="session")
def event_loop():
loop = asyncio.get_event_loop_policy().new_event_loop()
yield loop
loop.close()
@pytest.fixture
async def db_session():
engine = create_async_engine(TEST_DATABASE_URL, echo=True)
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
AsyncSessionLocal = sessionmaker(
engine, class_=AsyncSession, expire_on_commit=False
)
async with AsyncSessionLocal() as session:
yield session
@pytest.fixture
async def client(db_session):
async def override_get_db():
yield db_session
app.dependency_overrides[get_db] = override_get_db
async with AsyncClient(app=app, base_url="http://test") as client:
yield client
# tests/test_users.py
import pytest
@pytest.mark.asyncio
async def test_create_user(client):
response = await client.post(
"/api/v1/users/",
json={
"email": "test@example.com",
"password": "testpass123",
"name": "Test User"
}
)
assert response.status_code == 201
data = response.json()
assert data["email"] == "test@example.com"
assert "id" in data
```
## Resources
- **references/fastapi-architecture.md**: Detailed architecture guide
- **references/async-best-practices.md**: Async/await patterns
- **references/testing-strategies.md**: Comprehensive testing guide
- **assets/project-template/**: Complete FastAPI project
- **assets/docker-compose.yml**: Development environment setup
## Best Practices
1. **Async All The Way**: Use async for database, external APIs
2. **Dependency Injection**: Leverage FastAPI's DI system
3. **Repository Pattern**: Separate data access from business logic
4. **Service Layer**: Keep business logic out of routes
5. **Pydantic Schemas**: Strong typing for request/response
6. **Error Handling**: Consistent error responses
7. **Testing**: Test all layers independently
## Common Pitfalls
- **Blocking Code in Async**: Using synchronous database drivers
- **No Service Layer**: Business logic in route handlers
- **Missing Type Hints**: Loses FastAPI's benefits
- **Ignoring Sessions**: Not properly managing database sessions
- **No Testing**: Skipping integration tests
- **Tight Coupling**: Direct database access in routes

527
plugins/backend-development/skills/api-design-principles/SKILL.md Normal file
View File

@@ -0,0 +1,527 @@
---
name: api-design-principles
description: Master REST and GraphQL API design principles to build intuitive, scalable, and maintainable APIs that delight developers. Use when designing new APIs, reviewing API specifications, or establishing API design standards.
---
# API Design Principles
Master REST and GraphQL API design principles to build intuitive, scalable, and maintainable APIs that delight developers and stand the test of time.
## When to Use This Skill
- Designing new REST or GraphQL APIs
- Refactoring existing APIs for better usability
- Establishing API design standards for your team
- Reviewing API specifications before implementation
- Migrating between API paradigms (REST to GraphQL, etc.)
- Creating developer-friendly API documentation
- Optimizing APIs for specific use cases (mobile, third-party integrations)
## Core Concepts
### 1. RESTful Design Principles
**Resource-Oriented Architecture**
- Resources are nouns (users, orders, products), not verbs
- Use HTTP methods for actions (GET, POST, PUT, PATCH, DELETE)
- URLs represent resource hierarchies
- Consistent naming conventions
**HTTP Methods Semantics:**
- `GET`: Retrieve resources (idempotent, safe)
- `POST`: Create new resources
- `PUT`: Replace entire resource (idempotent)
- `PATCH`: Partial resource updates
- `DELETE`: Remove resources (idempotent)
### 2. GraphQL Design Principles
**Schema-First Development**
- Types define your domain model
- Queries for reading data
- Mutations for modifying data
- Subscriptions for real-time updates
**Query Structure:**
- Clients request exactly what they need
- Single endpoint, multiple operations
- Strongly typed schema
- Introspection built-in
### 3. API Versioning Strategies
**URL Versioning:**
```
/api/v1/users
/api/v2/users
```
**Header Versioning:**
```
Accept: application/vnd.api+json; version=1
```
**Query Parameter Versioning:**
```
/api/users?version=1
```
## REST API Design Patterns
### Pattern 1: Resource Collection Design
```python
# Good: Resource-oriented endpoints
GET /api/users # List users (with pagination)
POST /api/users # Create user
GET /api/users/{id} # Get specific user
PUT /api/users/{id} # Replace user
PATCH /api/users/{id} # Update user fields
DELETE /api/users/{id} # Delete user
# Nested resources
GET /api/users/{id}/orders # Get user's orders
POST /api/users/{id}/orders # Create order for user
# Bad: Action-oriented endpoints (avoid)
POST /api/createUser
POST /api/getUserById
POST /api/deleteUser
```
### Pattern 2: Pagination and Filtering
```python
from typing import List, Optional
from pydantic import BaseModel, Field
class PaginationParams(BaseModel):
page: int = Field(1, ge=1, description="Page number")
page_size: int = Field(20, ge=1, le=100, description="Items per page")
class FilterParams(BaseModel):
status: Optional[str] = None
created_after: Optional[str] = None
search: Optional[str] = None
class PaginatedResponse(BaseModel):
items: List[dict]
total: int
page: int
page_size: int
pages: int
@property
def has_next(self) -> bool:
return self.page < self.pages
@property
def has_prev(self) -> bool:
return self.page > 1
# FastAPI endpoint example
from fastapi import FastAPI, Query, Depends
app = FastAPI()
@app.get("/api/users", response_model=PaginatedResponse)
async def list_users(
page: int = Query(1, ge=1),
page_size: int = Query(20, ge=1, le=100),
status: Optional[str] = Query(None),
search: Optional[str] = Query(None)
):
# Apply filters
query = build_query(status=status, search=search)
# Count total
total = await count_users(query)
# Fetch page
offset = (page - 1) * page_size
users = await fetch_users(query, limit=page_size, offset=offset)
return PaginatedResponse(
items=users,
total=total,
page=page,
page_size=page_size,
pages=(total + page_size - 1) // page_size
)
```
### Pattern 3: Error Handling and Status Codes
```python
from fastapi import HTTPException, status
from pydantic import BaseModel
class ErrorResponse(BaseModel):
error: str
message: str
details: Optional[dict] = None
timestamp: str
path: str
class ValidationErrorDetail(BaseModel):
field: str
message: str
value: Any
# Consistent error responses
STATUS_CODES = {
"success": 200,
"created": 201,
"no_content": 204,
"bad_request": 400,
"unauthorized": 401,
"forbidden": 403,
"not_found": 404,
"conflict": 409,
"unprocessable": 422,
"internal_error": 500
}
def raise_not_found(resource: str, id: str):
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail={
"error": "NotFound",
"message": f"{resource} not found",
"details": {"id": id}
}
)
def raise_validation_error(errors: List[ValidationErrorDetail]):
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
detail={
"error": "ValidationError",
"message": "Request validation failed",
"details": {"errors": [e.dict() for e in errors]}
}
)
# Example usage
@app.get("/api/users/{user_id}")
async def get_user(user_id: str):
user = await fetch_user(user_id)
if not user:
raise_not_found("User", user_id)
return user
```
### Pattern 4: HATEOAS (Hypermedia as the Engine of Application State)
```python
class UserResponse(BaseModel):
id: str
name: str
email: str
_links: dict
@classmethod
def from_user(cls, user: User, base_url: str):
return cls(
id=user.id,
name=user.name,
email=user.email,
_links={
"self": {"href": f"{base_url}/api/users/{user.id}"},
"orders": {"href": f"{base_url}/api/users/{user.id}/orders"},
"update": {
"href": f"{base_url}/api/users/{user.id}",
"method": "PATCH"
},
"delete": {
"href": f"{base_url}/api/users/{user.id}",
"method": "DELETE"
}
}
)
```
## GraphQL Design Patterns
### Pattern 1: Schema Design
```graphql
# schema.graphql
# Clear type definitions
type User {
id: ID!
email: String!
name: String!
createdAt: DateTime!
# Relationships
orders(
first: Int = 20
after: String
status: OrderStatus
): OrderConnection!
profile: UserProfile
}
type Order {
id: ID!
status: OrderStatus!
total: Money!
items: [OrderItem!]!
createdAt: DateTime!
# Back-reference
user: User!
}
# Pagination pattern (Relay-style)
type OrderConnection {
edges: [OrderEdge!]!
pageInfo: PageInfo!
totalCount: Int!
}
type OrderEdge {
node: Order!
cursor: String!
}
type PageInfo {
hasNextPage: Boolean!
hasPreviousPage: Boolean!
startCursor: String
endCursor: String
}
# Enums for type safety
enum OrderStatus {
PENDING
CONFIRMED
SHIPPED
DELIVERED
CANCELLED
}
# Custom scalars
scalar DateTime
scalar Money
# Query root
type Query {
user(id: ID!): User
users(
first: Int = 20
after: String
search: String
): UserConnection!
order(id: ID!): Order
}
# Mutation root
type Mutation {
createUser(input: CreateUserInput!): CreateUserPayload!
updateUser(input: UpdateUserInput!): UpdateUserPayload!
deleteUser(id: ID!): DeleteUserPayload!
createOrder(input: CreateOrderInput!): CreateOrderPayload!
}
# Input types for mutations
input CreateUserInput {
email: String!
name: String!
password: String!
}
# Payload types for mutations
type CreateUserPayload {
user: User
errors: [Error!]
}
type Error {
field: String
message: String!
}
```
### Pattern 2: Resolver Design
```python
from typing import Optional, List
from ariadne import QueryType, MutationType, ObjectType
from dataclasses import dataclass
query = QueryType()
mutation = MutationType()
user_type = ObjectType("User")
@query.field("user")
async def resolve_user(obj, info, id: str) -> Optional[dict]:
"""Resolve single user by ID."""
return await fetch_user_by_id(id)
@query.field("users")
async def resolve_users(
obj,
info,
first: int = 20,
after: Optional[str] = None,
search: Optional[str] = None
) -> dict:
"""Resolve paginated user list."""
# Decode cursor
offset = decode_cursor(after) if after else 0
# Fetch users
users = await fetch_users(
limit=first + 1, # Fetch one extra to check hasNextPage
offset=offset,
search=search
)
# Pagination
has_next = len(users) > first
if has_next:
users = users[:first]
edges = [
{
"node": user,
"cursor": encode_cursor(offset + i)
}
for i, user in enumerate(users)
]
return {
"edges": edges,
"pageInfo": {
"hasNextPage": has_next,
"hasPreviousPage": offset > 0,
"startCursor": edges[0]["cursor"] if edges else None,
"endCursor": edges[-1]["cursor"] if edges else None
},
"totalCount": await count_users(search=search)
}
@user_type.field("orders")
async def resolve_user_orders(user: dict, info, first: int = 20) -> dict:
"""Resolve user's orders (N+1 prevention with DataLoader)."""
# Use DataLoader to batch requests
loader = info.context["loaders"]["orders_by_user"]
orders = await loader.load(user["id"])
return paginate_orders(orders, first)
@mutation.field("createUser")
async def resolve_create_user(obj, info, input: dict) -> dict:
"""Create new user."""
try:
# Validate input
validate_user_input(input)
# Create user
user = await create_user(
email=input["email"],
name=input["name"],
password=hash_password(input["password"])
)
return {
"user": user,
"errors": []
}
except ValidationError as e:
return {
"user": None,
"errors": [{"field": e.field, "message": e.message}]
}
```
### Pattern 3: DataLoader (N+1 Problem Prevention)
```python
from aiodataloader import DataLoader
from typing import List, Optional
class UserLoader(DataLoader):
"""Batch load users by ID."""
async def batch_load_fn(self, user_ids: List[str]) -> List[Optional[dict]]:
"""Load multiple users in single query."""
users = await fetch_users_by_ids(user_ids)
# Map results back to input order
user_map = {user["id"]: user for user in users}
return [user_map.get(user_id) for user_id in user_ids]
class OrdersByUserLoader(DataLoader):
"""Batch load orders by user ID."""
async def batch_load_fn(self, user_ids: List[str]) -> List[List[dict]]:
"""Load orders for multiple users in single query."""
orders = await fetch_orders_by_user_ids(user_ids)
# Group orders by user_id
orders_by_user = {}
for order in orders:
user_id = order["user_id"]
if user_id not in orders_by_user:
orders_by_user[user_id] = []
orders_by_user[user_id].append(order)
# Return in input order
return [orders_by_user.get(user_id, []) for user_id in user_ids]
# Context setup
def create_context():
return {
"loaders": {
"user": UserLoader(),
"orders_by_user": OrdersByUserLoader()
}
}
```
## Best Practices
### REST APIs
1. **Consistent Naming**: Use plural nouns for collections (`/users`, not `/user`)
2. **Stateless**: Each request contains all necessary information
3. **Use HTTP Status Codes Correctly**: 2xx success, 4xx client errors, 5xx server errors
4. **Version Your API**: Plan for breaking changes from day one
5. **Pagination**: Always paginate large collections
6. **Rate Limiting**: Protect your API with rate limits
7. **Documentation**: Use OpenAPI/Swagger for interactive docs
### GraphQL APIs
1. **Schema First**: Design schema before writing resolvers
2. **Avoid N+1**: Use DataLoaders for efficient data fetching
3. **Input Validation**: Validate at schema and resolver levels
4. **Error Handling**: Return structured errors in mutation payloads
5. **Pagination**: Use cursor-based pagination (Relay spec)
6. **Deprecation**: Use `@deprecated` directive for gradual migration
7. **Monitoring**: Track query complexity and execution time
## Common Pitfalls
- **Over-fetching/Under-fetching (REST)**: Fixed in GraphQL but requires DataLoaders
- **Breaking Changes**: Version APIs or use deprecation strategies
- **Inconsistent Error Formats**: Standardize error responses
- **Missing Rate Limits**: APIs without limits are vulnerable to abuse
- **Poor Documentation**: Undocumented APIs frustrate developers
- **Ignoring HTTP Semantics**: POST for idempotent operations breaks expectations
- **Tight Coupling**: API structure shouldn't mirror database schema
## Resources
- **references/rest-best-practices.md**: Comprehensive REST API design guide
- **references/graphql-schema-design.md**: GraphQL schema patterns and anti-patterns
- **references/api-versioning-strategies.md**: Versioning approaches and migration paths
- **assets/rest-api-template.py**: FastAPI REST API template
- **assets/graphql-schema-template.graphql**: Complete GraphQL schema example
- **assets/api-design-checklist.md**: Pre-implementation review checklist
- **scripts/openapi-generator.py**: Generate OpenAPI specs from code

136
plugins/backend-development/skills/api-design-principles/assets/api-design-checklist.md Normal file
View File

@@ -0,0 +1,136 @@
# API Design Checklist
## Pre-Implementation Review
### Resource Design
- [ ] Resources are nouns, not verbs
- [ ] Plural names for collections
- [ ] Consistent naming across all endpoints
- [ ] Clear resource hierarchy (avoid deep nesting >2 levels)
- [ ] All CRUD operations properly mapped to HTTP methods
### HTTP Methods
- [ ] GET for retrieval (safe, idempotent)
- [ ] POST for creation
- [ ] PUT for full replacement (idempotent)
- [ ] PATCH for partial updates
- [ ] DELETE for removal (idempotent)
### Status Codes
- [ ] 200 OK for successful GET/PATCH/PUT
- [ ] 201 Created for POST
- [ ] 204 No Content for DELETE
- [ ] 400 Bad Request for malformed requests
- [ ] 401 Unauthorized for missing auth
- [ ] 403 Forbidden for insufficient permissions
- [ ] 404 Not Found for missing resources
- [ ] 422 Unprocessable Entity for validation errors
- [ ] 429 Too Many Requests for rate limiting
- [ ] 500 Internal Server Error for server issues
### Pagination
- [ ] All collection endpoints paginated
- [ ] Default page size defined (e.g., 20)
- [ ] Maximum page size enforced (e.g., 100)
- [ ] Pagination metadata included (total, pages, etc.)
- [ ] Cursor-based or offset-based pattern chosen
### Filtering & Sorting
- [ ] Query parameters for filtering
- [ ] Sort parameter supported
- [ ] Search parameter for full-text search
- [ ] Field selection supported (sparse fieldsets)
### Versioning
- [ ] Versioning strategy defined (URL/header/query)
- [ ] Version included in all endpoints
- [ ] Deprecation policy documented
### Error Handling
- [ ] Consistent error response format
- [ ] Detailed error messages
- [ ] Field-level validation errors
- [ ] Error codes for client handling
- [ ] Timestamps in error responses
### Authentication & Authorization
- [ ] Authentication method defined (Bearer token, API key)
- [ ] Authorization checks on all endpoints
- [ ] 401 vs 403 used correctly
- [ ] Token expiration handled
### Rate Limiting
- [ ] Rate limits defined per endpoint/user
- [ ] Rate limit headers included
- [ ] 429 status code for exceeded limits
- [ ] Retry-After header provided
### Documentation
- [ ] OpenAPI/Swagger spec generated
- [ ] All endpoints documented
- [ ] Request/response examples provided
- [ ] Error responses documented
- [ ] Authentication flow documented
### Testing
- [ ] Unit tests for business logic
- [ ] Integration tests for endpoints
- [ ] Error scenarios tested
- [ ] Edge cases covered
- [ ] Performance tests for heavy endpoints
### Security
- [ ] Input validation on all fields
- [ ] SQL injection prevention
- [ ] XSS prevention
- [ ] CORS configured correctly
- [ ] HTTPS enforced
- [ ] Sensitive data not in URLs
- [ ] No secrets in responses
### Performance
- [ ] Database queries optimized
- [ ] N+1 queries prevented
- [ ] Caching strategy defined
- [ ] Cache headers set appropriately
- [ ] Large responses paginated
### Monitoring
- [ ] Logging implemented
- [ ] Error tracking configured
- [ ] Performance metrics collected
- [ ] Health check endpoint available
- [ ] Alerts configured for errors
## GraphQL-Specific Checks
### Schema Design
- [ ] Schema-first approach used
- [ ] Types properly defined
- [ ] Non-null vs nullable decided
- [ ] Interfaces/unions used appropriately
- [ ] Custom scalars defined
### Queries
- [ ] Query depth limiting
- [ ] Query complexity analysis
- [ ] DataLoaders prevent N+1
- [ ] Pagination pattern chosen (Relay/offset)
### Mutations
- [ ] Input types defined
- [ ] Payload types with errors
- [ ] Optimistic response support
- [ ] Idempotency considered
### Performance
- [ ] DataLoader for all relationships
- [ ] Query batching enabled
- [ ] Persisted queries considered
- [ ] Response caching implemented
### Documentation
- [ ] All fields documented
- [ ] Deprecations marked
- [ ] Examples provided
- [ ] Schema introspection enabled

165
plugins/backend-development/skills/api-design-principles/assets/rest-api-template.py Normal file
View File

@@ -0,0 +1,165 @@
"""
Production-ready REST API template using FastAPI.
Includes pagination, filtering, error handling, and best practices.
"""
from fastapi import FastAPI, HTTPException, Query, Path, Depends, status
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field, EmailStr
from typing import Optional, List, Any
from datetime import datetime
from enum import Enum
app = FastAPI(
title="API Template",
version="1.0.0",
docs_url="/api/docs"
)
# Models
class UserStatus(str, Enum):
ACTIVE = "active"
INACTIVE = "inactive"
SUSPENDED = "suspended"
class UserBase(BaseModel):
email: EmailStr
name: str = Field(..., min_length=1, max_length=100)
status: UserStatus = UserStatus.ACTIVE
class UserCreate(UserBase):
password: str = Field(..., min_length=8)
class UserUpdate(BaseModel):
email: Optional[EmailStr] = None
name: Optional[str] = Field(None, min_length=1, max_length=100)
status: Optional[UserStatus] = None
class User(UserBase):
id: str
created_at: datetime
updated_at: datetime
class Config:
from_attributes = True
# Pagination
class PaginationParams(BaseModel):
page: int = Field(1, ge=1)
page_size: int = Field(20, ge=1, le=100)
class PaginatedResponse(BaseModel):
items: List[Any]
total: int
page: int
page_size: int
pages: int
# Error handling
class ErrorDetail(BaseModel):
field: Optional[str] = None
message: str
code: str
class ErrorResponse(BaseModel):
error: str
message: str
details: Optional[List[ErrorDetail]] = None
@app.exception_handler(HTTPException)
async def http_exception_handler(request, exc):
return JSONResponse(
status_code=exc.status_code,
content=ErrorResponse(
error=exc.__class__.__name__,
message=exc.detail if isinstance(exc.detail, str) else exc.detail.get("message", "Error"),
details=exc.detail.get("details") if isinstance(exc.detail, dict) else None
).dict()
)
# Endpoints
@app.get("/api/users", response_model=PaginatedResponse, tags=["Users"])
async def list_users(
page: int = Query(1, ge=1),
page_size: int = Query(20, ge=1, le=100),
status: Optional[UserStatus] = Query(None),
search: Optional[str] = Query(None)
):
"""List users with pagination and filtering."""
# Mock implementation
total = 100
items = [
User(
id=str(i),
email=f"user{i}@example.com",
name=f"User {i}",
status=UserStatus.ACTIVE,
created_at=datetime.now(),
updated_at=datetime.now()
).dict()
for i in range((page-1)*page_size, min(page*page_size, total))
]
return PaginatedResponse(
items=items,
total=total,
page=page,
page_size=page_size,
pages=(total + page_size - 1) // page_size
)
@app.post("/api/users", response_model=User, status_code=status.HTTP_201_CREATED, tags=["Users"])
async def create_user(user: UserCreate):
"""Create a new user."""
# Mock implementation
return User(
id="123",
email=user.email,
name=user.name,
status=user.status,
created_at=datetime.now(),
updated_at=datetime.now()
)
@app.get("/api/users/{user_id}", response_model=User, tags=["Users"])
async def get_user(user_id: str = Path(..., description="User ID")):
"""Get user by ID."""
# Mock: Check if exists
if user_id == "999":
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail={"message": "User not found", "details": {"id": user_id}}
)
return User(
id=user_id,
email="user@example.com",
name="User Name",
status=UserStatus.ACTIVE,
created_at=datetime.now(),
updated_at=datetime.now()
)
@app.patch("/api/users/{user_id}", response_model=User, tags=["Users"])
async def update_user(user_id: str, update: UserUpdate):
"""Partially update user."""
# Validate user exists
existing = await get_user(user_id)
# Apply updates
update_data = update.dict(exclude_unset=True)
for field, value in update_data.items():
setattr(existing, field, value)
existing.updated_at = datetime.now()
return existing
@app.delete("/api/users/{user_id}", status_code=status.HTTP_204_NO_CONTENT, tags=["Users"])
async def delete_user(user_id: str):
"""Delete user."""
await get_user(user_id) # Verify exists
return None
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)

566
plugins/backend-development/skills/api-design-principles/references/graphql-schema-design.md Normal file
View File

@@ -0,0 +1,566 @@
# GraphQL Schema Design Patterns
## Schema Organization
### Modular Schema Structure
```graphql
# user.graphql
type User {
id: ID!
email: String!
name: String!
posts: [Post!]!
}
extend type Query {
user(id: ID!): User
users(first: Int, after: String): UserConnection!
}
extend type Mutation {
createUser(input: CreateUserInput!): CreateUserPayload!
}
# post.graphql
type Post {
id: ID!
title: String!
content: String!
author: User!
}
extend type Query {
post(id: ID!): Post
}
```
## Type Design Patterns
### 1. Non-Null Types
```graphql
type User {
id: ID! # Always required
email: String! # Required
phone: String # Optional (nullable)
posts: [Post!]! # Non-null array of non-null posts
tags: [String!] # Nullable array of non-null strings
}
```
### 2. Interfaces for Polymorphism
```graphql
interface Node {
id: ID!
createdAt: DateTime!
}
type User implements Node {
id: ID!
createdAt: DateTime!
email: String!
}
type Post implements Node {
id: ID!
createdAt: DateTime!
title: String!
}
type Query {
node(id: ID!): Node
}
```
### 3. Unions for Heterogeneous Results
```graphql
union SearchResult = User | Post | Comment
type Query {
search(query: String!): [SearchResult!]!
}
# Query example
{
search(query: "graphql") {
... on User {
name
email
}
... on Post {
title
content
}
... on Comment {
text
author { name }
}
}
}
```
### 4. Input Types
```graphql
input CreateUserInput {
email: String!
name: String!
password: String!
profileInput: ProfileInput
}
input ProfileInput {
bio: String
avatar: String
website: String
}
input UpdateUserInput {
id: ID!
email: String
name: String
profileInput: ProfileInput
}
```
## Pagination Patterns
### Relay Cursor Pagination (Recommended)
```graphql
type UserConnection {
edges: [UserEdge!]!
pageInfo: PageInfo!
totalCount: Int!
}
type UserEdge {
node: User!
cursor: String!
}
type PageInfo {
hasNextPage: Boolean!
hasPreviousPage: Boolean!
startCursor: String
endCursor: String
}
type Query {
users(
first: Int
after: String
last: Int
before: String
): UserConnection!
}
# Usage
{
users(first: 10, after: "cursor123") {
edges {
cursor
node {
id
name
}
}
pageInfo {
hasNextPage
endCursor
}
}
}
```
### Offset Pagination (Simpler)
```graphql
type UserList {
items: [User!]!
total: Int!
page: Int!
pageSize: Int!
}
type Query {
users(page: Int = 1, pageSize: Int = 20): UserList!
}
```
## Mutation Design Patterns
### 1. Input/Payload Pattern
```graphql
input CreatePostInput {
title: String!
content: String!
tags: [String!]
}
type CreatePostPayload {
post: Post
errors: [Error!]
success: Boolean!
}
type Error {
field: String
message: String!
code: String!
}
type Mutation {
createPost(input: CreatePostInput!): CreatePostPayload!
}
```
### 2. Optimistic Response Support
```graphql
type UpdateUserPayload {
user: User
clientMutationId: String
errors: [Error!]
}
input UpdateUserInput {
id: ID!
name: String
clientMutationId: String
}
type Mutation {
updateUser(input: UpdateUserInput!): UpdateUserPayload!
}
```
### 3. Batch Mutations
```graphql
input BatchCreateUserInput {
users: [CreateUserInput!]!
}
type BatchCreateUserPayload {
results: [CreateUserResult!]!
successCount: Int!
errorCount: Int!
}
type CreateUserResult {
user: User
errors: [Error!]
index: Int!
}
type Mutation {
batchCreateUsers(input: BatchCreateUserInput!): BatchCreateUserPayload!
}
```
## Field Design
### Arguments and Filtering
```graphql
type Query {
posts(
# Pagination
first: Int = 20
after: String
# Filtering
status: PostStatus
authorId: ID
tag: String
# Sorting
orderBy: PostOrderBy = CREATED_AT
orderDirection: OrderDirection = DESC
# Searching
search: String
): PostConnection!
}
enum PostStatus {
DRAFT
PUBLISHED
ARCHIVED
}
enum PostOrderBy {
CREATED_AT
UPDATED_AT
TITLE
}
enum OrderDirection {
ASC
DESC
}
```
### Computed Fields
```graphql
type User {
firstName: String!
lastName: String!
fullName: String! # Computed in resolver
posts: [Post!]!
postCount: Int! # Computed, doesn't load all posts
}
type Post {
likeCount: Int!
commentCount: Int!
isLikedByViewer: Boolean! # Context-dependent
}
```
## Subscriptions
```graphql
type Subscription {
postAdded: Post!
postUpdated(postId: ID!): Post!
userStatusChanged(userId: ID!): UserStatus!
}
type UserStatus {
userId: ID!
online: Boolean!
lastSeen: DateTime!
}
# Client usage
subscription {
postAdded {
id
title
author {
name
}
}
}
```
## Custom Scalars
```graphql
scalar DateTime
scalar Email
scalar URL
scalar JSON
scalar Money
type User {
email: Email!
website: URL
createdAt: DateTime!
metadata: JSON
}
type Product {
price: Money!
}
```
## Directives
### Built-in Directives
```graphql
type User {
name: String!
email: String! @deprecated(reason: "Use emails field instead")
emails: [String!]!
# Conditional inclusion
privateData: PrivateData @include(if: $isOwner)
}
# Query
query GetUser($isOwner: Boolean!) {
user(id: "123") {
name
privateData @include(if: $isOwner) {
ssn
}
}
}
```
### Custom Directives
```graphql
directive @auth(requires: Role = USER) on FIELD_DEFINITION
enum Role {
USER
ADMIN
MODERATOR
}
type Mutation {
deleteUser(id: ID!): Boolean! @auth(requires: ADMIN)
updateProfile(input: ProfileInput!): User! @auth
}
```
## Error Handling
### Union Error Pattern
```graphql
type User {
id: ID!
email: String!
}
type ValidationError {
field: String!
message: String!
}
type NotFoundError {
message: String!
resourceType: String!
resourceId: ID!
}
type AuthorizationError {
message: String!
}
union UserResult = User | ValidationError | NotFoundError | AuthorizationError
type Query {
user(id: ID!): UserResult!
}
# Usage
{
user(id: "123") {
... on User {
id
email
}
... on NotFoundError {
message
resourceType
}
... on AuthorizationError {
message
}
}
}
```
### Errors in Payload
```graphql
type CreateUserPayload {
user: User
errors: [Error!]
success: Boolean!
}
type Error {
field: String
message: String!
code: ErrorCode!
}
enum ErrorCode {
VALIDATION_ERROR
UNAUTHORIZED
NOT_FOUND
INTERNAL_ERROR
}
```
## N+1 Query Problem Solutions
### DataLoader Pattern
```python
from aiodataloader import DataLoader
class PostLoader(DataLoader):
async def batch_load_fn(self, post_ids):
posts = await db.posts.find({"id": {"$in": post_ids}})
post_map = {post["id"]: post for post in posts}
return [post_map.get(pid) for pid in post_ids]
# Resolver
@user_type.field("posts")
async def resolve_posts(user, info):
loader = info.context["loaders"]["post"]
return await loader.load_many(user["post_ids"])
```
### Query Depth Limiting
```python
from graphql import GraphQLError
def depth_limit_validator(max_depth: int):
def validate(context, node, ancestors):
depth = len(ancestors)
if depth > max_depth:
raise GraphQLError(
f"Query depth {depth} exceeds maximum {max_depth}"
)
return validate
```
### Query Complexity Analysis
```python
def complexity_limit_validator(max_complexity: int):
def calculate_complexity(node):
# Each field = 1, lists multiply
complexity = 1
if is_list_field(node):
complexity *= get_list_size_arg(node)
return complexity
return validate_complexity
```
## Schema Versioning
### Field Deprecation
```graphql
type User {
name: String! @deprecated(reason: "Use firstName and lastName")
firstName: String!
lastName: String!
}
```
### Schema Evolution
```graphql
# v1 - Initial
type User {
name: String!
}
# v2 - Add optional field (backward compatible)
type User {
name: String!
email: String
}
# v3 - Deprecate and add new field
type User {
name: String! @deprecated(reason: "Use firstName/lastName")
firstName: String!
lastName: String!
email: String
}
```
## Best Practices Summary
1. **Nullable vs Non-Null**: Start nullable, make non-null when guaranteed
2. **Input Types**: Always use input types for mutations
3. **Payload Pattern**: Return errors in mutation payloads
4. **Pagination**: Use cursor-based for infinite scroll, offset for simple cases
5. **Naming**: Use camelCase for fields, PascalCase for types
6. **Deprecation**: Use `@deprecated` instead of removing fields
7. **DataLoaders**: Always use for relationships to prevent N+1
8. **Complexity Limits**: Protect against expensive queries
9. **Custom Scalars**: Use for domain-specific types (Email, DateTime)
10. **Documentation**: Document all fields with descriptions

385
plugins/backend-development/skills/api-design-principles/references/rest-best-practices.md Normal file
View File

@@ -0,0 +1,385 @@
# REST API Best Practices
## URL Structure
### Resource Naming
```
# Good - Plural nouns
GET /api/users
GET /api/orders
GET /api/products
# Bad - Verbs or mixed conventions
GET /api/getUser
GET /api/user (inconsistent singular)
POST /api/createOrder
```
### Nested Resources
```
# Shallow nesting (preferred)
GET /api/users/{id}/orders
GET /api/orders/{id}
# Deep nesting (avoid)
GET /api/users/{id}/orders/{orderId}/items/{itemId}/reviews
# Better:
GET /api/order-items/{id}/reviews
```
## HTTP Methods and Status Codes
### GET - Retrieve Resources
```
GET /api/users → 200 OK (with list)
GET /api/users/{id} → 200 OK or 404 Not Found
GET /api/users?page=2 → 200 OK (paginated)
```
### POST - Create Resources
```
POST /api/users
Body: {"name": "John", "email": "john@example.com"}
→ 201 Created
Location: /api/users/123
Body: {"id": "123", "name": "John", ...}
POST /api/users (validation error)
→ 422 Unprocessable Entity
Body: {"errors": [...]}
```
### PUT - Replace Resources
```
PUT /api/users/{id}
Body: {complete user object}
→ 200 OK (updated)
→ 404 Not Found (doesn't exist)
# Must include ALL fields
```
### PATCH - Partial Update
```
PATCH /api/users/{id}
Body: {"name": "Jane"} (only changed fields)
→ 200 OK
→ 404 Not Found
```
### DELETE - Remove Resources
```
DELETE /api/users/{id}
→ 204 No Content (deleted)
→ 404 Not Found
→ 409 Conflict (can't delete due to references)
```
## Filtering, Sorting, and Searching
### Query Parameters
```
# Filtering
GET /api/users?status=active
GET /api/users?role=admin&status=active
# Sorting
GET /api/users?sort=created_at
GET /api/users?sort=-created_at (descending)
GET /api/users?sort=name,created_at
# Searching
GET /api/users?search=john
GET /api/users?q=john
# Field selection (sparse fieldsets)
GET /api/users?fields=id,name,email
```
## Pagination Patterns
### Offset-Based Pagination
```python
GET /api/users?page=2&page_size=20
Response:
{
"items": [...],
"page": 2,
"page_size": 20,
"total": 150,
"pages": 8
}
```
### Cursor-Based Pagination (for large datasets)
```python
GET /api/users?limit=20&cursor=eyJpZCI6MTIzfQ
Response:
{
"items": [...],
"next_cursor": "eyJpZCI6MTQzfQ",
"has_more": true
}
```
### Link Header Pagination (RESTful)
```
GET /api/users?page=2
Response Headers:
Link: <https://api.example.com/users?page=3>; rel="next",
<https://api.example.com/users?page=1>; rel="prev",
<https://api.example.com/users?page=1>; rel="first",
<https://api.example.com/users?page=8>; rel="last"
```
## Versioning Strategies
### URL Versioning (Recommended)
```
/api/v1/users
/api/v2/users
Pros: Clear, easy to route
Cons: Multiple URLs for same resource
```
### Header Versioning
```
GET /api/users
Accept: application/vnd.api+json; version=2
Pros: Clean URLs
Cons: Less visible, harder to test
```
### Query Parameter
```
GET /api/users?version=2
Pros: Easy to test
Cons: Optional parameter can be forgotten
```
## Rate Limiting
### Headers
```
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 742
X-RateLimit-Reset: 1640000000
Response when limited:
429 Too Many Requests
Retry-After: 3600
```
### Implementation Pattern
```python
from fastapi import HTTPException, Request
from datetime import datetime, timedelta
class RateLimiter:
def __init__(self, calls: int, period: int):
self.calls = calls
self.period = period
self.cache = {}
def check(self, key: str) -> bool:
now = datetime.now()
if key not in self.cache:
self.cache[key] = []
# Remove old requests
self.cache[key] = [
ts for ts in self.cache[key]
if now - ts < timedelta(seconds=self.period)
]
if len(self.cache[key]) >= self.calls:
return False
self.cache[key].append(now)
return True
limiter = RateLimiter(calls=100, period=60)
@app.get("/api/users")
async def get_users(request: Request):
if not limiter.check(request.client.host):
raise HTTPException(
status_code=429,
headers={"Retry-After": "60"}
)
return {"users": [...]}
```
## Authentication and Authorization
### Bearer Token
```
Authorization: Bearer eyJhbGciOiJIUzI1NiIs...
401 Unauthorized - Missing/invalid token
403 Forbidden - Valid token, insufficient permissions
```
### API Keys
```
X-API-Key: your-api-key-here
```
## Error Response Format
### Consistent Structure
```json
{
"error": {
"code": "VALIDATION_ERROR",
"message": "Request validation failed",
"details": [
{
"field": "email",
"message": "Invalid email format",
"value": "not-an-email"
}
],
"timestamp": "2025-10-16T12:00:00Z",
"path": "/api/users"
}
}
```
### Status Code Guidelines
- `200 OK`: Successful GET, PATCH, PUT
- `201 Created`: Successful POST
- `204 No Content`: Successful DELETE
- `400 Bad Request`: Malformed request
- `401 Unauthorized`: Authentication required
- `403 Forbidden`: Authenticated but not authorized
- `404 Not Found`: Resource doesn't exist
- `409 Conflict`: State conflict (duplicate email, etc.)
- `422 Unprocessable Entity`: Validation errors
- `429 Too Many Requests`: Rate limited
- `500 Internal Server Error`: Server error
- `503 Service Unavailable`: Temporary downtime
## Caching
### Cache Headers
```
# Client caching
Cache-Control: public, max-age=3600
# No caching
Cache-Control: no-cache, no-store, must-revalidate
# Conditional requests
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"
If-None-Match: "33a64df551425fcc55e4d42a148795d9f25f89d4"
→ 304 Not Modified
```
## Bulk Operations
### Batch Endpoints
```python
POST /api/users/batch
{
"items": [
{"name": "User1", "email": "user1@example.com"},
{"name": "User2", "email": "user2@example.com"}
]
}
Response:
{
"results": [
{"id": "1", "status": "created"},
{"id": null, "status": "failed", "error": "Email already exists"}
]
}
```
## Idempotency
### Idempotency Keys
```
POST /api/orders
Idempotency-Key: unique-key-123
If duplicate request:
→ 200 OK (return cached response)
```
## CORS Configuration
```python
from fastapi.middleware.cors import CORSMiddleware
app.add_middleware(
CORSMiddleware,
allow_origins=["https://example.com"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
```
## Documentation with OpenAPI
```python
from fastapi import FastAPI
app = FastAPI(
title="My API",
description="API for managing users",
version="1.0.0",
docs_url="/docs",
redoc_url="/redoc"
)
@app.get(
"/api/users/{user_id}",
summary="Get user by ID",
response_description="User details",
tags=["Users"]
)
async def get_user(
user_id: str = Path(..., description="The user ID")
):
"""
Retrieve user by ID.
Returns full user profile including:
- Basic information
- Contact details
- Account status
"""
pass
```
## Health and Monitoring Endpoints
```python
@app.get("/health")
async def health_check():
return {
"status": "healthy",
"version": "1.0.0",
"timestamp": datetime.now().isoformat()
}
@app.get("/health/detailed")
async def detailed_health():
return {
"status": "healthy",
"checks": {
"database": await check_database(),
"redis": await check_redis(),
"external_api": await check_external_api()
}
}
```

487
plugins/backend-development/skills/architecture-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,487 @@
---
name: architecture-patterns
description: Implement proven backend architecture patterns including Clean Architecture, Hexagonal Architecture, and Domain-Driven Design. Use when architecting complex backend systems or refactoring existing applications for better maintainability.
---
# Architecture Patterns
Master proven backend architecture patterns including Clean Architecture, Hexagonal Architecture, and Domain-Driven Design to build maintainable, testable, and scalable systems.
## When to Use This Skill
- Designing new backend systems from scratch
- Refactoring monolithic applications for better maintainability
- Establishing architecture standards for your team
- Migrating from tightly coupled to loosely coupled architectures
- Implementing domain-driven design principles
- Creating testable and mockable codebases
- Planning microservices decomposition
## Core Concepts
### 1. Clean Architecture (Uncle Bob)
**Layers (dependency flows inward):**
- **Entities**: Core business models
- **Use Cases**: Application business rules
- **Interface Adapters**: Controllers, presenters, gateways
- **Frameworks & Drivers**: UI, database, external services
**Key Principles:**
- Dependencies point inward
- Inner layers know nothing about outer layers
- Business logic independent of frameworks
- Testable without UI, database, or external services
### 2. Hexagonal Architecture (Ports and Adapters)
**Components:**
- **Domain Core**: Business logic
- **Ports**: Interfaces defining interactions
- **Adapters**: Implementations of ports (database, REST, message queue)
**Benefits:**
- Swap implementations easily (mock for testing)
- Technology-agnostic core
- Clear separation of concerns
### 3. Domain-Driven Design (DDD)
**Strategic Patterns:**
- **Bounded Contexts**: Separate models for different domains
- **Context Mapping**: How contexts relate
- **Ubiquitous Language**: Shared terminology
**Tactical Patterns:**
- **Entities**: Objects with identity
- **Value Objects**: Immutable objects defined by attributes
- **Aggregates**: Consistency boundaries
- **Repositories**: Data access abstraction
- **Domain Events**: Things that happened
## Clean Architecture Pattern
### Directory Structure
```
app/
├── domain/ # Entities & business rules
│ ├── entities/
│ │ ├── user.py
│ │ └── order.py
│ ├── value_objects/
│ │ ├── email.py
│ │ └── money.py
│ └── interfaces/ # Abstract interfaces
│ ├── user_repository.py
│ └── payment_gateway.py
├── use_cases/ # Application business rules
│ ├── create_user.py
│ ├── process_order.py
│ └── send_notification.py
├── adapters/ # Interface implementations
│ ├── repositories/
│ │ ├── postgres_user_repository.py
│ │ └── redis_cache_repository.py
│ ├── controllers/
│ │ └── user_controller.py
│ └── gateways/
│ ├── stripe_payment_gateway.py
│ └── sendgrid_email_gateway.py
└── infrastructure/ # Framework & external concerns
├── database.py
├── config.py
└── logging.py
```
### Implementation Example
```python
# domain/entities/user.py
from dataclasses import dataclass
from datetime import datetime
from typing import Optional
@dataclass
class User:
"""Core user entity - no framework dependencies."""
id: str
email: str
name: str
created_at: datetime
is_active: bool = True
def deactivate(self):
"""Business rule: deactivating user."""
self.is_active = False
def can_place_order(self) -> bool:
"""Business rule: active users can order."""
return self.is_active
# domain/interfaces/user_repository.py
from abc import ABC, abstractmethod
from typing import Optional, List
from domain.entities.user import User
class IUserRepository(ABC):
"""Port: defines contract, no implementation."""
@abstractmethod
async def find_by_id(self, user_id: str) -> Optional[User]:
pass
@abstractmethod
async def find_by_email(self, email: str) -> Optional[User]:
pass
@abstractmethod
async def save(self, user: User) -> User:
pass
@abstractmethod
async def delete(self, user_id: str) -> bool:
pass
# use_cases/create_user.py
from domain.entities.user import User
from domain.interfaces.user_repository import IUserRepository
from dataclasses import dataclass
from datetime import datetime
import uuid
@dataclass
class CreateUserRequest:
email: str
name: str
@dataclass
class CreateUserResponse:
user: User
success: bool
error: Optional[str] = None
class CreateUserUseCase:
"""Use case: orchestrates business logic."""
def __init__(self, user_repository: IUserRepository):
self.user_repository = user_repository
async def execute(self, request: CreateUserRequest) -> CreateUserResponse:
# Business validation
existing = await self.user_repository.find_by_email(request.email)
if existing:
return CreateUserResponse(
user=None,
success=False,
error="Email already exists"
)
# Create entity
user = User(
id=str(uuid.uuid4()),
email=request.email,
name=request.name,
created_at=datetime.now(),
is_active=True
)
# Persist
saved_user = await self.user_repository.save(user)
return CreateUserResponse(
user=saved_user,
success=True
)
# adapters/repositories/postgres_user_repository.py
from domain.interfaces.user_repository import IUserRepository
from domain.entities.user import User
from typing import Optional
import asyncpg
class PostgresUserRepository(IUserRepository):
"""Adapter: PostgreSQL implementation."""
def __init__(self, pool: asyncpg.Pool):
self.pool = pool
async def find_by_id(self, user_id: str) -> Optional[User]:
async with self.pool.acquire() as conn:
row = await conn.fetchrow(
"SELECT * FROM users WHERE id = $1", user_id
)
return self._to_entity(row) if row else None
async def find_by_email(self, email: str) -> Optional[User]:
async with self.pool.acquire() as conn:
row = await conn.fetchrow(
"SELECT * FROM users WHERE email = $1", email
)
return self._to_entity(row) if row else None
async def save(self, user: User) -> User:
async with self.pool.acquire() as conn:
await conn.execute(
"""
INSERT INTO users (id, email, name, created_at, is_active)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (id) DO UPDATE
SET email = $2, name = $3, is_active = $5
""",
user.id, user.email, user.name, user.created_at, user.is_active
)
return user
async def delete(self, user_id: str) -> bool:
async with self.pool.acquire() as conn:
result = await conn.execute(
"DELETE FROM users WHERE id = $1", user_id
)
return result == "DELETE 1"
def _to_entity(self, row) -> User:
"""Map database row to entity."""
return User(
id=row["id"],
email=row["email"],
name=row["name"],
created_at=row["created_at"],
is_active=row["is_active"]
)
# adapters/controllers/user_controller.py
from fastapi import APIRouter, Depends, HTTPException
from use_cases.create_user import CreateUserUseCase, CreateUserRequest
from pydantic import BaseModel
router = APIRouter()
class CreateUserDTO(BaseModel):
email: str
name: str
@router.post("/users")
async def create_user(
dto: CreateUserDTO,
use_case: CreateUserUseCase = Depends(get_create_user_use_case)
):
"""Controller: handles HTTP concerns only."""
request = CreateUserRequest(email=dto.email, name=dto.name)
response = await use_case.execute(request)
if not response.success:
raise HTTPException(status_code=400, detail=response.error)
return {"user": response.user}
```
## Hexagonal Architecture Pattern
```python
# Core domain (hexagon center)
class OrderService:
"""Domain service - no infrastructure dependencies."""
def __init__(
self,
order_repository: OrderRepositoryPort,
payment_gateway: PaymentGatewayPort,
notification_service: NotificationPort
):
self.orders = order_repository
self.payments = payment_gateway
self.notifications = notification_service
async def place_order(self, order: Order) -> OrderResult:
# Business logic
if not order.is_valid():
return OrderResult(success=False, error="Invalid order")
# Use ports (interfaces)
payment = await self.payments.charge(
amount=order.total,
customer=order.customer_id
)
if not payment.success:
return OrderResult(success=False, error="Payment failed")
order.mark_as_paid()
saved_order = await self.orders.save(order)
await self.notifications.send(
to=order.customer_email,
subject="Order confirmed",
body=f"Order {order.id} confirmed"
)
return OrderResult(success=True, order=saved_order)
# Ports (interfaces)
class OrderRepositoryPort(ABC):
@abstractmethod
async def save(self, order: Order) -> Order:
pass
class PaymentGatewayPort(ABC):
@abstractmethod
async def charge(self, amount: Money, customer: str) -> PaymentResult:
pass
class NotificationPort(ABC):
@abstractmethod
async def send(self, to: str, subject: str, body: str):
pass
# Adapters (implementations)
class StripePaymentAdapter(PaymentGatewayPort):
"""Primary adapter: connects to Stripe API."""
def __init__(self, api_key: str):
self.stripe = stripe
self.stripe.api_key = api_key
async def charge(self, amount: Money, customer: str) -> PaymentResult:
try:
charge = self.stripe.Charge.create(
amount=amount.cents,
currency=amount.currency,
customer=customer
)
return PaymentResult(success=True, transaction_id=charge.id)
except stripe.error.CardError as e:
return PaymentResult(success=False, error=str(e))
class MockPaymentAdapter(PaymentGatewayPort):
"""Test adapter: no external dependencies."""
async def charge(self, amount: Money, customer: str) -> PaymentResult:
return PaymentResult(success=True, transaction_id="mock-123")
```
## Domain-Driven Design Pattern
```python
# Value Objects (immutable)
from dataclasses import dataclass
from typing import Optional
@dataclass(frozen=True)
class Email:
"""Value object: validated email."""
value: str
def __post_init__(self):
if "@" not in self.value:
raise ValueError("Invalid email")
@dataclass(frozen=True)
class Money:
"""Value object: amount with currency."""
amount: int # cents
currency: str
def add(self, other: "Money") -> "Money":
if self.currency != other.currency:
raise ValueError("Currency mismatch")
return Money(self.amount + other.amount, self.currency)
# Entities (with identity)
class Order:
"""Entity: has identity, mutable state."""
def __init__(self, id: str, customer: Customer):
self.id = id
self.customer = customer
self.items: List[OrderItem] = []
self.status = OrderStatus.PENDING
self._events: List[DomainEvent] = []
def add_item(self, product: Product, quantity: int):
"""Business logic in entity."""
item = OrderItem(product, quantity)
self.items.append(item)
self._events.append(ItemAddedEvent(self.id, item))
def total(self) -> Money:
"""Calculated property."""
return sum(item.subtotal() for item in self.items)
def submit(self):
"""State transition with business rules."""
if not self.items:
raise ValueError("Cannot submit empty order")
if self.status != OrderStatus.PENDING:
raise ValueError("Order already submitted")
self.status = OrderStatus.SUBMITTED
self._events.append(OrderSubmittedEvent(self.id))
# Aggregates (consistency boundary)
class Customer:
"""Aggregate root: controls access to entities."""
def __init__(self, id: str, email: Email):
self.id = id
self.email = email
self._addresses: List[Address] = []
self._orders: List[str] = [] # Order IDs, not full objects
def add_address(self, address: Address):
"""Aggregate enforces invariants."""
if len(self._addresses) >= 5:
raise ValueError("Maximum 5 addresses allowed")
self._addresses.append(address)
@property
def primary_address(self) -> Optional[Address]:
return next((a for a in self._addresses if a.is_primary), None)
# Domain Events
@dataclass
class OrderSubmittedEvent:
order_id: str
occurred_at: datetime = field(default_factory=datetime.now)
# Repository (aggregate persistence)
class OrderRepository:
"""Repository: persist/retrieve aggregates."""
async def find_by_id(self, order_id: str) -> Optional[Order]:
"""Reconstitute aggregate from storage."""
pass
async def save(self, order: Order):
"""Persist aggregate and publish events."""
await self._persist(order)
await self._publish_events(order._events)
order._events.clear()
```
## Resources
- **references/clean-architecture-guide.md**: Detailed layer breakdown
- **references/hexagonal-architecture-guide.md**: Ports and adapters patterns
- **references/ddd-tactical-patterns.md**: Entities, value objects, aggregates
- **assets/clean-architecture-template/**: Complete project structure
- **assets/ddd-examples/**: Domain modeling examples
## Best Practices
1. **Dependency Rule**: Dependencies always point inward
2. **Interface Segregation**: Small, focused interfaces
3. **Business Logic in Domain**: Keep frameworks out of core
4. **Test Independence**: Core testable without infrastructure
5. **Bounded Contexts**: Clear domain boundaries
6. **Ubiquitous Language**: Consistent terminology
7. **Thin Controllers**: Delegate to use cases
8. **Rich Domain Models**: Behavior with data
## Common Pitfalls
- **Anemic Domain**: Entities with only data, no behavior
- **Framework Coupling**: Business logic depends on frameworks
- **Fat Controllers**: Business logic in controllers
- **Repository Leakage**: Exposing ORM objects
- **Missing Abstractions**: Concrete dependencies in core
- **Over-Engineering**: Clean architecture for simple CRUD

585
plugins/backend-development/skills/microservices-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,585 @@
---
name: microservices-patterns
description: Design microservices architectures with service boundaries, event-driven communication, and resilience patterns. Use when building distributed systems, decomposing monoliths, or implementing microservices.
---
# Microservices Patterns
Master microservices architecture patterns including service boundaries, inter-service communication, data management, and resilience patterns for building distributed systems.
## When to Use This Skill
- Decomposing monoliths into microservices
- Designing service boundaries and contracts
- Implementing inter-service communication
- Managing distributed data and transactions
- Building resilient distributed systems
- Implementing service discovery and load balancing
- Designing event-driven architectures
## Core Concepts
### 1. Service Decomposition Strategies
**By Business Capability**
- Organize services around business functions
- Each service owns its domain
- Example: OrderService, PaymentService, InventoryService
**By Subdomain (DDD)**
- Core domain, supporting subdomains
- Bounded contexts map to services
- Clear ownership and responsibility
**Strangler Fig Pattern**
- Gradually extract from monolith
- New functionality as microservices
- Proxy routes to old/new systems
### 2. Communication Patterns
**Synchronous (Request/Response)**
- REST APIs
- gRPC
- GraphQL
**Asynchronous (Events/Messages)**
- Event streaming (Kafka)
- Message queues (RabbitMQ, SQS)
- Pub/Sub patterns
### 3. Data Management
**Database Per Service**
- Each service owns its data
- No shared databases
- Loose coupling
**Saga Pattern**
- Distributed transactions
- Compensating actions
- Eventual consistency
### 4. Resilience Patterns
**Circuit Breaker**
- Fail fast on repeated errors
- Prevent cascade failures
**Retry with Backoff**
- Transient fault handling
- Exponential backoff
**Bulkhead**
- Isolate resources
- Limit impact of failures
## Service Decomposition Patterns
### Pattern 1: By Business Capability
```python
# E-commerce example
# Order Service
class OrderService:
"""Handles order lifecycle."""
async def create_order(self, order_data: dict) -> Order:
order = Order.create(order_data)
# Publish event for other services
await self.event_bus.publish(
OrderCreatedEvent(
order_id=order.id,
customer_id=order.customer_id,
items=order.items,
total=order.total
)
)
return order
# Payment Service (separate service)
class PaymentService:
"""Handles payment processing."""
async def process_payment(self, payment_request: PaymentRequest) -> PaymentResult:
# Process payment
result = await self.payment_gateway.charge(
amount=payment_request.amount,
customer=payment_request.customer_id
)
if result.success:
await self.event_bus.publish(
PaymentCompletedEvent(
order_id=payment_request.order_id,
transaction_id=result.transaction_id
)
)
return result
# Inventory Service (separate service)
class InventoryService:
"""Handles inventory management."""
async def reserve_items(self, order_id: str, items: List[OrderItem]) -> ReservationResult:
# Check availability
for item in items:
available = await self.inventory_repo.get_available(item.product_id)
if available < item.quantity:
return ReservationResult(
success=False,
error=f"Insufficient inventory for {item.product_id}"
)
# Reserve items
reservation = await self.create_reservation(order_id, items)
await self.event_bus.publish(
InventoryReservedEvent(
order_id=order_id,
reservation_id=reservation.id
)
)
return ReservationResult(success=True, reservation=reservation)
```
### Pattern 2: API Gateway
```python
from fastapi import FastAPI, HTTPException, Depends
import httpx
from circuitbreaker import circuit
app = FastAPI()
class APIGateway:
"""Central entry point for all client requests."""
def __init__(self):
self.order_service_url = "http://order-service:8000"
self.payment_service_url = "http://payment-service:8001"
self.inventory_service_url = "http://inventory-service:8002"
self.http_client = httpx.AsyncClient(timeout=5.0)
@circuit(failure_threshold=5, recovery_timeout=30)
async def call_order_service(self, path: str, method: str = "GET", **kwargs):
"""Call order service with circuit breaker."""
response = await self.http_client.request(
method,
f"{self.order_service_url}{path}",
**kwargs
)
response.raise_for_status()
return response.json()
async def create_order_aggregate(self, order_id: str) -> dict:
"""Aggregate data from multiple services."""
# Parallel requests
order, payment, inventory = await asyncio.gather(
self.call_order_service(f"/orders/{order_id}"),
self.call_payment_service(f"/payments/order/{order_id}"),
self.call_inventory_service(f"/reservations/order/{order_id}"),
return_exceptions=True
)
# Handle partial failures
result = {"order": order}
if not isinstance(payment, Exception):
result["payment"] = payment
if not isinstance(inventory, Exception):
result["inventory"] = inventory
return result
@app.post("/api/orders")
async def create_order(
order_data: dict,
gateway: APIGateway = Depends()
):
"""API Gateway endpoint."""
try:
# Route to order service
order = await gateway.call_order_service(
"/orders",
method="POST",
json=order_data
)
return {"order": order}
except httpx.HTTPError as e:
raise HTTPException(status_code=503, detail="Order service unavailable")
```
## Communication Patterns
### Pattern 1: Synchronous REST Communication
```python
# Service A calls Service B
import httpx
from tenacity import retry, stop_after_attempt, wait_exponential
class ServiceClient:
"""HTTP client with retries and timeout."""
def __init__(self, base_url: str):
self.base_url = base_url
self.client = httpx.AsyncClient(
timeout=httpx.Timeout(5.0, connect=2.0),
limits=httpx.Limits(max_keepalive_connections=20)
)
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=10)
)
async def get(self, path: str, **kwargs):
"""GET with automatic retries."""
response = await self.client.get(f"{self.base_url}{path}", **kwargs)
response.raise_for_status()
return response.json()
async def post(self, path: str, **kwargs):
"""POST request."""
response = await self.client.post(f"{self.base_url}{path}", **kwargs)
response.raise_for_status()
return response.json()
# Usage
payment_client = ServiceClient("http://payment-service:8001")
result = await payment_client.post("/payments", json=payment_data)
```
### Pattern 2: Asynchronous Event-Driven
```python
# Event-driven communication with Kafka
from aiokafka import AIOKafkaProducer, AIOKafkaConsumer
import json
from dataclasses import dataclass, asdict
from datetime import datetime
@dataclass
class DomainEvent:
event_id: str
event_type: str
aggregate_id: str
occurred_at: datetime
data: dict
class EventBus:
"""Event publishing and subscription."""
def __init__(self, bootstrap_servers: List[str]):
self.bootstrap_servers = bootstrap_servers
self.producer = None
async def start(self):
self.producer = AIOKafkaProducer(
bootstrap_servers=self.bootstrap_servers,
value_serializer=lambda v: json.dumps(v).encode()
)
await self.producer.start()
async def publish(self, event: DomainEvent):
"""Publish event to Kafka topic."""
topic = event.event_type
await self.producer.send_and_wait(
topic,
value=asdict(event),
key=event.aggregate_id.encode()
)
async def subscribe(self, topic: str, handler: callable):
"""Subscribe to events."""
consumer = AIOKafkaConsumer(
topic,
bootstrap_servers=self.bootstrap_servers,
value_deserializer=lambda v: json.loads(v.decode()),
group_id="my-service"
)
await consumer.start()
try:
async for message in consumer:
event_data = message.value
await handler(event_data)
finally:
await consumer.stop()
# Order Service publishes event
async def create_order(order_data: dict):
order = await save_order(order_data)
event = DomainEvent(
event_id=str(uuid.uuid4()),
event_type="OrderCreated",
aggregate_id=order.id,
occurred_at=datetime.now(),
data={
"order_id": order.id,
"customer_id": order.customer_id,
"total": order.total
}
)
await event_bus.publish(event)
# Inventory Service listens for OrderCreated
async def handle_order_created(event_data: dict):
"""React to order creation."""
order_id = event_data["data"]["order_id"]
items = event_data["data"]["items"]
# Reserve inventory
await reserve_inventory(order_id, items)
```
### Pattern 3: Saga Pattern (Distributed Transactions)
```python
# Saga orchestration for order fulfillment
from enum import Enum
from typing import List, Callable
class SagaStep:
"""Single step in saga."""
def __init__(
self,
name: str,
action: Callable,
compensation: Callable
):
self.name = name
self.action = action
self.compensation = compensation
class SagaStatus(Enum):
PENDING = "pending"
COMPLETED = "completed"
COMPENSATING = "compensating"
FAILED = "failed"
class OrderFulfillmentSaga:
"""Orchestrated saga for order fulfillment."""
def __init__(self):
self.steps: List[SagaStep] = [
SagaStep(
"create_order",
action=self.create_order,
compensation=self.cancel_order
),
SagaStep(
"reserve_inventory",
action=self.reserve_inventory,
compensation=self.release_inventory
),
SagaStep(
"process_payment",
action=self.process_payment,
compensation=self.refund_payment
),
SagaStep(
"confirm_order",
action=self.confirm_order,
compensation=self.cancel_order_confirmation
)
]
async def execute(self, order_data: dict) -> SagaResult:
"""Execute saga steps."""
completed_steps = []
context = {"order_data": order_data}
try:
for step in self.steps:
# Execute step
result = await step.action(context)
if not result.success:
# Compensate
await self.compensate(completed_steps, context)
return SagaResult(
status=SagaStatus.FAILED,
error=result.error
)
completed_steps.append(step)
context.update(result.data)
return SagaResult(status=SagaStatus.COMPLETED, data=context)
except Exception as e:
# Compensate on error
await self.compensate(completed_steps, context)
return SagaResult(status=SagaStatus.FAILED, error=str(e))
async def compensate(self, completed_steps: List[SagaStep], context: dict):
"""Execute compensating actions in reverse order."""
for step in reversed(completed_steps):
try:
await step.compensation(context)
except Exception as e:
# Log compensation failure
print(f"Compensation failed for {step.name}: {e}")
# Step implementations
async def create_order(self, context: dict) -> StepResult:
order = await order_service.create(context["order_data"])
return StepResult(success=True, data={"order_id": order.id})
async def cancel_order(self, context: dict):
await order_service.cancel(context["order_id"])
async def reserve_inventory(self, context: dict) -> StepResult:
result = await inventory_service.reserve(
context["order_id"],
context["order_data"]["items"]
)
return StepResult(
success=result.success,
data={"reservation_id": result.reservation_id}
)
async def release_inventory(self, context: dict):
await inventory_service.release(context["reservation_id"])
async def process_payment(self, context: dict) -> StepResult:
result = await payment_service.charge(
context["order_id"],
context["order_data"]["total"]
)
return StepResult(
success=result.success,
data={"transaction_id": result.transaction_id},
error=result.error
)
async def refund_payment(self, context: dict):
await payment_service.refund(context["transaction_id"])
```
## Resilience Patterns
### Circuit Breaker Pattern
```python
from enum import Enum
from datetime import datetime, timedelta
from typing import Callable, Any
class CircuitState(Enum):
CLOSED = "closed" # Normal operation
OPEN = "open" # Failing, reject requests
HALF_OPEN = "half_open" # Testing if recovered
class CircuitBreaker:
"""Circuit breaker for service calls."""
def __init__(
self,
failure_threshold: int = 5,
recovery_timeout: int = 30,
success_threshold: int = 2
):
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.success_threshold = success_threshold
self.failure_count = 0
self.success_count = 0
self.state = CircuitState.CLOSED
self.opened_at = None
async def call(self, func: Callable, *args, **kwargs) -> Any:
"""Execute function with circuit breaker."""
if self.state == CircuitState.OPEN:
if self._should_attempt_reset():
self.state = CircuitState.HALF_OPEN
else:
raise CircuitBreakerOpenError("Circuit breaker is open")
try:
result = await func(*args, **kwargs)
self._on_success()
return result
except Exception as e:
self._on_failure()
raise
def _on_success(self):
"""Handle successful call."""
self.failure_count = 0
if self.state == CircuitState.HALF_OPEN:
self.success_count += 1
if self.success_count >= self.success_threshold:
self.state = CircuitState.CLOSED
self.success_count = 0
def _on_failure(self):
"""Handle failed call."""
self.failure_count += 1
if self.failure_count >= self.failure_threshold:
self.state = CircuitState.OPEN
self.opened_at = datetime.now()
if self.state == CircuitState.HALF_OPEN:
self.state = CircuitState.OPEN
self.opened_at = datetime.now()
def _should_attempt_reset(self) -> bool:
"""Check if enough time passed to try again."""
return (
datetime.now() - self.opened_at
> timedelta(seconds=self.recovery_timeout)
)
# Usage
breaker = CircuitBreaker(failure_threshold=5, recovery_timeout=30)
async def call_payment_service(payment_data: dict):
return await breaker.call(
payment_client.process_payment,
payment_data
)
```
## Resources
- **references/service-decomposition-guide.md**: Breaking down monoliths
- **references/communication-patterns.md**: Sync vs async patterns
- **references/saga-implementation.md**: Distributed transactions
- **assets/circuit-breaker.py**: Production circuit breaker
- **assets/event-bus-template.py**: Kafka event bus implementation
- **assets/api-gateway-template.py**: Complete API gateway
## Best Practices
1. **Service Boundaries**: Align with business capabilities
2. **Database Per Service**: No shared databases
3. **API Contracts**: Versioned, backward compatible
4. **Async When Possible**: Events over direct calls
5. **Circuit Breakers**: Fail fast on service failures
6. **Distributed Tracing**: Track requests across services
7. **Service Registry**: Dynamic service discovery
8. **Health Checks**: Liveness and readiness probes
## Common Pitfalls
- **Distributed Monolith**: Tightly coupled services
- **Chatty Services**: Too many inter-service calls
- **Shared Databases**: Tight coupling through data
- **No Circuit Breakers**: Cascade failures
- **Synchronous Everything**: Tight coupling, poor resilience
- **Premature Microservices**: Starting with microservices
- **Ignoring Network Failures**: Assuming reliable network
- **No Compensation Logic**: Can't undo failed transactions

454
plugins/blockchain-web3/skills/defi-protocol-templates/SKILL.md Normal file
View File

@@ -0,0 +1,454 @@
---
name: defi-protocol-templates
description: Implement DeFi protocols with production-ready templates for staking, AMMs, governance, and lending systems. Use when building decentralized finance applications or smart contract protocols.
---
# DeFi Protocol Templates
Production-ready templates for common DeFi protocols including staking, AMMs, governance, lending, and flash loans.
## When to Use This Skill
- Building staking platforms with reward distribution
- Implementing AMM (Automated Market Maker) protocols
- Creating governance token systems
- Developing lending/borrowing protocols
- Integrating flash loan functionality
- Launching yield farming platforms
## Staking Contract
```solidity
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
import "@openzeppelin/contracts/token/ERC20/IERC20.sol";
import "@openzeppelin/contracts/security/ReentrancyGuard.sol";
import "@openzeppelin/contracts/access/Ownable.sol";
contract StakingRewards is ReentrancyGuard, Ownable {
IERC20 public stakingToken;
IERC20 public rewardsToken;
uint256 public rewardRate = 100; // Rewards per second
uint256 public lastUpdateTime;
uint256 public rewardPerTokenStored;
mapping(address => uint256) public userRewardPerTokenPaid;
mapping(address => uint256) public rewards;
mapping(address => uint256) public balances;
uint256 private _totalSupply;
event Staked(address indexed user, uint256 amount);
event Withdrawn(address indexed user, uint256 amount);
event RewardPaid(address indexed user, uint256 reward);
constructor(address _stakingToken, address _rewardsToken) {
stakingToken = IERC20(_stakingToken);
rewardsToken = IERC20(_rewardsToken);
}
modifier updateReward(address account) {
rewardPerTokenStored = rewardPerToken();
lastUpdateTime = block.timestamp;
if (account != address(0)) {
rewards[account] = earned(account);
userRewardPerTokenPaid[account] = rewardPerTokenStored;
}
_;
}
function rewardPerToken() public view returns (uint256) {
if (_totalSupply == 0) {
return rewardPerTokenStored;
}
return rewardPerTokenStored +
((block.timestamp - lastUpdateTime) * rewardRate * 1e18) / _totalSupply;
}
function earned(address account) public view returns (uint256) {
return (balances[account] *
(rewardPerToken() - userRewardPerTokenPaid[account])) / 1e18 +
rewards[account];
}
function stake(uint256 amount) external nonReentrant updateReward(msg.sender) {
require(amount > 0, "Cannot stake 0");
_totalSupply += amount;
balances[msg.sender] += amount;
stakingToken.transferFrom(msg.sender, address(this), amount);
emit Staked(msg.sender, amount);
}
function withdraw(uint256 amount) public nonReentrant updateReward(msg.sender) {
require(amount > 0, "Cannot withdraw 0");
_totalSupply -= amount;
balances[msg.sender] -= amount;
stakingToken.transfer(msg.sender, amount);
emit Withdrawn(msg.sender, amount);
}
function getReward() public nonReentrant updateReward(msg.sender) {
uint256 reward = rewards[msg.sender];
if (reward > 0) {
rewards[msg.sender] = 0;
rewardsToken.transfer(msg.sender, reward);
emit RewardPaid(msg.sender, reward);
}
}
function exit() external {
withdraw(balances[msg.sender]);
getReward();
}
}
```
## AMM (Automated Market Maker)
```solidity
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
import "@openzeppelin/contracts/token/ERC20/IERC20.sol";
contract SimpleAMM {
IERC20 public token0;
IERC20 public token1;
uint256 public reserve0;
uint256 public reserve1;
uint256 public totalSupply;
mapping(address => uint256) public balanceOf;
event Mint(address indexed to, uint256 amount);
event Burn(address indexed from, uint256 amount);
event Swap(address indexed trader, uint256 amount0In, uint256 amount1In, uint256 amount0Out, uint256 amount1Out);
constructor(address _token0, address _token1) {
token0 = IERC20(_token0);
token1 = IERC20(_token1);
}
function addLiquidity(uint256 amount0, uint256 amount1) external returns (uint256 shares) {
token0.transferFrom(msg.sender, address(this), amount0);
token1.transferFrom(msg.sender, address(this), amount1);
if (totalSupply == 0) {
shares = sqrt(amount0 * amount1);
} else {
shares = min(
(amount0 * totalSupply) / reserve0,
(amount1 * totalSupply) / reserve1
);
}
require(shares > 0, "Shares = 0");
_mint(msg.sender, shares);
_update(
token0.balanceOf(address(this)),
token1.balanceOf(address(this))
);
emit Mint(msg.sender, shares);
}
function removeLiquidity(uint256 shares) external returns (uint256 amount0, uint256 amount1) {
uint256 bal0 = token0.balanceOf(address(this));
uint256 bal1 = token1.balanceOf(address(this));
amount0 = (shares * bal0) / totalSupply;
amount1 = (shares * bal1) / totalSupply;
require(amount0 > 0 && amount1 > 0, "Amount0 or amount1 = 0");
_burn(msg.sender, shares);
_update(bal0 - amount0, bal1 - amount1);
token0.transfer(msg.sender, amount0);
token1.transfer(msg.sender, amount1);
emit Burn(msg.sender, shares);
}
function swap(address tokenIn, uint256 amountIn) external returns (uint256 amountOut) {
require(tokenIn == address(token0) || tokenIn == address(token1), "Invalid token");
bool isToken0 = tokenIn == address(token0);
(IERC20 tokenIn_, IERC20 tokenOut, uint256 resIn, uint256 resOut) = isToken0
? (token0, token1, reserve0, reserve1)
: (token1, token0, reserve1, reserve0);
tokenIn_.transferFrom(msg.sender, address(this), amountIn);
// 0.3% fee
uint256 amountInWithFee = (amountIn * 997) / 1000;
amountOut = (resOut * amountInWithFee) / (resIn + amountInWithFee);
tokenOut.transfer(msg.sender, amountOut);
_update(
token0.balanceOf(address(this)),
token1.balanceOf(address(this))
);
emit Swap(msg.sender, isToken0 ? amountIn : 0, isToken0 ? 0 : amountIn, isToken0 ? 0 : amountOut, isToken0 ? amountOut : 0);
}
function _mint(address to, uint256 amount) private {
balanceOf[to] += amount;
totalSupply += amount;
}
function _burn(address from, uint256 amount) private {
balanceOf[from] -= amount;
totalSupply -= amount;
}
function _update(uint256 res0, uint256 res1) private {
reserve0 = res0;
reserve1 = res1;
}
function sqrt(uint256 y) private pure returns (uint256 z) {
if (y > 3) {
z = y;
uint256 x = y / 2 + 1;
while (x < z) {
z = x;
x = (y / x + x) / 2;
}
} else if (y != 0) {
z = 1;
}
}
function min(uint256 x, uint256 y) private pure returns (uint256) {
return x <= y ? x : y;
}
}
```
## Governance Token
```solidity
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
import "@openzeppelin/contracts/token/ERC20/extensions/ERC20Votes.sol";
import "@openzeppelin/contracts/access/Ownable.sol";
contract GovernanceToken is ERC20Votes, Ownable {
constructor() ERC20("Governance Token", "GOV") ERC20Permit("Governance Token") {
_mint(msg.sender, 1000000 * 10**decimals());
}
function _afterTokenTransfer(
address from,
address to,
uint256 amount
) internal override(ERC20Votes) {
super._afterTokenTransfer(from, to, amount);
}
function _mint(address to, uint256 amount) internal override(ERC20Votes) {
super._mint(to, amount);
}
function _burn(address account, uint256 amount) internal override(ERC20Votes) {
super._burn(account, amount);
}
}
contract Governor is Ownable {
GovernanceToken public governanceToken;
struct Proposal {
uint256 id;
address proposer;
string description;
uint256 forVotes;
uint256 againstVotes;
uint256 startBlock;
uint256 endBlock;
bool executed;
mapping(address => bool) hasVoted;
}
uint256 public proposalCount;
mapping(uint256 => Proposal) public proposals;
uint256 public votingPeriod = 17280; // ~3 days in blocks
uint256 public proposalThreshold = 100000 * 10**18;
event ProposalCreated(uint256 indexed proposalId, address proposer, string description);
event VoteCast(address indexed voter, uint256 indexed proposalId, bool support, uint256 weight);
event ProposalExecuted(uint256 indexed proposalId);
constructor(address _governanceToken) {
governanceToken = GovernanceToken(_governanceToken);
}
function propose(string memory description) external returns (uint256) {
require(
governanceToken.getPastVotes(msg.sender, block.number - 1) >= proposalThreshold,
"Proposer votes below threshold"
);
proposalCount++;
Proposal storage newProposal = proposals[proposalCount];
newProposal.id = proposalCount;
newProposal.proposer = msg.sender;
newProposal.description = description;
newProposal.startBlock = block.number;
newProposal.endBlock = block.number + votingPeriod;
emit ProposalCreated(proposalCount, msg.sender, description);
return proposalCount;
}
function vote(uint256 proposalId, bool support) external {
Proposal storage proposal = proposals[proposalId];
require(block.number >= proposal.startBlock, "Voting not started");
require(block.number <= proposal.endBlock, "Voting ended");
require(!proposal.hasVoted[msg.sender], "Already voted");
uint256 weight = governanceToken.getPastVotes(msg.sender, proposal.startBlock);
require(weight > 0, "No voting power");
proposal.hasVoted[msg.sender] = true;
if (support) {
proposal.forVotes += weight;
} else {
proposal.againstVotes += weight;
}
emit VoteCast(msg.sender, proposalId, support, weight);
}
function execute(uint256 proposalId) external {
Proposal storage proposal = proposals[proposalId];
require(block.number > proposal.endBlock, "Voting not ended");
require(!proposal.executed, "Already executed");
require(proposal.forVotes > proposal.againstVotes, "Proposal failed");
proposal.executed = true;
// Execute proposal logic here
emit ProposalExecuted(proposalId);
}
}
```
## Flash Loan
```solidity
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
import "@openzeppelin/contracts/token/ERC20/IERC20.sol";
interface IFlashLoanReceiver {
function executeOperation(
address asset,
uint256 amount,
uint256 fee,
bytes calldata params
) external returns (bool);
}
contract FlashLoanProvider {
IERC20 public token;
uint256 public feePercentage = 9; // 0.09% fee
event FlashLoan(address indexed borrower, uint256 amount, uint256 fee);
constructor(address _token) {
token = IERC20(_token);
}
function flashLoan(
address receiver,
uint256 amount,
bytes calldata params
) external {
uint256 balanceBefore = token.balanceOf(address(this));
require(balanceBefore >= amount, "Insufficient liquidity");
uint256 fee = (amount * feePercentage) / 10000;
// Send tokens to receiver
token.transfer(receiver, amount);
// Execute callback
require(
IFlashLoanReceiver(receiver).executeOperation(
address(token),
amount,
fee,
params
),
"Flash loan failed"
);
// Verify repayment
uint256 balanceAfter = token.balanceOf(address(this));
require(balanceAfter >= balanceBefore + fee, "Flash loan not repaid");
emit FlashLoan(receiver, amount, fee);
}
}
// Example flash loan receiver
contract FlashLoanReceiver is IFlashLoanReceiver {
function executeOperation(
address asset,
uint256 amount,
uint256 fee,
bytes calldata params
) external override returns (bool) {
// Decode params and execute arbitrage, liquidation, etc.
// ...
// Approve repayment
IERC20(asset).approve(msg.sender, amount + fee);
return true;
}
}
```
## Resources
- **references/staking.md**: Staking mechanics and reward distribution
- **references/liquidity-pools.md**: AMM mathematics and pricing
- **references/governance-tokens.md**: Governance and voting systems
- **references/lending-protocols.md**: Lending/borrowing implementation
- **references/flash-loans.md**: Flash loan security and use cases
- **assets/staking-contract.sol**: Production staking template
- **assets/amm-contract.sol**: Full AMM implementation
- **assets/governance-token.sol**: Governance system
- **assets/lending-protocol.sol**: Lending platform template
## Best Practices
1. **Use Established Libraries**: OpenZeppelin, Solmate
2. **Test Thoroughly**: Unit tests, integration tests, fuzzing
3. **Audit Before Launch**: Professional security audits
4. **Start Simple**: MVP first, add features incrementally
5. **Monitor**: Track contract health and user activity
6. **Upgradability**: Consider proxy patterns for upgrades
7. **Emergency Controls**: Pause mechanisms for critical issues
## Common DeFi Patterns
- **Time-Weighted Average Price (TWAP)**: Price oracle resistance
- **Liquidity Mining**: Incentivize liquidity provision
- **Vesting**: Lock tokens with gradual release
- **Multisig**: Require multiple signatures for critical operations
- **Timelocks**: Delay execution of governance decisions

381
plugins/blockchain-web3/skills/nft-standards/SKILL.md Normal file
View File

@@ -0,0 +1,381 @@
---
name: nft-standards
description: Implement NFT standards (ERC-721, ERC-1155) with proper metadata handling, minting strategies, and marketplace integration. Use when creating NFT contracts, building NFT marketplaces, or implementing digital asset systems.
---
# NFT Standards
Master ERC-721 and ERC-1155 NFT standards, metadata best practices, and advanced NFT features.
## When to Use This Skill
- Creating NFT collections (art, gaming, collectibles)
- Implementing marketplace functionality
- Building on-chain or off-chain metadata
- Creating soulbound tokens (non-transferable)
- Implementing royalties and revenue sharing
- Developing dynamic/evolving NFTs
## ERC-721 (Non-Fungible Token Standard)
```solidity
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
import "@openzeppelin/contracts/token/ERC721/extensions/ERC721URIStorage.sol";
import "@openzeppelin/contracts/token/ERC721/extensions/ERC721Enumerable.sol";
import "@openzeppelin/contracts/access/Ownable.sol";
import "@openzeppelin/contracts/utils/Counters.sol";
contract MyNFT is ERC721URIStorage, ERC721Enumerable, Ownable {
using Counters for Counters.Counter;
Counters.Counter private _tokenIds;
uint256 public constant MAX_SUPPLY = 10000;
uint256 public constant MINT_PRICE = 0.08 ether;
uint256 public constant MAX_PER_MINT = 20;
constructor() ERC721("MyNFT", "MNFT") {}
function mint(uint256 quantity) external payable {
require(quantity > 0 && quantity <= MAX_PER_MINT, "Invalid quantity");
require(_tokenIds.current() + quantity <= MAX_SUPPLY, "Exceeds max supply");
require(msg.value >= MINT_PRICE * quantity, "Insufficient payment");
for (uint256 i = 0; i < quantity; i++) {
_tokenIds.increment();
uint256 newTokenId = _tokenIds.current();
_safeMint(msg.sender, newTokenId);
_setTokenURI(newTokenId, generateTokenURI(newTokenId));
}
}
function generateTokenURI(uint256 tokenId) internal pure returns (string memory) {
// Return IPFS URI or on-chain metadata
return string(abi.encodePacked("ipfs://QmHash/", Strings.toString(tokenId), ".json"));
}
// Required overrides
function _beforeTokenTransfer(
address from,
address to,
uint256 tokenId,
uint256 batchSize
) internal override(ERC721, ERC721Enumerable) {
super._beforeTokenTransfer(from, to, tokenId, batchSize);
}
function _burn(uint256 tokenId) internal override(ERC721, ERC721URIStorage) {
super._burn(tokenId);
}
function tokenURI(uint256 tokenId) public view override(ERC721, ERC721URIStorage) returns (string memory) {
return super.tokenURI(tokenId);
}
function supportsInterface(bytes4 interfaceId)
public
view
override(ERC721, ERC721Enumerable)
returns (bool)
{
return super.supportsInterface(interfaceId);
}
function withdraw() external onlyOwner {
payable(owner()).transfer(address(this).balance);
}
}
```
## ERC-1155 (Multi-Token Standard)
```solidity
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
import "@openzeppelin/contracts/token/ERC1155/ERC1155.sol";
import "@openzeppelin/contracts/access/Ownable.sol";
contract GameItems is ERC1155, Ownable {
uint256 public constant SWORD = 1;
uint256 public constant SHIELD = 2;
uint256 public constant POTION = 3;
mapping(uint256 => uint256) public tokenSupply;
mapping(uint256 => uint256) public maxSupply;
constructor() ERC1155("ipfs://QmBaseHash/{id}.json") {
maxSupply[SWORD] = 1000;
maxSupply[SHIELD] = 500;
maxSupply[POTION] = 10000;
}
function mint(
address to,
uint256 id,
uint256 amount
) external onlyOwner {
require(tokenSupply[id] + amount <= maxSupply[id], "Exceeds max supply");
_mint(to, id, amount, "");
tokenSupply[id] += amount;
}
function mintBatch(
address to,
uint256[] memory ids,
uint256[] memory amounts
) external onlyOwner {
for (uint256 i = 0; i < ids.length; i++) {
require(tokenSupply[ids[i]] + amounts[i] <= maxSupply[ids[i]], "Exceeds max supply");
tokenSupply[ids[i]] += amounts[i];
}
_mintBatch(to, ids, amounts, "");
}
function burn(
address from,
uint256 id,
uint256 amount
) external {
require(from == msg.sender || isApprovedForAll(from, msg.sender), "Not authorized");
_burn(from, id, amount);
tokenSupply[id] -= amount;
}
}
```
## Metadata Standards
### Off-Chain Metadata (IPFS)
```json
{
"name": "NFT #1",
"description": "Description of the NFT",
"image": "ipfs://QmImageHash",
"attributes": [
{
"trait_type": "Background",
"value": "Blue"
},
{
"trait_type": "Rarity",
"value": "Legendary"
},
{
"trait_type": "Power",
"value": 95,
"display_type": "number",
"max_value": 100
}
]
}
```
### On-Chain Metadata
```solidity
contract OnChainNFT is ERC721 {
struct Traits {
uint8 background;
uint8 body;
uint8 head;
uint8 rarity;
}
mapping(uint256 => Traits) public tokenTraits;
function tokenURI(uint256 tokenId) public view override returns (string memory) {
Traits memory traits = tokenTraits[tokenId];
string memory json = Base64.encode(
bytes(
string(
abi.encodePacked(
'{"name": "NFT #', Strings.toString(tokenId), '",',
'"description": "On-chain NFT",',
'"image": "data:image/svg+xml;base64,', generateSVG(traits), '",',
'"attributes": [',
'{"trait_type": "Background", "value": "', Strings.toString(traits.background), '"},',
'{"trait_type": "Rarity", "value": "', getRarityName(traits.rarity), '"}',
']}'
)
)
)
);
return string(abi.encodePacked("data:application/json;base64,", json));
}
function generateSVG(Traits memory traits) internal pure returns (string memory) {
// Generate SVG based on traits
return "...";
}
}
```
## Royalties (EIP-2981)
```solidity
import "@openzeppelin/contracts/interfaces/IERC2981.sol";
contract NFTWithRoyalties is ERC721, IERC2981 {
address public royaltyRecipient;
uint96 public royaltyFee = 500; // 5%
constructor() ERC721("Royalty NFT", "RNFT") {
royaltyRecipient = msg.sender;
}
function royaltyInfo(uint256 tokenId, uint256 salePrice)
external
view
override
returns (address receiver, uint256 royaltyAmount)
{
return (royaltyRecipient, (salePrice * royaltyFee) / 10000);
}
function setRoyalty(address recipient, uint96 fee) external onlyOwner {
require(fee <= 1000, "Royalty fee too high"); // Max 10%
royaltyRecipient = recipient;
royaltyFee = fee;
}
function supportsInterface(bytes4 interfaceId)
public
view
override(ERC721, IERC165)
returns (bool)
{
return interfaceId == type(IERC2981).interfaceId ||
super.supportsInterface(interfaceId);
}
}
```
## Soulbound Tokens (Non-Transferable)
```solidity
contract SoulboundToken is ERC721 {
constructor() ERC721("Soulbound", "SBT") {}
function _beforeTokenTransfer(
address from,
address to,
uint256 tokenId,
uint256 batchSize
) internal virtual override {
require(from == address(0) || to == address(0), "Token is soulbound");
super._beforeTokenTransfer(from, to, tokenId, batchSize);
}
function mint(address to) external {
uint256 tokenId = totalSupply() + 1;
_safeMint(to, tokenId);
}
// Burn is allowed (user can destroy their SBT)
function burn(uint256 tokenId) external {
require(ownerOf(tokenId) == msg.sender, "Not token owner");
_burn(tokenId);
}
}
```
## Dynamic NFTs
```solidity
contract DynamicNFT is ERC721 {
struct TokenState {
uint256 level;
uint256 experience;
uint256 lastUpdated;
}
mapping(uint256 => TokenState) public tokenStates;
function gainExperience(uint256 tokenId, uint256 exp) external {
require(ownerOf(tokenId) == msg.sender, "Not token owner");
TokenState storage state = tokenStates[tokenId];
state.experience += exp;
// Level up logic
if (state.experience >= state.level * 100) {
state.level++;
}
state.lastUpdated = block.timestamp;
}
function tokenURI(uint256 tokenId) public view override returns (string memory) {
TokenState memory state = tokenStates[tokenId];
// Generate metadata based on current state
return generateMetadata(tokenId, state);
}
function generateMetadata(uint256 tokenId, TokenState memory state)
internal
pure
returns (string memory)
{
// Dynamic metadata generation
return "";
}
}
```
## Gas-Optimized Minting (ERC721A)
```solidity
import "erc721a/contracts/ERC721A.sol";
contract OptimizedNFT is ERC721A {
uint256 public constant MAX_SUPPLY = 10000;
uint256 public constant MINT_PRICE = 0.05 ether;
constructor() ERC721A("Optimized NFT", "ONFT") {}
function mint(uint256 quantity) external payable {
require(_totalMinted() + quantity <= MAX_SUPPLY, "Exceeds max supply");
require(msg.value >= MINT_PRICE * quantity, "Insufficient payment");
_mint(msg.sender, quantity);
}
function _baseURI() internal pure override returns (string memory) {
return "ipfs://QmBaseHash/";
}
}
```
## Resources
- **references/erc721.md**: ERC-721 specification details
- **references/erc1155.md**: ERC-1155 multi-token standard
- **references/metadata-standards.md**: Metadata best practices
- **references/enumeration.md**: Token enumeration patterns
- **assets/erc721-contract.sol**: Production ERC-721 template
- **assets/erc1155-contract.sol**: Production ERC-1155 template
- **assets/metadata-schema.json**: Standard metadata format
- **assets/metadata-uploader.py**: IPFS upload utility
## Best Practices
1. **Use OpenZeppelin**: Battle-tested implementations
2. **Pin Metadata**: Use IPFS with pinning service
3. **Implement Royalties**: EIP-2981 for marketplace compatibility
4. **Gas Optimization**: Use ERC721A for batch minting
5. **Reveal Mechanism**: Placeholder → reveal pattern
6. **Enumeration**: Support walletOfOwner for marketplaces
7. **Whitelist**: Merkle trees for efficient whitelisting
## Marketplace Integration
- OpenSea: ERC-721/1155, metadata standards
- LooksRare: Royalty enforcement
- Rarible: Protocol fees, lazy minting
- Blur: Gas-optimized trading

507
plugins/blockchain-web3/skills/solidity-security/SKILL.md Normal file
View File

@@ -0,0 +1,507 @@
---
name: solidity-security
description: Master smart contract security best practices to prevent common vulnerabilities and implement secure Solidity patterns. Use when writing smart contracts, auditing existing contracts, or implementing security measures for blockchain applications.
---
# Solidity Security
Master smart contract security best practices, vulnerability prevention, and secure Solidity development patterns.
## When to Use This Skill
- Writing secure smart contracts
- Auditing existing contracts for vulnerabilities
- Implementing secure DeFi protocols
- Preventing reentrancy, overflow, and access control issues
- Optimizing gas usage while maintaining security
- Preparing contracts for professional audits
- Understanding common attack vectors
## Critical Vulnerabilities
### 1. Reentrancy
Attacker calls back into your contract before state is updated.
**Vulnerable Code:**
```solidity
// VULNERABLE TO REENTRANCY
contract VulnerableBank {
mapping(address => uint256) public balances;
function withdraw() public {
uint256 amount = balances[msg.sender];
// DANGER: External call before state update
(bool success, ) = msg.sender.call{value: amount}("");
require(success);
balances[msg.sender] = 0; // Too late!
}
}
```
**Secure Pattern (Checks-Effects-Interactions):**
```solidity
contract SecureBank {
mapping(address => uint256) public balances;
function withdraw() public {
uint256 amount = balances[msg.sender];
require(amount > 0, "Insufficient balance");
// EFFECTS: Update state BEFORE external call
balances[msg.sender] = 0;
// INTERACTIONS: External call last
(bool success, ) = msg.sender.call{value: amount}("");
require(success, "Transfer failed");
}
}
```
**Alternative: ReentrancyGuard**
```solidity
import "@openzeppelin/contracts/security/ReentrancyGuard.sol";
contract SecureBank is ReentrancyGuard {
mapping(address => uint256) public balances;
function withdraw() public nonReentrant {
uint256 amount = balances[msg.sender];
require(amount > 0, "Insufficient balance");
balances[msg.sender] = 0;
(bool success, ) = msg.sender.call{value: amount}("");
require(success, "Transfer failed");
}
}
```
### 2. Integer Overflow/Underflow
**Vulnerable Code (Solidity < 0.8.0):**
```solidity
// VULNERABLE
contract VulnerableToken {
mapping(address => uint256) public balances;
function transfer(address to, uint256 amount) public {
// No overflow check - can wrap around
balances[msg.sender] -= amount; // Can underflow!
balances[to] += amount; // Can overflow!
}
}
```
**Secure Pattern (Solidity >= 0.8.0):**
```solidity
// Solidity 0.8+ has built-in overflow/underflow checks
contract SecureToken {
mapping(address => uint256) public balances;
function transfer(address to, uint256 amount) public {
// Automatically reverts on overflow/underflow
balances[msg.sender] -= amount;
balances[to] += amount;
}
}
```
**For Solidity < 0.8.0, use SafeMath:**
```solidity
import "@openzeppelin/contracts/utils/math/SafeMath.sol";
contract SecureToken {
using SafeMath for uint256;
mapping(address => uint256) public balances;
function transfer(address to, uint256 amount) public {
balances[msg.sender] = balances[msg.sender].sub(amount);
balances[to] = balances[to].add(amount);
}
}
```
### 3. Access Control
**Vulnerable Code:**
```solidity
// VULNERABLE: Anyone can call critical functions
contract VulnerableContract {
address public owner;
function withdraw(uint256 amount) public {
// No access control!
payable(msg.sender).transfer(amount);
}
}
```
**Secure Pattern:**
```solidity
import "@openzeppelin/contracts/access/Ownable.sol";
contract SecureContract is Ownable {
function withdraw(uint256 amount) public onlyOwner {
payable(owner()).transfer(amount);
}
}
// Or implement custom role-based access
contract RoleBasedContract {
mapping(address => bool) public admins;
modifier onlyAdmin() {
require(admins[msg.sender], "Not an admin");
_;
}
function criticalFunction() public onlyAdmin {
// Protected function
}
}
```
### 4. Front-Running
**Vulnerable:**
```solidity
// VULNERABLE TO FRONT-RUNNING
contract VulnerableDEX {
function swap(uint256 amount, uint256 minOutput) public {
// Attacker sees this in mempool and front-runs
uint256 output = calculateOutput(amount);
require(output >= minOutput, "Slippage too high");
// Perform swap
}
}
```
**Mitigation:**
```solidity
contract SecureDEX {
mapping(bytes32 => bool) public usedCommitments;
// Step 1: Commit to trade
function commitTrade(bytes32 commitment) public {
usedCommitments[commitment] = true;
}
// Step 2: Reveal trade (next block)
function revealTrade(
uint256 amount,
uint256 minOutput,
bytes32 secret
) public {
bytes32 commitment = keccak256(abi.encodePacked(
msg.sender, amount, minOutput, secret
));
require(usedCommitments[commitment], "Invalid commitment");
// Perform swap
}
}
```
## Security Best Practices
### Checks-Effects-Interactions Pattern
```solidity
contract SecurePattern {
mapping(address => uint256) public balances;
function withdraw(uint256 amount) public {
// 1. CHECKS: Validate conditions
require(amount <= balances[msg.sender], "Insufficient balance");
require(amount > 0, "Amount must be positive");
// 2. EFFECTS: Update state
balances[msg.sender] -= amount;
// 3. INTERACTIONS: External calls last
(bool success, ) = msg.sender.call{value: amount}("");
require(success, "Transfer failed");
}
}
```
### Pull Over Push Pattern
```solidity
// Prefer this (pull)
contract SecurePayment {
mapping(address => uint256) public pendingWithdrawals;
function recordPayment(address recipient, uint256 amount) internal {
pendingWithdrawals[recipient] += amount;
}
function withdraw() public {
uint256 amount = pendingWithdrawals[msg.sender];
require(amount > 0, "Nothing to withdraw");
pendingWithdrawals[msg.sender] = 0;
payable(msg.sender).transfer(amount);
}
}
// Over this (push)
contract RiskyPayment {
function distributePayments(address[] memory recipients, uint256[] memory amounts) public {
for (uint i = 0; i < recipients.length; i++) {
// If any transfer fails, entire batch fails
payable(recipients[i]).transfer(amounts[i]);
}
}
}
```
### Input Validation
```solidity
contract SecureContract {
function transfer(address to, uint256 amount) public {
// Validate inputs
require(to != address(0), "Invalid recipient");
require(to != address(this), "Cannot send to contract");
require(amount > 0, "Amount must be positive");
require(amount <= balances[msg.sender], "Insufficient balance");
// Proceed with transfer
balances[msg.sender] -= amount;
balances[to] += amount;
}
}
```
### Emergency Stop (Circuit Breaker)
```solidity
import "@openzeppelin/contracts/security/Pausable.sol";
contract EmergencyStop is Pausable, Ownable {
function criticalFunction() public whenNotPaused {
// Function logic
}
function emergencyStop() public onlyOwner {
_pause();
}
function resume() public onlyOwner {
_unpause();
}
}
```
## Gas Optimization
### Use `uint256` Instead of Smaller Types
```solidity
// More gas efficient
contract GasEfficient {
uint256 public value; // Optimal
function set(uint256 _value) public {
value = _value;
}
}
// Less efficient
contract GasInefficient {
uint8 public value; // Still uses 256-bit slot
function set(uint8 _value) public {
value = _value; // Extra gas for type conversion
}
}
```
### Pack Storage Variables
```solidity
// Gas efficient (3 variables in 1 slot)
contract PackedStorage {
uint128 public a; // Slot 0
uint64 public b; // Slot 0
uint64 public c; // Slot 0
uint256 public d; // Slot 1
}
// Gas inefficient (each variable in separate slot)
contract UnpackedStorage {
uint256 public a; // Slot 0
uint256 public b; // Slot 1
uint256 public c; // Slot 2
uint256 public d; // Slot 3
}
```
### Use `calldata` Instead of `memory` for Function Arguments
```solidity
contract GasOptimized {
// More gas efficient
function processData(uint256[] calldata data) public pure returns (uint256) {
return data[0];
}
// Less efficient
function processDataMemory(uint256[] memory data) public pure returns (uint256) {
return data[0];
}
}
```
### Use Events for Data Storage (When Appropriate)
```solidity
contract EventStorage {
// Emitting events is cheaper than storage
event DataStored(address indexed user, uint256 indexed id, bytes data);
function storeData(uint256 id, bytes calldata data) public {
emit DataStored(msg.sender, id, data);
// Don't store in contract storage unless needed
}
}
```
## Common Vulnerabilities Checklist
```solidity
// Security Checklist Contract
contract SecurityChecklist {
/**
* [ ] Reentrancy protection (ReentrancyGuard or CEI pattern)
* [ ] Integer overflow/underflow (Solidity 0.8+ or SafeMath)
* [ ] Access control (Ownable, roles, modifiers)
* [ ] Input validation (require statements)
* [ ] Front-running mitigation (commit-reveal if applicable)
* [ ] Gas optimization (packed storage, calldata)
* [ ] Emergency stop mechanism (Pausable)
* [ ] Pull over push pattern for payments
* [ ] No delegatecall to untrusted contracts
* [ ] No tx.origin for authentication (use msg.sender)
* [ ] Proper event emission
* [ ] External calls at end of function
* [ ] Check return values of external calls
* [ ] No hardcoded addresses
* [ ] Upgrade mechanism (if proxy pattern)
*/
}
```
## Testing for Security
```javascript
// Hardhat test example
const { expect } = require("chai");
const { ethers } = require("hardhat");
describe("Security Tests", function () {
it("Should prevent reentrancy attack", async function () {
const [attacker] = await ethers.getSigners();
const VictimBank = await ethers.getContractFactory("SecureBank");
const bank = await VictimBank.deploy();
const Attacker = await ethers.getContractFactory("ReentrancyAttacker");
const attackerContract = await Attacker.deploy(bank.address);
// Deposit funds
await bank.deposit({value: ethers.utils.parseEther("10")});
// Attempt reentrancy attack
await expect(
attackerContract.attack({value: ethers.utils.parseEther("1")})
).to.be.revertedWith("ReentrancyGuard: reentrant call");
});
it("Should prevent integer overflow", async function () {
const Token = await ethers.getContractFactory("SecureToken");
const token = await Token.deploy();
// Attempt overflow
await expect(
token.transfer(attacker.address, ethers.constants.MaxUint256)
).to.be.reverted;
});
it("Should enforce access control", async function () {
const [owner, attacker] = await ethers.getSigners();
const Contract = await ethers.getContractFactory("SecureContract");
const contract = await Contract.deploy();
// Attempt unauthorized withdrawal
await expect(
contract.connect(attacker).withdraw(100)
).to.be.revertedWith("Ownable: caller is not the owner");
});
});
```
## Audit Preparation
```solidity
contract WellDocumentedContract {
/**
* @title Well Documented Contract
* @dev Example of proper documentation for audits
* @notice This contract handles user deposits and withdrawals
*/
/// @notice Mapping of user balances
mapping(address => uint256) public balances;
/**
* @dev Deposits ETH into the contract
* @notice Anyone can deposit funds
*/
function deposit() public payable {
require(msg.value > 0, "Must send ETH");
balances[msg.sender] += msg.value;
}
/**
* @dev Withdraws user's balance
* @notice Follows CEI pattern to prevent reentrancy
* @param amount Amount to withdraw in wei
*/
function withdraw(uint256 amount) public {
// CHECKS
require(amount <= balances[msg.sender], "Insufficient balance");
// EFFECTS
balances[msg.sender] -= amount;
// INTERACTIONS
(bool success, ) = msg.sender.call{value: amount}("");
require(success, "Transfer failed");
}
}
```
## Resources
- **references/reentrancy.md**: Comprehensive reentrancy prevention
- **references/access-control.md**: Role-based access patterns
- **references/overflow-underflow.md**: SafeMath and integer safety
- **references/gas-optimization.md**: Gas saving techniques
- **references/vulnerability-patterns.md**: Common vulnerability catalog
- **assets/solidity-contracts-templates.sol**: Secure contract templates
- **assets/security-checklist.md**: Pre-audit checklist
- **scripts/analyze-contract.sh**: Static analysis tools
## Tools for Security Analysis
- **Slither**: Static analysis tool
- **Mythril**: Security analysis tool
- **Echidna**: Fuzzing tool
- **Manticore**: Symbolic execution
- **Securify**: Automated security scanner
## Common Pitfalls
1. **Using `tx.origin` for Authentication**: Use `msg.sender` instead
2. **Unchecked External Calls**: Always check return values
3. **Delegatecall to Untrusted Contracts**: Can hijack your contract
4. **Floating Pragma**: Pin to specific Solidity version
5. **Missing Events**: Emit events for state changes
6. **Excessive Gas in Loops**: Can hit block gas limit
7. **No Upgrade Path**: Consider proxy patterns if upgrades needed

399
plugins/blockchain-web3/skills/web3-testing/SKILL.md Normal file
View File

@@ -0,0 +1,399 @@
---
name: web3-testing
description: Test smart contracts comprehensively using Hardhat and Foundry with unit tests, integration tests, and mainnet forking. Use when testing Solidity contracts, setting up blockchain test suites, or validating DeFi protocols.
---
# Web3 Smart Contract Testing
Master comprehensive testing strategies for smart contracts using Hardhat, Foundry, and advanced testing patterns.
## When to Use This Skill
- Writing unit tests for smart contracts
- Setting up integration test suites
- Performing gas optimization testing
- Fuzzing for edge cases
- Forking mainnet for realistic testing
- Automating test coverage reporting
- Verifying contracts on Etherscan
## Hardhat Testing Setup
```javascript
// hardhat.config.js
require("@nomicfoundation/hardhat-toolbox");
require("@nomiclabs/hardhat-etherscan");
require("hardhat-gas-reporter");
require("solidity-coverage");
module.exports = {
solidity: {
version: "0.8.19",
settings: {
optimizer: {
enabled: true,
runs: 200
}
}
},
networks: {
hardhat: {
forking: {
url: process.env.MAINNET_RPC_URL,
blockNumber: 15000000
}
},
goerli: {
url: process.env.GOERLI_RPC_URL,
accounts: [process.env.PRIVATE_KEY]
}
},
gasReporter: {
enabled: true,
currency: 'USD',
coinmarketcap: process.env.COINMARKETCAP_API_KEY
},
etherscan: {
apiKey: process.env.ETHERSCAN_API_KEY
}
};
```
## Unit Testing Patterns
```javascript
const { expect } = require("chai");
const { ethers } = require("hardhat");
const { loadFixture, time } = require("@nomicfoundation/hardhat-network-helpers");
describe("Token Contract", function () {
// Fixture for test setup
async function deployTokenFixture() {
const [owner, addr1, addr2] = await ethers.getSigners();
const Token = await ethers.getContractFactory("Token");
const token = await Token.deploy();
return { token, owner, addr1, addr2 };
}
describe("Deployment", function () {
it("Should set the right owner", async function () {
const { token, owner } = await loadFixture(deployTokenFixture);
expect(await token.owner()).to.equal(owner.address);
});
it("Should assign total supply to owner", async function () {
const { token, owner } = await loadFixture(deployTokenFixture);
const ownerBalance = await token.balanceOf(owner.address);
expect(await token.totalSupply()).to.equal(ownerBalance);
});
});
describe("Transactions", function () {
it("Should transfer tokens between accounts", async function () {
const { token, owner, addr1 } = await loadFixture(deployTokenFixture);
await expect(token.transfer(addr1.address, 50))
.to.changeTokenBalances(token, [owner, addr1], [-50, 50]);
});
it("Should fail if sender doesn't have enough tokens", async function () {
const { token, addr1 } = await loadFixture(deployTokenFixture);
const initialBalance = await token.balanceOf(addr1.address);
await expect(
token.connect(addr1).transfer(owner.address, 1)
).to.be.revertedWith("Insufficient balance");
});
it("Should emit Transfer event", async function () {
const { token, owner, addr1 } = await loadFixture(deployTokenFixture);
await expect(token.transfer(addr1.address, 50))
.to.emit(token, "Transfer")
.withArgs(owner.address, addr1.address, 50);
});
});
describe("Time-based tests", function () {
it("Should handle time-locked operations", async function () {
const { token } = await loadFixture(deployTokenFixture);
// Increase time by 1 day
await time.increase(86400);
// Test time-dependent functionality
});
});
describe("Gas optimization", function () {
it("Should use gas efficiently", async function () {
const { token } = await loadFixture(deployTokenFixture);
const tx = await token.transfer(addr1.address, 100);
const receipt = await tx.wait();
expect(receipt.gasUsed).to.be.lessThan(50000);
});
});
});
```
## Foundry Testing (Forge)
```solidity
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
import "forge-std/Test.sol";
import "../src/Token.sol";
contract TokenTest is Test {
Token token;
address owner = address(1);
address user1 = address(2);
address user2 = address(3);
function setUp() public {
vm.prank(owner);
token = new Token();
}
function testInitialSupply() public {
assertEq(token.totalSupply(), 1000000 * 10**18);
}
function testTransfer() public {
vm.prank(owner);
token.transfer(user1, 100);
assertEq(token.balanceOf(user1), 100);
assertEq(token.balanceOf(owner), token.totalSupply() - 100);
}
function testFailTransferInsufficientBalance() public {
vm.prank(user1);
token.transfer(user2, 100); // Should fail
}
function testCannotTransferToZeroAddress() public {
vm.prank(owner);
vm.expectRevert("Invalid recipient");
token.transfer(address(0), 100);
}
// Fuzzing test
function testFuzzTransfer(uint256 amount) public {
vm.assume(amount > 0 && amount <= token.totalSupply());
vm.prank(owner);
token.transfer(user1, amount);
assertEq(token.balanceOf(user1), amount);
}
// Test with cheatcodes
function testDealAndPrank() public {
// Give ETH to address
vm.deal(user1, 10 ether);
// Impersonate address
vm.prank(user1);
// Test functionality
assertEq(user1.balance, 10 ether);
}
// Mainnet fork test
function testForkMainnet() public {
vm.createSelectFork("https://eth-mainnet.alchemyapi.io/v2/...");
// Interact with mainnet contracts
address dai = 0x6B175474E89094C44Da98b954EedeAC495271d0F;
assertEq(IERC20(dai).symbol(), "DAI");
}
}
```
## Advanced Testing Patterns
### Snapshot and Revert
```javascript
describe("Complex State Changes", function () {
let snapshotId;
beforeEach(async function () {
snapshotId = await network.provider.send("evm_snapshot");
});
afterEach(async function () {
await network.provider.send("evm_revert", [snapshotId]);
});
it("Test 1", async function () {
// Make state changes
});
it("Test 2", async function () {
// State reverted, clean slate
});
});
```
### Mainnet Forking
```javascript
describe("Mainnet Fork Tests", function () {
let uniswapRouter, dai, usdc;
before(async function () {
await network.provider.request({
method: "hardhat_reset",
params: [{
forking: {
jsonRpcUrl: process.env.MAINNET_RPC_URL,
blockNumber: 15000000
}
}]
});
// Connect to existing mainnet contracts
uniswapRouter = await ethers.getContractAt(
"IUniswapV2Router",
"0x7a250d5630B4cF539739dF2C5dAcb4c659F2488D"
);
dai = await ethers.getContractAt(
"IERC20",
"0x6B175474E89094C44Da98b954EedeAC495271d0F"
);
});
it("Should swap on Uniswap", async function () {
// Test with real Uniswap contracts
});
});
```
### Impersonating Accounts
```javascript
it("Should impersonate whale account", async function () {
const whaleAddress = "0x...";
await network.provider.request({
method: "hardhat_impersonateAccount",
params: [whaleAddress]
});
const whale = await ethers.getSigner(whaleAddress);
// Use whale's tokens
await dai.connect(whale).transfer(addr1.address, ethers.utils.parseEther("1000"));
});
```
## Gas Optimization Testing
```javascript
const { expect } = require("chai");
describe("Gas Optimization", function () {
it("Compare gas usage between implementations", async function () {
const Implementation1 = await ethers.getContractFactory("OptimizedContract");
const Implementation2 = await ethers.getContractFactory("UnoptimizedContract");
const contract1 = await Implementation1.deploy();
const contract2 = await Implementation2.deploy();
const tx1 = await contract1.doSomething();
const receipt1 = await tx1.wait();
const tx2 = await contract2.doSomething();
const receipt2 = await tx2.wait();
console.log("Optimized gas:", receipt1.gasUsed.toString());
console.log("Unoptimized gas:", receipt2.gasUsed.toString());
expect(receipt1.gasUsed).to.be.lessThan(receipt2.gasUsed);
});
});
```
## Coverage Reporting
```bash
# Generate coverage report
npx hardhat coverage
# Output shows:
# File | % Stmts | % Branch | % Funcs | % Lines |
# -------------------|---------|----------|---------|---------|
# contracts/Token.sol | 100 | 90 | 100 | 95 |
```
## Contract Verification
```javascript
// Verify on Etherscan
await hre.run("verify:verify", {
address: contractAddress,
constructorArguments: [arg1, arg2]
});
```
```bash
# Or via CLI
npx hardhat verify --network mainnet CONTRACT_ADDRESS "Constructor arg1" "arg2"
```
## CI/CD Integration
```yaml
# .github/workflows/test.yml
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
with:
node-version: '16'
- run: npm install
- run: npx hardhat compile
- run: npx hardhat test
- run: npx hardhat coverage
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v2
```
## Resources
- **references/hardhat-setup.md**: Hardhat configuration guide
- **references/foundry-setup.md**: Foundry testing framework
- **references/test-patterns.md**: Testing best practices
- **references/mainnet-forking.md**: Fork testing strategies
- **references/contract-verification.md**: Etherscan verification
- **assets/hardhat-config.js**: Complete Hardhat configuration
- **assets/test-suite.js**: Comprehensive test examples
- **assets/foundry.toml**: Foundry configuration
- **scripts/test-contract.sh**: Automated testing script
## Best Practices
1. **Test Coverage**: Aim for >90% coverage
2. **Edge Cases**: Test boundary conditions
3. **Gas Limits**: Verify functions don't hit block gas limit
4. **Reentrancy**: Test for reentrancy vulnerabilities
5. **Access Control**: Test unauthorized access attempts
6. **Events**: Verify event emissions
7. **Fixtures**: Use fixtures to avoid code duplication
8. **Mainnet Fork**: Test with real contracts
9. **Fuzzing**: Use property-based testing
10. **CI/CD**: Automate testing on every commit

359
plugins/cicd-automation/skills/deployment-pipeline-design/SKILL.md Normal file
View File

@@ -0,0 +1,359 @@
---
name: deployment-pipeline-design
description: Design multi-stage CI/CD pipelines with approval gates, security checks, and deployment orchestration. Use when architecting deployment workflows, setting up continuous delivery, or implementing GitOps practices.
---
# Deployment Pipeline Design
Architecture patterns for multi-stage CI/CD pipelines with approval gates and deployment strategies.
## Purpose
Design robust, secure deployment pipelines that balance speed with safety through proper stage organization and approval workflows.
## When to Use
- Design CI/CD architecture
- Implement deployment gates
- Configure multi-environment pipelines
- Establish deployment best practices
- Implement progressive delivery
## Pipeline Stages
### Standard Pipeline Flow
```
┌─────────┐ ┌──────┐ ┌─────────┐ ┌────────┐ ┌──────────┐
│ Build │ → │ Test │ → │ Staging │ → │ Approve│ → │Production│
└─────────┘ └──────┘ └─────────┘ └────────┘ └──────────┘
```
### Detailed Stage Breakdown
1. **Source** - Code checkout
2. **Build** - Compile, package, containerize
3. **Test** - Unit, integration, security scans
4. **Staging Deploy** - Deploy to staging environment
5. **Integration Tests** - E2E, smoke tests
6. **Approval Gate** - Manual approval required
7. **Production Deploy** - Canary, blue-green, rolling
8. **Verification** - Health checks, monitoring
9. **Rollback** - Automated rollback on failure
## Approval Gate Patterns
### Pattern 1: Manual Approval
```yaml
# GitHub Actions
production-deploy:
needs: staging-deploy
environment:
name: production
url: https://app.example.com
runs-on: ubuntu-latest
steps:
- name: Deploy to production
run: |
# Deployment commands
```
### Pattern 2: Time-Based Approval
```yaml
# GitLab CI
deploy:production:
stage: deploy
script:
- deploy.sh production
environment:
name: production
when: delayed
start_in: 30 minutes
only:
- main
```
### Pattern 3: Multi-Approver
```yaml
# Azure Pipelines
stages:
- stage: Production
dependsOn: Staging
jobs:
- deployment: Deploy
environment:
name: production
resourceType: Kubernetes
strategy:
runOnce:
preDeploy:
steps:
- task: ManualValidation@0
inputs:
notifyUsers: 'team-leads@example.com'
instructions: 'Review staging metrics before approving'
```
**Reference:** See `assets/approval-gate-template.yml`
## Deployment Strategies
### 1. Rolling Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 10
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2
maxUnavailable: 1
```
**Characteristics:**
- Gradual rollout
- Zero downtime
- Easy rollback
- Best for most applications
### 2. Blue-Green Deployment
```yaml
# Blue (current)
kubectl apply -f blue-deployment.yaml
kubectl label service my-app version=blue
# Green (new)
kubectl apply -f green-deployment.yaml
# Test green environment
kubectl label service my-app version=green
# Rollback if needed
kubectl label service my-app version=blue
```
**Characteristics:**
- Instant switchover
- Easy rollback
- Doubles infrastructure cost temporarily
- Good for high-risk deployments
### 3. Canary Deployment
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: my-app
spec:
replicas: 10
strategy:
canary:
steps:
- setWeight: 10
- pause: {duration: 5m}
- setWeight: 25
- pause: {duration: 5m}
- setWeight: 50
- pause: {duration: 5m}
- setWeight: 100
```
**Characteristics:**
- Gradual traffic shift
- Risk mitigation
- Real user testing
- Requires service mesh or similar
### 4. Feature Flags
```python
from flagsmith import Flagsmith
flagsmith = Flagsmith(environment_key="API_KEY")
if flagsmith.has_feature("new_checkout_flow"):
# New code path
process_checkout_v2()
else:
# Existing code path
process_checkout_v1()
```
**Characteristics:**
- Deploy without releasing
- A/B testing
- Instant rollback
- Granular control
## Pipeline Orchestration
### Multi-Stage Pipeline Example
```yaml
name: Production Pipeline
on:
push:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build application
run: make build
- name: Build Docker image
run: docker build -t myapp:${{ github.sha }} .
- name: Push to registry
run: docker push myapp:${{ github.sha }}
test:
needs: build
runs-on: ubuntu-latest
steps:
- name: Unit tests
run: make test
- name: Security scan
run: trivy image myapp:${{ github.sha }}
deploy-staging:
needs: test
runs-on: ubuntu-latest
environment:
name: staging
steps:
- name: Deploy to staging
run: kubectl apply -f k8s/staging/
integration-test:
needs: deploy-staging
runs-on: ubuntu-latest
steps:
- name: Run E2E tests
run: npm run test:e2e
deploy-production:
needs: integration-test
runs-on: ubuntu-latest
environment:
name: production
steps:
- name: Canary deployment
run: |
kubectl apply -f k8s/production/
kubectl argo rollouts promote my-app
verify:
needs: deploy-production
runs-on: ubuntu-latest
steps:
- name: Health check
run: curl -f https://app.example.com/health
- name: Notify team
run: |
curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
-d '{"text":"Production deployment successful!"}'
```
## Pipeline Best Practices
1. **Fail fast** - Run quick tests first
2. **Parallel execution** - Run independent jobs concurrently
3. **Caching** - Cache dependencies between runs
4. **Artifact management** - Store build artifacts
5. **Environment parity** - Keep environments consistent
6. **Secrets management** - Use secret stores (Vault, etc.)
7. **Deployment windows** - Schedule deployments appropriately
8. **Monitoring integration** - Track deployment metrics
9. **Rollback automation** - Auto-rollback on failures
10. **Documentation** - Document pipeline stages
## Rollback Strategies
### Automated Rollback
```yaml
deploy-and-verify:
steps:
- name: Deploy new version
run: kubectl apply -f k8s/
- name: Wait for rollout
run: kubectl rollout status deployment/my-app
- name: Health check
id: health
run: |
for i in {1..10}; do
if curl -sf https://app.example.com/health; then
exit 0
fi
sleep 10
done
exit 1
- name: Rollback on failure
if: failure()
run: kubectl rollout undo deployment/my-app
```
### Manual Rollback
```bash
# List revision history
kubectl rollout history deployment/my-app
# Rollback to previous version
kubectl rollout undo deployment/my-app
# Rollback to specific revision
kubectl rollout undo deployment/my-app --to-revision=3
```
## Monitoring and Metrics
### Key Pipeline Metrics
- **Deployment Frequency** - How often deployments occur
- **Lead Time** - Time from commit to production
- **Change Failure Rate** - Percentage of failed deployments
- **Mean Time to Recovery (MTTR)** - Time to recover from failure
- **Pipeline Success Rate** - Percentage of successful runs
- **Average Pipeline Duration** - Time to complete pipeline
### Integration with Monitoring
```yaml
- name: Post-deployment verification
run: |
# Wait for metrics stabilization
sleep 60
# Check error rate
ERROR_RATE=$(curl -s "$PROMETHEUS_URL/api/v1/query?query=rate(http_errors_total[5m])" | jq '.data.result[0].value[1]')
if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
echo "Error rate too high: $ERROR_RATE"
exit 1
fi
```
## Reference Files
- `references/pipeline-orchestration.md` - Complex pipeline patterns
- `assets/approval-gate-template.yml` - Approval workflow templates
## Related Skills
- `github-actions-templates` - For GitHub Actions implementation
- `gitlab-ci-patterns` - For GitLab CI implementation
- `secrets-management` - For secrets handling

333
plugins/cicd-automation/skills/github-actions-templates/SKILL.md Normal file
View File

@@ -0,0 +1,333 @@
---
name: github-actions-templates
description: Create production-ready GitHub Actions workflows for automated testing, building, and deploying applications. Use when setting up CI/CD with GitHub Actions, automating development workflows, or creating reusable workflow templates.
---
# GitHub Actions Templates
Production-ready GitHub Actions workflow patterns for testing, building, and deploying applications.
## Purpose
Create efficient, secure GitHub Actions workflows for continuous integration and deployment across various tech stacks.
## When to Use
- Automate testing and deployment
- Build Docker images and push to registries
- Deploy to Kubernetes clusters
- Run security scans
- Implement matrix builds for multiple environments
## Common Workflow Patterns
### Pattern 1: Test Workflow
```yaml
name: Test
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main ]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [18.x, 20.x]
steps:
- uses: actions/checkout@v4
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run linter
run: npm run lint
- name: Run tests
run: npm test
- name: Upload coverage
uses: codecov/codecov-action@v3
with:
files: ./coverage/lcov.info
```
**Reference:** See `assets/test-workflow.yml`
### Pattern 2: Build and Push Docker Image
```yaml
name: Build and Push
on:
push:
branches: [ main ]
tags: [ 'v*' ]
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
```
**Reference:** See `assets/deploy-workflow.yml`
### Pattern 3: Deploy to Kubernetes
```yaml
name: Deploy to Kubernetes
on:
push:
branches: [ main ]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Update kubeconfig
run: |
aws eks update-kubeconfig --name production-cluster --region us-west-2
- name: Deploy to Kubernetes
run: |
kubectl apply -f k8s/
kubectl rollout status deployment/my-app -n production
kubectl get services -n production
- name: Verify deployment
run: |
kubectl get pods -n production
kubectl describe deployment my-app -n production
```
### Pattern 4: Matrix Build
```yaml
name: Matrix Build
on: [push, pull_request]
jobs:
build:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ['3.9', '3.10', '3.11', '3.12']
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
run: pytest
```
**Reference:** See `assets/matrix-build.yml`
## Workflow Best Practices
1. **Use specific action versions** (@v4, not @latest)
2. **Cache dependencies** to speed up builds
3. **Use secrets** for sensitive data
4. **Implement status checks** on PRs
5. **Use matrix builds** for multi-version testing
6. **Set appropriate permissions**
7. **Use reusable workflows** for common patterns
8. **Implement approval gates** for production
9. **Add notification steps** for failures
10. **Use self-hosted runners** for sensitive workloads
## Reusable Workflows
```yaml
# .github/workflows/reusable-test.yml
name: Reusable Test Workflow
on:
workflow_call:
inputs:
node-version:
required: true
type: string
secrets:
NPM_TOKEN:
required: true
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ inputs.node-version }}
- run: npm ci
- run: npm test
```
**Use reusable workflow:**
```yaml
jobs:
call-test:
uses: ./.github/workflows/reusable-test.yml
with:
node-version: '20.x'
secrets:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
```
## Security Scanning
```yaml
name: Security Scan
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
scan-type: 'fs'
scan-ref: '.'
format: 'sarif'
output: 'trivy-results.sarif'
- name: Upload Trivy results to GitHub Security
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: 'trivy-results.sarif'
- name: Run Snyk Security Scan
uses: snyk/actions/node@master
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
```
## Deployment with Approvals
```yaml
name: Deploy to Production
on:
push:
tags: [ 'v*' ]
jobs:
deploy:
runs-on: ubuntu-latest
environment:
name: production
url: https://app.example.com
steps:
- uses: actions/checkout@v4
- name: Deploy application
run: |
echo "Deploying to production..."
# Deployment commands here
- name: Notify Slack
if: success()
uses: slackapi/slack-github-action@v1
with:
webhook-url: ${{ secrets.SLACK_WEBHOOK }}
payload: |
{
"text": "Deployment to production completed successfully!"
}
```
## Reference Files
- `assets/test-workflow.yml` - Testing workflow template
- `assets/deploy-workflow.yml` - Deployment workflow template
- `assets/matrix-build.yml` - Matrix build template
- `references/common-workflows.md` - Common workflow patterns
## Related Skills
- `gitlab-ci-patterns` - For GitLab CI workflows
- `deployment-pipeline-design` - For pipeline architecture
- `secrets-management` - For secrets handling

271
plugins/cicd-automation/skills/gitlab-ci-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,271 @@
---
name: gitlab-ci-patterns
description: Build GitLab CI/CD pipelines with multi-stage workflows, caching, and distributed runners for scalable automation. Use when implementing GitLab CI/CD, optimizing pipeline performance, or setting up automated testing and deployment.
---
# GitLab CI Patterns
Comprehensive GitLab CI/CD pipeline patterns for automated testing, building, and deployment.
## Purpose
Create efficient GitLab CI pipelines with proper stage organization, caching, and deployment strategies.
## When to Use
- Automate GitLab-based CI/CD
- Implement multi-stage pipelines
- Configure GitLab Runners
- Deploy to Kubernetes from GitLab
- Implement GitOps workflows
## Basic Pipeline Structure
```yaml
stages:
- build
- test
- deploy
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
build:
stage: build
image: node:20
script:
- npm ci
- npm run build
artifacts:
paths:
- dist/
expire_in: 1 hour
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- node_modules/
test:
stage: test
image: node:20
script:
- npm ci
- npm run lint
- npm test
coverage: '/Lines\s*:\s*(\d+\.\d+)%/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
deploy:
stage: deploy
image: bitnami/kubectl:latest
script:
- kubectl apply -f k8s/
- kubectl rollout status deployment/my-app
only:
- main
environment:
name: production
url: https://app.example.com
```
## Docker Build and Push
```yaml
build-docker:
stage: build
image: docker:24
services:
- docker:24-dind
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker build -t $CI_REGISTRY_IMAGE:latest .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- docker push $CI_REGISTRY_IMAGE:latest
only:
- main
- tags
```
## Multi-Environment Deployment
```yaml
.deploy_template: &deploy_template
image: bitnami/kubectl:latest
before_script:
- kubectl config set-cluster k8s --server="$KUBE_URL" --insecure-skip-tls-verify=true
- kubectl config set-credentials admin --token="$KUBE_TOKEN"
- kubectl config set-context default --cluster=k8s --user=admin
- kubectl config use-context default
deploy:staging:
<<: *deploy_template
stage: deploy
script:
- kubectl apply -f k8s/ -n staging
- kubectl rollout status deployment/my-app -n staging
environment:
name: staging
url: https://staging.example.com
only:
- develop
deploy:production:
<<: *deploy_template
stage: deploy
script:
- kubectl apply -f k8s/ -n production
- kubectl rollout status deployment/my-app -n production
environment:
name: production
url: https://app.example.com
when: manual
only:
- main
```
## Terraform Pipeline
```yaml
stages:
- validate
- plan
- apply
variables:
TF_ROOT: ${CI_PROJECT_DIR}/terraform
TF_VERSION: "1.6.0"
before_script:
- cd ${TF_ROOT}
- terraform --version
validate:
stage: validate
image: hashicorp/terraform:${TF_VERSION}
script:
- terraform init -backend=false
- terraform validate
- terraform fmt -check
plan:
stage: plan
image: hashicorp/terraform:${TF_VERSION}
script:
- terraform init
- terraform plan -out=tfplan
artifacts:
paths:
- ${TF_ROOT}/tfplan
expire_in: 1 day
apply:
stage: apply
image: hashicorp/terraform:${TF_VERSION}
script:
- terraform init
- terraform apply -auto-approve tfplan
dependencies:
- plan
when: manual
only:
- main
```
## Security Scanning
```yaml
include:
- template: Security/SAST.gitlab-ci.yml
- template: Security/Dependency-Scanning.gitlab-ci.yml
- template: Security/Container-Scanning.gitlab-ci.yml
trivy-scan:
stage: test
image: aquasec/trivy:latest
script:
- trivy image --exit-code 1 --severity HIGH,CRITICAL $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
allow_failure: true
```
## Caching Strategies
```yaml
# Cache node_modules
build:
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- node_modules/
policy: pull-push
# Global cache
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .cache/
- vendor/
# Separate cache per job
job1:
cache:
key: job1-cache
paths:
- build/
job2:
cache:
key: job2-cache
paths:
- dist/
```
## Dynamic Child Pipelines
```yaml
generate-pipeline:
stage: build
script:
- python generate_pipeline.py > child-pipeline.yml
artifacts:
paths:
- child-pipeline.yml
trigger-child:
stage: deploy
trigger:
include:
- artifact: child-pipeline.yml
job: generate-pipeline
strategy: depend
```
## Reference Files
- `assets/gitlab-ci.yml.template` - Complete pipeline template
- `references/pipeline-stages.md` - Stage organization patterns
## Best Practices
1. **Use specific image tags** (node:20, not node:latest)
2. **Cache dependencies** appropriately
3. **Use artifacts** for build outputs
4. **Implement manual gates** for production
5. **Use environments** for deployment tracking
6. **Enable merge request pipelines**
7. **Use pipeline schedules** for recurring jobs
8. **Implement security scanning**
9. **Use CI/CD variables** for secrets
10. **Monitor pipeline performance**
## Related Skills
- `github-actions-templates` - For GitHub Actions
- `deployment-pipeline-design` - For architecture
- `secrets-management` - For secrets handling

346
plugins/cicd-automation/skills/secrets-management/SKILL.md Normal file
View File

@@ -0,0 +1,346 @@
---
name: secrets-management
description: Implement secure secrets management for CI/CD pipelines using Vault, AWS Secrets Manager, or native platform solutions. Use when handling sensitive credentials, rotating secrets, or securing CI/CD environments.
---
# Secrets Management
Secure secrets management practices for CI/CD pipelines using Vault, AWS Secrets Manager, and other tools.
## Purpose
Implement secure secrets management in CI/CD pipelines without hardcoding sensitive information.
## When to Use
- Store API keys and credentials
- Manage database passwords
- Handle TLS certificates
- Rotate secrets automatically
- Implement least-privilege access
## Secrets Management Tools
### HashiCorp Vault
- Centralized secrets management
- Dynamic secrets generation
- Secret rotation
- Audit logging
- Fine-grained access control
### AWS Secrets Manager
- AWS-native solution
- Automatic rotation
- Integration with RDS
- CloudFormation support
### Azure Key Vault
- Azure-native solution
- HSM-backed keys
- Certificate management
- RBAC integration
### Google Secret Manager
- GCP-native solution
- Versioning
- IAM integration
## HashiCorp Vault Integration
### Setup Vault
```bash
# Start Vault dev server
vault server -dev
# Set environment
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'
# Enable secrets engine
vault secrets enable -path=secret kv-v2
# Store secret
vault kv put secret/database/config username=admin password=secret
```
### GitHub Actions with Vault
```yaml
name: Deploy with Vault Secrets
on: [push]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Import Secrets from Vault
uses: hashicorp/vault-action@v2
with:
url: https://vault.example.com:8200
token: ${{ secrets.VAULT_TOKEN }}
secrets: |
secret/data/database username | DB_USERNAME ;
secret/data/database password | DB_PASSWORD ;
secret/data/api key | API_KEY
- name: Use secrets
run: |
echo "Connecting to database as $DB_USERNAME"
# Use $DB_PASSWORD, $API_KEY
```
### GitLab CI with Vault
```yaml
deploy:
image: vault:latest
before_script:
- export VAULT_ADDR=https://vault.example.com:8200
- export VAULT_TOKEN=$VAULT_TOKEN
- apk add curl jq
script:
- |
DB_PASSWORD=$(vault kv get -field=password secret/database/config)
API_KEY=$(vault kv get -field=key secret/api/credentials)
echo "Deploying with secrets..."
# Use $DB_PASSWORD, $API_KEY
```
**Reference:** See `references/vault-setup.md`
## AWS Secrets Manager
### Store Secret
```bash
aws secretsmanager create-secret \
--name production/database/password \
--secret-string "super-secret-password"
```
### Retrieve in GitHub Actions
```yaml
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Get secret from AWS
run: |
SECRET=$(aws secretsmanager get-secret-value \
--secret-id production/database/password \
--query SecretString \
--output text)
echo "::add-mask::$SECRET"
echo "DB_PASSWORD=$SECRET" >> $GITHUB_ENV
- name: Use secret
run: |
# Use $DB_PASSWORD
./deploy.sh
```
### Terraform with AWS Secrets Manager
```hcl
data "aws_secretsmanager_secret_version" "db_password" {
secret_id = "production/database/password"
}
resource "aws_db_instance" "main" {
allocated_storage = 100
engine = "postgres"
instance_class = "db.t3.large"
username = "admin"
password = jsondecode(data.aws_secretsmanager_secret_version.db_password.secret_string)["password"]
}
```
## GitHub Secrets
### Organization/Repository Secrets
```yaml
- name: Use GitHub secret
run: |
echo "API Key: ${{ secrets.API_KEY }}"
echo "Database URL: ${{ secrets.DATABASE_URL }}"
```
### Environment Secrets
```yaml
deploy:
runs-on: ubuntu-latest
environment: production
steps:
- name: Deploy
run: |
echo "Deploying with ${{ secrets.PROD_API_KEY }}"
```
**Reference:** See `references/github-secrets.md`
## GitLab CI/CD Variables
### Project Variables
```yaml
deploy:
script:
- echo "Deploying with $API_KEY"
- echo "Database: $DATABASE_URL"
```
### Protected and Masked Variables
- Protected: Only available in protected branches
- Masked: Hidden in job logs
- File type: Stored as file
## Best Practices
1. **Never commit secrets** to Git
2. **Use different secrets** per environment
3. **Rotate secrets regularly**
4. **Implement least-privilege access**
5. **Enable audit logging**
6. **Use secret scanning** (GitGuardian, TruffleHog)
7. **Mask secrets in logs**
8. **Encrypt secrets at rest**
9. **Use short-lived tokens** when possible
10. **Document secret requirements**
## Secret Rotation
### Automated Rotation with AWS
```python
import boto3
import json
def lambda_handler(event, context):
client = boto3.client('secretsmanager')
# Get current secret
response = client.get_secret_value(SecretId='my-secret')
current_secret = json.loads(response['SecretString'])
# Generate new password
new_password = generate_strong_password()
# Update database password
update_database_password(new_password)
# Update secret
client.put_secret_value(
SecretId='my-secret',
SecretString=json.dumps({
'username': current_secret['username'],
'password': new_password
})
)
return {'statusCode': 200}
```
### Manual Rotation Process
1. Generate new secret
2. Update secret in secret store
3. Update applications to use new secret
4. Verify functionality
5. Revoke old secret
## External Secrets Operator
### Kubernetes Integration
```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: production
spec:
provider:
vault:
server: "https://vault.example.com:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "production"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-credentials
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: database-credentials
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: database/config
property: username
- secretKey: password
remoteRef:
key: database/config
property: password
```
## Secret Scanning
### Pre-commit Hook
```bash
#!/bin/bash
# .git/hooks/pre-commit
# Check for secrets with TruffleHog
docker run --rm -v "$(pwd):/repo" \
trufflesecurity/trufflehog:latest \
filesystem --directory=/repo
if [ $? -ne 0 ]; then
echo "❌ Secret detected! Commit blocked."
exit 1
fi
```
### CI/CD Secret Scanning
```yaml
secret-scan:
stage: security
image: trufflesecurity/trufflehog:latest
script:
- trufflehog filesystem .
allow_failure: false
```
## Reference Files
- `references/vault-setup.md` - HashiCorp Vault configuration
- `references/github-secrets.md` - GitHub Secrets best practices
## Related Skills
- `github-actions-templates` - For GitHub Actions integration
- `gitlab-ci-patterns` - For GitLab CI integration
- `deployment-pipeline-design` - For pipeline architecture

274
plugins/cloud-infrastructure/skills/cost-optimization/SKILL.md Normal file
View File

@@ -0,0 +1,274 @@
---
name: cost-optimization
description: Optimize cloud costs through resource rightsizing, tagging strategies, reserved instances, and spending analysis. Use when reducing cloud expenses, analyzing infrastructure costs, or implementing cost governance policies.
---
# Cloud Cost Optimization
Strategies and patterns for optimizing cloud costs across AWS, Azure, and GCP.
## Purpose
Implement systematic cost optimization strategies to reduce cloud spending while maintaining performance and reliability.
## When to Use
- Reduce cloud spending
- Right-size resources
- Implement cost governance
- Optimize multi-cloud costs
- Meet budget constraints
## Cost Optimization Framework
### 1. Visibility
- Implement cost allocation tags
- Use cloud cost management tools
- Set up budget alerts
- Create cost dashboards
### 2. Right-Sizing
- Analyze resource utilization
- Downsize over-provisioned resources
- Use auto-scaling
- Remove idle resources
### 3. Pricing Models
- Use reserved capacity
- Leverage spot/preemptible instances
- Implement savings plans
- Use committed use discounts
### 4. Architecture Optimization
- Use managed services
- Implement caching
- Optimize data transfer
- Use lifecycle policies
## AWS Cost Optimization
### Reserved Instances
```
Savings: 30-72% vs On-Demand
Term: 1 or 3 years
Payment: All/Partial/No upfront
Flexibility: Standard or Convertible
```
### Savings Plans
```
Compute Savings Plans: 66% savings
EC2 Instance Savings Plans: 72% savings
Applies to: EC2, Fargate, Lambda
Flexible across: Instance families, regions, OS
```
### Spot Instances
```
Savings: Up to 90% vs On-Demand
Best for: Batch jobs, CI/CD, stateless workloads
Risk: 2-minute interruption notice
Strategy: Mix with On-Demand for resilience
```
### S3 Cost Optimization
```hcl
resource "aws_s3_bucket_lifecycle_configuration" "example" {
bucket = aws_s3_bucket.example.id
rule {
id = "transition-to-ia"
status = "Enabled"
transition {
days = 30
storage_class = "STANDARD_IA"
}
transition {
days = 90
storage_class = "GLACIER"
}
expiration {
days = 365
}
}
}
```
## Azure Cost Optimization
### Reserved VM Instances
- 1 or 3 year terms
- Up to 72% savings
- Flexible sizing
- Exchangeable
### Azure Hybrid Benefit
- Use existing Windows Server licenses
- Up to 80% savings with RI
- Available for Windows and SQL Server
### Azure Advisor Recommendations
- Right-size VMs
- Delete unused resources
- Use reserved capacity
- Optimize storage
## GCP Cost Optimization
### Committed Use Discounts
- 1 or 3 year commitment
- Up to 57% savings
- Applies to vCPUs and memory
- Resource-based or spend-based
### Sustained Use Discounts
- Automatic discounts
- Up to 30% for running instances
- No commitment required
- Applies to Compute Engine, GKE
### Preemptible VMs
- Up to 80% savings
- 24-hour maximum runtime
- Best for batch workloads
## Tagging Strategy
### AWS Tagging
```hcl
locals {
common_tags = {
Environment = "production"
Project = "my-project"
CostCenter = "engineering"
Owner = "team@example.com"
ManagedBy = "terraform"
}
}
resource "aws_instance" "example" {
ami = "ami-12345678"
instance_type = "t3.medium"
tags = merge(
local.common_tags,
{
Name = "web-server"
}
)
}
```
**Reference:** See `references/tagging-standards.md`
## Cost Monitoring
### Budget Alerts
```hcl
# AWS Budget
resource "aws_budgets_budget" "monthly" {
name = "monthly-budget"
budget_type = "COST"
limit_amount = "1000"
limit_unit = "USD"
time_period_start = "2024-01-01_00:00"
time_unit = "MONTHLY"
notification {
comparison_operator = "GREATER_THAN"
threshold = 80
threshold_type = "PERCENTAGE"
notification_type = "ACTUAL"
subscriber_email_addresses = ["team@example.com"]
}
}
```
### Cost Anomaly Detection
- AWS Cost Anomaly Detection
- Azure Cost Management alerts
- GCP Budget alerts
## Architecture Patterns
### Pattern 1: Serverless First
- Use Lambda/Functions for event-driven
- Pay only for execution time
- Auto-scaling included
- No idle costs
### Pattern 2: Right-Sized Databases
```
Development: t3.small RDS
Staging: t3.large RDS
Production: r6g.2xlarge RDS with read replicas
```
### Pattern 3: Multi-Tier Storage
```
Hot data: S3 Standard
Warm data: S3 Standard-IA (30 days)
Cold data: S3 Glacier (90 days)
Archive: S3 Deep Archive (365 days)
```
### Pattern 4: Auto-Scaling
```hcl
resource "aws_autoscaling_policy" "scale_up" {
name = "scale-up"
scaling_adjustment = 2
adjustment_type = "ChangeInCapacity"
cooldown = 300
autoscaling_group_name = aws_autoscaling_group.main.name
}
resource "aws_cloudwatch_metric_alarm" "cpu_high" {
alarm_name = "cpu-high"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = "2"
metric_name = "CPUUtilization"
namespace = "AWS/EC2"
period = "60"
statistic = "Average"
threshold = "80"
alarm_actions = [aws_autoscaling_policy.scale_up.arn]
}
```
## Cost Optimization Checklist
- [ ] Implement cost allocation tags
- [ ] Delete unused resources (EBS, EIPs, snapshots)
- [ ] Right-size instances based on utilization
- [ ] Use reserved capacity for steady workloads
- [ ] Implement auto-scaling
- [ ] Optimize storage classes
- [ ] Use lifecycle policies
- [ ] Enable cost anomaly detection
- [ ] Set budget alerts
- [ ] Review costs weekly
- [ ] Use spot/preemptible instances
- [ ] Optimize data transfer costs
- [ ] Implement caching layers
- [ ] Use managed services
- [ ] Monitor and optimize continuously
## Tools
- **AWS:** Cost Explorer, Cost Anomaly Detection, Compute Optimizer
- **Azure:** Cost Management, Advisor
- **GCP:** Cost Management, Recommender
- **Multi-cloud:** CloudHealth, Cloudability, Kubecost
## Reference Files
- `references/tagging-standards.md` - Tagging conventions
- `assets/cost-analysis-template.xlsx` - Cost analysis spreadsheet
## Related Skills
- `terraform-module-library` - For resource provisioning
- `multi-cloud-architecture` - For cloud selection

226
plugins/cloud-infrastructure/skills/hybrid-cloud-networking/SKILL.md Normal file
View File

@@ -0,0 +1,226 @@
---
name: hybrid-cloud-networking
description: Configure secure, high-performance connectivity between on-premises infrastructure and cloud platforms using VPN and dedicated connections. Use when building hybrid cloud architectures, connecting data centers to cloud, or implementing secure cross-premises networking.
---
# Hybrid Cloud Networking
Configure secure, high-performance connectivity between on-premises and cloud environments using VPN, Direct Connect, and ExpressRoute.
## Purpose
Establish secure, reliable network connectivity between on-premises data centers and cloud providers (AWS, Azure, GCP).
## When to Use
- Connect on-premises to cloud
- Extend datacenter to cloud
- Implement hybrid active-active setups
- Meet compliance requirements
- Migrate to cloud gradually
## Connection Options
### AWS Connectivity
#### 1. Site-to-Site VPN
- IPSec VPN over internet
- Up to 1.25 Gbps per tunnel
- Cost-effective for moderate bandwidth
- Higher latency, internet-dependent
```hcl
resource "aws_vpn_gateway" "main" {
vpc_id = aws_vpc.main.id
tags = {
Name = "main-vpn-gateway"
}
}
resource "aws_customer_gateway" "main" {
bgp_asn = 65000
ip_address = "203.0.113.1"
type = "ipsec.1"
}
resource "aws_vpn_connection" "main" {
vpn_gateway_id = aws_vpn_gateway.main.id
customer_gateway_id = aws_customer_gateway.main.id
type = "ipsec.1"
static_routes_only = false
}
```
#### 2. AWS Direct Connect
- Dedicated network connection
- 1 Gbps to 100 Gbps
- Lower latency, consistent bandwidth
- More expensive, setup time required
**Reference:** See `references/direct-connect.md`
### Azure Connectivity
#### 1. Site-to-Site VPN
```hcl
resource "azurerm_virtual_network_gateway" "vpn" {
name = "vpn-gateway"
location = azurerm_resource_group.main.location
resource_group_name = azurerm_resource_group.main.name
type = "Vpn"
vpn_type = "RouteBased"
sku = "VpnGw1"
ip_configuration {
name = "vnetGatewayConfig"
public_ip_address_id = azurerm_public_ip.vpn.id
private_ip_address_allocation = "Dynamic"
subnet_id = azurerm_subnet.gateway.id
}
}
```
#### 2. Azure ExpressRoute
- Private connection via connectivity provider
- Up to 100 Gbps
- Low latency, high reliability
- Premium for global connectivity
### GCP Connectivity
#### 1. Cloud VPN
- IPSec VPN (Classic or HA VPN)
- HA VPN: 99.99% SLA
- Up to 3 Gbps per tunnel
#### 2. Cloud Interconnect
- Dedicated (10 Gbps, 100 Gbps)
- Partner (50 Mbps to 50 Gbps)
- Lower latency than VPN
## Hybrid Network Patterns
### Pattern 1: Hub-and-Spoke
```
On-Premises Datacenter
↓
VPN/Direct Connect
↓
Transit Gateway (AWS) / vWAN (Azure)
↓
├─ Production VPC/VNet
├─ Staging VPC/VNet
└─ Development VPC/VNet
```
### Pattern 2: Multi-Region Hybrid
```
On-Premises
├─ Direct Connect → us-east-1
└─ Direct Connect → us-west-2
↓
Cross-Region Peering
```
### Pattern 3: Multi-Cloud Hybrid
```
On-Premises Datacenter
├─ Direct Connect → AWS
├─ ExpressRoute → Azure
└─ Interconnect → GCP
```
## Routing Configuration
### BGP Configuration
```
On-Premises Router:
- AS Number: 65000
- Advertise: 10.0.0.0/8
Cloud Router:
- AS Number: 64512 (AWS), 65515 (Azure)
- Advertise: Cloud VPC/VNet CIDRs
```
### Route Propagation
- Enable route propagation on route tables
- Use BGP for dynamic routing
- Implement route filtering
- Monitor route advertisements
## Security Best Practices
1. **Use private connectivity** (Direct Connect/ExpressRoute)
2. **Implement encryption** for VPN tunnels
3. **Use VPC endpoints** to avoid internet routing
4. **Configure network ACLs** and security groups
5. **Enable VPC Flow Logs** for monitoring
6. **Implement DDoS protection**
7. **Use PrivateLink/Private Endpoints**
8. **Monitor connections** with CloudWatch/Monitor
9. **Implement redundancy** (dual tunnels)
10. **Regular security audits**
## High Availability
### Dual VPN Tunnels
```hcl
resource "aws_vpn_connection" "primary" {
vpn_gateway_id = aws_vpn_gateway.main.id
customer_gateway_id = aws_customer_gateway.primary.id
type = "ipsec.1"
}
resource "aws_vpn_connection" "secondary" {
vpn_gateway_id = aws_vpn_gateway.main.id
customer_gateway_id = aws_customer_gateway.secondary.id
type = "ipsec.1"
}
```
### Active-Active Configuration
- Multiple connections from different locations
- BGP for automatic failover
- Equal-cost multi-path (ECMP) routing
- Monitor health of all connections
## Monitoring and Troubleshooting
### Key Metrics
- Tunnel status (up/down)
- Bytes in/out
- Packet loss
- Latency
- BGP session status
### Troubleshooting
```bash
# AWS VPN
aws ec2 describe-vpn-connections
aws ec2 get-vpn-connection-telemetry
# Azure VPN
az network vpn-connection show
az network vpn-connection show-device-config-script
```
## Cost Optimization
1. **Right-size connections** based on traffic
2. **Use VPN for low-bandwidth** workloads
3. **Consolidate traffic** through fewer connections
4. **Minimize data transfer** costs
5. **Use Direct Connect** for high bandwidth
6. **Implement caching** to reduce traffic
## Reference Files
- `references/vpn-setup.md` - VPN configuration guide
- `references/direct-connect.md` - Direct Connect setup
## Related Skills
- `multi-cloud-architecture` - For architecture decisions
- `terraform-module-library` - For IaC implementation

177
plugins/cloud-infrastructure/skills/multi-cloud-architecture/SKILL.md Normal file
View File

@@ -0,0 +1,177 @@
---
name: multi-cloud-architecture
description: Design multi-cloud architectures using a decision framework to select and integrate services across AWS, Azure, and GCP. Use when building multi-cloud systems, avoiding vendor lock-in, or leveraging best-of-breed services from multiple providers.
---
# Multi-Cloud Architecture
Decision framework and patterns for architecting applications across AWS, Azure, and GCP.
## Purpose
Design cloud-agnostic architectures and make informed decisions about service selection across cloud providers.
## When to Use
- Design multi-cloud strategies
- Migrate between cloud providers
- Select cloud services for specific workloads
- Implement cloud-agnostic architectures
- Optimize costs across providers
## Cloud Service Comparison
### Compute Services
| AWS | Azure | GCP | Use Case |
|-----|-------|-----|----------|
| EC2 | Virtual Machines | Compute Engine | IaaS VMs |
| ECS | Container Instances | Cloud Run | Containers |
| EKS | AKS | GKE | Kubernetes |
| Lambda | Functions | Cloud Functions | Serverless |
| Fargate | Container Apps | Cloud Run | Managed containers |
### Storage Services
| AWS | Azure | GCP | Use Case |
|-----|-------|-----|----------|
| S3 | Blob Storage | Cloud Storage | Object storage |
| EBS | Managed Disks | Persistent Disk | Block storage |
| EFS | Azure Files | Filestore | File storage |
| Glacier | Archive Storage | Archive Storage | Cold storage |
### Database Services
| AWS | Azure | GCP | Use Case |
|-----|-------|-----|----------|
| RDS | SQL Database | Cloud SQL | Managed SQL |
| DynamoDB | Cosmos DB | Firestore | NoSQL |
| Aurora | PostgreSQL/MySQL | Cloud Spanner | Distributed SQL |
| ElastiCache | Cache for Redis | Memorystore | Caching |
**Reference:** See `references/service-comparison.md` for complete comparison
## Multi-Cloud Patterns
### Pattern 1: Single Provider with DR
- Primary workload in one cloud
- Disaster recovery in another
- Database replication across clouds
- Automated failover
### Pattern 2: Best-of-Breed
- Use best service from each provider
- AI/ML on GCP
- Enterprise apps on Azure
- General compute on AWS
### Pattern 3: Geographic Distribution
- Serve users from nearest cloud region
- Data sovereignty compliance
- Global load balancing
- Regional failover
### Pattern 4: Cloud-Agnostic Abstraction
- Kubernetes for compute
- PostgreSQL for database
- S3-compatible storage (MinIO)
- Open source tools
## Cloud-Agnostic Architecture
### Use Cloud-Native Alternatives
- **Compute:** Kubernetes (EKS/AKS/GKE)
- **Database:** PostgreSQL/MySQL (RDS/SQL Database/Cloud SQL)
- **Message Queue:** Apache Kafka (MSK/Event Hubs/Confluent)
- **Cache:** Redis (ElastiCache/Azure Cache/Memorystore)
- **Object Storage:** S3-compatible API
- **Monitoring:** Prometheus/Grafana
- **Service Mesh:** Istio/Linkerd
### Abstraction Layers
```
Application Layer
↓
Infrastructure Abstraction (Terraform)
↓
Cloud Provider APIs
↓
AWS / Azure / GCP
```
## Cost Comparison
### Compute Pricing Factors
- **AWS:** On-demand, Reserved, Spot, Savings Plans
- **Azure:** Pay-as-you-go, Reserved, Spot
- **GCP:** On-demand, Committed use, Preemptible
### Cost Optimization Strategies
1. Use reserved/committed capacity (30-70% savings)
2. Leverage spot/preemptible instances
3. Right-size resources
4. Use serverless for variable workloads
5. Optimize data transfer costs
6. Implement lifecycle policies
7. Use cost allocation tags
8. Monitor with cloud cost tools
**Reference:** See `references/multi-cloud-patterns.md`
## Migration Strategy
### Phase 1: Assessment
- Inventory current infrastructure
- Identify dependencies
- Assess cloud compatibility
- Estimate costs
### Phase 2: Pilot
- Select pilot workload
- Implement in target cloud
- Test thoroughly
- Document learnings
### Phase 3: Migration
- Migrate workloads incrementally
- Maintain dual-run period
- Monitor performance
- Validate functionality
### Phase 4: Optimization
- Right-size resources
- Implement cloud-native services
- Optimize costs
- Enhance security
## Best Practices
1. **Use infrastructure as code** (Terraform/OpenTofu)
2. **Implement CI/CD pipelines** for deployments
3. **Design for failure** across clouds
4. **Use managed services** when possible
5. **Implement comprehensive monitoring**
6. **Automate cost optimization**
7. **Follow security best practices**
8. **Document cloud-specific configurations**
9. **Test disaster recovery** procedures
10. **Train teams** on multiple clouds
## Reference Files
- `references/service-comparison.md` - Complete service comparison
- `references/multi-cloud-patterns.md` - Architecture patterns
## Related Skills
- `terraform-module-library` - For IaC implementation
- `cost-optimization` - For cost management
- `hybrid-cloud-networking` - For connectivity

249
plugins/cloud-infrastructure/skills/terraform-module-library/SKILL.md Normal file
View File

@@ -0,0 +1,249 @@
---
name: terraform-module-library
description: Build reusable Terraform modules for AWS, Azure, and GCP infrastructure following infrastructure-as-code best practices. Use when creating infrastructure modules, standardizing cloud provisioning, or implementing reusable IaC components.
---
# Terraform Module Library
Production-ready Terraform module patterns for AWS, Azure, and GCP infrastructure.
## Purpose
Create reusable, well-tested Terraform modules for common cloud infrastructure patterns across multiple cloud providers.
## When to Use
- Build reusable infrastructure components
- Standardize cloud resource provisioning
- Implement infrastructure as code best practices
- Create multi-cloud compatible modules
- Establish organizational Terraform standards
## Module Structure
```
terraform-modules/
├── aws/
│ ├── vpc/
│ ├── eks/
│ ├── rds/
│ └── s3/
├── azure/
│ ├── vnet/
│ ├── aks/
│ └── storage/
└── gcp/
├── vpc/
├── gke/
└── cloud-sql/
```
## Standard Module Pattern
```
module-name/
├── main.tf # Main resources
├── variables.tf # Input variables
├── outputs.tf # Output values
├── versions.tf # Provider versions
├── README.md # Documentation
├── examples/ # Usage examples
│ └── complete/
│ ├── main.tf
│ └── variables.tf
└── tests/ # Terratest files
└── module_test.go
```
## AWS VPC Module Example
**main.tf:**
```hcl
resource "aws_vpc" "main" {
cidr_block = var.cidr_block
enable_dns_hostnames = var.enable_dns_hostnames
enable_dns_support = var.enable_dns_support
tags = merge(
{
Name = var.name
},
var.tags
)
}
resource "aws_subnet" "private" {
count = length(var.private_subnet_cidrs)
vpc_id = aws_vpc.main.id
cidr_block = var.private_subnet_cidrs[count.index]
availability_zone = var.availability_zones[count.index]
tags = merge(
{
Name = "${var.name}-private-${count.index + 1}"
Tier = "private"
},
var.tags
)
}
resource "aws_internet_gateway" "main" {
count = var.create_internet_gateway ? 1 : 0
vpc_id = aws_vpc.main.id
tags = merge(
{
Name = "${var.name}-igw"
},
var.tags
)
}
```
**variables.tf:**
```hcl
variable "name" {
description = "Name of the VPC"
type = string
}
variable "cidr_block" {
description = "CIDR block for VPC"
type = string
validation {
condition = can(regex("^([0-9]{1,3}\\.){3}[0-9]{1,3}/[0-9]{1,2}$", var.cidr_block))
error_message = "CIDR block must be valid IPv4 CIDR notation."
}
}
variable "availability_zones" {
description = "List of availability zones"
type = list(string)
}
variable "private_subnet_cidrs" {
description = "CIDR blocks for private subnets"
type = list(string)
default = []
}
variable "enable_dns_hostnames" {
description = "Enable DNS hostnames in VPC"
type = bool
default = true
}
variable "tags" {
description = "Additional tags"
type = map(string)
default = {}
}
```
**outputs.tf:**
```hcl
output "vpc_id" {
description = "ID of the VPC"
value = aws_vpc.main.id
}
output "private_subnet_ids" {
description = "IDs of private subnets"
value = aws_subnet.private[*].id
}
output "vpc_cidr_block" {
description = "CIDR block of VPC"
value = aws_vpc.main.cidr_block
}
```
## Best Practices
1. **Use semantic versioning** for modules
2. **Document all variables** with descriptions
3. **Provide examples** in examples/ directory
4. **Use validation blocks** for input validation
5. **Output important attributes** for module composition
6. **Pin provider versions** in versions.tf
7. **Use locals** for computed values
8. **Implement conditional resources** with count/for_each
9. **Test modules** with Terratest
10. **Tag all resources** consistently
## Module Composition
```hcl
module "vpc" {
source = "../../modules/aws/vpc"
name = "production"
cidr_block = "10.0.0.0/16"
availability_zones = ["us-west-2a", "us-west-2b", "us-west-2c"]
private_subnet_cidrs = [
"10.0.1.0/24",
"10.0.2.0/24",
"10.0.3.0/24"
]
tags = {
Environment = "production"
ManagedBy = "terraform"
}
}
module "rds" {
source = "../../modules/aws/rds"
identifier = "production-db"
engine = "postgres"
engine_version = "15.3"
instance_class = "db.t3.large"
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnet_ids
tags = {
Environment = "production"
}
}
```
## Reference Files
- `assets/vpc-module/` - Complete VPC module example
- `assets/rds-module/` - RDS module example
- `references/aws-modules.md` - AWS module patterns
- `references/azure-modules.md` - Azure module patterns
- `references/gcp-modules.md` - GCP module patterns
## Testing
```go
// tests/vpc_test.go
package test
import (
"testing"
"github.com/gruntwork-io/terratest/modules/terraform"
"github.com/stretchr/testify/assert"
)
func TestVPCModule(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/complete",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
vpcID := terraform.Output(t, terraformOptions, "vpc_id")
assert.NotEmpty(t, vpcID)
}
```
## Related Skills
- `multi-cloud-architecture` - For architectural decisions
- `cost-optimization` - For cost-effective designs

63
plugins/cloud-infrastructure/skills/terraform-module-library/references/aws-modules.md Normal file
View File

@@ -0,0 +1,63 @@
# AWS Terraform Module Patterns
## VPC Module
- VPC with public/private subnets
- Internet Gateway and NAT Gateways
- Route tables and associations
- Network ACLs
- VPC Flow Logs
## EKS Module
- EKS cluster with managed node groups
- IRSA (IAM Roles for Service Accounts)
- Cluster autoscaler
- VPC CNI configuration
- Cluster logging
## RDS Module
- RDS instance or cluster
- Automated backups
- Read replicas
- Parameter groups
- Subnet groups
- Security groups
## S3 Module
- S3 bucket with versioning
- Encryption at rest
- Bucket policies
- Lifecycle rules
- Replication configuration
## ALB Module
- Application Load Balancer
- Target groups
- Listener rules
- SSL/TLS certificates
- Access logs
## Lambda Module
- Lambda function
- IAM execution role
- CloudWatch Logs
- Environment variables
- VPC configuration (optional)
## Security Group Module
- Reusable security group rules
- Ingress/egress rules
- Dynamic rule creation
- Rule descriptions
## Best Practices
1. Use AWS provider version ~> 5.0
2. Enable encryption by default
3. Use least-privilege IAM
4. Tag all resources consistently
5. Enable logging and monitoring
6. Use KMS for encryption
7. Implement backup strategies
8. Use PrivateLink when possible
9. Enable GuardDuty/SecurityHub
10. Follow AWS Well-Architected Framework

410
plugins/framework-migration/skills/angular-migration/SKILL.md Normal file
View File

@@ -0,0 +1,410 @@
---
name: angular-migration
description: Migrate from AngularJS to Angular using hybrid mode, incremental component rewriting, and dependency injection updates. Use when upgrading AngularJS applications, planning framework migrations, or modernizing legacy Angular code.
---
# Angular Migration
Master AngularJS to Angular migration, including hybrid apps, component conversion, dependency injection changes, and routing migration.
## When to Use This Skill
- Migrating AngularJS (1.x) applications to Angular (2+)
- Running hybrid AngularJS/Angular applications
- Converting directives to components
- Modernizing dependency injection
- Migrating routing systems
- Updating to latest Angular versions
- Implementing Angular best practices
## Migration Strategies
### 1. Big Bang (Complete Rewrite)
- Rewrite entire app in Angular
- Parallel development
- Switch over at once
- **Best for:** Small apps, green field projects
### 2. Incremental (Hybrid Approach)
- Run AngularJS and Angular side-by-side
- Migrate feature by feature
- ngUpgrade for interop
- **Best for:** Large apps, continuous delivery
### 3. Vertical Slice
- Migrate one feature completely
- New features in Angular, maintain old in AngularJS
- Gradually replace
- **Best for:** Medium apps, distinct features
## Hybrid App Setup
```typescript
// main.ts - Bootstrap hybrid app
import { platformBrowserDynamic } from '@angular/platform-browser-dynamic';
import { UpgradeModule } from '@angular/upgrade/static';
import { AppModule } from './app/app.module';
platformBrowserDynamic()
.bootstrapModule(AppModule)
.then(platformRef => {
const upgrade = platformRef.injector.get(UpgradeModule);
// Bootstrap AngularJS
upgrade.bootstrap(document.body, ['myAngularJSApp'], { strictDi: true });
});
```
```typescript
// app.module.ts
import { NgModule } from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';
import { UpgradeModule } from '@angular/upgrade/static';
@NgModule({
imports: [
BrowserModule,
UpgradeModule
]
})
export class AppModule {
constructor(private upgrade: UpgradeModule) {}
ngDoBootstrap() {
// Bootstrapped manually in main.ts
}
}
```
## Component Migration
### AngularJS Controller → Angular Component
```javascript
// Before: AngularJS controller
angular.module('myApp').controller('UserController', function($scope, UserService) {
$scope.user = {};
$scope.loadUser = function(id) {
UserService.getUser(id).then(function(user) {
$scope.user = user;
});
};
$scope.saveUser = function() {
UserService.saveUser($scope.user);
};
});
```
```typescript
// After: Angular component
import { Component, OnInit } from '@angular/core';
import { UserService } from './user.service';
@Component({
selector: 'app-user',
template: `
<div>
<h2>{{ user.name }}</h2>
<button (click)="saveUser()">Save</button>
</div>
`
})
export class UserComponent implements OnInit {
user: any = {};
constructor(private userService: UserService) {}
ngOnInit() {
this.loadUser(1);
}
loadUser(id: number) {
this.userService.getUser(id).subscribe(user => {
this.user = user;
});
}
saveUser() {
this.userService.saveUser(this.user);
}
}
```
### AngularJS Directive → Angular Component
```javascript
// Before: AngularJS directive
angular.module('myApp').directive('userCard', function() {
return {
restrict: 'E',
scope: {
user: '=',
onDelete: '&'
},
template: `
<div class="card">
<h3>{{ user.name }}</h3>
<button ng-click="onDelete()">Delete</button>
</div>
`
};
});
```
```typescript
// After: Angular component
import { Component, Input, Output, EventEmitter } from '@angular/core';
@Component({
selector: 'app-user-card',
template: `
<div class="card">
<h3>{{ user.name }}</h3>
<button (click)="delete.emit()">Delete</button>
</div>
`
})
export class UserCardComponent {
@Input() user: any;
@Output() delete = new EventEmitter<void>();
}
// Usage: <app-user-card [user]="user" (delete)="handleDelete()"></app-user-card>
```
## Service Migration
```javascript
// Before: AngularJS service
angular.module('myApp').factory('UserService', function($http) {
return {
getUser: function(id) {
return $http.get('/api/users/' + id);
},
saveUser: function(user) {
return $http.post('/api/users', user);
}
};
});
```
```typescript
// After: Angular service
import { Injectable } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { Observable } from 'rxjs';
@Injectable({
providedIn: 'root'
})
export class UserService {
constructor(private http: HttpClient) {}
getUser(id: number): Observable<any> {
return this.http.get(`/api/users/${id}`);
}
saveUser(user: any): Observable<any> {
return this.http.post('/api/users', user);
}
}
```
## Dependency Injection Changes
### Downgrading Angular → AngularJS
```typescript
// Angular service
import { Injectable } from '@angular/core';
@Injectable({ providedIn: 'root' })
export class NewService {
getData() {
return 'data from Angular';
}
}
// Make available to AngularJS
import { downgradeInjectable } from '@angular/upgrade/static';
angular.module('myApp')
.factory('newService', downgradeInjectable(NewService));
// Use in AngularJS
angular.module('myApp').controller('OldController', function(newService) {
console.log(newService.getData());
});
```
### Upgrading AngularJS → Angular
```typescript
// AngularJS service
angular.module('myApp').factory('oldService', function() {
return {
getData: function() {
return 'data from AngularJS';
}
};
});
// Make available to Angular
import { InjectionToken } from '@angular/core';
export const OLD_SERVICE = new InjectionToken<any>('oldService');
@NgModule({
providers: [
{
provide: OLD_SERVICE,
useFactory: (i: any) => i.get('oldService'),
deps: ['$injector']
}
]
})
// Use in Angular
@Component({...})
export class NewComponent {
constructor(@Inject(OLD_SERVICE) private oldService: any) {
console.log(this.oldService.getData());
}
}
```
## Routing Migration
```javascript
// Before: AngularJS routing
angular.module('myApp').config(function($routeProvider) {
$routeProvider
.when('/users', {
template: '<user-list></user-list>'
})
.when('/users/:id', {
template: '<user-detail></user-detail>'
});
});
```
```typescript
// After: Angular routing
import { NgModule } from '@angular/core';
import { RouterModule, Routes } from '@angular/router';
const routes: Routes = [
{ path: 'users', component: UserListComponent },
{ path: 'users/:id', component: UserDetailComponent }
];
@NgModule({
imports: [RouterModule.forRoot(routes)],
exports: [RouterModule]
})
export class AppRoutingModule {}
```
## Forms Migration
```html
<!-- Before: AngularJS -->
<form name="userForm" ng-submit="saveUser()">
<input type="text" ng-model="user.name" required>
<input type="email" ng-model="user.email" required>
<button ng-disabled="userForm.$invalid">Save</button>
</form>
```
```typescript
// After: Angular (Template-driven)
@Component({
template: `
<form #userForm="ngForm" (ngSubmit)="saveUser()">
<input type="text" [(ngModel)]="user.name" name="name" required>
<input type="email" [(ngModel)]="user.email" name="email" required>
<button [disabled]="userForm.invalid">Save</button>
</form>
`
})
// Or Reactive Forms (preferred)
import { FormBuilder, FormGroup, Validators } from '@angular/forms';
@Component({
template: `
<form [formGroup]="userForm" (ngSubmit)="saveUser()">
<input formControlName="name">
<input formControlName="email">
<button [disabled]="userForm.invalid">Save</button>
</form>
`
})
export class UserFormComponent {
userForm: FormGroup;
constructor(private fb: FormBuilder) {
this.userForm = this.fb.group({
name: ['', Validators.required],
email: ['', [Validators.required, Validators.email]]
});
}
saveUser() {
console.log(this.userForm.value);
}
}
```
## Migration Timeline
```
Phase 1: Setup (1-2 weeks)
- Install Angular CLI
- Set up hybrid app
- Configure build tools
- Set up testing
Phase 2: Infrastructure (2-4 weeks)
- Migrate services
- Migrate utilities
- Set up routing
- Migrate shared components
Phase 3: Feature Migration (varies)
- Migrate feature by feature
- Test thoroughly
- Deploy incrementally
Phase 4: Cleanup (1-2 weeks)
- Remove AngularJS code
- Remove ngUpgrade
- Optimize bundle
- Final testing
```
## Resources
- **references/hybrid-mode.md**: Hybrid app patterns
- **references/component-migration.md**: Component conversion guide
- **references/dependency-injection.md**: DI migration strategies
- **references/routing.md**: Routing migration
- **assets/hybrid-bootstrap.ts**: Hybrid app template
- **assets/migration-timeline.md**: Project planning
- **scripts/analyze-angular-app.sh**: App analysis script
## Best Practices
1. **Start with Services**: Migrate services first (easier)
2. **Incremental Approach**: Feature-by-feature migration
3. **Test Continuously**: Test at every step
4. **Use TypeScript**: Migrate to TypeScript early
5. **Follow Style Guide**: Angular style guide from day 1
6. **Optimize Later**: Get it working, then optimize
7. **Document**: Keep migration notes
## Common Pitfalls
- Not setting up hybrid app correctly
- Migrating UI before logic
- Ignoring change detection differences
- Not handling scope properly
- Mixing patterns (AngularJS + Angular)
- Inadequate testing

424
plugins/framework-migration/skills/database-migration/SKILL.md Normal file
View File

@@ -0,0 +1,424 @@
---
name: database-migration
description: Execute database migrations across ORMs and platforms with zero-downtime strategies, data transformation, and rollback procedures. Use when migrating databases, changing schemas, performing data transformations, or implementing zero-downtime deployment strategies.
---
# Database Migration
Master database schema and data migrations across ORMs (Sequelize, TypeORM, Prisma), including rollback strategies and zero-downtime deployments.
## When to Use This Skill
- Migrating between different ORMs
- Performing schema transformations
- Moving data between databases
- Implementing rollback procedures
- Zero-downtime deployments
- Database version upgrades
- Data model refactoring
## ORM Migrations
### Sequelize Migrations
```javascript
// migrations/20231201-create-users.js
module.exports = {
up: async (queryInterface, Sequelize) => {
await queryInterface.createTable('users', {
id: {
type: Sequelize.INTEGER,
primaryKey: true,
autoIncrement: true
},
email: {
type: Sequelize.STRING,
unique: true,
allowNull: false
},
createdAt: Sequelize.DATE,
updatedAt: Sequelize.DATE
});
},
down: async (queryInterface, Sequelize) => {
await queryInterface.dropTable('users');
}
};
// Run: npx sequelize-cli db:migrate
// Rollback: npx sequelize-cli db:migrate:undo
```
### TypeORM Migrations
```typescript
// migrations/1701234567-CreateUsers.ts
import { MigrationInterface, QueryRunner, Table } from 'typeorm';
export class CreateUsers1701234567 implements MigrationInterface {
public async up(queryRunner: QueryRunner): Promise<void> {
await queryRunner.createTable(
new Table({
name: 'users',
columns: [
{
name: 'id',
type: 'int',
isPrimary: true,
isGenerated: true,
generationStrategy: 'increment'
},
{
name: 'email',
type: 'varchar',
isUnique: true
},
{
name: 'created_at',
type: 'timestamp',
default: 'CURRENT_TIMESTAMP'
}
]
})
);
}
public async down(queryRunner: QueryRunner): Promise<void> {
await queryRunner.dropTable('users');
}
}
// Run: npm run typeorm migration:run
// Rollback: npm run typeorm migration:revert
```
### Prisma Migrations
```prisma
// schema.prisma
model User {
id Int @id @default(autoincrement())
email String @unique
createdAt DateTime @default(now())
}
// Generate migration: npx prisma migrate dev --name create_users
// Apply: npx prisma migrate deploy
```
## Schema Transformations
### Adding Columns with Defaults
```javascript
// Safe migration: add column with default
module.exports = {
up: async (queryInterface, Sequelize) => {
await queryInterface.addColumn('users', 'status', {
type: Sequelize.STRING,
defaultValue: 'active',
allowNull: false
});
},
down: async (queryInterface) => {
await queryInterface.removeColumn('users', 'status');
}
};
```
### Renaming Columns (Zero Downtime)
```javascript
// Step 1: Add new column
module.exports = {
up: async (queryInterface, Sequelize) => {
await queryInterface.addColumn('users', 'full_name', {
type: Sequelize.STRING
});
// Copy data from old column
await queryInterface.sequelize.query(
'UPDATE users SET full_name = name'
);
},
down: async (queryInterface) => {
await queryInterface.removeColumn('users', 'full_name');
}
};
// Step 2: Update application to use new column
// Step 3: Remove old column
module.exports = {
up: async (queryInterface) => {
await queryInterface.removeColumn('users', 'name');
},
down: async (queryInterface, Sequelize) => {
await queryInterface.addColumn('users', 'name', {
type: Sequelize.STRING
});
}
};
```
### Changing Column Types
```javascript
module.exports = {
up: async (queryInterface, Sequelize) => {
// For large tables, use multi-step approach
// 1. Add new column
await queryInterface.addColumn('users', 'age_new', {
type: Sequelize.INTEGER
});
// 2. Copy and transform data
await queryInterface.sequelize.query(`
UPDATE users
SET age_new = CAST(age AS INTEGER)
WHERE age IS NOT NULL
`);
// 3. Drop old column
await queryInterface.removeColumn('users', 'age');
// 4. Rename new column
await queryInterface.renameColumn('users', 'age_new', 'age');
},
down: async (queryInterface, Sequelize) => {
await queryInterface.changeColumn('users', 'age', {
type: Sequelize.STRING
});
}
};
```
## Data Transformations
### Complex Data Migration
```javascript
module.exports = {
up: async (queryInterface, Sequelize) => {
// Get all records
const [users] = await queryInterface.sequelize.query(
'SELECT id, address_string FROM users'
);
// Transform each record
for (const user of users) {
const addressParts = user.address_string.split(',');
await queryInterface.sequelize.query(
`UPDATE users
SET street = :street,
city = :city,
state = :state
WHERE id = :id`,
{
replacements: {
id: user.id,
street: addressParts[0]?.trim(),
city: addressParts[1]?.trim(),
state: addressParts[2]?.trim()
}
}
);
}
// Drop old column
await queryInterface.removeColumn('users', 'address_string');
},
down: async (queryInterface, Sequelize) => {
// Reconstruct original column
await queryInterface.addColumn('users', 'address_string', {
type: Sequelize.STRING
});
await queryInterface.sequelize.query(`
UPDATE users
SET address_string = CONCAT(street, ', ', city, ', ', state)
`);
await queryInterface.removeColumn('users', 'street');
await queryInterface.removeColumn('users', 'city');
await queryInterface.removeColumn('users', 'state');
}
};
```
## Rollback Strategies
### Transaction-Based Migrations
```javascript
module.exports = {
up: async (queryInterface, Sequelize) => {
const transaction = await queryInterface.sequelize.transaction();
try {
await queryInterface.addColumn(
'users',
'verified',
{ type: Sequelize.BOOLEAN, defaultValue: false },
{ transaction }
);
await queryInterface.sequelize.query(
'UPDATE users SET verified = true WHERE email_verified_at IS NOT NULL',
{ transaction }
);
await transaction.commit();
} catch (error) {
await transaction.rollback();
throw error;
}
},
down: async (queryInterface) => {
await queryInterface.removeColumn('users', 'verified');
}
};
```
### Checkpoint-Based Rollback
```javascript
module.exports = {
up: async (queryInterface, Sequelize) => {
// Create backup table
await queryInterface.sequelize.query(
'CREATE TABLE users_backup AS SELECT * FROM users'
);
try {
// Perform migration
await queryInterface.addColumn('users', 'new_field', {
type: Sequelize.STRING
});
// Verify migration
const [result] = await queryInterface.sequelize.query(
"SELECT COUNT(*) as count FROM users WHERE new_field IS NULL"
);
if (result[0].count > 0) {
throw new Error('Migration verification failed');
}
// Drop backup
await queryInterface.dropTable('users_backup');
} catch (error) {
// Restore from backup
await queryInterface.sequelize.query('DROP TABLE users');
await queryInterface.sequelize.query(
'CREATE TABLE users AS SELECT * FROM users_backup'
);
await queryInterface.dropTable('users_backup');
throw error;
}
}
};
```
## Zero-Downtime Migrations
### Blue-Green Deployment Strategy
```javascript
// Phase 1: Make changes backward compatible
module.exports = {
up: async (queryInterface, Sequelize) => {
// Add new column (both old and new code can work)
await queryInterface.addColumn('users', 'email_new', {
type: Sequelize.STRING
});
}
};
// Phase 2: Deploy code that writes to both columns
// Phase 3: Backfill data
module.exports = {
up: async (queryInterface) => {
await queryInterface.sequelize.query(`
UPDATE users
SET email_new = email
WHERE email_new IS NULL
`);
}
};
// Phase 4: Deploy code that reads from new column
// Phase 5: Remove old column
module.exports = {
up: async (queryInterface) => {
await queryInterface.removeColumn('users', 'email');
}
};
```
## Cross-Database Migrations
### PostgreSQL to MySQL
```javascript
// Handle differences
module.exports = {
up: async (queryInterface, Sequelize) => {
const dialectName = queryInterface.sequelize.getDialect();
if (dialectName === 'mysql') {
await queryInterface.createTable('users', {
id: {
type: Sequelize.INTEGER,
primaryKey: true,
autoIncrement: true
},
data: {
type: Sequelize.JSON // MySQL JSON type
}
});
} else if (dialectName === 'postgres') {
await queryInterface.createTable('users', {
id: {
type: Sequelize.INTEGER,
primaryKey: true,
autoIncrement: true
},
data: {
type: Sequelize.JSONB // PostgreSQL JSONB type
}
});
}
}
};
```
## Resources
- **references/orm-switching.md**: ORM migration guides
- **references/schema-migration.md**: Schema transformation patterns
- **references/data-transformation.md**: Data migration scripts
- **references/rollback-strategies.md**: Rollback procedures
- **assets/schema-migration-template.sql**: SQL migration templates
- **assets/data-migration-script.py**: Data migration utilities
- **scripts/test-migration.sh**: Migration testing script
## Best Practices
1. **Always Provide Rollback**: Every up() needs a down()
2. **Test Migrations**: Test on staging first
3. **Use Transactions**: Atomic migrations when possible
4. **Backup First**: Always backup before migration
5. **Small Changes**: Break into small, incremental steps
6. **Monitor**: Watch for errors during deployment
7. **Document**: Explain why and how
8. **Idempotent**: Migrations should be rerunnable
## Common Pitfalls
- Not testing rollback procedures
- Making breaking changes without downtime strategy
- Forgetting to handle NULL values
- Not considering index performance
- Ignoring foreign key constraints
- Migrating too much data at once

409
plugins/framework-migration/skills/dependency-upgrade/SKILL.md Normal file
View File

@@ -0,0 +1,409 @@
---
name: dependency-upgrade
description: Manage major dependency version upgrades with compatibility analysis, staged rollout, and comprehensive testing. Use when upgrading framework versions, updating major dependencies, or managing breaking changes in libraries.
---
# Dependency Upgrade
Master major dependency version upgrades, compatibility analysis, staged upgrade strategies, and comprehensive testing approaches.
## When to Use This Skill
- Upgrading major framework versions
- Updating security-vulnerable dependencies
- Modernizing legacy dependencies
- Resolving dependency conflicts
- Planning incremental upgrade paths
- Testing compatibility matrices
- Automating dependency updates
## Semantic Versioning Review
```
MAJOR.MINOR.PATCH (e.g., 2.3.1)
MAJOR: Breaking changes
MINOR: New features, backward compatible
PATCH: Bug fixes, backward compatible
^2.3.1 = >=2.3.1 <3.0.0 (minor updates)
~2.3.1 = >=2.3.1 <2.4.0 (patch updates)
2.3.1 = exact version
```
## Dependency Analysis
### Audit Dependencies
```bash
# npm
npm outdated
npm audit
npm audit fix
# yarn
yarn outdated
yarn audit
# Check for major updates
npx npm-check-updates
npx npm-check-updates -u # Update package.json
```
### Analyze Dependency Tree
```bash
# See why a package is installed
npm ls package-name
yarn why package-name
# Find duplicate packages
npm dedupe
yarn dedupe
# Visualize dependencies
npx madge --image graph.png src/
```
## Compatibility Matrix
```javascript
// compatibility-matrix.js
const compatibilityMatrix = {
'react': {
'16.x': {
'react-dom': '^16.0.0',
'react-router-dom': '^5.0.0',
'@testing-library/react': '^11.0.0'
},
'17.x': {
'react-dom': '^17.0.0',
'react-router-dom': '^5.0.0 || ^6.0.0',
'@testing-library/react': '^12.0.0'
},
'18.x': {
'react-dom': '^18.0.0',
'react-router-dom': '^6.0.0',
'@testing-library/react': '^13.0.0'
}
}
};
function checkCompatibility(packages) {
// Validate package versions against matrix
}
```
## Staged Upgrade Strategy
### Phase 1: Planning
```bash
# 1. Identify current versions
npm list --depth=0
# 2. Check for breaking changes
# Read CHANGELOG.md and MIGRATION.md
# 3. Create upgrade plan
echo "Upgrade order:
1. TypeScript
2. React
3. React Router
4. Testing libraries
5. Build tools" > UPGRADE_PLAN.md
```
### Phase 2: Incremental Updates
```bash
# Don't upgrade everything at once!
# Step 1: Update TypeScript
npm install typescript@latest
# Test
npm run test
npm run build
# Step 2: Update React (one major version at a time)
npm install react@17 react-dom@17
# Test again
npm run test
# Step 3: Continue with other packages
npm install react-router-dom@6
# And so on...
```
### Phase 3: Validation
```javascript
// tests/compatibility.test.js
describe('Dependency Compatibility', () => {
it('should have compatible React versions', () => {
const reactVersion = require('react/package.json').version;
const reactDomVersion = require('react-dom/package.json').version;
expect(reactVersion).toBe(reactDomVersion);
});
it('should not have peer dependency warnings', () => {
// Run npm ls and check for warnings
});
});
```
## Breaking Change Handling
### Identifying Breaking Changes
```bash
# Use changelog parsers
npx changelog-parser react 16.0.0 17.0.0
# Or manually check
curl https://raw.githubusercontent.com/facebook/react/main/CHANGELOG.md
```
### Codemod for Automated Fixes
```bash
# React upgrade codemods
npx react-codeshift <transform> <path>
# Example: Update lifecycle methods
npx react-codeshift \
--parser tsx \
--transform react-codeshift/transforms/rename-unsafe-lifecycles.js \
src/
```
### Custom Migration Script
```javascript
// migration-script.js
const fs = require('fs');
const glob = require('glob');
glob('src/**/*.tsx', (err, files) => {
files.forEach(file => {
let content = fs.readFileSync(file, 'utf8');
// Replace old API with new API
content = content.replace(
/componentWillMount/g,
'UNSAFE_componentWillMount'
);
// Update imports
content = content.replace(
/import { Component } from 'react'/g,
"import React, { Component } from 'react'"
);
fs.writeFileSync(file, content);
});
});
```
## Testing Strategy
### Unit Tests
```javascript
// Ensure tests pass before and after upgrade
npm run test
// Update test utilities if needed
npm install @testing-library/react@latest
```
### Integration Tests
```javascript
// tests/integration/app.test.js
describe('App Integration', () => {
it('should render without crashing', () => {
render(<App />);
});
it('should handle navigation', () => {
const { getByText } = render(<App />);
fireEvent.click(getByText('Navigate'));
expect(screen.getByText('New Page')).toBeInTheDocument();
});
});
```
### Visual Regression Tests
```javascript
// visual-regression.test.js
describe('Visual Regression', () => {
it('should match snapshot', () => {
const { container } = render(<App />);
expect(container.firstChild).toMatchSnapshot();
});
});
```
### E2E Tests
```javascript
// cypress/e2e/app.cy.js
describe('E2E Tests', () => {
it('should complete user flow', () => {
cy.visit('/');
cy.get('[data-testid="login"]').click();
cy.get('input[name="email"]').type('user@example.com');
cy.get('button[type="submit"]').click();
cy.url().should('include', '/dashboard');
});
});
```
## Automated Dependency Updates
### Renovate Configuration
```json
// renovate.json
{
"extends": ["config:base"],
"packageRules": [
{
"matchUpdateTypes": ["minor", "patch"],
"automerge": true
},
{
"matchUpdateTypes": ["major"],
"automerge": false,
"labels": ["major-update"]
}
],
"schedule": ["before 3am on Monday"],
"timezone": "America/New_York"
}
```
### Dependabot Configuration
```yaml
# .github/dependabot.yml
version: 2
updates:
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: "weekly"
open-pull-requests-limit: 5
reviewers:
- "team-leads"
commit-message:
prefix: "chore"
include: "scope"
```
## Rollback Plan
```javascript
// rollback.sh
#!/bin/bash
# Save current state
git stash
git checkout -b upgrade-branch
# Attempt upgrade
npm install package@latest
# Run tests
if npm run test; then
echo "Upgrade successful"
git add package.json package-lock.json
git commit -m "chore: upgrade package"
else
echo "Upgrade failed, rolling back"
git checkout main
git branch -D upgrade-branch
npm install # Restore from package-lock.json
fi
```
## Common Upgrade Patterns
### Lock File Management
```bash
# npm
npm install --package-lock-only # Update lock file only
npm ci # Clean install from lock file
# yarn
yarn install --frozen-lockfile # CI mode
yarn upgrade-interactive # Interactive upgrades
```
### Peer Dependency Resolution
```bash
# npm 7+: strict peer dependencies
npm install --legacy-peer-deps # Ignore peer deps
# npm 8+: override peer dependencies
npm install --force
```
### Workspace Upgrades
```bash
# Update all workspace packages
npm install --workspaces
# Update specific workspace
npm install package@latest --workspace=packages/app
```
## Resources
- **references/semver.md**: Semantic versioning guide
- **references/compatibility-matrix.md**: Common compatibility issues
- **references/staged-upgrades.md**: Incremental upgrade strategies
- **references/testing-strategy.md**: Comprehensive testing approaches
- **assets/upgrade-checklist.md**: Step-by-step checklist
- **assets/compatibility-matrix.csv**: Version compatibility table
- **scripts/audit-dependencies.sh**: Dependency audit script
## Best Practices
1. **Read Changelogs**: Understand what changed
2. **Upgrade Incrementally**: One major version at a time
3. **Test Thoroughly**: Unit, integration, E2E tests
4. **Check Peer Dependencies**: Resolve conflicts early
5. **Use Lock Files**: Ensure reproducible installs
6. **Automate Updates**: Use Renovate or Dependabot
7. **Monitor**: Watch for runtime errors post-upgrade
8. **Document**: Keep upgrade notes
## Upgrade Checklist
```markdown
Pre-Upgrade:
- [ ] Review current dependency versions
- [ ] Read changelogs for breaking changes
- [ ] Create feature branch
- [ ] Backup current state (git tag)
- [ ] Run full test suite (baseline)
During Upgrade:
- [ ] Upgrade one dependency at a time
- [ ] Update peer dependencies
- [ ] Fix TypeScript errors
- [ ] Update tests if needed
- [ ] Run test suite after each upgrade
- [ ] Check bundle size impact
Post-Upgrade:
- [ ] Full regression testing
- [ ] Performance testing
- [ ] Update documentation
- [ ] Deploy to staging
- [ ] Monitor for errors
- [ ] Deploy to production
```
## Common Pitfalls
- Upgrading all dependencies at once
- Not testing after each upgrade
- Ignoring peer dependency warnings
- Forgetting to update lock file
- Not reading breaking change notes
- Skipping major versions
- Not having rollback plan

513
plugins/framework-migration/skills/react-modernization/SKILL.md Normal file
View File

@@ -0,0 +1,513 @@
---
name: react-modernization
description: Upgrade React applications to latest versions, migrate from class components to hooks, and adopt concurrent features. Use when modernizing React codebases, migrating to React Hooks, or upgrading to latest React versions.
---
# React Modernization
Master React version upgrades, class to hooks migration, concurrent features adoption, and codemods for automated transformation.
## When to Use This Skill
- Upgrading React applications to latest versions
- Migrating class components to functional components with hooks
- Adopting concurrent React features (Suspense, transitions)
- Applying codemods for automated refactoring
- Modernizing state management patterns
- Updating to TypeScript
- Improving performance with React 18+ features
## Version Upgrade Path
### React 16 → 17 → 18
**Breaking Changes by Version:**
**React 17:**
- Event delegation changes
- No event pooling
- Effect cleanup timing
- JSX transform (no React import needed)
**React 18:**
- Automatic batching
- Concurrent rendering
- Strict Mode changes (double invocation)
- New root API
- Suspense on server
## Class to Hooks Migration
### State Management
```javascript
// Before: Class component
class Counter extends React.Component {
constructor(props) {
super(props);
this.state = {
count: 0,
name: ''
};
}
increment = () => {
this.setState({ count: this.state.count + 1 });
}
render() {
return (
<div>
<p>Count: {this.state.count}</p>
<button onClick={this.increment}>Increment</button>
</div>
);
}
}
// After: Functional component with hooks
function Counter() {
const [count, setCount] = useState(0);
const [name, setName] = useState('');
const increment = () => {
setCount(count + 1);
};
return (
<div>
<p>Count: {count}</p>
<button onClick={increment}>Increment</button>
</div>
);
}
```
### Lifecycle Methods to Hooks
```javascript
// Before: Lifecycle methods
class DataFetcher extends React.Component {
state = { data: null, loading: true };
componentDidMount() {
this.fetchData();
}
componentDidUpdate(prevProps) {
if (prevProps.id !== this.props.id) {
this.fetchData();
}
}
componentWillUnmount() {
this.cancelRequest();
}
fetchData = async () => {
const data = await fetch(`/api/${this.props.id}`);
this.setState({ data, loading: false });
};
cancelRequest = () => {
// Cleanup
};
render() {
if (this.state.loading) return <div>Loading...</div>;
return <div>{this.state.data}</div>;
}
}
// After: useEffect hook
function DataFetcher({ id }) {
const [data, setData] = useState(null);
const [loading, setLoading] = useState(true);
useEffect(() => {
let cancelled = false;
const fetchData = async () => {
try {
const response = await fetch(`/api/${id}`);
const result = await response.json();
if (!cancelled) {
setData(result);
setLoading(false);
}
} catch (error) {
if (!cancelled) {
console.error(error);
}
}
};
fetchData();
// Cleanup function
return () => {
cancelled = true;
};
}, [id]); // Re-run when id changes
if (loading) return <div>Loading...</div>;
return <div>{data}</div>;
}
```
### Context and HOCs to Hooks
```javascript
// Before: Context consumer and HOC
const ThemeContext = React.createContext();
class ThemedButton extends React.Component {
static contextType = ThemeContext;
render() {
return (
<button style={{ background: this.context.theme }}>
{this.props.children}
</button>
);
}
}
// After: useContext hook
function ThemedButton({ children }) {
const { theme } = useContext(ThemeContext);
return (
<button style={{ background: theme }}>
{children}
</button>
);
}
// Before: HOC for data fetching
function withUser(Component) {
return class extends React.Component {
state = { user: null };
componentDidMount() {
fetchUser().then(user => this.setState({ user }));
}
render() {
return <Component {...this.props} user={this.state.user} />;
}
};
}
// After: Custom hook
function useUser() {
const [user, setUser] = useState(null);
useEffect(() => {
fetchUser().then(setUser);
}, []);
return user;
}
function UserProfile() {
const user = useUser();
if (!user) return <div>Loading...</div>;
return <div>{user.name}</div>;
}
```
## React 18 Concurrent Features
### New Root API
```javascript
// Before: React 17
import ReactDOM from 'react-dom';
ReactDOM.render(<App />, document.getElementById('root'));
// After: React 18
import { createRoot } from 'react-dom/client';
const root = createRoot(document.getElementById('root'));
root.render(<App />);
```
### Automatic Batching
```javascript
// React 18: All updates are batched
function handleClick() {
setCount(c => c + 1);
setFlag(f => !f);
// Only one re-render (batched)
}
// Even in async:
setTimeout(() => {
setCount(c => c + 1);
setFlag(f => !f);
// Still batched in React 18!
}, 1000);
// Opt out if needed
import { flushSync } from 'react-dom';
flushSync(() => {
setCount(c => c + 1);
});
// Re-render happens here
setFlag(f => !f);
// Another re-render
```
### Transitions
```javascript
import { useState, useTransition } from 'react';
function SearchResults() {
const [query, setQuery] = useState('');
const [results, setResults] = useState([]);
const [isPending, startTransition] = useTransition();
const handleChange = (e) => {
// Urgent: Update input immediately
setQuery(e.target.value);
// Non-urgent: Update results (can be interrupted)
startTransition(() => {
setResults(searchResults(e.target.value));
});
};
return (
<>
<input value={query} onChange={handleChange} />
{isPending && <Spinner />}
<Results data={results} />
</>
);
}
```
### Suspense for Data Fetching
```javascript
import { Suspense } from 'react';
// Resource-based data fetching (with React 18)
const resource = fetchProfileData();
function ProfilePage() {
return (
<Suspense fallback={<Loading />}>
<ProfileDetails />
<Suspense fallback={<Loading />}>
<ProfileTimeline />
</Suspense>
</Suspense>
);
}
function ProfileDetails() {
// This will suspend if data not ready
const user = resource.user.read();
return <h1>{user.name}</h1>;
}
function ProfileTimeline() {
const posts = resource.posts.read();
return <Timeline posts={posts} />;
}
```
## Codemods for Automation
### Run React Codemods
```bash
# Install jscodeshift
npm install -g jscodeshift
# React 16.9 codemod (rename unsafe lifecycle methods)
npx react-codeshift <transform> <path>
# Example: Rename UNSAFE_ methods
npx react-codeshift --parser=tsx \
--transform=react-codeshift/transforms/rename-unsafe-lifecycles.js \
src/
# Update to new JSX Transform (React 17+)
npx react-codeshift --parser=tsx \
--transform=react-codeshift/transforms/new-jsx-transform.js \
src/
# Class to Hooks (third-party)
npx codemod react/hooks/convert-class-to-function src/
```
### Custom Codemod Example
```javascript
// custom-codemod.js
module.exports = function(file, api) {
const j = api.jscodeshift;
const root = j(file.source);
// Find setState calls
root.find(j.CallExpression, {
callee: {
type: 'MemberExpression',
property: { name: 'setState' }
}
}).forEach(path => {
// Transform to useState
// ... transformation logic
});
return root.toSource();
};
// Run: jscodeshift -t custom-codemod.js src/
```
## Performance Optimization
### useMemo and useCallback
```javascript
function ExpensiveComponent({ items, filter }) {
// Memoize expensive calculation
const filteredItems = useMemo(() => {
return items.filter(item => item.category === filter);
}, [items, filter]);
// Memoize callback to prevent child re-renders
const handleClick = useCallback((id) => {
console.log('Clicked:', id);
}, []); // No dependencies, never changes
return (
<List items={filteredItems} onClick={handleClick} />
);
}
// Child component with memo
const List = React.memo(({ items, onClick }) => {
return items.map(item => (
<Item key={item.id} item={item} onClick={onClick} />
));
});
```
### Code Splitting
```javascript
import { lazy, Suspense } from 'react';
// Lazy load components
const Dashboard = lazy(() => import('./Dashboard'));
const Settings = lazy(() => import('./Settings'));
function App() {
return (
<Suspense fallback={<Loading />}>
<Routes>
<Route path="/dashboard" element={<Dashboard />} />
<Route path="/settings" element={<Settings />} />
</Routes>
</Suspense>
);
}
```
## TypeScript Migration
```typescript
// Before: JavaScript
function Button({ onClick, children }) {
return <button onClick={onClick}>{children}</button>;
}
// After: TypeScript
interface ButtonProps {
onClick: () => void;
children: React.ReactNode;
}
function Button({ onClick, children }: ButtonProps) {
return <button onClick={onClick}>{children}</button>;
}
// Generic components
interface ListProps<T> {
items: T[];
renderItem: (item: T) => React.ReactNode;
}
function List<T>({ items, renderItem }: ListProps<T>) {
return <>{items.map(renderItem)}</>;
}
```
## Migration Checklist
```markdown
### Pre-Migration
- [ ] Update dependencies incrementally (not all at once)
- [ ] Review breaking changes in release notes
- [ ] Set up testing suite
- [ ] Create feature branch
### Class → Hooks Migration
- [ ] Identify class components to migrate
- [ ] Start with leaf components (no children)
- [ ] Convert state to useState
- [ ] Convert lifecycle to useEffect
- [ ] Convert context to useContext
- [ ] Extract custom hooks
- [ ] Test thoroughly
### React 18 Upgrade
- [ ] Update to React 17 first (if needed)
- [ ] Update react and react-dom to 18
- [ ] Update @types/react if using TypeScript
- [ ] Change to createRoot API
- [ ] Test with StrictMode (double invocation)
- [ ] Address concurrent rendering issues
- [ ] Adopt Suspense/Transitions where beneficial
### Performance
- [ ] Identify performance bottlenecks
- [ ] Add React.memo where appropriate
- [ ] Use useMemo/useCallback for expensive operations
- [ ] Implement code splitting
- [ ] Optimize re-renders
### Testing
- [ ] Update test utilities (React Testing Library)
- [ ] Test with React 18 features
- [ ] Check for warnings in console
- [ ] Performance testing
```
## Resources
- **references/breaking-changes.md**: Version-specific breaking changes
- **references/codemods.md**: Codemod usage guide
- **references/hooks-migration.md**: Comprehensive hooks patterns
- **references/concurrent-features.md**: React 18 concurrent features
- **assets/codemod-config.json**: Codemod configurations
- **assets/migration-checklist.md**: Step-by-step checklist
- **scripts/apply-codemods.sh**: Automated codemod script
## Best Practices
1. **Incremental Migration**: Don't migrate everything at once
2. **Test Thoroughly**: Comprehensive testing at each step
3. **Use Codemods**: Automate repetitive transformations
4. **Start Simple**: Begin with leaf components
5. **Leverage StrictMode**: Catch issues early
6. **Monitor Performance**: Measure before and after
7. **Document Changes**: Keep migration log
## Common Pitfalls
- Forgetting useEffect dependencies
- Over-using useMemo/useCallback
- Not handling cleanup in useEffect
- Mixing class and functional patterns
- Ignoring StrictMode warnings
- Breaking change assumptions

1025
plugins/javascript-typescript/skills/javascript-testing-patterns/SKILL.md Normal file
View File

File diff suppressed because it is too large Load Diff

911
plugins/javascript-typescript/skills/modern-javascript-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,911 @@
---
name: modern-javascript-patterns
description: Master ES6+ features including async/await, destructuring, spread operators, arrow functions, promises, modules, iterators, generators, and functional programming patterns for writing clean, efficient JavaScript code. Use when refactoring legacy code, implementing modern patterns, or optimizing JavaScript applications.
---
# Modern JavaScript Patterns
Comprehensive guide for mastering modern JavaScript (ES6+) features, functional programming patterns, and best practices for writing clean, maintainable, and performant code.
## When to Use This Skill
- Refactoring legacy JavaScript to modern syntax
- Implementing functional programming patterns
- Optimizing JavaScript performance
- Writing maintainable and readable code
- Working with asynchronous operations
- Building modern web applications
- Migrating from callbacks to Promises/async-await
- Implementing data transformation pipelines
## ES6+ Core Features
### 1. Arrow Functions
**Syntax and Use Cases:**
```javascript
// Traditional function
function add(a, b) {
return a + b;
}
// Arrow function
const add = (a, b) => a + b;
// Single parameter (parentheses optional)
const double = x => x * 2;
// No parameters
const getRandom = () => Math.random();
// Multiple statements (need curly braces)
const processUser = user => {
const normalized = user.name.toLowerCase();
return { ...user, name: normalized };
};
// Returning objects (wrap in parentheses)
const createUser = (name, age) => ({ name, age });
```
**Lexical 'this' Binding:**
```javascript
class Counter {
constructor() {
this.count = 0;
}
// Arrow function preserves 'this' context
increment = () => {
this.count++;
};
// Traditional function loses 'this' in callbacks
incrementTraditional() {
setTimeout(function() {
this.count++; // 'this' is undefined
}, 1000);
}
// Arrow function maintains 'this'
incrementArrow() {
setTimeout(() => {
this.count++; // 'this' refers to Counter instance
}, 1000);
}
}
```
### 2. Destructuring
**Object Destructuring:**
```javascript
const user = {
id: 1,
name: 'John Doe',
email: 'john@example.com',
address: {
city: 'New York',
country: 'USA'
}
};
// Basic destructuring
const { name, email } = user;
// Rename variables
const { name: userName, email: userEmail } = user;
// Default values
const { age = 25 } = user;
// Nested destructuring
const { address: { city, country } } = user;
// Rest operator
const { id, ...userWithoutId } = user;
// Function parameters
function greet({ name, age = 18 }) {
console.log(`Hello ${name}, you are ${age}`);
}
greet(user);
```
**Array Destructuring:**
```javascript
const numbers = [1, 2, 3, 4, 5];
// Basic destructuring
const [first, second] = numbers;
// Skip elements
const [, , third] = numbers;
// Rest operator
const [head, ...tail] = numbers;
// Swapping variables
let a = 1, b = 2;
[a, b] = [b, a];
// Function return values
function getCoordinates() {
return [10, 20];
}
const [x, y] = getCoordinates();
// Default values
const [one, two, three = 0] = [1, 2];
```
### 3. Spread and Rest Operators
**Spread Operator:**
```javascript
// Array spreading
const arr1 = [1, 2, 3];
const arr2 = [4, 5, 6];
const combined = [...arr1, ...arr2];
// Object spreading
const defaults = { theme: 'dark', lang: 'en' };
const userPrefs = { theme: 'light' };
const settings = { ...defaults, ...userPrefs };
// Function arguments
const numbers = [1, 2, 3];
Math.max(...numbers);
// Copying arrays/objects (shallow copy)
const copy = [...arr1];
const objCopy = { ...user };
// Adding items immutably
const newArr = [...arr1, 4, 5];
const newObj = { ...user, age: 30 };
```
**Rest Parameters:**
```javascript
// Collect function arguments
function sum(...numbers) {
return numbers.reduce((total, num) => total + num, 0);
}
sum(1, 2, 3, 4, 5);
// With regular parameters
function greet(greeting, ...names) {
return `${greeting} ${names.join(', ')}`;
}
greet('Hello', 'John', 'Jane', 'Bob');
// Object rest
const { id, ...userData } = user;
// Array rest
const [first, ...rest] = [1, 2, 3, 4, 5];
```
### 4. Template Literals
```javascript
// Basic usage
const name = 'John';
const greeting = `Hello, ${name}!`;
// Multi-line strings
const html = `
<div>
<h1>${title}</h1>
<p>${content}</p>
</div>
`;
// Expression evaluation
const price = 19.99;
const total = `Total: $${(price * 1.2).toFixed(2)}`;
// Tagged template literals
function highlight(strings, ...values) {
return strings.reduce((result, str, i) => {
const value = values[i] || '';
return result + str + `<mark>${value}</mark>`;
}, '');
}
const name = 'John';
const age = 30;
const html = highlight`Name: ${name}, Age: ${age}`;
// Output: "Name: <mark>John</mark>, Age: <mark>30</mark>"
```
### 5. Enhanced Object Literals
```javascript
const name = 'John';
const age = 30;
// Shorthand property names
const user = { name, age };
// Shorthand method names
const calculator = {
add(a, b) {
return a + b;
},
subtract(a, b) {
return a - b;
}
};
// Computed property names
const field = 'email';
const user = {
name: 'John',
[field]: 'john@example.com',
[`get${field.charAt(0).toUpperCase()}${field.slice(1)}`]() {
return this[field];
}
};
// Dynamic property creation
const createUser = (name, ...props) => {
return props.reduce((user, [key, value]) => ({
...user,
[key]: value
}), { name });
};
const user = createUser('John', ['age', 30], ['email', 'john@example.com']);
```
## Asynchronous Patterns
### 1. Promises
**Creating and Using Promises:**
```javascript
// Creating a promise
const fetchUser = (id) => {
return new Promise((resolve, reject) => {
setTimeout(() => {
if (id > 0) {
resolve({ id, name: 'John' });
} else {
reject(new Error('Invalid ID'));
}
}, 1000);
});
};
// Using promises
fetchUser(1)
.then(user => console.log(user))
.catch(error => console.error(error))
.finally(() => console.log('Done'));
// Chaining promises
fetchUser(1)
.then(user => fetchUserPosts(user.id))
.then(posts => processPosts(posts))
.then(result => console.log(result))
.catch(error => console.error(error));
```
**Promise Combinators:**
```javascript
// Promise.all - Wait for all promises
const promises = [
fetchUser(1),
fetchUser(2),
fetchUser(3)
];
Promise.all(promises)
.then(users => console.log(users))
.catch(error => console.error('At least one failed:', error));
// Promise.allSettled - Wait for all, regardless of outcome
Promise.allSettled(promises)
.then(results => {
results.forEach(result => {
if (result.status === 'fulfilled') {
console.log('Success:', result.value);
} else {
console.log('Error:', result.reason);
}
});
});
// Promise.race - First to complete
Promise.race(promises)
.then(winner => console.log('First:', winner))
.catch(error => console.error(error));
// Promise.any - First to succeed
Promise.any(promises)
.then(first => console.log('First success:', first))
.catch(error => console.error('All failed:', error));
```
### 2. Async/Await
**Basic Usage:**
```javascript
// Async function always returns a Promise
async function fetchUser(id) {
const response = await fetch(`/api/users/${id}`);
const user = await response.json();
return user;
}
// Error handling with try/catch
async function getUserData(id) {
try {
const user = await fetchUser(id);
const posts = await fetchUserPosts(user.id);
return { user, posts };
} catch (error) {
console.error('Error fetching data:', error);
throw error;
}
}
// Sequential vs Parallel execution
async function sequential() {
const user1 = await fetchUser(1); // Wait
const user2 = await fetchUser(2); // Then wait
return [user1, user2];
}
async function parallel() {
const [user1, user2] = await Promise.all([
fetchUser(1),
fetchUser(2)
]);
return [user1, user2];
}
```
**Advanced Patterns:**
```javascript
// Async IIFE
(async () => {
const result = await someAsyncOperation();
console.log(result);
})();
// Async iteration
async function processUsers(userIds) {
for (const id of userIds) {
const user = await fetchUser(id);
await processUser(user);
}
}
// Top-level await (ES2022)
const config = await fetch('/config.json').then(r => r.json());
// Retry logic
async function fetchWithRetry(url, retries = 3) {
for (let i = 0; i < retries; i++) {
try {
return await fetch(url);
} catch (error) {
if (i === retries - 1) throw error;
await new Promise(resolve => setTimeout(resolve, 1000 * (i + 1)));
}
}
}
// Timeout wrapper
async function withTimeout(promise, ms) {
const timeout = new Promise((_, reject) =>
setTimeout(() => reject(new Error('Timeout')), ms)
);
return Promise.race([promise, timeout]);
}
```
## Functional Programming Patterns
### 1. Array Methods
**Map, Filter, Reduce:**
```javascript
const users = [
{ id: 1, name: 'John', age: 30, active: true },
{ id: 2, name: 'Jane', age: 25, active: false },
{ id: 3, name: 'Bob', age: 35, active: true }
];
// Map - Transform array
const names = users.map(user => user.name);
const upperNames = users.map(user => user.name.toUpperCase());
// Filter - Select elements
const activeUsers = users.filter(user => user.active);
const adults = users.filter(user => user.age >= 18);
// Reduce - Aggregate data
const totalAge = users.reduce((sum, user) => sum + user.age, 0);
const avgAge = totalAge / users.length;
// Group by property
const byActive = users.reduce((groups, user) => {
const key = user.active ? 'active' : 'inactive';
return {
...groups,
[key]: [...(groups[key] || []), user]
};
}, {});
// Chaining methods
const result = users
.filter(user => user.active)
.map(user => user.name)
.sort()
.join(', ');
```
**Advanced Array Methods:**
```javascript
// Find - First matching element
const user = users.find(u => u.id === 2);
// FindIndex - Index of first match
const index = users.findIndex(u => u.name === 'Jane');
// Some - At least one matches
const hasActive = users.some(u => u.active);
// Every - All match
const allAdults = users.every(u => u.age >= 18);
// FlatMap - Map and flatten
const userTags = [
{ name: 'John', tags: ['admin', 'user'] },
{ name: 'Jane', tags: ['user'] }
];
const allTags = userTags.flatMap(u => u.tags);
// From - Create array from iterable
const str = 'hello';
const chars = Array.from(str);
const numbers = Array.from({ length: 5 }, (_, i) => i + 1);
// Of - Create array from arguments
const arr = Array.of(1, 2, 3);
```
### 2. Higher-Order Functions
**Functions as Arguments:**
```javascript
// Custom forEach
function forEach(array, callback) {
for (let i = 0; i < array.length; i++) {
callback(array[i], i, array);
}
}
// Custom map
function map(array, transform) {
const result = [];
for (const item of array) {
result.push(transform(item));
}
return result;
}
// Custom filter
function filter(array, predicate) {
const result = [];
for (const item of array) {
if (predicate(item)) {
result.push(item);
}
}
return result;
}
```
**Functions Returning Functions:**
```javascript
// Currying
const multiply = a => b => a * b;
const double = multiply(2);
const triple = multiply(3);
console.log(double(5)); // 10
console.log(triple(5)); // 15
// Partial application
function partial(fn, ...args) {
return (...moreArgs) => fn(...args, ...moreArgs);
}
const add = (a, b, c) => a + b + c;
const add5 = partial(add, 5);
console.log(add5(3, 2)); // 10
// Memoization
function memoize(fn) {
const cache = new Map();
return (...args) => {
const key = JSON.stringify(args);
if (cache.has(key)) {
return cache.get(key);
}
const result = fn(...args);
cache.set(key, result);
return result;
};
}
const fibonacci = memoize((n) => {
if (n <= 1) return n;
return fibonacci(n - 1) + fibonacci(n - 2);
});
```
### 3. Composition and Piping
```javascript
// Function composition
const compose = (...fns) => x =>
fns.reduceRight((acc, fn) => fn(acc), x);
const pipe = (...fns) => x =>
fns.reduce((acc, fn) => fn(acc), x);
// Example usage
const addOne = x => x + 1;
const double = x => x * 2;
const square = x => x * x;
const composed = compose(square, double, addOne);
console.log(composed(3)); // ((3 + 1) * 2)^2 = 64
const piped = pipe(addOne, double, square);
console.log(piped(3)); // ((3 + 1) * 2)^2 = 64
// Practical example
const processUser = pipe(
user => ({ ...user, name: user.name.trim() }),
user => ({ ...user, email: user.email.toLowerCase() }),
user => ({ ...user, age: parseInt(user.age) })
);
const user = processUser({
name: ' John ',
email: 'JOHN@EXAMPLE.COM',
age: '30'
});
```
### 4. Pure Functions and Immutability
```javascript
// Impure function (modifies input)
function addItemImpure(cart, item) {
cart.items.push(item);
cart.total += item.price;
return cart;
}
// Pure function (no side effects)
function addItemPure(cart, item) {
return {
...cart,
items: [...cart.items, item],
total: cart.total + item.price
};
}
// Immutable array operations
const numbers = [1, 2, 3, 4, 5];
// Add to array
const withSix = [...numbers, 6];
// Remove from array
const withoutThree = numbers.filter(n => n !== 3);
// Update array element
const doubled = numbers.map(n => n === 3 ? n * 2 : n);
// Immutable object operations
const user = { name: 'John', age: 30 };
// Update property
const olderUser = { ...user, age: 31 };
// Add property
const withEmail = { ...user, email: 'john@example.com' };
// Remove property
const { age, ...withoutAge } = user;
// Deep cloning (simple approach)
const deepClone = obj => JSON.parse(JSON.stringify(obj));
// Better deep cloning
const structuredClone = obj => globalThis.structuredClone(obj);
```
## Modern Class Features
```javascript
// Class syntax
class User {
// Private fields
#password;
// Public fields
id;
name;
// Static field
static count = 0;
constructor(id, name, password) {
this.id = id;
this.name = name;
this.#password = password;
User.count++;
}
// Public method
greet() {
return `Hello, ${this.name}`;
}
// Private method
#hashPassword(password) {
return `hashed_${password}`;
}
// Getter
get displayName() {
return this.name.toUpperCase();
}
// Setter
set password(newPassword) {
this.#password = this.#hashPassword(newPassword);
}
// Static method
static create(id, name, password) {
return new User(id, name, password);
}
}
// Inheritance
class Admin extends User {
constructor(id, name, password, role) {
super(id, name, password);
this.role = role;
}
greet() {
return `${super.greet()}, I'm an admin`;
}
}
```
## Modules (ES6)
```javascript
// Exporting
// math.js
export const PI = 3.14159;
export function add(a, b) {
return a + b;
}
export class Calculator {
// ...
}
// Default export
export default function multiply(a, b) {
return a * b;
}
// Importing
// app.js
import multiply, { PI, add, Calculator } from './math.js';
// Rename imports
import { add as sum } from './math.js';
// Import all
import * as Math from './math.js';
// Dynamic imports
const module = await import('./math.js');
const { add } = await import('./math.js');
// Conditional loading
if (condition) {
const module = await import('./feature.js');
module.init();
}
```
## Iterators and Generators
```javascript
// Custom iterator
const range = {
from: 1,
to: 5,
[Symbol.iterator]() {
return {
current: this.from,
last: this.to,
next() {
if (this.current <= this.last) {
return { done: false, value: this.current++ };
} else {
return { done: true };
}
}
};
}
};
for (const num of range) {
console.log(num); // 1, 2, 3, 4, 5
}
// Generator function
function* rangeGenerator(from, to) {
for (let i = from; i <= to; i++) {
yield i;
}
}
for (const num of rangeGenerator(1, 5)) {
console.log(num);
}
// Infinite generator
function* fibonacci() {
let [prev, curr] = [0, 1];
while (true) {
yield curr;
[prev, curr] = [curr, prev + curr];
}
}
// Async generator
async function* fetchPages(url) {
let page = 1;
while (true) {
const response = await fetch(`${url}?page=${page}`);
const data = await response.json();
if (data.length === 0) break;
yield data;
page++;
}
}
for await (const page of fetchPages('/api/users')) {
console.log(page);
}
```
## Modern Operators
```javascript
// Optional chaining
const user = { name: 'John', address: { city: 'NYC' } };
const city = user?.address?.city;
const zipCode = user?.address?.zipCode; // undefined
// Function call
const result = obj.method?.();
// Array access
const first = arr?.[0];
// Nullish coalescing
const value = null ?? 'default'; // 'default'
const value = undefined ?? 'default'; // 'default'
const value = 0 ?? 'default'; // 0 (not 'default')
const value = '' ?? 'default'; // '' (not 'default')
// Logical assignment
let a = null;
a ??= 'default'; // a = 'default'
let b = 5;
b ??= 10; // b = 5 (unchanged)
let obj = { count: 0 };
obj.count ||= 1; // obj.count = 1
obj.count &&= 2; // obj.count = 2
```
## Performance Optimization
```javascript
// Debounce
function debounce(fn, delay) {
let timeoutId;
return (...args) => {
clearTimeout(timeoutId);
timeoutId = setTimeout(() => fn(...args), delay);
};
}
const searchDebounced = debounce(search, 300);
// Throttle
function throttle(fn, limit) {
let inThrottle;
return (...args) => {
if (!inThrottle) {
fn(...args);
inThrottle = true;
setTimeout(() => inThrottle = false, limit);
}
};
}
const scrollThrottled = throttle(handleScroll, 100);
// Lazy evaluation
function* lazyMap(iterable, transform) {
for (const item of iterable) {
yield transform(item);
}
}
// Use only what you need
const numbers = [1, 2, 3, 4, 5];
const doubled = lazyMap(numbers, x => x * 2);
const first = doubled.next().value; // Only computes first value
```
## Best Practices
1. **Use const by default**: Only use let when reassignment is needed
2. **Prefer arrow functions**: Especially for callbacks
3. **Use template literals**: Instead of string concatenation
4. **Destructure objects and arrays**: For cleaner code
5. **Use async/await**: Instead of Promise chains
6. **Avoid mutating data**: Use spread operator and array methods
7. **Use optional chaining**: Prevent "Cannot read property of undefined"
8. **Use nullish coalescing**: For default values
9. **Prefer array methods**: Over traditional loops
10. **Use modules**: For better code organization
11. **Write pure functions**: Easier to test and reason about
12. **Use meaningful variable names**: Self-documenting code
13. **Keep functions small**: Single responsibility principle
14. **Handle errors properly**: Use try/catch with async/await
15. **Use strict mode**: `'use strict'` for better error catching
## Common Pitfalls
1. **this binding confusion**: Use arrow functions or bind()
2. **Async/await without error handling**: Always use try/catch
3. **Promise creation unnecessary**: Don't wrap already async functions
4. **Mutation of objects**: Use spread operator or Object.assign()
5. **Forgetting await**: Async functions return promises
6. **Blocking event loop**: Avoid synchronous operations
7. **Memory leaks**: Clean up event listeners and timers
8. **Not handling promise rejections**: Use catch() or try/catch
## Resources
- **MDN Web Docs**: https://developer.mozilla.org/en-US/docs/Web/JavaScript
- **JavaScript.info**: https://javascript.info/
- **You Don't Know JS**: https://github.com/getify/You-Dont-Know-JS
- **Eloquent JavaScript**: https://eloquentjavascript.net/
- **ES6 Features**: http://es6-features.org/

1020
plugins/javascript-typescript/skills/nodejs-backend-patterns/SKILL.md Normal file
View File

File diff suppressed because it is too large Load Diff

717
plugins/javascript-typescript/skills/typescript-advanced-types/SKILL.md Normal file
View File

@@ -0,0 +1,717 @@
---
name: typescript-advanced-types
description: Master TypeScript's advanced type system including generics, conditional types, mapped types, template literals, and utility types for building type-safe applications. Use when implementing complex type logic, creating reusable type utilities, or ensuring compile-time type safety in TypeScript projects.
---
# TypeScript Advanced Types
Comprehensive guidance for mastering TypeScript's advanced type system including generics, conditional types, mapped types, template literal types, and utility types for building robust, type-safe applications.
## When to Use This Skill
- Building type-safe libraries or frameworks
- Creating reusable generic components
- Implementing complex type inference logic
- Designing type-safe API clients
- Building form validation systems
- Creating strongly-typed configuration objects
- Implementing type-safe state management
- Migrating JavaScript codebases to TypeScript
## Core Concepts
### 1. Generics
**Purpose:** Create reusable, type-flexible components while maintaining type safety.
**Basic Generic Function:**
```typescript
function identity<T>(value: T): T {
return value;
}
const num = identity<number>(42); // Type: number
const str = identity<string>("hello"); // Type: string
const auto = identity(true); // Type inferred: boolean
```
**Generic Constraints:**
```typescript
interface HasLength {
length: number;
}
function logLength<T extends HasLength>(item: T): T {
console.log(item.length);
return item;
}
logLength("hello"); // OK: string has length
logLength([1, 2, 3]); // OK: array has length
logLength({ length: 10 }); // OK: object has length
// logLength(42); // Error: number has no length
```
**Multiple Type Parameters:**
```typescript
function merge<T, U>(obj1: T, obj2: U): T & U {
return { ...obj1, ...obj2 };
}
const merged = merge(
{ name: "John" },
{ age: 30 }
);
// Type: { name: string } & { age: number }
```
### 2. Conditional Types
**Purpose:** Create types that depend on conditions, enabling sophisticated type logic.
**Basic Conditional Type:**
```typescript
type IsString<T> = T extends string ? true : false;
type A = IsString<string>; // true
type B = IsString<number>; // false
```
**Extracting Return Types:**
```typescript
type ReturnType<T> = T extends (...args: any[]) => infer R ? R : never;
function getUser() {
return { id: 1, name: "John" };
}
type User = ReturnType<typeof getUser>;
// Type: { id: number; name: string; }
```
**Distributive Conditional Types:**
```typescript
type ToArray<T> = T extends any ? T[] : never;
type StrOrNumArray = ToArray<string | number>;
// Type: string[] | number[]
```
**Nested Conditions:**
```typescript
type TypeName<T> =
T extends string ? "string" :
T extends number ? "number" :
T extends boolean ? "boolean" :
T extends undefined ? "undefined" :
T extends Function ? "function" :
"object";
type T1 = TypeName<string>; // "string"
type T2 = TypeName<() => void>; // "function"
```
### 3. Mapped Types
**Purpose:** Transform existing types by iterating over their properties.
**Basic Mapped Type:**
```typescript
type Readonly<T> = {
readonly [P in keyof T]: T[P];
};
interface User {
id: number;
name: string;
}
type ReadonlyUser = Readonly<User>;
// Type: { readonly id: number; readonly name: string; }
```
**Optional Properties:**
```typescript
type Partial<T> = {
[P in keyof T]?: T[P];
};
type PartialUser = Partial<User>;
// Type: { id?: number; name?: string; }
```
**Key Remapping:**
```typescript
type Getters<T> = {
[K in keyof T as `get${Capitalize<string & K>}`]: () => T[K]
};
interface Person {
name: string;
age: number;
}
type PersonGetters = Getters<Person>;
// Type: { getName: () => string; getAge: () => number; }
```
**Filtering Properties:**
```typescript
type PickByType<T, U> = {
[K in keyof T as T[K] extends U ? K : never]: T[K]
};
interface Mixed {
id: number;
name: string;
age: number;
active: boolean;
}
type OnlyNumbers = PickByType<Mixed, number>;
// Type: { id: number; age: number; }
```
### 4. Template Literal Types
**Purpose:** Create string-based types with pattern matching and transformation.
**Basic Template Literal:**
```typescript
type EventName = "click" | "focus" | "blur";
type EventHandler = `on${Capitalize<EventName>}`;
// Type: "onClick" | "onFocus" | "onBlur"
```
**String Manipulation:**
```typescript
type UppercaseGreeting = Uppercase<"hello">; // "HELLO"
type LowercaseGreeting = Lowercase<"HELLO">; // "hello"
type CapitalizedName = Capitalize<"john">; // "John"
type UncapitalizedName = Uncapitalize<"John">; // "john"
```
**Path Building:**
```typescript
type Path<T> = T extends object
? { [K in keyof T]: K extends string
? `${K}` | `${K}.${Path<T[K]>}`
: never
}[keyof T]
: never;
interface Config {
server: {
host: string;
port: number;
};
database: {
url: string;
};
}
type ConfigPath = Path<Config>;
// Type: "server" | "database" | "server.host" | "server.port" | "database.url"
```
### 5. Utility Types
**Built-in Utility Types:**
```typescript
// Partial<T> - Make all properties optional
type PartialUser = Partial<User>;
// Required<T> - Make all properties required
type RequiredUser = Required<PartialUser>;
// Readonly<T> - Make all properties readonly
type ReadonlyUser = Readonly<User>;
// Pick<T, K> - Select specific properties
type UserName = Pick<User, "name" | "email">;
// Omit<T, K> - Remove specific properties
type UserWithoutPassword = Omit<User, "password">;
// Exclude<T, U> - Exclude types from union
type T1 = Exclude<"a" | "b" | "c", "a">; // "b" | "c"
// Extract<T, U> - Extract types from union
type T2 = Extract<"a" | "b" | "c", "a" | "b">; // "a" | "b"
// NonNullable<T> - Exclude null and undefined
type T3 = NonNullable<string | null | undefined>; // string
// Record<K, T> - Create object type with keys K and values T
type PageInfo = Record<"home" | "about", { title: string }>;
```
## Advanced Patterns
### Pattern 1: Type-Safe Event Emitter
```typescript
type EventMap = {
"user:created": { id: string; name: string };
"user:updated": { id: string };
"user:deleted": { id: string };
};
class TypedEventEmitter<T extends Record<string, any>> {
private listeners: {
[K in keyof T]?: Array<(data: T[K]) => void>;
} = {};
on<K extends keyof T>(event: K, callback: (data: T[K]) => void): void {
if (!this.listeners[event]) {
this.listeners[event] = [];
}
this.listeners[event]!.push(callback);
}
emit<K extends keyof T>(event: K, data: T[K]): void {
const callbacks = this.listeners[event];
if (callbacks) {
callbacks.forEach(callback => callback(data));
}
}
}
const emitter = new TypedEventEmitter<EventMap>();
emitter.on("user:created", (data) => {
console.log(data.id, data.name); // Type-safe!
});
emitter.emit("user:created", { id: "1", name: "John" });
// emitter.emit("user:created", { id: "1" }); // Error: missing 'name'
```
### Pattern 2: Type-Safe API Client
```typescript
type HTTPMethod = "GET" | "POST" | "PUT" | "DELETE";
type EndpointConfig = {
"/users": {
GET: { response: User[] };
POST: { body: { name: string; email: string }; response: User };
};
"/users/:id": {
GET: { params: { id: string }; response: User };
PUT: { params: { id: string }; body: Partial<User>; response: User };
DELETE: { params: { id: string }; response: void };
};
};
type ExtractParams<T> = T extends { params: infer P } ? P : never;
type ExtractBody<T> = T extends { body: infer B } ? B : never;
type ExtractResponse<T> = T extends { response: infer R } ? R : never;
class APIClient<Config extends Record<string, Record<HTTPMethod, any>>> {
async request<
Path extends keyof Config,
Method extends keyof Config[Path]
>(
path: Path,
method: Method,
...[options]: ExtractParams<Config[Path][Method]> extends never
? ExtractBody<Config[Path][Method]> extends never
? []
: [{ body: ExtractBody<Config[Path][Method]> }]
: [{
params: ExtractParams<Config[Path][Method]>;
body?: ExtractBody<Config[Path][Method]>;
}]
): Promise<ExtractResponse<Config[Path][Method]>> {
// Implementation here
return {} as any;
}
}
const api = new APIClient<EndpointConfig>();
// Type-safe API calls
const users = await api.request("/users", "GET");
// Type: User[]
const newUser = await api.request("/users", "POST", {
body: { name: "John", email: "john@example.com" }
});
// Type: User
const user = await api.request("/users/:id", "GET", {
params: { id: "123" }
});
// Type: User
```
### Pattern 3: Builder Pattern with Type Safety
```typescript
type BuilderState<T> = {
[K in keyof T]: T[K] | undefined;
};
type RequiredKeys<T> = {
[K in keyof T]-?: {} extends Pick<T, K> ? never : K;
}[keyof T];
type OptionalKeys<T> = {
[K in keyof T]-?: {} extends Pick<T, K> ? K : never;
}[keyof T];
type IsComplete<T, S> =
RequiredKeys<T> extends keyof S
? S[RequiredKeys<T>] extends undefined
? false
: true
: false;
class Builder<T, S extends BuilderState<T> = {}> {
private state: S = {} as S;
set<K extends keyof T>(
key: K,
value: T[K]
): Builder<T, S & Record<K, T[K]>> {
this.state[key] = value;
return this as any;
}
build(
this: IsComplete<T, S> extends true ? this : never
): T {
return this.state as T;
}
}
interface User {
id: string;
name: string;
email: string;
age?: number;
}
const builder = new Builder<User>();
const user = builder
.set("id", "1")
.set("name", "John")
.set("email", "john@example.com")
.build(); // OK: all required fields set
// const incomplete = builder
// .set("id", "1")
// .build(); // Error: missing required fields
```
### Pattern 4: Deep Readonly/Partial
```typescript
type DeepReadonly<T> = {
readonly [P in keyof T]: T[P] extends object
? T[P] extends Function
? T[P]
: DeepReadonly<T[P]>
: T[P];
};
type DeepPartial<T> = {
[P in keyof T]?: T[P] extends object
? T[P] extends Array<infer U>
? Array<DeepPartial<U>>
: DeepPartial<T[P]>
: T[P];
};
interface Config {
server: {
host: string;
port: number;
ssl: {
enabled: boolean;
cert: string;
};
};
database: {
url: string;
pool: {
min: number;
max: number;
};
};
}
type ReadonlyConfig = DeepReadonly<Config>;
// All nested properties are readonly
type PartialConfig = DeepPartial<Config>;
// All nested properties are optional
```
### Pattern 5: Type-Safe Form Validation
```typescript
type ValidationRule<T> = {
validate: (value: T) => boolean;
message: string;
};
type FieldValidation<T> = {
[K in keyof T]?: ValidationRule<T[K]>[];
};
type ValidationErrors<T> = {
[K in keyof T]?: string[];
};
class FormValidator<T extends Record<string, any>> {
constructor(private rules: FieldValidation<T>) {}
validate(data: T): ValidationErrors<T> | null {
const errors: ValidationErrors<T> = {};
let hasErrors = false;
for (const key in this.rules) {
const fieldRules = this.rules[key];
const value = data[key];
if (fieldRules) {
const fieldErrors: string[] = [];
for (const rule of fieldRules) {
if (!rule.validate(value)) {
fieldErrors.push(rule.message);
}
}
if (fieldErrors.length > 0) {
errors[key] = fieldErrors;
hasErrors = true;
}
}
}
return hasErrors ? errors : null;
}
}
interface LoginForm {
email: string;
password: string;
}
const validator = new FormValidator<LoginForm>({
email: [
{
validate: (v) => v.includes("@"),
message: "Email must contain @"
},
{
validate: (v) => v.length > 0,
message: "Email is required"
}
],
password: [
{
validate: (v) => v.length >= 8,
message: "Password must be at least 8 characters"
}
]
});
const errors = validator.validate({
email: "invalid",
password: "short"
});
// Type: { email?: string[]; password?: string[]; } | null
```
### Pattern 6: Discriminated Unions
```typescript
type Success<T> = {
status: "success";
data: T;
};
type Error = {
status: "error";
error: string;
};
type Loading = {
status: "loading";
};
type AsyncState<T> = Success<T> | Error | Loading;
function handleState<T>(state: AsyncState<T>): void {
switch (state.status) {
case "success":
console.log(state.data); // Type: T
break;
case "error":
console.log(state.error); // Type: string
break;
case "loading":
console.log("Loading...");
break;
}
}
// Type-safe state machine
type State =
| { type: "idle" }
| { type: "fetching"; requestId: string }
| { type: "success"; data: any }
| { type: "error"; error: Error };
type Event =
| { type: "FETCH"; requestId: string }
| { type: "SUCCESS"; data: any }
| { type: "ERROR"; error: Error }
| { type: "RESET" };
function reducer(state: State, event: Event): State {
switch (state.type) {
case "idle":
return event.type === "FETCH"
? { type: "fetching", requestId: event.requestId }
: state;
case "fetching":
if (event.type === "SUCCESS") {
return { type: "success", data: event.data };
}
if (event.type === "ERROR") {
return { type: "error", error: event.error };
}
return state;
case "success":
case "error":
return event.type === "RESET" ? { type: "idle" } : state;
}
}
```
## Type Inference Techniques
### 1. Infer Keyword
```typescript
// Extract array element type
type ElementType<T> = T extends (infer U)[] ? U : never;
type NumArray = number[];
type Num = ElementType<NumArray>; // number
// Extract promise type
type PromiseType<T> = T extends Promise<infer U> ? U : never;
type AsyncNum = PromiseType<Promise<number>>; // number
// Extract function parameters
type Parameters<T> = T extends (...args: infer P) => any ? P : never;
function foo(a: string, b: number) {}
type FooParams = Parameters<typeof foo>; // [string, number]
```
### 2. Type Guards
```typescript
function isString(value: unknown): value is string {
return typeof value === "string";
}
function isArrayOf<T>(
value: unknown,
guard: (item: unknown) => item is T
): value is T[] {
return Array.isArray(value) && value.every(guard);
}
const data: unknown = ["a", "b", "c"];
if (isArrayOf(data, isString)) {
data.forEach(s => s.toUpperCase()); // Type: string[]
}
```
### 3. Assertion Functions
```typescript
function assertIsString(value: unknown): asserts value is string {
if (typeof value !== "string") {
throw new Error("Not a string");
}
}
function processValue(value: unknown) {
assertIsString(value);
// value is now typed as string
console.log(value.toUpperCase());
}
```
## Best Practices
1. **Use `unknown` over `any`**: Enforce type checking
2. **Prefer `interface` for object shapes**: Better error messages
3. **Use `type` for unions and complex types**: More flexible
4. **Leverage type inference**: Let TypeScript infer when possible
5. **Create helper types**: Build reusable type utilities
6. **Use const assertions**: Preserve literal types
7. **Avoid type assertions**: Use type guards instead
8. **Document complex types**: Add JSDoc comments
9. **Use strict mode**: Enable all strict compiler options
10. **Test your types**: Use type tests to verify type behavior
## Type Testing
```typescript
// Type assertion tests
type AssertEqual<T, U> =
[T] extends [U]
? [U] extends [T]
? true
: false
: false;
type Test1 = AssertEqual<string, string>; // true
type Test2 = AssertEqual<string, number>; // false
type Test3 = AssertEqual<string | number, string>; // false
// Expect error helper
type ExpectError<T extends never> = T;
// Example usage
type ShouldError = ExpectError<AssertEqual<string, number>>;
```
## Common Pitfalls
1. **Over-using `any`**: Defeats the purpose of TypeScript
2. **Ignoring strict null checks**: Can lead to runtime errors
3. **Too complex types**: Can slow down compilation
4. **Not using discriminated unions**: Misses type narrowing opportunities
5. **Forgetting readonly modifiers**: Allows unintended mutations
6. **Circular type references**: Can cause compiler errors
7. **Not handling edge cases**: Like empty arrays or null values
## Performance Considerations
- Avoid deeply nested conditional types
- Use simple types when possible
- Cache complex type computations
- Limit recursion depth in recursive types
- Use build tools to skip type checking in production
## Resources
- **TypeScript Handbook**: https://www.typescriptlang.org/docs/handbook/
- **Type Challenges**: https://github.com/type-challenges/type-challenges
- **TypeScript Deep Dive**: https://basarat.gitbook.io/typescript/
- **Effective TypeScript**: Book by Dan Vanderkam

285
plugins/kubernetes-operations/skills/gitops-workflow/SKILL.md Normal file
View File

@@ -0,0 +1,285 @@
---
name: gitops-workflow
description: Implement GitOps workflows with ArgoCD and Flux for automated, declarative Kubernetes deployments with continuous reconciliation. Use when implementing GitOps practices, automating Kubernetes deployments, or setting up declarative infrastructure management.
---
# GitOps Workflow
Complete guide to implementing GitOps workflows with ArgoCD and Flux for automated Kubernetes deployments.
## Purpose
Implement declarative, Git-based continuous delivery for Kubernetes using ArgoCD or Flux CD, following OpenGitOps principles.
## When to Use This Skill
- Set up GitOps for Kubernetes clusters
- Automate application deployments from Git
- Implement progressive delivery strategies
- Manage multi-cluster deployments
- Configure automated sync policies
- Set up secret management in GitOps
## OpenGitOps Principles
1. **Declarative** - Entire system described declaratively
2. **Versioned and Immutable** - Desired state stored in Git
3. **Pulled Automatically** - Software agents pull desired state
4. **Continuously Reconciled** - Agents reconcile actual vs desired state
## ArgoCD Setup
### 1. Installation
```bash
# Create namespace
kubectl create namespace argocd
# Install ArgoCD
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# Get admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
```
**Reference:** See `references/argocd-setup.md` for detailed setup
### 2. Repository Structure
```
gitops-repo/
├── apps/
│ ├── production/
│ │ ├── app1/
│ │ │ ├── kustomization.yaml
│ │ │ └── deployment.yaml
│ │ └── app2/
│ └── staging/
├── infrastructure/
│ ├── ingress-nginx/
│ ├── cert-manager/
│ └── monitoring/
└── argocd/
├── applications/
└── projects/
```
### 3. Create Application
```yaml
# argocd/applications/my-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/org/gitops-repo
targetRevision: main
path: apps/production/my-app
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
```
### 4. App of Apps Pattern
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: applications
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/org/gitops-repo
targetRevision: main
path: argocd/applications
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated: {}
```
## Flux CD Setup
### 1. Installation
```bash
# Install Flux CLI
curl -s https://fluxcd.io/install.sh | sudo bash
# Bootstrap Flux
flux bootstrap github \
--owner=org \
--repository=gitops-repo \
--branch=main \
--path=clusters/production \
--personal
```
### 2. Create GitRepository
```yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: my-app
namespace: flux-system
spec:
interval: 1m
url: https://github.com/org/my-app
ref:
branch: main
```
### 3. Create Kustomization
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: my-app
namespace: flux-system
spec:
interval: 5m
path: ./deploy
prune: true
sourceRef:
kind: GitRepository
name: my-app
```
## Sync Policies
### Auto-Sync Configuration
**ArgoCD:**
```yaml
syncPolicy:
automated:
prune: true # Delete resources not in Git
selfHeal: true # Reconcile manual changes
allowEmpty: false
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
```
**Flux:**
```yaml
spec:
interval: 1m
prune: true
wait: true
timeout: 5m
```
**Reference:** See `references/sync-policies.md`
## Progressive Delivery
### Canary Deployment with ArgoCD Rollouts
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: my-app
spec:
replicas: 5
strategy:
canary:
steps:
- setWeight: 20
- pause: {duration: 1m}
- setWeight: 50
- pause: {duration: 2m}
- setWeight: 100
```
### Blue-Green Deployment
```yaml
strategy:
blueGreen:
activeService: my-app
previewService: my-app-preview
autoPromotionEnabled: false
```
## Secret Management
### External Secrets Operator
```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: SecretStore
target:
name: db-credentials
data:
- secretKey: password
remoteRef:
key: prod/db/password
```
### Sealed Secrets
```bash
# Encrypt secret
kubeseal --format yaml < secret.yaml > sealed-secret.yaml
# Commit sealed-secret.yaml to Git
```
## Best Practices
1. **Use separate repos or branches** for different environments
2. **Implement RBAC** for Git repositories
3. **Enable notifications** for sync failures
4. **Use health checks** for custom resources
5. **Implement approval gates** for production
6. **Keep secrets out of Git** (use External Secrets)
7. **Use App of Apps pattern** for organization
8. **Tag releases** for easy rollback
9. **Monitor sync status** with alerts
10. **Test changes** in staging first
## Troubleshooting
**Sync failures:**
```bash
argocd app get my-app
argocd app sync my-app --prune
```
**Out of sync status:**
```bash
argocd app diff my-app
argocd app sync my-app --force
```
## Related Skills
- `k8s-manifest-generator` - For creating manifests
- `helm-chart-scaffolding` - For packaging applications

134
plugins/kubernetes-operations/skills/gitops-workflow/references/argocd-setup.md Normal file
View File

@@ -0,0 +1,134 @@
# ArgoCD Setup and Configuration
## Installation Methods
### 1. Standard Installation
```bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```
### 2. High Availability Installation
```bash
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml
```
### 3. Helm Installation
```bash
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd --create-namespace
```
## Initial Configuration
### Access ArgoCD UI
```bash
# Port forward
kubectl port-forward svc/argocd-server -n argocd 8080:443
# Get initial admin password
argocd admin initial-password -n argocd
```
### Configure Ingress
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: argocd-server-ingress
namespace: argocd
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
ingressClassName: nginx
rules:
- host: argocd.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: argocd-server
port:
number: 443
tls:
- hosts:
- argocd.example.com
secretName: argocd-secret
```
## CLI Configuration
### Login
```bash
argocd login argocd.example.com --username admin
```
### Add Repository
```bash
argocd repo add https://github.com/org/repo --username user --password token
```
### Create Application
```bash
argocd app create my-app \
--repo https://github.com/org/repo \
--path apps/my-app \
--dest-server https://kubernetes.default.svc \
--dest-namespace production
```
## SSO Configuration
### GitHub OAuth
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
namespace: argocd
data:
url: https://argocd.example.com
dex.config: |
connectors:
- type: github
id: github
name: GitHub
config:
clientID: $GITHUB_CLIENT_ID
clientSecret: $GITHUB_CLIENT_SECRET
orgs:
- name: my-org
```
## RBAC Configuration
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-rbac-cm
namespace: argocd
data:
policy.default: role:readonly
policy.csv: |
p, role:developers, applications, *, */dev, allow
p, role:operators, applications, *, */*, allow
g, my-org:devs, role:developers
g, my-org:ops, role:operators
```
## Best Practices
1. Enable SSO for production
2. Implement RBAC policies
3. Use separate projects for teams
4. Enable audit logging
5. Configure notifications
6. Use ApplicationSets for multi-cluster
7. Implement resource hooks
8. Configure health checks
9. Use sync windows for maintenance
10. Monitor with Prometheus metrics

131
plugins/kubernetes-operations/skills/gitops-workflow/references/sync-policies.md Normal file
View File

@@ -0,0 +1,131 @@
# GitOps Sync Policies
## ArgoCD Sync Policies
### Automated Sync
```yaml
syncPolicy:
automated:
prune: true # Delete resources removed from Git
selfHeal: true # Reconcile manual changes
allowEmpty: false # Prevent empty sync
```
### Manual Sync
```yaml
syncPolicy:
syncOptions:
- PrunePropagationPolicy=foreground
- CreateNamespace=true
```
### Sync Windows
```yaml
syncWindows:
- kind: allow
schedule: "0 8 * * *"
duration: 1h
applications:
- my-app
- kind: deny
schedule: "0 22 * * *"
duration: 8h
applications:
- '*'
```
### Retry Policy
```yaml
syncPolicy:
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
```
## Flux Sync Policies
### Kustomization Sync
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: my-app
spec:
interval: 5m
prune: true
wait: true
timeout: 5m
retryInterval: 1m
force: false
```
### Source Sync Interval
```yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: my-app
spec:
interval: 1m
timeout: 60s
```
## Health Assessment
### Custom Health Checks
```yaml
# ArgoCD
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
namespace: argocd
data:
resource.customizations.health.MyCustomResource: |
hs = {}
if obj.status ~= nil then
if obj.status.conditions ~= nil then
for i, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
end
end
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
```
## Sync Options
### Common Sync Options
- `PrunePropagationPolicy=foreground` - Wait for pruned resources to be deleted
- `CreateNamespace=true` - Auto-create namespace
- `Validate=false` - Skip kubectl validation
- `PruneLast=true` - Prune resources after sync
- `RespectIgnoreDifferences=true` - Honor ignore differences
- `ApplyOutOfSyncOnly=true` - Only apply out-of-sync resources
## Best Practices
1. Use automated sync for non-production
2. Require manual approval for production
3. Configure sync windows for maintenance
4. Implement health checks for custom resources
5. Use selective sync for large applications
6. Configure appropriate retry policies
7. Monitor sync failures with alerts
8. Use prune with caution in production
9. Test sync policies in staging
10. Document sync behavior for teams

544
plugins/kubernetes-operations/skills/helm-chart-scaffolding/SKILL.md Normal file
View File

@@ -0,0 +1,544 @@
---
name: helm-chart-scaffolding
description: Design, organize, and manage Helm charts for templating and packaging Kubernetes applications with reusable configurations. Use when creating Helm charts, packaging Kubernetes applications, or implementing templated deployments.
---
# Helm Chart Scaffolding
Comprehensive guidance for creating, organizing, and managing Helm charts for packaging and deploying Kubernetes applications.
## Purpose
This skill provides step-by-step instructions for building production-ready Helm charts, including chart structure, templating patterns, values management, and validation strategies.
## When to Use This Skill
Use this skill when you need to:
- Create new Helm charts from scratch
- Package Kubernetes applications for distribution
- Manage multi-environment deployments with Helm
- Implement templating for reusable Kubernetes manifests
- Set up Helm chart repositories
- Follow Helm best practices and conventions
## Helm Overview
**Helm** is the package manager for Kubernetes that:
- Templates Kubernetes manifests for reusability
- Manages application releases and rollbacks
- Handles dependencies between charts
- Provides version control for deployments
- Simplifies configuration management across environments
## Step-by-Step Workflow
### 1. Initialize Chart Structure
**Create new chart:**
```bash
helm create my-app
```
**Standard chart structure:**
```
my-app/
├── Chart.yaml # Chart metadata
├── values.yaml # Default configuration values
├── charts/ # Chart dependencies
├── templates/ # Kubernetes manifest templates
│ ├── NOTES.txt # Post-install notes
│ ├── _helpers.tpl # Template helpers
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── ingress.yaml
│ ├── serviceaccount.yaml
│ ├── hpa.yaml
│ └── tests/
│ └── test-connection.yaml
└── .helmignore # Files to ignore
```
### 2. Configure Chart.yaml
**Chart metadata defines the package:**
```yaml
apiVersion: v2
name: my-app
description: A Helm chart for My Application
type: application
version: 1.0.0 # Chart version
appVersion: "2.1.0" # Application version
# Keywords for chart discovery
keywords:
- web
- api
- backend
# Maintainer information
maintainers:
- name: DevOps Team
email: devops@example.com
url: https://github.com/example/my-app
# Source code repository
sources:
- https://github.com/example/my-app
# Homepage
home: https://example.com
# Chart icon
icon: https://example.com/icon.png
# Dependencies
dependencies:
- name: postgresql
version: "12.0.0"
repository: "https://charts.bitnami.com/bitnami"
condition: postgresql.enabled
- name: redis
version: "17.0.0"
repository: "https://charts.bitnami.com/bitnami"
condition: redis.enabled
```
**Reference:** See `assets/Chart.yaml.template` for complete example
### 3. Design values.yaml Structure
**Organize values hierarchically:**
```yaml
# Image configuration
image:
repository: myapp
tag: "1.0.0"
pullPolicy: IfNotPresent
# Number of replicas
replicaCount: 3
# Service configuration
service:
type: ClusterIP
port: 80
targetPort: 8080
# Ingress configuration
ingress:
enabled: false
className: nginx
hosts:
- host: app.example.com
paths:
- path: /
pathType: Prefix
# Resources
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
# Autoscaling
autoscaling:
enabled: false
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 80
# Environment variables
env:
- name: LOG_LEVEL
value: "info"
# ConfigMap data
configMap:
data:
APP_MODE: production
# Dependencies
postgresql:
enabled: true
auth:
database: myapp
username: myapp
redis:
enabled: false
```
**Reference:** See `assets/values.yaml.template` for complete structure
### 4. Create Template Files
**Use Go templating with Helm functions:**
**templates/deployment.yaml:**
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "my-app.fullname" . }}
labels:
{{- include "my-app.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "my-app.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "my-app.selectorLabels" . | nindent 8 }}
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.service.targetPort }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
env:
{{- toYaml .Values.env | nindent 12 }}
```
### 5. Create Template Helpers
**templates/_helpers.tpl:**
```yaml
{{/*
Expand the name of the chart.
*/}}
{{- define "my-app.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
*/}}
{{- define "my-app.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "my-app.labels" -}}
helm.sh/chart: {{ include "my-app.chart" . }}
{{ include "my-app.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "my-app.selectorLabels" -}}
app.kubernetes.io/name: {{ include "my-app.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
```
### 6. Manage Dependencies
**Add dependencies in Chart.yaml:**
```yaml
dependencies:
- name: postgresql
version: "12.0.0"
repository: "https://charts.bitnami.com/bitnami"
condition: postgresql.enabled
```
**Update dependencies:**
```bash
helm dependency update
helm dependency build
```
**Override dependency values:**
```yaml
# values.yaml
postgresql:
enabled: true
auth:
database: myapp
username: myapp
password: changeme
primary:
persistence:
enabled: true
size: 10Gi
```
### 7. Test and Validate
**Validation commands:**
```bash
# Lint the chart
helm lint my-app/
# Dry-run installation
helm install my-app ./my-app --dry-run --debug
# Template rendering
helm template my-app ./my-app
# Template with values
helm template my-app ./my-app -f values-prod.yaml
# Show computed values
helm show values ./my-app
```
**Validation script:**
```bash
#!/bin/bash
set -e
echo "Linting chart..."
helm lint .
echo "Testing template rendering..."
helm template test-release . --dry-run
echo "Checking for required values..."
helm template test-release . --validate
echo "All validations passed!"
```
**Reference:** See `scripts/validate-chart.sh`
### 8. Package and Distribute
**Package the chart:**
```bash
helm package my-app/
# Creates: my-app-1.0.0.tgz
```
**Create chart repository:**
```bash
# Create index
helm repo index .
# Upload to repository
# AWS S3 example
aws s3 sync . s3://my-helm-charts/ --exclude "*" --include "*.tgz" --include "index.yaml"
```
**Use the chart:**
```bash
helm repo add my-repo https://charts.example.com
helm repo update
helm install my-app my-repo/my-app
```
### 9. Multi-Environment Configuration
**Environment-specific values files:**
```
my-app/
├── values.yaml # Defaults
├── values-dev.yaml # Development
├── values-staging.yaml # Staging
└── values-prod.yaml # Production
```
**values-prod.yaml:**
```yaml
replicaCount: 5
image:
tag: "2.1.0"
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "1Gi"
cpu: "1000m"
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 20
ingress:
enabled: true
hosts:
- host: app.example.com
paths:
- path: /
pathType: Prefix
postgresql:
enabled: true
primary:
persistence:
size: 100Gi
```
**Install with environment:**
```bash
helm install my-app ./my-app -f values-prod.yaml --namespace production
```
### 10. Implement Hooks and Tests
**Pre-install hook:**
```yaml
# templates/pre-install-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "my-app.fullname" . }}-db-setup
annotations:
"helm.sh/hook": pre-install
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": hook-succeeded
spec:
template:
spec:
containers:
- name: db-setup
image: postgres:15
command: ["psql", "-c", "CREATE DATABASE myapp"]
restartPolicy: Never
```
**Test connection:**
```yaml
# templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
name: "{{ include "my-app.fullname" . }}-test-connection"
annotations:
"helm.sh/hook": test
spec:
containers:
- name: wget
image: busybox
command: ['wget']
args: ['{{ include "my-app.fullname" . }}:{{ .Values.service.port }}']
restartPolicy: Never
```
**Run tests:**
```bash
helm test my-app
```
## Common Patterns
### Pattern 1: Conditional Resources
```yaml
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "my-app.fullname" . }}
spec:
# ...
{{- end }}
```
### Pattern 2: Iterating Over Lists
```yaml
env:
{{- range .Values.env }}
- name: {{ .name }}
value: {{ .value | quote }}
{{- end }}
```
### Pattern 3: Including Files
```yaml
data:
config.yaml: |
{{- .Files.Get "config/application.yaml" | nindent 4 }}
```
### Pattern 4: Global Values
```yaml
global:
imageRegistry: docker.io
imagePullSecrets:
- name: regcred
# Use in templates:
image: {{ .Values.global.imageRegistry }}/{{ .Values.image.repository }}
```
## Best Practices
1. **Use semantic versioning** for chart and app versions
2. **Document all values** in values.yaml with comments
3. **Use template helpers** for repeated logic
4. **Validate charts** before packaging
5. **Pin dependency versions** explicitly
6. **Use conditions** for optional resources
7. **Follow naming conventions** (lowercase, hyphens)
8. **Include NOTES.txt** with usage instructions
9. **Add labels** consistently using helpers
10. **Test installations** in all environments
## Troubleshooting
**Template rendering errors:**
```bash
helm template my-app ./my-app --debug
```
**Dependency issues:**
```bash
helm dependency update
helm dependency list
```
**Installation failures:**
```bash
helm install my-app ./my-app --dry-run --debug
kubectl get events --sort-by='.lastTimestamp'
```
## Reference Files
- `assets/Chart.yaml.template` - Chart metadata template
- `assets/values.yaml.template` - Values structure template
- `scripts/validate-chart.sh` - Validation script
- `references/chart-structure.md` - Detailed chart organization
## Related Skills
- `k8s-manifest-generator` - For creating base Kubernetes manifests
- `gitops-workflow` - For automated Helm chart deployments

42
plugins/kubernetes-operations/skills/helm-chart-scaffolding/assets/Chart.yaml.template Normal file
View File

@@ -0,0 +1,42 @@
apiVersion: v2
name: <chart-name>
description: <Chart description>
type: application
version: 0.1.0
appVersion: "1.0.0"
keywords:
- <keyword1>
- <keyword2>
home: https://github.com/<org>/<repo>
sources:
- https://github.com/<org>/<repo>
maintainers:
- name: <Maintainer Name>
email: <maintainer@example.com>
url: https://github.com/<username>
icon: https://example.com/icon.png
kubeVersion: ">=1.24.0"
dependencies:
- name: postgresql
version: "12.0.0"
repository: "https://charts.bitnami.com/bitnami"
condition: postgresql.enabled
tags:
- database
- name: redis
version: "17.0.0"
repository: "https://charts.bitnami.com/bitnami"
condition: redis.enabled
tags:
- cache
annotations:
category: Application
licenses: Apache-2.0

185
plugins/kubernetes-operations/skills/helm-chart-scaffolding/assets/values.yaml.template Normal file
View File

@@ -0,0 +1,185 @@
# Global values shared with subcharts
global:
imageRegistry: docker.io
imagePullSecrets: []
storageClass: ""
# Image configuration
image:
registry: docker.io
repository: myapp/web
tag: "" # Defaults to .Chart.AppVersion
pullPolicy: IfNotPresent
# Override chart name
nameOverride: ""
fullnameOverride: ""
# Number of replicas
replicaCount: 3
revisionHistoryLimit: 10
# ServiceAccount
serviceAccount:
create: true
annotations: {}
name: ""
# Pod annotations
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/metrics"
# Pod security context
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
# Container security context
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
# Service configuration
service:
type: ClusterIP
port: 80
targetPort: http
annotations: {}
sessionAffinity: None
# Ingress configuration
ingress:
enabled: false
className: nginx
annotations: {}
hosts:
- host: app.example.com
paths:
- path: /
pathType: Prefix
tls: []
# Resources
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 256Mi
# Liveness probe
livenessProbe:
httpGet:
path: /health/live
port: http
initialDelaySeconds: 30
periodSeconds: 10
# Readiness probe
readinessProbe:
httpGet:
path: /health/ready
port: http
initialDelaySeconds: 5
periodSeconds: 5
# Autoscaling
autoscaling:
enabled: false
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
# Pod Disruption Budget
podDisruptionBudget:
enabled: true
minAvailable: 1
# Node selection
nodeSelector: {}
tolerations: []
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- '{{ include "my-app.name" . }}'
topologyKey: kubernetes.io/hostname
# Environment variables
env: []
# - name: LOG_LEVEL
# value: "info"
# ConfigMap data
configMap:
enabled: true
data: {}
# APP_MODE: production
# DATABASE_HOST: postgres.example.com
# Secrets (use external secret management in production)
secrets:
enabled: false
data: {}
# Persistent Volume
persistence:
enabled: false
storageClass: ""
accessMode: ReadWriteOnce
size: 10Gi
annotations: {}
# PostgreSQL dependency
postgresql:
enabled: false
auth:
database: myapp
username: myapp
password: changeme
primary:
persistence:
enabled: true
size: 10Gi
# Redis dependency
redis:
enabled: false
auth:
enabled: false
master:
persistence:
enabled: false
# ServiceMonitor for Prometheus Operator
serviceMonitor:
enabled: false
interval: 30s
scrapeTimeout: 10s
labels: {}
# Network Policy
networkPolicy:
enabled: false
policyTypes:
- Ingress
- Egress
ingress: []
egress: []

500
plugins/kubernetes-operations/skills/helm-chart-scaffolding/references/chart-structure.md Normal file
View File

@@ -0,0 +1,500 @@
# Helm Chart Structure Reference
Complete guide to Helm chart organization, file conventions, and best practices.
## Standard Chart Directory Structure
```
my-app/
├── Chart.yaml # Chart metadata (required)
├── Chart.lock # Dependency lock file (generated)
├── values.yaml # Default configuration values (required)
├── values.schema.json # JSON schema for values validation
├── .helmignore # Patterns to ignore when packaging
├── README.md # Chart documentation
├── LICENSE # Chart license
├── charts/ # Chart dependencies (bundled)
│ └── postgresql-12.0.0.tgz
├── crds/ # Custom Resource Definitions
│ └── my-crd.yaml
├── templates/ # Kubernetes manifest templates (required)
│ ├── NOTES.txt # Post-install instructions
│ ├── _helpers.tpl # Template helper functions
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── ingress.yaml
│ ├── configmap.yaml
│ ├── secret.yaml
│ ├── serviceaccount.yaml
│ ├── hpa.yaml
│ ├── pdb.yaml
│ ├── networkpolicy.yaml
│ └── tests/
│ └── test-connection.yaml
└── files/ # Additional files to include
└── config/
└── app.conf
```
## Chart.yaml Specification
### API Version v2 (Helm 3+)
```yaml
apiVersion: v2 # Required: API version
name: my-application # Required: Chart name
version: 1.2.3 # Required: Chart version (SemVer)
appVersion: "2.5.0" # Application version
description: A Helm chart for my application # Required
type: application # Chart type: application or library
keywords: # Search keywords
- web
- api
- backend
home: https://example.com # Project home page
sources: # Source code URLs
- https://github.com/example/my-app
maintainers: # Maintainer list
- name: John Doe
email: john@example.com
url: https://github.com/johndoe
icon: https://example.com/icon.png # Chart icon URL
kubeVersion: ">=1.24.0" # Compatible Kubernetes versions
deprecated: false # Mark chart as deprecated
annotations: # Arbitrary annotations
example.com/release-notes: https://example.com/releases/v1.2.3
dependencies: # Chart dependencies
- name: postgresql
version: "12.0.0"
repository: "https://charts.bitnami.com/bitnami"
condition: postgresql.enabled
tags:
- database
import-values:
- child: database
parent: database
alias: db
```
## Chart Types
### Application Chart
```yaml
type: application
```
- Standard Kubernetes applications
- Can be installed and managed
- Contains templates for K8s resources
### Library Chart
```yaml
type: library
```
- Shared template helpers
- Cannot be installed directly
- Used as dependency by other charts
- No templates/ directory
## Values Files Organization
### values.yaml (defaults)
```yaml
# Global values (shared with subcharts)
global:
imageRegistry: docker.io
imagePullSecrets: []
# Image configuration
image:
registry: docker.io
repository: myapp/web
tag: "" # Defaults to .Chart.AppVersion
pullPolicy: IfNotPresent
# Deployment settings
replicaCount: 1
revisionHistoryLimit: 10
# Pod configuration
podAnnotations: {}
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
# Container security
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
# Service
service:
type: ClusterIP
port: 80
targetPort: http
annotations: {}
# Resources
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
# Autoscaling
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# Node selection
nodeSelector: {}
tolerations: []
affinity: {}
# Monitoring
serviceMonitor:
enabled: false
interval: 30s
```
### values.schema.json (validation)
```json
{
"$schema": "https://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"replicaCount": {
"type": "integer",
"minimum": 1
},
"image": {
"type": "object",
"required": ["repository"],
"properties": {
"repository": {
"type": "string"
},
"tag": {
"type": "string"
},
"pullPolicy": {
"type": "string",
"enum": ["Always", "IfNotPresent", "Never"]
}
}
}
},
"required": ["image"]
}
```
## Template Files
### Template Naming Conventions
- **Lowercase with hyphens**: `deployment.yaml`, `service-account.yaml`
- **Partial templates**: Prefix with underscore `_helpers.tpl`
- **Tests**: Place in `templates/tests/`
- **CRDs**: Place in `crds/` (not templated)
### Common Templates
#### _helpers.tpl
```yaml
{{/*
Standard naming helpers
*/}}
{{- define "my-app.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- define "my-app.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- define "my-app.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Common labels
*/}}
{{- define "my-app.labels" -}}
helm.sh/chart: {{ include "my-app.chart" . }}
{{ include "my-app.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}
{{- define "my-app.selectorLabels" -}}
app.kubernetes.io/name: {{ include "my-app.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}
{{/*
Image name helper
*/}}
{{- define "my-app.image" -}}
{{- $registry := .Values.global.imageRegistry | default .Values.image.registry -}}
{{- $repository := .Values.image.repository -}}
{{- $tag := .Values.image.tag | default .Chart.AppVersion -}}
{{- printf "%s/%s:%s" $registry $repository $tag -}}
{{- end -}}
```
#### NOTES.txt
```
Thank you for installing {{ .Chart.Name }}.
Your release is named {{ .Release.Name }}.
To learn more about the release, try:
$ helm status {{ .Release.Name }}
$ helm get all {{ .Release.Name }}
{{- if .Values.ingress.enabled }}
Application URL:
{{- range .Values.ingress.hosts }}
http{{ if $.Values.ingress.tls }}s{{ end }}://{{ .host }}{{ .path }}
{{- end }}
{{- else }}
Get the application URL by running:
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "my-app.name" . }}" -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward $POD_NAME 8080:80
echo "Visit http://127.0.0.1:8080"
{{- end }}
```
## Dependencies Management
### Declaring Dependencies
```yaml
# Chart.yaml
dependencies:
- name: postgresql
version: "12.0.0"
repository: "https://charts.bitnami.com/bitnami"
condition: postgresql.enabled # Enable/disable via values
tags: # Group dependencies
- database
import-values: # Import values from subchart
- child: database
parent: database
alias: db # Reference as .Values.db
```
### Managing Dependencies
```bash
# Update dependencies
helm dependency update
# List dependencies
helm dependency list
# Build dependencies
helm dependency build
```
### Chart.lock
Generated automatically by `helm dependency update`:
```yaml
dependencies:
- name: postgresql
repository: https://charts.bitnami.com/bitnami
version: 12.0.0
digest: sha256:abcd1234...
generated: "2024-01-01T00:00:00Z"
```
## .helmignore
Exclude files from chart package:
```
# Development files
.git/
.gitignore
*.md
docs/
# Build artifacts
*.swp
*.bak
*.tmp
*.orig
# CI/CD
.travis.yml
.gitlab-ci.yml
Jenkinsfile
# Testing
test/
*.test
# IDE
.vscode/
.idea/
*.iml
```
## Custom Resource Definitions (CRDs)
Place CRDs in `crds/` directory:
```
crds/
├── my-app-crd.yaml
└── another-crd.yaml
```
**Important CRD notes:**
- CRDs are installed before any templates
- CRDs are NOT templated (no `{{ }}` syntax)
- CRDs are NOT upgraded or deleted with chart
- Use `helm install --skip-crds` to skip installation
## Chart Versioning
### Semantic Versioning
- **Chart Version**: Increment when chart changes
- MAJOR: Breaking changes
- MINOR: New features, backward compatible
- PATCH: Bug fixes
- **App Version**: Application version being deployed
- Can be any string
- Not required to follow SemVer
```yaml
version: 2.3.1 # Chart version
appVersion: "1.5.0" # Application version
```
## Chart Testing
### Test Files
```yaml
# templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
name: "{{ include "my-app.fullname" . }}-test-connection"
annotations:
"helm.sh/hook": test
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
containers:
- name: wget
image: busybox
command: ['wget']
args: ['{{ include "my-app.fullname" . }}:{{ .Values.service.port }}']
restartPolicy: Never
```
### Running Tests
```bash
helm test my-release
helm test my-release --logs
```
## Hooks
Helm hooks allow intervention at specific points:
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "my-app.fullname" . }}-migration
annotations:
"helm.sh/hook": pre-upgrade,pre-install
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
```
### Hook Types
- `pre-install`: Before templates rendered
- `post-install`: After all resources loaded
- `pre-delete`: Before any resources deleted
- `post-delete`: After all resources deleted
- `pre-upgrade`: Before upgrade
- `post-upgrade`: After upgrade
- `pre-rollback`: Before rollback
- `post-rollback`: After rollback
- `test`: Run with `helm test`
### Hook Weight
Controls hook execution order (-5 to 5, lower runs first)
### Hook Deletion Policies
- `before-hook-creation`: Delete previous hook before new one
- `hook-succeeded`: Delete after successful execution
- `hook-failed`: Delete if hook fails
## Best Practices
1. **Use helpers** for repeated template logic
2. **Quote strings** in templates: `{{ .Values.name | quote }}`
3. **Validate values** with values.schema.json
4. **Document all values** in values.yaml
5. **Use semantic versioning** for chart versions
6. **Pin dependency versions** exactly
7. **Include NOTES.txt** with usage instructions
8. **Add tests** for critical functionality
9. **Use hooks** for database migrations
10. **Keep charts focused** - one application per chart
## Chart Repository Structure
```
helm-charts/
├── index.yaml
├── my-app-1.0.0.tgz
├── my-app-1.1.0.tgz
├── my-app-1.2.0.tgz
└── another-chart-2.0.0.tgz
```
### Creating Repository Index
```bash
helm repo index . --url https://charts.example.com
```
## Related Resources
- [Helm Documentation](https://helm.sh/docs/)
- [Chart Template Guide](https://helm.sh/docs/chart_template_guide/)
- [Best Practices](https://helm.sh/docs/chart_best_practices/)

244
plugins/kubernetes-operations/skills/helm-chart-scaffolding/scripts/validate-chart.sh Executable file
View File

@@ -0,0 +1,244 @@
#!/bin/bash
set -e
CHART_DIR="${1:-.}"
RELEASE_NAME="test-release"
echo "═══════════════════════════════════════════════════════"
echo " Helm Chart Validation"
echo "═══════════════════════════════════════════════════════"
echo ""
# Colors
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
RED='\033[0;31m'
NC='\033[0m' # No Color
success() {
echo -e "${GREEN}✓${NC} $1"
}
warning() {
echo -e "${YELLOW}⚠${NC} $1"
}
error() {
echo -e "${RED}✗${NC} $1"
}
# Check if Helm is installed
if ! command -v helm &> /dev/null; then
error "Helm is not installed"
exit 1
fi
echo "📦 Chart directory: $CHART_DIR"
echo ""
# 1. Check chart structure
echo "1️⃣ Checking chart structure..."
if [ ! -f "$CHART_DIR/Chart.yaml" ]; then
error "Chart.yaml not found"
exit 1
fi
success "Chart.yaml exists"
if [ ! -f "$CHART_DIR/values.yaml" ]; then
error "values.yaml not found"
exit 1
fi
success "values.yaml exists"
if [ ! -d "$CHART_DIR/templates" ]; then
error "templates/ directory not found"
exit 1
fi
success "templates/ directory exists"
echo ""
# 2. Lint the chart
echo "2️⃣ Linting chart..."
if helm lint "$CHART_DIR"; then
success "Chart passed lint"
else
error "Chart failed lint"
exit 1
fi
echo ""
# 3. Check Chart.yaml
echo "3️⃣ Validating Chart.yaml..."
CHART_NAME=$(grep "^name:" "$CHART_DIR/Chart.yaml" | awk '{print $2}')
CHART_VERSION=$(grep "^version:" "$CHART_DIR/Chart.yaml" | awk '{print $2}')
APP_VERSION=$(grep "^appVersion:" "$CHART_DIR/Chart.yaml" | awk '{print $2}' | tr -d '"')
if [ -z "$CHART_NAME" ]; then
error "Chart name not found"
exit 1
fi
success "Chart name: $CHART_NAME"
if [ -z "$CHART_VERSION" ]; then
error "Chart version not found"
exit 1
fi
success "Chart version: $CHART_VERSION"
if [ -z "$APP_VERSION" ]; then
warning "App version not specified"
else
success "App version: $APP_VERSION"
fi
echo ""
# 4. Test template rendering
echo "4️⃣ Testing template rendering..."
if helm template "$RELEASE_NAME" "$CHART_DIR" > /dev/null 2>&1; then
success "Templates rendered successfully"
else
error "Template rendering failed"
helm template "$RELEASE_NAME" "$CHART_DIR"
exit 1
fi
echo ""
# 5. Dry-run installation
echo "5️⃣ Testing dry-run installation..."
if helm install "$RELEASE_NAME" "$CHART_DIR" --dry-run --debug > /dev/null 2>&1; then
success "Dry-run installation successful"
else
error "Dry-run installation failed"
exit 1
fi
echo ""
# 6. Check for required Kubernetes resources
echo "6️⃣ Checking generated resources..."
MANIFESTS=$(helm template "$RELEASE_NAME" "$CHART_DIR")
if echo "$MANIFESTS" | grep -q "kind: Deployment"; then
success "Deployment found"
else
warning "No Deployment found"
fi
if echo "$MANIFESTS" | grep -q "kind: Service"; then
success "Service found"
else
warning "No Service found"
fi
if echo "$MANIFESTS" | grep -q "kind: ServiceAccount"; then
success "ServiceAccount found"
else
warning "No ServiceAccount found"
fi
echo ""
# 7. Check for security best practices
echo "7️⃣ Checking security best practices..."
if echo "$MANIFESTS" | grep -q "runAsNonRoot: true"; then
success "Running as non-root user"
else
warning "Not explicitly running as non-root"
fi
if echo "$MANIFESTS" | grep -q "readOnlyRootFilesystem: true"; then
success "Using read-only root filesystem"
else
warning "Not using read-only root filesystem"
fi
if echo "$MANIFESTS" | grep -q "allowPrivilegeEscalation: false"; then
success "Privilege escalation disabled"
else
warning "Privilege escalation not explicitly disabled"
fi
echo ""
# 8. Check for resource limits
echo "8️⃣ Checking resource configuration..."
if echo "$MANIFESTS" | grep -q "resources:"; then
if echo "$MANIFESTS" | grep -q "limits:"; then
success "Resource limits defined"
else
warning "No resource limits defined"
fi
if echo "$MANIFESTS" | grep -q "requests:"; then
success "Resource requests defined"
else
warning "No resource requests defined"
fi
else
warning "No resources defined"
fi
echo ""
# 9. Check for health probes
echo "9️⃣ Checking health probes..."
if echo "$MANIFESTS" | grep -q "livenessProbe:"; then
success "Liveness probe configured"
else
warning "No liveness probe found"
fi
if echo "$MANIFESTS" | grep -q "readinessProbe:"; then
success "Readiness probe configured"
else
warning "No readiness probe found"
fi
echo ""
# 10. Check dependencies
if [ -f "$CHART_DIR/Chart.yaml" ] && grep -q "^dependencies:" "$CHART_DIR/Chart.yaml"; then
echo "🔟 Checking dependencies..."
if helm dependency list "$CHART_DIR" > /dev/null 2>&1; then
success "Dependencies valid"
if [ -f "$CHART_DIR/Chart.lock" ]; then
success "Chart.lock file present"
else
warning "Chart.lock file missing (run 'helm dependency update')"
fi
else
error "Dependencies check failed"
fi
echo ""
fi
# 11. Check for values schema
if [ -f "$CHART_DIR/values.schema.json" ]; then
echo "1️⃣1️⃣ Validating values schema..."
success "values.schema.json present"
# Validate schema if jq is available
if command -v jq &> /dev/null; then
if jq empty "$CHART_DIR/values.schema.json" 2>/dev/null; then
success "values.schema.json is valid JSON"
else
error "values.schema.json contains invalid JSON"
exit 1
fi
fi
echo ""
fi
# Summary
echo "═══════════════════════════════════════════════════════"
echo " Validation Complete!"
echo "═══════════════════════════════════════════════════════"
echo ""
echo "Chart: $CHART_NAME"
echo "Version: $CHART_VERSION"
if [ -n "$APP_VERSION" ]; then
echo "App Version: $APP_VERSION"
fi
echo ""
success "All validations passed!"
echo ""
echo "Next steps:"
echo " • helm package $CHART_DIR"
echo " • helm install my-release $CHART_DIR"
echo " • helm test my-release"
echo ""

511
plugins/kubernetes-operations/skills/k8s-manifest-generator/SKILL.md Normal file
View File

@@ -0,0 +1,511 @@
---
name: k8s-manifest-generator
description: Create production-ready Kubernetes manifests for Deployments, Services, ConfigMaps, and Secrets following best practices and security standards. Use when generating Kubernetes YAML manifests, creating K8s resources, or implementing production-grade Kubernetes configurations.
---
# Kubernetes Manifest Generator
Step-by-step guidance for creating production-ready Kubernetes manifests including Deployments, Services, ConfigMaps, Secrets, and PersistentVolumeClaims.
## Purpose
This skill provides comprehensive guidance for generating well-structured, secure, and production-ready Kubernetes manifests following cloud-native best practices and Kubernetes conventions.
## When to Use This Skill
Use this skill when you need to:
- Create new Kubernetes Deployment manifests
- Define Service resources for network connectivity
- Generate ConfigMap and Secret resources for configuration management
- Create PersistentVolumeClaim manifests for stateful workloads
- Follow Kubernetes best practices and naming conventions
- Implement resource limits, health checks, and security contexts
- Design manifests for multi-environment deployments
## Step-by-Step Workflow
### 1. Gather Requirements
**Understand the workload:**
- Application type (stateless/stateful)
- Container image and version
- Environment variables and configuration needs
- Storage requirements
- Network exposure requirements (internal/external)
- Resource requirements (CPU, memory)
- Scaling requirements
- Health check endpoints
**Questions to ask:**
- What is the application name and purpose?
- What container image and tag will be used?
- Does the application need persistent storage?
- What ports does the application expose?
- Are there any secrets or configuration files needed?
- What are the CPU and memory requirements?
- Does the application need to be exposed externally?
### 2. Create Deployment Manifest
**Follow this structure:**
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: <app-name>
namespace: <namespace>
labels:
app: <app-name>
version: <version>
spec:
replicas: 3
selector:
matchLabels:
app: <app-name>
template:
metadata:
labels:
app: <app-name>
version: <version>
spec:
containers:
- name: <container-name>
image: <image>:<tag>
ports:
- containerPort: <port>
name: http
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 5
periodSeconds: 5
env:
- name: ENV_VAR
value: "value"
envFrom:
- configMapRef:
name: <app-name>-config
- secretRef:
name: <app-name>-secret
```
**Best practices to apply:**
- Always set resource requests and limits
- Implement both liveness and readiness probes
- Use specific image tags (never `:latest`)
- Apply security context for non-root users
- Use labels for organization and selection
- Set appropriate replica count based on availability needs
**Reference:** See `references/deployment-spec.md` for detailed deployment options
### 3. Create Service Manifest
**Choose the appropriate Service type:**
**ClusterIP (internal only):**
```yaml
apiVersion: v1
kind: Service
metadata:
name: <app-name>
namespace: <namespace>
labels:
app: <app-name>
spec:
type: ClusterIP
selector:
app: <app-name>
ports:
- name: http
port: 80
targetPort: 8080
protocol: TCP
```
**LoadBalancer (external access):**
```yaml
apiVersion: v1
kind: Service
metadata:
name: <app-name>
namespace: <namespace>
labels:
app: <app-name>
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
type: LoadBalancer
selector:
app: <app-name>
ports:
- name: http
port: 80
targetPort: 8080
protocol: TCP
```
**Reference:** See `references/service-spec.md` for service types and networking
### 4. Create ConfigMap
**For application configuration:**
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: <app-name>-config
namespace: <namespace>
data:
APP_MODE: production
LOG_LEVEL: info
DATABASE_HOST: db.example.com
# For config files
app.properties: |
server.port=8080
server.host=0.0.0.0
logging.level=INFO
```
**Best practices:**
- Use ConfigMaps for non-sensitive data only
- Organize related configuration together
- Use meaningful names for keys
- Consider using one ConfigMap per component
- Version ConfigMaps when making changes
**Reference:** See `assets/configmap-template.yaml` for examples
### 5. Create Secret
**For sensitive data:**
```yaml
apiVersion: v1
kind: Secret
metadata:
name: <app-name>-secret
namespace: <namespace>
type: Opaque
stringData:
DATABASE_PASSWORD: "changeme"
API_KEY: "secret-api-key"
# For certificate files
tls.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls.key: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
```
**Security considerations:**
- Never commit secrets to Git in plain text
- Use Sealed Secrets, External Secrets Operator, or Vault
- Rotate secrets regularly
- Use RBAC to limit secret access
- Consider using Secret type: `kubernetes.io/tls` for TLS secrets
### 6. Create PersistentVolumeClaim (if needed)
**For stateful applications:**
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: <app-name>-data
namespace: <namespace>
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp3
resources:
requests:
storage: 10Gi
```
**Mount in Deployment:**
```yaml
spec:
template:
spec:
containers:
- name: app
volumeMounts:
- name: data
mountPath: /var/lib/app
volumes:
- name: data
persistentVolumeClaim:
claimName: <app-name>-data
```
**Storage considerations:**
- Choose appropriate StorageClass for performance needs
- Use ReadWriteOnce for single-pod access
- Use ReadWriteMany for multi-pod shared storage
- Consider backup strategies
- Set appropriate retention policies
### 7. Apply Security Best Practices
**Add security context to Deployment:**
```yaml
spec:
template:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: app
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
```
**Security checklist:**
- [ ] Run as non-root user
- [ ] Drop all capabilities
- [ ] Use read-only root filesystem
- [ ] Disable privilege escalation
- [ ] Set seccomp profile
- [ ] Use Pod Security Standards
### 8. Add Labels and Annotations
**Standard labels (recommended):**
```yaml
metadata:
labels:
app.kubernetes.io/name: <app-name>
app.kubernetes.io/instance: <instance-name>
app.kubernetes.io/version: "1.0.0"
app.kubernetes.io/component: backend
app.kubernetes.io/part-of: <system-name>
app.kubernetes.io/managed-by: kubectl
```
**Useful annotations:**
```yaml
metadata:
annotations:
description: "Application description"
contact: "team@example.com"
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/metrics"
```
### 9. Organize Multi-Resource Manifests
**File organization options:**
**Option 1: Single file with `---` separator**
```yaml
# app-name.yaml
---
apiVersion: v1
kind: ConfigMap
...
---
apiVersion: v1
kind: Secret
...
---
apiVersion: apps/v1
kind: Deployment
...
---
apiVersion: v1
kind: Service
...
```
**Option 2: Separate files**
```
manifests/
├── configmap.yaml
├── secret.yaml
├── deployment.yaml
├── service.yaml
└── pvc.yaml
```
**Option 3: Kustomize structure**
```
base/
├── kustomization.yaml
├── deployment.yaml
├── service.yaml
└── configmap.yaml
overlays/
├── dev/
│ └── kustomization.yaml
└── prod/
└── kustomization.yaml
```
### 10. Validate and Test
**Validation steps:**
```bash
# Dry-run validation
kubectl apply -f manifest.yaml --dry-run=client
# Server-side validation
kubectl apply -f manifest.yaml --dry-run=server
# Validate with kubeval
kubeval manifest.yaml
# Validate with kube-score
kube-score score manifest.yaml
# Check with kube-linter
kube-linter lint manifest.yaml
```
**Testing checklist:**
- [ ] Manifest passes dry-run validation
- [ ] All required fields are present
- [ ] Resource limits are reasonable
- [ ] Health checks are configured
- [ ] Security context is set
- [ ] Labels follow conventions
- [ ] Namespace exists or is created
## Common Patterns
### Pattern 1: Simple Stateless Web Application
**Use case:** Standard web API or microservice
**Components needed:**
- Deployment (3 replicas for HA)
- ClusterIP Service
- ConfigMap for configuration
- Secret for API keys
- HorizontalPodAutoscaler (optional)
**Reference:** See `assets/deployment-template.yaml`
### Pattern 2: Stateful Database Application
**Use case:** Database or persistent storage application
**Components needed:**
- StatefulSet (not Deployment)
- Headless Service
- PersistentVolumeClaim template
- ConfigMap for DB configuration
- Secret for credentials
### Pattern 3: Background Job or Cron
**Use case:** Scheduled tasks or batch processing
**Components needed:**
- CronJob or Job
- ConfigMap for job parameters
- Secret for credentials
- ServiceAccount with RBAC
### Pattern 4: Multi-Container Pod
**Use case:** Application with sidecar containers
**Components needed:**
- Deployment with multiple containers
- Shared volumes between containers
- Init containers for setup
- Service (if needed)
## Templates
The following templates are available in the `assets/` directory:
- `deployment-template.yaml` - Standard deployment with best practices
- `service-template.yaml` - Service configurations (ClusterIP, LoadBalancer, NodePort)
- `configmap-template.yaml` - ConfigMap examples with different data types
- `secret-template.yaml` - Secret examples (to be generated, not committed)
- `pvc-template.yaml` - PersistentVolumeClaim templates
## Reference Documentation
- `references/deployment-spec.md` - Detailed Deployment specification
- `references/service-spec.md` - Service types and networking details
## Best Practices Summary
1. **Always set resource requests and limits** - Prevents resource starvation
2. **Implement health checks** - Ensures Kubernetes can manage your application
3. **Use specific image tags** - Avoid unpredictable deployments
4. **Apply security contexts** - Run as non-root, drop capabilities
5. **Use ConfigMaps and Secrets** - Separate config from code
6. **Label everything** - Enables filtering and organization
7. **Follow naming conventions** - Use standard Kubernetes labels
8. **Validate before applying** - Use dry-run and validation tools
9. **Version your manifests** - Keep in Git with version control
10. **Document with annotations** - Add context for other developers
## Troubleshooting
**Pods not starting:**
- Check image pull errors: `kubectl describe pod <pod-name>`
- Verify resource availability: `kubectl get nodes`
- Check events: `kubectl get events --sort-by='.lastTimestamp'`
**Service not accessible:**
- Verify selector matches pod labels: `kubectl get endpoints <service-name>`
- Check service type and port configuration
- Test from within cluster: `kubectl run debug --rm -it --image=busybox -- sh`
**ConfigMap/Secret not loading:**
- Verify names match in Deployment
- Check namespace
- Ensure resources exist: `kubectl get configmap,secret`
## Next Steps
After creating manifests:
1. Store in Git repository
2. Set up CI/CD pipeline for deployment
3. Consider using Helm or Kustomize for templating
4. Implement GitOps with ArgoCD or Flux
5. Add monitoring and observability
## Related Skills
- `helm-chart-scaffolding` - For templating and packaging
- `gitops-workflow` - For automated deployments
- `k8s-security-policies` - For advanced security configurations

296
plugins/kubernetes-operations/skills/k8s-manifest-generator/assets/configmap-template.yaml Normal file
View File

@@ -0,0 +1,296 @@
# Kubernetes ConfigMap Templates
---
# Template 1: Simple Key-Value Configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: <app-name>-config
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
app.kubernetes.io/instance: <instance-name>
data:
# Simple key-value pairs
APP_ENV: "production"
LOG_LEVEL: "info"
DATABASE_HOST: "db.example.com"
DATABASE_PORT: "5432"
CACHE_TTL: "3600"
MAX_CONNECTIONS: "100"
---
# Template 2: Configuration File
apiVersion: v1
kind: ConfigMap
metadata:
name: <app-name>-config-file
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
data:
# Application configuration file
application.yaml: |
server:
port: 8080
host: 0.0.0.0
logging:
level: INFO
format: json
database:
host: db.example.com
port: 5432
pool_size: 20
timeout: 30
cache:
enabled: true
ttl: 3600
max_entries: 10000
features:
new_ui: true
beta_features: false
---
# Template 3: Multiple Configuration Files
apiVersion: v1
kind: ConfigMap
metadata:
name: <app-name>-multi-config
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
data:
# Nginx configuration
nginx.conf: |
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
# Default site configuration
default.conf: |
server {
listen 80;
server_name _;
location / {
proxy_pass http://backend:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location /health {
access_log off;
return 200 "healthy\n";
}
}
---
# Template 4: JSON Configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: <app-name>-json-config
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
data:
config.json: |
{
"server": {
"port": 8080,
"host": "0.0.0.0",
"timeout": 30
},
"database": {
"host": "postgres.example.com",
"port": 5432,
"database": "myapp",
"pool": {
"min": 2,
"max": 20
}
},
"redis": {
"host": "redis.example.com",
"port": 6379,
"db": 0
},
"features": {
"auth": true,
"metrics": true,
"tracing": true
}
}
---
# Template 5: Environment-Specific Configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: <app-name>-prod-config
namespace: production
labels:
app.kubernetes.io/name: <app-name>
environment: production
data:
APP_ENV: "production"
LOG_LEVEL: "warn"
DEBUG: "false"
RATE_LIMIT: "1000"
CACHE_TTL: "3600"
DATABASE_POOL_SIZE: "50"
FEATURE_FLAG_NEW_UI: "true"
FEATURE_FLAG_BETA: "false"
---
# Template 6: Script Configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: <app-name>-scripts
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
data:
# Initialization script
init.sh: |
#!/bin/bash
set -e
echo "Running initialization..."
# Wait for database
until nc -z $DATABASE_HOST $DATABASE_PORT; do
echo "Waiting for database..."
sleep 2
done
echo "Database is ready!"
# Run migrations
if [ "$RUN_MIGRATIONS" = "true" ]; then
echo "Running database migrations..."
./migrate up
fi
echo "Initialization complete!"
# Health check script
healthcheck.sh: |
#!/bin/bash
# Check application health endpoint
response=$(curl -sf http://localhost:8080/health)
if [ $? -eq 0 ]; then
echo "Health check passed"
exit 0
else
echo "Health check failed"
exit 1
fi
---
# Template 7: Prometheus Configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: monitoring
labels:
app.kubernetes.io/name: prometheus
data:
prometheus.yml: |
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
cluster: 'production'
region: 'us-west-2'
alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager:9093
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
---
# Usage Examples:
#
# 1. Mount as environment variables:
# envFrom:
# - configMapRef:
# name: <app-name>-config
#
# 2. Mount as files:
# volumeMounts:
# - name: config
# mountPath: /etc/app
# volumes:
# - name: config
# configMap:
# name: <app-name>-config-file
#
# 3. Mount specific keys as files:
# volumes:
# - name: nginx-config
# configMap:
# name: <app-name>-multi-config
# items:
# - key: nginx.conf
# path: nginx.conf
#
# 4. Use individual environment variables:
# env:
# - name: LOG_LEVEL
# valueFrom:
# configMapKeyRef:
# name: <app-name>-config
# key: LOG_LEVEL

203
plugins/kubernetes-operations/skills/k8s-manifest-generator/assets/deployment-template.yaml Normal file
View File

@@ -0,0 +1,203 @@
# Production-Ready Kubernetes Deployment Template
# Replace all <placeholders> with actual values
apiVersion: apps/v1
kind: Deployment
metadata:
name: <app-name>
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
app.kubernetes.io/instance: <instance-name>
app.kubernetes.io/version: "<version>"
app.kubernetes.io/component: <component> # backend, frontend, database, cache
app.kubernetes.io/part-of: <system-name>
app.kubernetes.io/managed-by: kubectl
annotations:
description: "<application description>"
contact: "<team-email>"
spec:
replicas: 3 # Minimum 3 for production HA
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/name: <app-name>
app.kubernetes.io/instance: <instance-name>
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0 # Zero-downtime deployment
minReadySeconds: 10
progressDeadlineSeconds: 600
template:
metadata:
labels:
app.kubernetes.io/name: <app-name>
app.kubernetes.io/instance: <instance-name>
app.kubernetes.io/version: "<version>"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/metrics"
spec:
serviceAccountName: <app-name>
# Pod-level security context
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
# Init containers (optional)
initContainers:
- name: init-wait
image: busybox:1.36
command: ['sh', '-c', 'echo "Initializing..."']
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
containers:
- name: <container-name>
image: <registry>/<image>:<tag> # Never use :latest
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
protocol: TCP
- name: metrics
containerPort: 9090
protocol: TCP
# Environment variables
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
# Load from ConfigMap and Secret
envFrom:
- configMapRef:
name: <app-name>-config
- secretRef:
name: <app-name>-secret
# Resource limits
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
# Startup probe (for slow-starting apps)
startupProbe:
httpGet:
path: /health/startup
port: http
initialDelaySeconds: 0
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 30 # 5 minutes to start
# Liveness probe
livenessProbe:
httpGet:
path: /health/live
port: http
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
# Readiness probe
readinessProbe:
httpGet:
path: /health/ready
port: http
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3
# Volume mounts
volumeMounts:
- name: tmp
mountPath: /tmp
- name: cache
mountPath: /app/cache
# - name: data
# mountPath: /var/lib/app
# Container security context
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop:
- ALL
# Lifecycle hooks
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"] # Graceful shutdown
# Volumes
volumes:
- name: tmp
emptyDir: {}
- name: cache
emptyDir:
sizeLimit: 1Gi
# - name: data
# persistentVolumeClaim:
# claimName: <app-name>-data
# Scheduling
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: <app-name>
topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/name: <app-name>
terminationGracePeriodSeconds: 30
# Image pull secrets (if using private registry)
# imagePullSecrets:
# - name: regcred

171
plugins/kubernetes-operations/skills/k8s-manifest-generator/assets/service-template.yaml Normal file
View File

@@ -0,0 +1,171 @@
# Kubernetes Service Templates
---
# Template 1: ClusterIP Service (Internal Only)
apiVersion: v1
kind: Service
metadata:
name: <app-name>
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
app.kubernetes.io/instance: <instance-name>
annotations:
description: "Internal service for <app-name>"
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: <app-name>
app.kubernetes.io/instance: <instance-name>
ports:
- name: http
port: 80
targetPort: http # Named port from container
protocol: TCP
sessionAffinity: None
---
# Template 2: LoadBalancer Service (External Access)
apiVersion: v1
kind: Service
metadata:
name: <app-name>-lb
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
annotations:
# AWS NLB annotations
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
# SSL certificate (optional)
# service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:..."
spec:
type: LoadBalancer
externalTrafficPolicy: Local # Preserves client IP
selector:
app.kubernetes.io/name: <app-name>
ports:
- name: http
port: 80
targetPort: http
protocol: TCP
- name: https
port: 443
targetPort: https
protocol: TCP
# Restrict access to specific IPs (optional)
# loadBalancerSourceRanges:
# - 203.0.113.0/24
---
# Template 3: NodePort Service (Direct Node Access)
apiVersion: v1
kind: Service
metadata:
name: <app-name>-np
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
spec:
type: NodePort
selector:
app.kubernetes.io/name: <app-name>
ports:
- name: http
port: 80
targetPort: 8080
nodePort: 30080 # Optional, 30000-32767 range
protocol: TCP
---
# Template 4: Headless Service (StatefulSet)
apiVersion: v1
kind: Service
metadata:
name: <app-name>-headless
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
spec:
clusterIP: None # Headless
selector:
app.kubernetes.io/name: <app-name>
ports:
- name: client
port: 9042
targetPort: 9042
publishNotReadyAddresses: true # Include not-ready pods in DNS
---
# Template 5: Multi-Port Service with Metrics
apiVersion: v1
kind: Service
metadata:
name: <app-name>-multi
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/metrics"
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: <app-name>
ports:
- name: http
port: 80
targetPort: 8080
protocol: TCP
- name: https
port: 443
targetPort: 8443
protocol: TCP
- name: grpc
port: 9090
targetPort: 9090
protocol: TCP
- name: metrics
port: 9091
targetPort: 9091
protocol: TCP
---
# Template 6: Service with Session Affinity
apiVersion: v1
kind: Service
metadata:
name: <app-name>-sticky
namespace: <namespace>
labels:
app.kubernetes.io/name: <app-name>
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: <app-name>
ports:
- name: http
port: 80
targetPort: 8080
protocol: TCP
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800 # 3 hours
---
# Template 7: ExternalName Service (External Service Mapping)
apiVersion: v1
kind: Service
metadata:
name: external-db
namespace: <namespace>
spec:
type: ExternalName
externalName: db.example.com
ports:
- port: 5432
targetPort: 5432
protocol: TCP

753
plugins/kubernetes-operations/skills/k8s-manifest-generator/references/deployment-spec.md Normal file
View File

@@ -0,0 +1,753 @@
# Kubernetes Deployment Specification Reference
Comprehensive reference for Kubernetes Deployment resources, covering all key fields, best practices, and common patterns.
## Overview
A Deployment provides declarative updates for Pods and ReplicaSets. It manages the desired state of your application, handling rollouts, rollbacks, and scaling operations.
## Complete Deployment Specification
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
namespace: production
labels:
app.kubernetes.io/name: my-app
app.kubernetes.io/version: "1.0.0"
app.kubernetes.io/component: backend
app.kubernetes.io/part-of: my-system
annotations:
description: "Main application deployment"
contact: "backend-team@example.com"
spec:
# Replica management
replicas: 3
revisionHistoryLimit: 10
# Pod selection
selector:
matchLabels:
app: my-app
version: v1
# Update strategy
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
# Minimum time for pod to be ready
minReadySeconds: 10
# Deployment will fail if it doesn't progress in this time
progressDeadlineSeconds: 600
# Pod template
template:
metadata:
labels:
app: my-app
version: v1
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
spec:
# Service account for RBAC
serviceAccountName: my-app
# Security context for the pod
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
# Init containers run before main containers
initContainers:
- name: init-db
image: busybox:1.36
command: ['sh', '-c', 'until nc -z db-service 5432; do sleep 1; done']
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
# Main containers
containers:
- name: app
image: myapp:1.0.0
imagePullPolicy: IfNotPresent
# Container ports
ports:
- name: http
containerPort: 8080
protocol: TCP
- name: metrics
containerPort: 9090
protocol: TCP
# Environment variables
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: db-credentials
key: url
# ConfigMap and Secret references
envFrom:
- configMapRef:
name: app-config
- secretRef:
name: app-secrets
# Resource requests and limits
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
# Liveness probe
livenessProbe:
httpGet:
path: /health/live
port: http
httpHeaders:
- name: Custom-Header
value: Awesome
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
# Readiness probe
readinessProbe:
httpGet:
path: /health/ready
port: http
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 3
# Startup probe (for slow-starting containers)
startupProbe:
httpGet:
path: /health/startup
port: http
initialDelaySeconds: 0
periodSeconds: 10
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 30
# Volume mounts
volumeMounts:
- name: data
mountPath: /var/lib/app
- name: config
mountPath: /etc/app
readOnly: true
- name: tmp
mountPath: /tmp
# Security context for container
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop:
- ALL
# Lifecycle hooks
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Container started > /tmp/started"]
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]
# Volumes
volumes:
- name: data
persistentVolumeClaim:
claimName: app-data
- name: config
configMap:
name: app-config
- name: tmp
emptyDir: {}
# DNS configuration
dnsPolicy: ClusterFirst
dnsConfig:
options:
- name: ndots
value: "2"
# Scheduling
nodeSelector:
disktype: ssd
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- my-app
topologyKey: kubernetes.io/hostname
tolerations:
- key: "app"
operator: "Equal"
value: "my-app"
effect: "NoSchedule"
# Termination
terminationGracePeriodSeconds: 30
# Image pull secrets
imagePullSecrets:
- name: regcred
```
## Field Reference
### Metadata Fields
#### Required Fields
- `apiVersion`: `apps/v1` (current stable version)
- `kind`: `Deployment`
- `metadata.name`: Unique name within namespace
#### Recommended Metadata
- `metadata.namespace`: Target namespace (defaults to `default`)
- `metadata.labels`: Key-value pairs for organization
- `metadata.annotations`: Non-identifying metadata
### Spec Fields
#### Replica Management
**`replicas`** (integer, default: 1)
- Number of desired pod instances
- Best practice: Use 3+ for production high availability
- Can be scaled manually or via HorizontalPodAutoscaler
**`revisionHistoryLimit`** (integer, default: 10)
- Number of old ReplicaSets to retain for rollback
- Set to 0 to disable rollback capability
- Reduces storage overhead for long-running deployments
#### Update Strategy
**`strategy.type`** (string)
- `RollingUpdate` (default): Gradual pod replacement
- `Recreate`: Delete all pods before creating new ones
**`strategy.rollingUpdate.maxSurge`** (int or percent, default: 25%)
- Maximum pods above desired replicas during update
- Example: With 3 replicas and maxSurge=1, up to 4 pods during update
**`strategy.rollingUpdate.maxUnavailable`** (int or percent, default: 25%)
- Maximum pods below desired replicas during update
- Set to 0 for zero-downtime deployments
- Cannot be 0 if maxSurge is 0
**Best practices:**
```yaml
# Zero-downtime deployment
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
# Fast deployment (can have brief downtime)
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2
maxUnavailable: 1
# Complete replacement
strategy:
type: Recreate
```
#### Pod Template
**`template.metadata.labels`**
- Must include labels matching `spec.selector.matchLabels`
- Add version labels for blue/green deployments
- Include standard Kubernetes labels
**`template.spec.containers`** (required)
- Array of container specifications
- At least one container required
- Each container needs unique name
#### Container Configuration
**Image Management:**
```yaml
containers:
- name: app
image: registry.example.com/myapp:1.0.0
imagePullPolicy: IfNotPresent # or Always, Never
```
Image pull policies:
- `IfNotPresent`: Pull if not cached (default for tagged images)
- `Always`: Always pull (default for :latest)
- `Never`: Never pull, fail if not cached
**Port Declarations:**
```yaml
ports:
- name: http # Named for referencing in Service
containerPort: 8080
protocol: TCP # TCP (default), UDP, or SCTP
hostPort: 8080 # Optional: Bind to host port (rarely used)
```
#### Resource Management
**Requests vs Limits:**
```yaml
resources:
requests:
memory: "256Mi" # Guaranteed resources
cpu: "250m" # 0.25 CPU cores
limits:
memory: "512Mi" # Maximum allowed
cpu: "500m" # 0.5 CPU cores
```
**QoS Classes (determined automatically):**
1. **Guaranteed**: requests = limits for all containers
- Highest priority
- Last to be evicted
2. **Burstable**: requests < limits or only requests set
- Medium priority
- Evicted before Guaranteed
3. **BestEffort**: No requests or limits set
- Lowest priority
- First to be evicted
**Best practices:**
- Always set requests in production
- Set limits to prevent resource monopolization
- Memory limits should be 1.5-2x requests
- CPU limits can be higher for bursty workloads
#### Health Checks
**Probe Types:**
1. **startupProbe** - For slow-starting applications
```yaml
startupProbe:
httpGet:
path: /health/startup
port: 8080
initialDelaySeconds: 0
periodSeconds: 10
failureThreshold: 30 # 5 minutes to start (10s * 30)
```
2. **livenessProbe** - Restarts unhealthy containers
```yaml
livenessProbe:
httpGet:
path: /health/live
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3 # Restart after 3 failures
```
3. **readinessProbe** - Controls traffic routing
```yaml
readinessProbe:
httpGet:
path: /health/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3 # Remove from service after 3 failures
```
**Probe Mechanisms:**
```yaml
# HTTP GET
httpGet:
path: /health
port: 8080
httpHeaders:
- name: Authorization
value: Bearer token
# TCP Socket
tcpSocket:
port: 3306
# Command execution
exec:
command:
- cat
- /tmp/healthy
# gRPC (Kubernetes 1.24+)
grpc:
port: 9090
service: my.service.health.v1.Health
```
**Probe Timing Parameters:**
- `initialDelaySeconds`: Wait before first probe
- `periodSeconds`: How often to probe
- `timeoutSeconds`: Probe timeout
- `successThreshold`: Successes needed to mark healthy (1 for liveness/startup)
- `failureThreshold`: Failures before taking action
#### Security Context
**Pod-level security context:**
```yaml
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
```
**Container-level security context:**
```yaml
containers:
- name: app
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE # Only if needed
```
**Security best practices:**
- Always run as non-root (`runAsNonRoot: true`)
- Drop all capabilities and add only needed ones
- Use read-only root filesystem when possible
- Enable seccomp profile
- Disable privilege escalation
#### Volumes
**Volume Types:**
```yaml
volumes:
# PersistentVolumeClaim
- name: data
persistentVolumeClaim:
claimName: app-data
# ConfigMap
- name: config
configMap:
name: app-config
items:
- key: app.properties
path: application.properties
# Secret
- name: secrets
secret:
secretName: app-secrets
defaultMode: 0400
# EmptyDir (ephemeral)
- name: cache
emptyDir:
sizeLimit: 1Gi
# HostPath (avoid in production)
- name: host-data
hostPath:
path: /data
type: DirectoryOrCreate
```
#### Scheduling
**Node Selection:**
```yaml
# Simple node selector
nodeSelector:
disktype: ssd
zone: us-west-1a
# Node affinity (more expressive)
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- amd64
- arm64
```
**Pod Affinity/Anti-Affinity:**
```yaml
# Spread pods across nodes
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: my-app
topologyKey: kubernetes.io/hostname
# Co-locate with database
affinity:
podAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: database
topologyKey: kubernetes.io/hostname
```
**Tolerations:**
```yaml
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 30
- key: "dedicated"
operator: "Equal"
value: "database"
effect: "NoSchedule"
```
## Common Patterns
### High Availability Deployment
```yaml
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: my-app
topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: my-app
```
### Sidecar Container Pattern
```yaml
spec:
template:
spec:
containers:
- name: app
image: myapp:1.0.0
volumeMounts:
- name: shared-logs
mountPath: /var/log
- name: log-forwarder
image: fluent-bit:2.0
volumeMounts:
- name: shared-logs
mountPath: /var/log
readOnly: true
volumes:
- name: shared-logs
emptyDir: {}
```
### Init Container for Dependencies
```yaml
spec:
template:
spec:
initContainers:
- name: wait-for-db
image: busybox:1.36
command:
- sh
- -c
- |
until nc -z database-service 5432; do
echo "Waiting for database..."
sleep 2
done
- name: run-migrations
image: myapp:1.0.0
command: ["./migrate", "up"]
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: db-credentials
key: url
containers:
- name: app
image: myapp:1.0.0
```
## Best Practices
### Production Checklist
- [ ] Set resource requests and limits
- [ ] Implement all three probe types (startup, liveness, readiness)
- [ ] Use specific image tags (not :latest)
- [ ] Configure security context (non-root, read-only filesystem)
- [ ] Set replica count >= 3 for HA
- [ ] Configure pod anti-affinity for spread
- [ ] Set appropriate update strategy (maxUnavailable: 0 for zero-downtime)
- [ ] Use ConfigMaps and Secrets for configuration
- [ ] Add standard labels and annotations
- [ ] Configure graceful shutdown (preStop hook, terminationGracePeriodSeconds)
- [ ] Set revisionHistoryLimit for rollback capability
- [ ] Use ServiceAccount with minimal RBAC permissions
### Performance Tuning
**Fast startup:**
```yaml
spec:
minReadySeconds: 5
strategy:
rollingUpdate:
maxSurge: 2
maxUnavailable: 1
```
**Zero-downtime updates:**
```yaml
spec:
minReadySeconds: 10
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
```
**Graceful shutdown:**
```yaml
spec:
template:
spec:
terminationGracePeriodSeconds: 60
containers:
- name: app
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15 && kill -SIGTERM 1"]
```
## Troubleshooting
### Common Issues
**Pods not starting:**
```bash
kubectl describe deployment <name>
kubectl get pods -l app=<app-name>
kubectl describe pod <pod-name>
kubectl logs <pod-name>
```
**ImagePullBackOff:**
- Check image name and tag
- Verify imagePullSecrets
- Check registry credentials
**CrashLoopBackOff:**
- Check container logs
- Verify liveness probe is not too aggressive
- Check resource limits
- Verify application dependencies
**Deployment stuck in progress:**
- Check progressDeadlineSeconds
- Verify readiness probes
- Check resource availability
## Related Resources
- [Kubernetes Deployment API Reference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#deployment-v1-apps)
- [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/)
- [Resource Management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)

724
plugins/kubernetes-operations/skills/k8s-manifest-generator/references/service-spec.md Normal file
View File

@@ -0,0 +1,724 @@
# Kubernetes Service Specification Reference
Comprehensive reference for Kubernetes Service resources, covering service types, networking, load balancing, and service discovery patterns.
## Overview
A Service provides stable network endpoints for accessing Pods. Services enable loose coupling between microservices by providing service discovery and load balancing.
## Service Types
### 1. ClusterIP (Default)
Exposes the service on an internal cluster IP. Only reachable from within the cluster.
```yaml
apiVersion: v1
kind: Service
metadata:
name: backend-service
namespace: production
spec:
type: ClusterIP
selector:
app: backend
ports:
- name: http
port: 80
targetPort: 8080
protocol: TCP
sessionAffinity: None
```
**Use cases:**
- Internal microservice communication
- Database services
- Internal APIs
- Message queues
### 2. NodePort
Exposes the service on each Node's IP at a static port (30000-32767 range).
```yaml
apiVersion: v1
kind: Service
metadata:
name: frontend-service
spec:
type: NodePort
selector:
app: frontend
ports:
- name: http
port: 80
targetPort: 8080
nodePort: 30080 # Optional, auto-assigned if omitted
protocol: TCP
```
**Use cases:**
- Development/testing external access
- Small deployments without load balancer
- Direct node access requirements
**Limitations:**
- Limited port range (30000-32767)
- Must handle node failures
- No built-in load balancing across nodes
### 3. LoadBalancer
Exposes the service using a cloud provider's load balancer.
```yaml
apiVersion: v1
kind: Service
metadata:
name: public-api
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
spec:
type: LoadBalancer
selector:
app: api
ports:
- name: https
port: 443
targetPort: 8443
protocol: TCP
loadBalancerSourceRanges:
- 203.0.113.0/24
```
**Cloud-specific annotations:**
**AWS:**
```yaml
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb" # or "external"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:..."
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
```
**Azure:**
```yaml
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
service.beta.kubernetes.io/azure-pip-name: "my-public-ip"
```
**GCP:**
```yaml
annotations:
cloud.google.com/load-balancer-type: "Internal"
cloud.google.com/backend-config: '{"default": "my-backend-config"}'
```
### 4. ExternalName
Maps service to external DNS name (CNAME record).
```yaml
apiVersion: v1
kind: Service
metadata:
name: external-db
spec:
type: ExternalName
externalName: db.external.example.com
ports:
- port: 5432
```
**Use cases:**
- Accessing external services
- Service migration scenarios
- Multi-cluster service references
## Complete Service Specification
```yaml
apiVersion: v1
kind: Service
metadata:
name: my-service
namespace: production
labels:
app: my-app
tier: backend
annotations:
description: "Main application service"
prometheus.io/scrape: "true"
spec:
# Service type
type: ClusterIP
# Pod selector
selector:
app: my-app
version: v1
# Ports configuration
ports:
- name: http
port: 80 # Service port
targetPort: 8080 # Container port (or named port)
protocol: TCP # TCP, UDP, or SCTP
# Session affinity
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
# IP configuration
clusterIP: 10.0.0.10 # Optional: specific IP
clusterIPs:
- 10.0.0.10
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
# External traffic policy
externalTrafficPolicy: Local
# Internal traffic policy
internalTrafficPolicy: Local
# Health check
healthCheckNodePort: 30000
# Load balancer config (for type: LoadBalancer)
loadBalancerIP: 203.0.113.100
loadBalancerSourceRanges:
- 203.0.113.0/24
# External IPs
externalIPs:
- 80.11.12.10
# Publishing strategy
publishNotReadyAddresses: false
```
## Port Configuration
### Named Ports
Use named ports in Pods for flexibility:
**Deployment:**
```yaml
spec:
template:
spec:
containers:
- name: app
ports:
- name: http
containerPort: 8080
- name: metrics
containerPort: 9090
```
**Service:**
```yaml
spec:
ports:
- name: http
port: 80
targetPort: http # References named port
- name: metrics
port: 9090
targetPort: metrics
```
### Multiple Ports
```yaml
spec:
ports:
- name: http
port: 80
targetPort: 8080
protocol: TCP
- name: https
port: 443
targetPort: 8443
protocol: TCP
- name: grpc
port: 9090
targetPort: 9090
protocol: TCP
```
## Session Affinity
### None (Default)
Distributes requests randomly across pods.
```yaml
spec:
sessionAffinity: None
```
### ClientIP
Routes requests from same client IP to same pod.
```yaml
spec:
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800 # 3 hours
```
**Use cases:**
- Stateful applications
- Session-based applications
- WebSocket connections
## Traffic Policies
### External Traffic Policy
**Cluster (Default):**
```yaml
spec:
externalTrafficPolicy: Cluster
```
- Load balances across all nodes
- May add extra network hop
- Source IP is masked
**Local:**
```yaml
spec:
externalTrafficPolicy: Local
```
- Traffic goes only to pods on receiving node
- Preserves client source IP
- Better performance (no extra hop)
- May cause imbalanced load
### Internal Traffic Policy
```yaml
spec:
internalTrafficPolicy: Local # or Cluster
```
Controls traffic routing for cluster-internal clients.
## Headless Services
Service without cluster IP for direct pod access.
```yaml
apiVersion: v1
kind: Service
metadata:
name: database
spec:
clusterIP: None # Headless
selector:
app: database
ports:
- port: 5432
targetPort: 5432
```
**Use cases:**
- StatefulSet pod discovery
- Direct pod-to-pod communication
- Custom load balancing
- Database clusters
**DNS returns:**
- Individual pod IPs instead of service IP
- Format: `<pod-name>.<service-name>.<namespace>.svc.cluster.local`
## Service Discovery
### DNS
**ClusterIP Service:**
```
<service-name>.<namespace>.svc.cluster.local
```
Example:
```bash
curl http://backend-service.production.svc.cluster.local
```
**Within same namespace:**
```bash
curl http://backend-service
```
**Headless Service (returns pod IPs):**
```
<pod-name>.<service-name>.<namespace>.svc.cluster.local
```
### Environment Variables
Kubernetes injects service info into pods:
```bash
# Service host and port
BACKEND_SERVICE_SERVICE_HOST=10.0.0.100
BACKEND_SERVICE_SERVICE_PORT=80
# For named ports
BACKEND_SERVICE_SERVICE_PORT_HTTP=80
```
**Note:** Pods must be created after the service for env vars to be injected.
## Load Balancing
### Algorithms
Kubernetes uses random selection by default. For advanced load balancing:
**Service Mesh (Istio example):**
```yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: my-destination-rule
spec:
host: my-service
trafficPolicy:
loadBalancer:
simple: LEAST_REQUEST # or ROUND_ROBIN, RANDOM, PASSTHROUGH
connectionPool:
tcp:
maxConnections: 100
```
### Connection Limits
Use pod disruption budgets and resource limits:
```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: my-app-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: my-app
```
## Service Mesh Integration
### Istio Virtual Service
```yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: my-service
spec:
hosts:
- my-service
http:
- match:
- headers:
version:
exact: v2
route:
- destination:
host: my-service
subset: v2
- route:
- destination:
host: my-service
subset: v1
weight: 90
- destination:
host: my-service
subset: v2
weight: 10
```
## Common Patterns
### Pattern 1: Internal Microservice
```yaml
apiVersion: v1
kind: Service
metadata:
name: user-service
namespace: backend
labels:
app: user-service
tier: backend
spec:
type: ClusterIP
selector:
app: user-service
ports:
- name: http
port: 8080
targetPort: http
protocol: TCP
- name: grpc
port: 9090
targetPort: grpc
protocol: TCP
```
### Pattern 2: Public API with Load Balancer
```yaml
apiVersion: v1
kind: Service
metadata:
name: api-gateway
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:..."
spec:
type: LoadBalancer
externalTrafficPolicy: Local
selector:
app: api-gateway
ports:
- name: https
port: 443
targetPort: 8443
protocol: TCP
loadBalancerSourceRanges:
- 0.0.0.0/0
```
### Pattern 3: StatefulSet with Headless Service
```yaml
apiVersion: v1
kind: Service
metadata:
name: cassandra
spec:
clusterIP: None
selector:
app: cassandra
ports:
- port: 9042
targetPort: 9042
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cassandra
spec:
serviceName: cassandra
replicas: 3
selector:
matchLabels:
app: cassandra
template:
metadata:
labels:
app: cassandra
spec:
containers:
- name: cassandra
image: cassandra:4.0
```
### Pattern 4: External Service Mapping
```yaml
apiVersion: v1
kind: Service
metadata:
name: external-database
spec:
type: ExternalName
externalName: prod-db.cxyz.us-west-2.rds.amazonaws.com
---
# Or with Endpoints for IP-based external service
apiVersion: v1
kind: Service
metadata:
name: external-api
spec:
ports:
- port: 443
targetPort: 443
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: external-api
subsets:
- addresses:
- ip: 203.0.113.100
ports:
- port: 443
```
### Pattern 5: Multi-Port Service with Metrics
```yaml
apiVersion: v1
kind: Service
metadata:
name: web-app
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/metrics"
spec:
type: ClusterIP
selector:
app: web-app
ports:
- name: http
port: 80
targetPort: 8080
- name: metrics
port: 9090
targetPort: 9090
```
## Network Policies
Control traffic to services:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-backend
spec:
podSelector:
matchLabels:
app: backend
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080
```
## Best Practices
### Service Configuration
1. **Use named ports** for flexibility
2. **Set appropriate service type** based on exposure needs
3. **Use labels and selectors consistently** across Deployments and Services
4. **Configure session affinity** for stateful apps
5. **Set external traffic policy to Local** for IP preservation
6. **Use headless services** for StatefulSets
7. **Implement network policies** for security
8. **Add monitoring annotations** for observability
### Production Checklist
- [ ] Service type appropriate for use case
- [ ] Selector matches pod labels
- [ ] Named ports used for clarity
- [ ] Session affinity configured if needed
- [ ] Traffic policy set appropriately
- [ ] Load balancer annotations configured (if applicable)
- [ ] Source IP ranges restricted (for public services)
- [ ] Health check configuration validated
- [ ] Monitoring annotations added
- [ ] Network policies defined
### Performance Tuning
**For high traffic:**
```yaml
spec:
externalTrafficPolicy: Local
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 3600
```
**For WebSocket/long connections:**
```yaml
spec:
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 86400 # 24 hours
```
## Troubleshooting
### Service not accessible
```bash
# Check service exists
kubectl get service <service-name>
# Check endpoints (should show pod IPs)
kubectl get endpoints <service-name>
# Describe service
kubectl describe service <service-name>
# Check if pods match selector
kubectl get pods -l app=<app-name>
```
**Common issues:**
- Selector doesn't match pod labels
- No pods running (endpoints empty)
- Ports misconfigured
- Network policy blocking traffic
### DNS resolution failing
```bash
# Test DNS from pod
kubectl run debug --rm -it --image=busybox -- nslookup <service-name>
# Check CoreDNS
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns
```
### Load balancer issues
```bash
# Check load balancer status
kubectl describe service <service-name>
# Check events
kubectl get events --sort-by='.lastTimestamp'
# Verify cloud provider configuration
kubectl describe node
```
## Related Resources
- [Kubernetes Service API Reference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#service-v1-core)
- [Service Networking](https://kubernetes.io/docs/concepts/services-networking/service/)
- [DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/)

334
plugins/kubernetes-operations/skills/k8s-security-policies/SKILL.md Normal file
View File

@@ -0,0 +1,334 @@
---
name: k8s-security-policies
description: Implement Kubernetes security policies including NetworkPolicy, PodSecurityPolicy, and RBAC for production-grade security. Use when securing Kubernetes clusters, implementing network isolation, or enforcing pod security standards.
---
# Kubernetes Security Policies
Comprehensive guide for implementing NetworkPolicy, PodSecurityPolicy, RBAC, and Pod Security Standards in Kubernetes.
## Purpose
Implement defense-in-depth security for Kubernetes clusters using network policies, pod security standards, and RBAC.
## When to Use This Skill
- Implement network segmentation
- Configure pod security standards
- Set up RBAC for least-privilege access
- Create security policies for compliance
- Implement admission control
- Secure multi-tenant clusters
## Pod Security Standards
### 1. Privileged (Unrestricted)
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: privileged-ns
labels:
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/audit: privileged
pod-security.kubernetes.io/warn: privileged
```
### 2. Baseline (Minimally restrictive)
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: baseline-ns
labels:
pod-security.kubernetes.io/enforce: baseline
pod-security.kubernetes.io/audit: baseline
pod-security.kubernetes.io/warn: baseline
```
### 3. Restricted (Most restrictive)
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: restricted-ns
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
```
## Network Policies
### Default Deny All
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
```
### Allow Frontend to Backend
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-backend
namespace: production
spec:
podSelector:
matchLabels:
app: backend
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080
```
### Allow DNS
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: production
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
name: kube-system
ports:
- protocol: UDP
port: 53
```
**Reference:** See `assets/network-policy-template.yaml`
## RBAC Configuration
### Role (Namespace-scoped)
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: production
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]
```
### ClusterRole (Cluster-wide)
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "watch", "list"]
```
### RoleBinding
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: production
subjects:
- kind: User
name: jane
apiGroup: rbac.authorization.k8s.io
- kind: ServiceAccount
name: default
namespace: production
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
```
**Reference:** See `references/rbac-patterns.md`
## Pod Security Context
### Restricted Pod
```yaml
apiVersion: v1
kind: Pod
metadata:
name: secure-pod
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: myapp:1.0
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
```
## Policy Enforcement with OPA Gatekeeper
### ConstraintTemplate
```yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredlabels
spec:
crd:
spec:
names:
kind: K8sRequiredLabels
validation:
openAPIV3Schema:
type: object
properties:
labels:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlabels
violation[{"msg": msg, "details": {"missing_labels": missing}}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[_]}
missing := required - provided
count(missing) > 0
msg := sprintf("missing required labels: %v", [missing])
}
```
### Constraint
```yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: require-app-label
spec:
match:
kinds:
- apiGroups: ["apps"]
kinds: ["Deployment"]
parameters:
labels: ["app", "environment"]
```
## Service Mesh Security (Istio)
### PeerAuthentication (mTLS)
```yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: production
spec:
mtls:
mode: STRICT
```
### AuthorizationPolicy
```yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: allow-frontend
namespace: production
spec:
selector:
matchLabels:
app: backend
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/production/sa/frontend"]
```
## Best Practices
1. **Implement Pod Security Standards** at namespace level
2. **Use Network Policies** for network segmentation
3. **Apply least-privilege RBAC** for all service accounts
4. **Enable admission control** (OPA Gatekeeper/Kyverno)
5. **Run containers as non-root**
6. **Use read-only root filesystem**
7. **Drop all capabilities** unless needed
8. **Implement resource quotas** and limit ranges
9. **Enable audit logging** for security events
10. **Regular security scanning** of images
## Compliance Frameworks
### CIS Kubernetes Benchmark
- Use RBAC authorization
- Enable audit logging
- Use Pod Security Standards
- Configure network policies
- Implement secrets encryption at rest
- Enable node authentication
### NIST Cybersecurity Framework
- Implement defense in depth
- Use network segmentation
- Configure security monitoring
- Implement access controls
- Enable logging and monitoring
## Troubleshooting
**NetworkPolicy not working:**
```bash
# Check if CNI supports NetworkPolicy
kubectl get nodes -o wide
kubectl describe networkpolicy <name>
```
**RBAC permission denied:**
```bash
# Check effective permissions
kubectl auth can-i list pods --as system:serviceaccount:default:my-sa
kubectl auth can-i '*' '*' --as system:serviceaccount:default:my-sa
```
## Reference Files
- `assets/network-policy-template.yaml` - Network policy examples
- `assets/pod-security-template.yaml` - Pod security policies
- `references/rbac-patterns.md` - RBAC configuration patterns
## Related Skills
- `k8s-manifest-generator` - For creating secure manifests
- `gitops-workflow` - For automated policy deployment

177
plugins/kubernetes-operations/skills/k8s-security-policies/assets/network-policy-template.yaml Normal file
View File

@@ -0,0 +1,177 @@
# Network Policy Templates
---
# Template 1: Default Deny All (Start Here)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: <namespace>
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# Template 2: Allow DNS (Essential)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: <namespace>
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
name: kube-system
ports:
- protocol: UDP
port: 53
---
# Template 3: Frontend to Backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-backend
namespace: <namespace>
spec:
podSelector:
matchLabels:
app: backend
tier: backend
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
tier: frontend
ports:
- protocol: TCP
port: 8080
- protocol: TCP
port: 9090
---
# Template 4: Allow Ingress Controller
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-controller
namespace: <namespace>
spec:
podSelector:
matchLabels:
app: web
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: ingress-nginx
ports:
- protocol: TCP
port: 80
- protocol: TCP
port: 443
---
# Template 5: Allow Monitoring (Prometheus)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-prometheus-scraping
namespace: <namespace>
spec:
podSelector:
matchLabels:
prometheus.io/scrape: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: monitoring
ports:
- protocol: TCP
port: 9090
---
# Template 6: Allow External HTTPS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-external-https
namespace: <namespace>
spec:
podSelector:
matchLabels:
app: api-client
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32 # Block metadata service
ports:
- protocol: TCP
port: 443
---
# Template 7: Database Access
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-app-to-database
namespace: <namespace>
spec:
podSelector:
matchLabels:
app: postgres
tier: database
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
tier: backend
ports:
- protocol: TCP
port: 5432
---
# Template 8: Cross-Namespace Communication
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-prod-namespace
namespace: <namespace>
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
environment: production
podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080

187
plugins/kubernetes-operations/skills/k8s-security-policies/references/rbac-patterns.md Normal file
View File

@@ -0,0 +1,187 @@
# RBAC Patterns and Best Practices
## Common RBAC Patterns
### Pattern 1: Read-Only Access
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: read-only
rules:
- apiGroups: ["", "apps", "batch"]
resources: ["*"]
verbs: ["get", "list", "watch"]
```
### Pattern 2: Namespace Admin
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: namespace-admin
namespace: production
rules:
- apiGroups: ["", "apps", "batch", "extensions"]
resources: ["*"]
verbs: ["*"]
```
### Pattern 3: Deployment Manager
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: deployment-manager
namespace: production
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
```
### Pattern 4: Secret Reader (ServiceAccount)
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: secret-reader
namespace: production
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
resourceNames: ["app-secrets"] # Specific secret only
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: app-secret-reader
namespace: production
subjects:
- kind: ServiceAccount
name: my-app
namespace: production
roleRef:
kind: Role
name: secret-reader
apiGroup: rbac.authorization.k8s.io
```
### Pattern 5: CI/CD Pipeline Access
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cicd-deployer
rules:
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list", "create", "update", "patch"]
- apiGroups: [""]
resources: ["services", "configmaps"]
verbs: ["get", "list", "create", "update", "patch"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
```
## ServiceAccount Best Practices
### Create Dedicated ServiceAccounts
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-app
namespace: production
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
spec:
serviceAccountName: my-app
automountServiceAccountToken: false # Disable if not needed
```
### Least-Privilege ServiceAccount
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: my-app-role
namespace: production
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get"]
resourceNames: ["my-app-config"]
```
## Security Best Practices
1. **Use Roles over ClusterRoles** when possible
2. **Specify resourceNames** for fine-grained access
3. **Avoid wildcard permissions** (`*`) in production
4. **Create dedicated ServiceAccounts** for each app
5. **Disable token auto-mounting** if not needed
6. **Regular RBAC audits** to remove unused permissions
7. **Use groups** for user management
8. **Implement namespace isolation**
9. **Monitor RBAC usage** with audit logs
10. **Document role purposes** in metadata
## Troubleshooting RBAC
### Check User Permissions
```bash
kubectl auth can-i list pods --as john@example.com
kubectl auth can-i '*' '*' --as system:serviceaccount:default:my-app
```
### View Effective Permissions
```bash
kubectl describe clusterrole cluster-admin
kubectl describe rolebinding -n production
```
### Debug Access Issues
```bash
kubectl get rolebindings,clusterrolebindings --all-namespaces -o wide | grep my-user
```
## Common RBAC Verbs
- `get` - Read a specific resource
- `list` - List all resources of a type
- `watch` - Watch for resource changes
- `create` - Create new resources
- `update` - Update existing resources
- `patch` - Partially update resources
- `delete` - Delete resources
- `deletecollection` - Delete multiple resources
- `*` - All verbs (avoid in production)
## Resource Scope
### Cluster-Scoped Resources
- Nodes
- PersistentVolumes
- ClusterRoles
- ClusterRoleBindings
- Namespaces
### Namespace-Scoped Resources
- Pods
- Services
- Deployments
- ConfigMaps
- Secrets
- Roles
- RoleBindings

338
plugins/llm-application-dev/skills/langchain-architecture/SKILL.md Normal file
View File

@@ -0,0 +1,338 @@
---
name: langchain-architecture
description: Design LLM applications using the LangChain framework with agents, memory, and tool integration patterns. Use when building LangChain applications, implementing AI agents, or creating complex LLM workflows.
---
# LangChain Architecture
Master the LangChain framework for building sophisticated LLM applications with agents, chains, memory, and tool integration.
## When to Use This Skill
- Building autonomous AI agents with tool access
- Implementing complex multi-step LLM workflows
- Managing conversation memory and state
- Integrating LLMs with external data sources and APIs
- Creating modular, reusable LLM application components
- Implementing document processing pipelines
- Building production-grade LLM applications
## Core Concepts
### 1. Agents
Autonomous systems that use LLMs to decide which actions to take.
**Agent Types:**
- **ReAct**: Reasoning + Acting in interleaved manner
- **OpenAI Functions**: Leverages function calling API
- **Structured Chat**: Handles multi-input tools
- **Conversational**: Optimized for chat interfaces
- **Self-Ask with Search**: Decomposes complex queries
### 2. Chains
Sequences of calls to LLMs or other utilities.
**Chain Types:**
- **LLMChain**: Basic prompt + LLM combination
- **SequentialChain**: Multiple chains in sequence
- **RouterChain**: Routes inputs to specialized chains
- **TransformChain**: Data transformations between steps
- **MapReduceChain**: Parallel processing with aggregation
### 3. Memory
Systems for maintaining context across interactions.
**Memory Types:**
- **ConversationBufferMemory**: Stores all messages
- **ConversationSummaryMemory**: Summarizes older messages
- **ConversationBufferWindowMemory**: Keeps last N messages
- **EntityMemory**: Tracks information about entities
- **VectorStoreMemory**: Semantic similarity retrieval
### 4. Document Processing
Loading, transforming, and storing documents for retrieval.
**Components:**
- **Document Loaders**: Load from various sources
- **Text Splitters**: Chunk documents intelligently
- **Vector Stores**: Store and retrieve embeddings
- **Retrievers**: Fetch relevant documents
- **Indexes**: Organize documents for efficient access
### 5. Callbacks
Hooks for logging, monitoring, and debugging.
**Use Cases:**
- Request/response logging
- Token usage tracking
- Latency monitoring
- Error handling
- Custom metrics collection
## Quick Start
```python
from langchain.agents import AgentType, initialize_agent, load_tools
from langchain.llms import OpenAI
from langchain.memory import ConversationBufferMemory
# Initialize LLM
llm = OpenAI(temperature=0)
# Load tools
tools = load_tools(["serpapi", "llm-math"], llm=llm)
# Add memory
memory = ConversationBufferMemory(memory_key="chat_history")
# Create agent
agent = initialize_agent(
tools,
llm,
agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION,
memory=memory,
verbose=True
)
# Run agent
result = agent.run("What's the weather in SF? Then calculate 25 * 4")
```
## Architecture Patterns
### Pattern 1: RAG with LangChain
```python
from langchain.chains import RetrievalQA
from langchain.document_loaders import TextLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
# Load and process documents
loader = TextLoader('documents.txt')
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
texts = text_splitter.split_documents(documents)
# Create vector store
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(texts, embeddings)
# Create retrieval chain
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vectorstore.as_retriever(),
return_source_documents=True
)
# Query
result = qa_chain({"query": "What is the main topic?"})
```
### Pattern 2: Custom Agent with Tools
```python
from langchain.agents import Tool, AgentExecutor
from langchain.agents.react.base import ReActDocstoreAgent
from langchain.tools import tool
@tool
def search_database(query: str) -> str:
"""Search internal database for information."""
# Your database search logic
return f"Results for: {query}"
@tool
def send_email(recipient: str, content: str) -> str:
"""Send an email to specified recipient."""
# Email sending logic
return f"Email sent to {recipient}"
tools = [search_database, send_email]
agent = initialize_agent(
tools,
llm,
agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
verbose=True
)
```
### Pattern 3: Multi-Step Chain
```python
from langchain.chains import LLMChain, SequentialChain
from langchain.prompts import PromptTemplate
# Step 1: Extract key information
extract_prompt = PromptTemplate(
input_variables=["text"],
template="Extract key entities from: {text}\n\nEntities:"
)
extract_chain = LLMChain(llm=llm, prompt=extract_prompt, output_key="entities")
# Step 2: Analyze entities
analyze_prompt = PromptTemplate(
input_variables=["entities"],
template="Analyze these entities: {entities}\n\nAnalysis:"
)
analyze_chain = LLMChain(llm=llm, prompt=analyze_prompt, output_key="analysis")
# Step 3: Generate summary
summary_prompt = PromptTemplate(
input_variables=["entities", "analysis"],
template="Summarize:\nEntities: {entities}\nAnalysis: {analysis}\n\nSummary:"
)
summary_chain = LLMChain(llm=llm, prompt=summary_prompt, output_key="summary")
# Combine into sequential chain
overall_chain = SequentialChain(
chains=[extract_chain, analyze_chain, summary_chain],
input_variables=["text"],
output_variables=["entities", "analysis", "summary"],
verbose=True
)
```
## Memory Management Best Practices
### Choosing the Right Memory Type
```python
# For short conversations (< 10 messages)
from langchain.memory import ConversationBufferMemory
memory = ConversationBufferMemory()
# For long conversations (summarize old messages)
from langchain.memory import ConversationSummaryMemory
memory = ConversationSummaryMemory(llm=llm)
# For sliding window (last N messages)
from langchain.memory import ConversationBufferWindowMemory
memory = ConversationBufferWindowMemory(k=5)
# For entity tracking
from langchain.memory import ConversationEntityMemory
memory = ConversationEntityMemory(llm=llm)
# For semantic retrieval of relevant history
from langchain.memory import VectorStoreRetrieverMemory
memory = VectorStoreRetrieverMemory(retriever=retriever)
```
## Callback System
### Custom Callback Handler
```python
from langchain.callbacks.base import BaseCallbackHandler
class CustomCallbackHandler(BaseCallbackHandler):
def on_llm_start(self, serialized, prompts, **kwargs):
print(f"LLM started with prompts: {prompts}")
def on_llm_end(self, response, **kwargs):
print(f"LLM ended with response: {response}")
def on_llm_error(self, error, **kwargs):
print(f"LLM error: {error}")
def on_chain_start(self, serialized, inputs, **kwargs):
print(f"Chain started with inputs: {inputs}")
def on_agent_action(self, action, **kwargs):
print(f"Agent taking action: {action}")
# Use callback
agent.run("query", callbacks=[CustomCallbackHandler()])
```
## Testing Strategies
```python
import pytest
from unittest.mock import Mock
def test_agent_tool_selection():
# Mock LLM to return specific tool selection
mock_llm = Mock()
mock_llm.predict.return_value = "Action: search_database\nAction Input: test query"
agent = initialize_agent(tools, mock_llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION)
result = agent.run("test query")
# Verify correct tool was selected
assert "search_database" in str(mock_llm.predict.call_args)
def test_memory_persistence():
memory = ConversationBufferMemory()
memory.save_context({"input": "Hi"}, {"output": "Hello!"})
assert "Hi" in memory.load_memory_variables({})['history']
assert "Hello!" in memory.load_memory_variables({})['history']
```
## Performance Optimization
### 1. Caching
```python
from langchain.cache import InMemoryCache
import langchain
langchain.llm_cache = InMemoryCache()
```
### 2. Batch Processing
```python
# Process multiple documents in parallel
from langchain.document_loaders import DirectoryLoader
from concurrent.futures import ThreadPoolExecutor
loader = DirectoryLoader('./docs')
docs = loader.load()
def process_doc(doc):
return text_splitter.split_documents([doc])
with ThreadPoolExecutor(max_workers=4) as executor:
split_docs = list(executor.map(process_doc, docs))
```
### 3. Streaming Responses
```python
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
llm = OpenAI(streaming=True, callbacks=[StreamingStdOutCallbackHandler()])
```
## Resources
- **references/agents.md**: Deep dive on agent architectures
- **references/memory.md**: Memory system patterns
- **references/chains.md**: Chain composition strategies
- **references/document-processing.md**: Document loading and indexing
- **references/callbacks.md**: Monitoring and observability
- **assets/agent-template.py**: Production-ready agent template
- **assets/memory-config.yaml**: Memory configuration examples
- **assets/chain-example.py**: Complex chain examples
## Common Pitfalls
1. **Memory Overflow**: Not managing conversation history length
2. **Tool Selection Errors**: Poor tool descriptions confuse agents
3. **Context Window Exceeded**: Exceeding LLM token limits
4. **No Error Handling**: Not catching and handling agent failures
5. **Inefficient Retrieval**: Not optimizing vector store queries
## Production Checklist
- [ ] Implement proper error handling
- [ ] Add request/response logging
- [ ] Monitor token usage and costs
- [ ] Set timeout limits for agent execution
- [ ] Implement rate limiting
- [ ] Add input validation
- [ ] Test with edge cases
- [ ] Set up observability (callbacks)
- [ ] Implement fallback strategies
- [ ] Version control prompts and configurations

471
plugins/llm-application-dev/skills/llm-evaluation/SKILL.md Normal file
View File

@@ -0,0 +1,471 @@
---
name: llm-evaluation
description: Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.
---
# LLM Evaluation
Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.
## When to Use This Skill
- Measuring LLM application performance systematically
- Comparing different models or prompts
- Detecting performance regressions before deployment
- Validating improvements from prompt changes
- Building confidence in production systems
- Establishing baselines and tracking progress over time
- Debugging unexpected model behavior
## Core Evaluation Types
### 1. Automated Metrics
Fast, repeatable, scalable evaluation using computed scores.
**Text Generation:**
- **BLEU**: N-gram overlap (translation)
- **ROUGE**: Recall-oriented (summarization)
- **METEOR**: Semantic similarity
- **BERTScore**: Embedding-based similarity
- **Perplexity**: Language model confidence
**Classification:**
- **Accuracy**: Percentage correct
- **Precision/Recall/F1**: Class-specific performance
- **Confusion Matrix**: Error patterns
- **AUC-ROC**: Ranking quality
**Retrieval (RAG):**
- **MRR**: Mean Reciprocal Rank
- **NDCG**: Normalized Discounted Cumulative Gain
- **Precision@K**: Relevant in top K
- **Recall@K**: Coverage in top K
### 2. Human Evaluation
Manual assessment for quality aspects difficult to automate.
**Dimensions:**
- **Accuracy**: Factual correctness
- **Coherence**: Logical flow
- **Relevance**: Answers the question
- **Fluency**: Natural language quality
- **Safety**: No harmful content
- **Helpfulness**: Useful to the user
### 3. LLM-as-Judge
Use stronger LLMs to evaluate weaker model outputs.
**Approaches:**
- **Pointwise**: Score individual responses
- **Pairwise**: Compare two responses
- **Reference-based**: Compare to gold standard
- **Reference-free**: Judge without ground truth
## Quick Start
```python
from llm_eval import EvaluationSuite, Metric
# Define evaluation suite
suite = EvaluationSuite([
Metric.accuracy(),
Metric.bleu(),
Metric.bertscore(),
Metric.custom(name="groundedness", fn=check_groundedness)
])
# Prepare test cases
test_cases = [
{
"input": "What is the capital of France?",
"expected": "Paris",
"context": "France is a country in Europe. Paris is its capital."
},
# ... more test cases
]
# Run evaluation
results = suite.evaluate(
model=your_model,
test_cases=test_cases
)
print(f"Overall Accuracy: {results.metrics['accuracy']}")
print(f"BLEU Score: {results.metrics['bleu']}")
```
## Automated Metrics Implementation
### BLEU Score
```python
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
def calculate_bleu(reference, hypothesis):
"""Calculate BLEU score between reference and hypothesis."""
smoothie = SmoothingFunction().method4
return sentence_bleu(
[reference.split()],
hypothesis.split(),
smoothing_function=smoothie
)
# Usage
bleu = calculate_bleu(
reference="The cat sat on the mat",
hypothesis="A cat is sitting on the mat"
)
```
### ROUGE Score
```python
from rouge_score import rouge_scorer
def calculate_rouge(reference, hypothesis):
"""Calculate ROUGE scores."""
scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
scores = scorer.score(reference, hypothesis)
return {
'rouge1': scores['rouge1'].fmeasure,
'rouge2': scores['rouge2'].fmeasure,
'rougeL': scores['rougeL'].fmeasure
}
```
### BERTScore
```python
from bert_score import score
def calculate_bertscore(references, hypotheses):
"""Calculate BERTScore using pre-trained BERT."""
P, R, F1 = score(
hypotheses,
references,
lang='en',
model_type='microsoft/deberta-xlarge-mnli'
)
return {
'precision': P.mean().item(),
'recall': R.mean().item(),
'f1': F1.mean().item()
}
```
### Custom Metrics
```python
def calculate_groundedness(response, context):
"""Check if response is grounded in provided context."""
# Use NLI model to check entailment
from transformers import pipeline
nli = pipeline("text-classification", model="microsoft/deberta-large-mnli")
result = nli(f"{context} [SEP] {response}")[0]
# Return confidence that response is entailed by context
return result['score'] if result['label'] == 'ENTAILMENT' else 0.0
def calculate_toxicity(text):
"""Measure toxicity in generated text."""
from detoxify import Detoxify
results = Detoxify('original').predict(text)
return max(results.values()) # Return highest toxicity score
def calculate_factuality(claim, knowledge_base):
"""Verify factual claims against knowledge base."""
# Implementation depends on your knowledge base
# Could use retrieval + NLI, or fact-checking API
pass
```
## LLM-as-Judge Patterns
### Single Output Evaluation
```python
def llm_judge_quality(response, question):
"""Use GPT-4 to judge response quality."""
prompt = f"""Rate the following response on a scale of 1-10 for:
1. Accuracy (factually correct)
2. Helpfulness (answers the question)
3. Clarity (well-written and understandable)
Question: {question}
Response: {response}
Provide ratings in JSON format:
{{
"accuracy": <1-10>,
"helpfulness": <1-10>,
"clarity": <1-10>,
"reasoning": "<brief explanation>"
}}
"""
result = openai.ChatCompletion.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}],
temperature=0
)
return json.loads(result.choices[0].message.content)
```
### Pairwise Comparison
```python
def compare_responses(question, response_a, response_b):
"""Compare two responses using LLM judge."""
prompt = f"""Compare these two responses to the question and determine which is better.
Question: {question}
Response A: {response_a}
Response B: {response_b}
Which response is better and why? Consider accuracy, helpfulness, and clarity.
Answer with JSON:
{{
"winner": "A" or "B" or "tie",
"reasoning": "<explanation>",
"confidence": <1-10>
}}
"""
result = openai.ChatCompletion.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}],
temperature=0
)
return json.loads(result.choices[0].message.content)
```
## Human Evaluation Frameworks
### Annotation Guidelines
```python
class AnnotationTask:
"""Structure for human annotation task."""
def __init__(self, response, question, context=None):
self.response = response
self.question = question
self.context = context
def get_annotation_form(self):
return {
"question": self.question,
"context": self.context,
"response": self.response,
"ratings": {
"accuracy": {
"scale": "1-5",
"description": "Is the response factually correct?"
},
"relevance": {
"scale": "1-5",
"description": "Does it answer the question?"
},
"coherence": {
"scale": "1-5",
"description": "Is it logically consistent?"
}
},
"issues": {
"factual_error": False,
"hallucination": False,
"off_topic": False,
"unsafe_content": False
},
"feedback": ""
}
```
### Inter-Rater Agreement
```python
from sklearn.metrics import cohen_kappa_score
def calculate_agreement(rater1_scores, rater2_scores):
"""Calculate inter-rater agreement."""
kappa = cohen_kappa_score(rater1_scores, rater2_scores)
interpretation = {
kappa < 0: "Poor",
kappa < 0.2: "Slight",
kappa < 0.4: "Fair",
kappa < 0.6: "Moderate",
kappa < 0.8: "Substantial",
kappa <= 1.0: "Almost Perfect"
}
return {
"kappa": kappa,
"interpretation": interpretation[True]
}
```
## A/B Testing
### Statistical Testing Framework
```python
from scipy import stats
import numpy as np
class ABTest:
def __init__(self, variant_a_name="A", variant_b_name="B"):
self.variant_a = {"name": variant_a_name, "scores": []}
self.variant_b = {"name": variant_b_name, "scores": []}
def add_result(self, variant, score):
"""Add evaluation result for a variant."""
if variant == "A":
self.variant_a["scores"].append(score)
else:
self.variant_b["scores"].append(score)
def analyze(self, alpha=0.05):
"""Perform statistical analysis."""
a_scores = self.variant_a["scores"]
b_scores = self.variant_b["scores"]
# T-test
t_stat, p_value = stats.ttest_ind(a_scores, b_scores)
# Effect size (Cohen's d)
pooled_std = np.sqrt((np.std(a_scores)**2 + np.std(b_scores)**2) / 2)
cohens_d = (np.mean(b_scores) - np.mean(a_scores)) / pooled_std
return {
"variant_a_mean": np.mean(a_scores),
"variant_b_mean": np.mean(b_scores),
"difference": np.mean(b_scores) - np.mean(a_scores),
"relative_improvement": (np.mean(b_scores) - np.mean(a_scores)) / np.mean(a_scores),
"p_value": p_value,
"statistically_significant": p_value < alpha,
"cohens_d": cohens_d,
"effect_size": self.interpret_cohens_d(cohens_d),
"winner": "B" if np.mean(b_scores) > np.mean(a_scores) else "A"
}
@staticmethod
def interpret_cohens_d(d):
"""Interpret Cohen's d effect size."""
abs_d = abs(d)
if abs_d < 0.2:
return "negligible"
elif abs_d < 0.5:
return "small"
elif abs_d < 0.8:
return "medium"
else:
return "large"
```
## Regression Testing
### Regression Detection
```python
class RegressionDetector:
def __init__(self, baseline_results, threshold=0.05):
self.baseline = baseline_results
self.threshold = threshold
def check_for_regression(self, new_results):
"""Detect if new results show regression."""
regressions = []
for metric in self.baseline.keys():
baseline_score = self.baseline[metric]
new_score = new_results.get(metric)
if new_score is None:
continue
# Calculate relative change
relative_change = (new_score - baseline_score) / baseline_score
# Flag if significant decrease
if relative_change < -self.threshold:
regressions.append({
"metric": metric,
"baseline": baseline_score,
"current": new_score,
"change": relative_change
})
return {
"has_regression": len(regressions) > 0,
"regressions": regressions
}
```
## Benchmarking
### Running Benchmarks
```python
class BenchmarkRunner:
def __init__(self, benchmark_dataset):
self.dataset = benchmark_dataset
def run_benchmark(self, model, metrics):
"""Run model on benchmark and calculate metrics."""
results = {metric.name: [] for metric in metrics}
for example in self.dataset:
# Generate prediction
prediction = model.predict(example["input"])
# Calculate each metric
for metric in metrics:
score = metric.calculate(
prediction=prediction,
reference=example["reference"],
context=example.get("context")
)
results[metric.name].append(score)
# Aggregate results
return {
metric: {
"mean": np.mean(scores),
"std": np.std(scores),
"min": min(scores),
"max": max(scores)
}
for metric, scores in results.items()
}
```
## Resources
- **references/metrics.md**: Comprehensive metric guide
- **references/human-evaluation.md**: Annotation best practices
- **references/benchmarking.md**: Standard benchmarks
- **references/a-b-testing.md**: Statistical testing guide
- **references/regression-testing.md**: CI/CD integration
- **assets/evaluation-framework.py**: Complete evaluation harness
- **assets/benchmark-dataset.jsonl**: Example datasets
- **scripts/evaluate-model.py**: Automated evaluation runner
## Best Practices
1. **Multiple Metrics**: Use diverse metrics for comprehensive view
2. **Representative Data**: Test on real-world, diverse examples
3. **Baselines**: Always compare against baseline performance
4. **Statistical Rigor**: Use proper statistical tests for comparisons
5. **Continuous Evaluation**: Integrate into CI/CD pipeline
6. **Human Validation**: Combine automated metrics with human judgment
7. **Error Analysis**: Investigate failures to understand weaknesses
8. **Version Control**: Track evaluation results over time
## Common Pitfalls
- **Single Metric Obsession**: Optimizing for one metric at the expense of others
- **Small Sample Size**: Drawing conclusions from too few examples
- **Data Contamination**: Testing on training data
- **Ignoring Variance**: Not accounting for statistical uncertainty
- **Metric Mismatch**: Using metrics not aligned with business goals

201
plugins/llm-application-dev/skills/prompt-engineering-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,201 @@
---
name: prompt-engineering-patterns
description: Master advanced prompt engineering techniques to maximize LLM performance, reliability, and controllability in production. Use when optimizing prompts, improving LLM outputs, or designing production prompt templates.
---
# Prompt Engineering Patterns
Master advanced prompt engineering techniques to maximize LLM performance, reliability, and controllability.
## When to Use This Skill
- Designing complex prompts for production LLM applications
- Optimizing prompt performance and consistency
- Implementing structured reasoning patterns (chain-of-thought, tree-of-thought)
- Building few-shot learning systems with dynamic example selection
- Creating reusable prompt templates with variable interpolation
- Debugging and refining prompts that produce inconsistent outputs
- Implementing system prompts for specialized AI assistants
## Core Capabilities
### 1. Few-Shot Learning
- Example selection strategies (semantic similarity, diversity sampling)
- Balancing example count with context window constraints
- Constructing effective demonstrations with input-output pairs
- Dynamic example retrieval from knowledge bases
- Handling edge cases through strategic example selection
### 2. Chain-of-Thought Prompting
- Step-by-step reasoning elicitation
- Zero-shot CoT with "Let's think step by step"
- Few-shot CoT with reasoning traces
- Self-consistency techniques (sampling multiple reasoning paths)
- Verification and validation steps
### 3. Prompt Optimization
- Iterative refinement workflows
- A/B testing prompt variations
- Measuring prompt performance metrics (accuracy, consistency, latency)
- Reducing token usage while maintaining quality
- Handling edge cases and failure modes
### 4. Template Systems
- Variable interpolation and formatting
- Conditional prompt sections
- Multi-turn conversation templates
- Role-based prompt composition
- Modular prompt components
### 5. System Prompt Design
- Setting model behavior and constraints
- Defining output formats and structure
- Establishing role and expertise
- Safety guidelines and content policies
- Context setting and background information
## Quick Start
```python
from prompt_optimizer import PromptTemplate, FewShotSelector
# Define a structured prompt template
template = PromptTemplate(
system="You are an expert SQL developer. Generate efficient, secure SQL queries.",
instruction="Convert the following natural language query to SQL:\n{query}",
few_shot_examples=True,
output_format="SQL code block with explanatory comments"
)
# Configure few-shot learning
selector = FewShotSelector(
examples_db="sql_examples.jsonl",
selection_strategy="semantic_similarity",
max_examples=3
)
# Generate optimized prompt
prompt = template.render(
query="Find all users who registered in the last 30 days",
examples=selector.select(query="user registration date filter")
)
```
## Key Patterns
### Progressive Disclosure
Start with simple prompts, add complexity only when needed:
1. **Level 1**: Direct instruction
- "Summarize this article"
2. **Level 2**: Add constraints
- "Summarize this article in 3 bullet points, focusing on key findings"
3. **Level 3**: Add reasoning
- "Read this article, identify the main findings, then summarize in 3 bullet points"
4. **Level 4**: Add examples
- Include 2-3 example summaries with input-output pairs
### Instruction Hierarchy
```
[System Context] → [Task Instruction] → [Examples] → [Input Data] → [Output Format]
```
### Error Recovery
Build prompts that gracefully handle failures:
- Include fallback instructions
- Request confidence scores
- Ask for alternative interpretations when uncertain
- Specify how to indicate missing information
## Best Practices
1. **Be Specific**: Vague prompts produce inconsistent results
2. **Show, Don't Tell**: Examples are more effective than descriptions
3. **Test Extensively**: Evaluate on diverse, representative inputs
4. **Iterate Rapidly**: Small changes can have large impacts
5. **Monitor Performance**: Track metrics in production
6. **Version Control**: Treat prompts as code with proper versioning
7. **Document Intent**: Explain why prompts are structured as they are
## Common Pitfalls
- **Over-engineering**: Starting with complex prompts before trying simple ones
- **Example pollution**: Using examples that don't match the target task
- **Context overflow**: Exceeding token limits with excessive examples
- **Ambiguous instructions**: Leaving room for multiple interpretations
- **Ignoring edge cases**: Not testing on unusual or boundary inputs
## Integration Patterns
### With RAG Systems
```python
# Combine retrieved context with prompt engineering
prompt = f"""Given the following context:
{retrieved_context}
{few_shot_examples}
Question: {user_question}
Provide a detailed answer based solely on the context above. If the context doesn't contain enough information, explicitly state what's missing."""
```
### With Validation
```python
# Add self-verification step
prompt = f"""{main_task_prompt}
After generating your response, verify it meets these criteria:
1. Answers the question directly
2. Uses only information from provided context
3. Cites specific sources
4. Acknowledges any uncertainty
If verification fails, revise your response."""
```
## Performance Optimization
### Token Efficiency
- Remove redundant words and phrases
- Use abbreviations consistently after first definition
- Consolidate similar instructions
- Move stable content to system prompts
### Latency Reduction
- Minimize prompt length without sacrificing quality
- Use streaming for long-form outputs
- Cache common prompt prefixes
- Batch similar requests when possible
## Resources
- **references/few-shot-learning.md**: Deep dive on example selection and construction
- **references/chain-of-thought.md**: Advanced reasoning elicitation techniques
- **references/prompt-optimization.md**: Systematic refinement workflows
- **references/prompt-templates.md**: Reusable template patterns
- **references/system-prompts.md**: System-level prompt design
- **assets/prompt-template-library.md**: Battle-tested prompt templates
- **assets/few-shot-examples.json**: Curated example datasets
- **scripts/optimize-prompt.py**: Automated prompt optimization tool
## Success Metrics
Track these KPIs for your prompts:
- **Accuracy**: Correctness of outputs
- **Consistency**: Reproducibility across similar inputs
- **Latency**: Response time (P50, P95, P99)
- **Token Usage**: Average tokens per request
- **Success Rate**: Percentage of valid outputs
- **User Satisfaction**: Ratings and feedback
## Next Steps
1. Review the prompt template library for common patterns
2. Experiment with few-shot learning for your specific use case
3. Implement prompt versioning and A/B testing
4. Set up automated evaluation pipelines
5. Document your prompt engineering decisions and learnings

106
plugins/llm-application-dev/skills/prompt-engineering-patterns/assets/few-shot-examples.json Normal file
View File

@@ -0,0 +1,106 @@
{
"sentiment_analysis": [
{
"input": "This product exceeded my expectations! The quality is outstanding.",
"output": "Positive"
},
{
"input": "Terrible experience. The item arrived damaged and customer service was unhelpful.",
"output": "Negative"
},
{
"input": "The product works as described. Nothing special, but does the job.",
"output": "Neutral"
}
],
"entity_extraction": [
{
"input": "Apple CEO Tim Cook announced the new iPhone at an event in Cupertino on September 12th.",
"output": {
"persons": ["Tim Cook"],
"organizations": ["Apple"],
"products": ["iPhone"],
"locations": ["Cupertino"],
"dates": ["September 12th"]
}
},
{
"input": "Microsoft acquired GitHub for $7.5 billion in 2018.",
"output": {
"persons": [],
"organizations": ["Microsoft", "GitHub"],
"products": [],
"locations": [],
"dates": ["2018"],
"monetary_values": ["$7.5 billion"]
}
}
],
"code_generation": [
{
"input": "Write a Python function to check if a string is a palindrome",
"output": "def is_palindrome(s: str) -> bool:\n \"\"\"Check if string is palindrome, ignoring case and spaces.\"\"\"\n # Remove spaces and convert to lowercase\n cleaned = s.replace(' ', '').lower()\n # Compare with reversed string\n return cleaned == cleaned[::-1]"
}
],
"text_classification": [
{
"input": "How do I reset my password?",
"output": "account_management"
},
{
"input": "My order hasn't arrived yet. Where is it?",
"output": "shipping_inquiry"
},
{
"input": "I'd like to cancel my subscription.",
"output": "subscription_cancellation"
},
{
"input": "The app keeps crashing when I try to log in.",
"output": "technical_support"
}
],
"data_transformation": [
{
"input": "John Smith, john@email.com, (555) 123-4567",
"output": {
"name": "John Smith",
"email": "john@email.com",
"phone": "(555) 123-4567"
}
},
{
"input": "Jane Doe | jane.doe@company.com | +1-555-987-6543",
"output": {
"name": "Jane Doe",
"email": "jane.doe@company.com",
"phone": "+1-555-987-6543"
}
}
],
"question_answering": [
{
"context": "The Eiffel Tower is a wrought-iron lattice tower in Paris, France. It was constructed from 1887 to 1889 and stands 324 meters (1,063 ft) tall.",
"question": "When was the Eiffel Tower built?",
"answer": "The Eiffel Tower was constructed from 1887 to 1889."
},
{
"context": "Python 3.11 was released on October 24, 2022. It includes performance improvements and new features like exception groups and improved error messages.",
"question": "What are the new features in Python 3.11?",
"answer": "Python 3.11 includes exception groups, improved error messages, and performance improvements."
}
],
"summarization": [
{
"input": "Climate change refers to long-term shifts in global temperatures and weather patterns. While climate change is natural, human activities have been the main driver since the 1800s, primarily due to the burning of fossil fuels like coal, oil and gas which produces heat-trapping greenhouse gases. The consequences include rising sea levels, more extreme weather events, and threats to biodiversity.",
"output": "Climate change involves long-term alterations in global temperatures and weather patterns, primarily driven by human fossil fuel consumption since the 1800s, resulting in rising sea levels, extreme weather, and biodiversity threats."
}
],
"sql_generation": [
{
"schema": "users (id, name, email, created_at)\norders (id, user_id, total, order_date)",
"request": "Find all users who have placed orders totaling more than $1000",
"output": "SELECT u.id, u.name, u.email, SUM(o.total) as total_spent\nFROM users u\nJOIN orders o ON u.id = o.user_id\nGROUP BY u.id, u.name, u.email\nHAVING SUM(o.total) > 1000;"
}
]
}

246
plugins/llm-application-dev/skills/prompt-engineering-patterns/assets/prompt-template-library.md Normal file
View File

@@ -0,0 +1,246 @@
# Prompt Template Library
## Classification Templates
### Sentiment Analysis
```
Classify the sentiment of the following text as Positive, Negative, or Neutral.
Text: {text}
Sentiment:
```
### Intent Detection
```
Determine the user's intent from the following message.
Possible intents: {intent_list}
Message: {message}
Intent:
```
### Topic Classification
```
Classify the following article into one of these categories: {categories}
Article:
{article}
Category:
```
## Extraction Templates
### Named Entity Recognition
```
Extract all named entities from the text and categorize them.
Text: {text}
Entities (JSON format):
{
"persons": [],
"organizations": [],
"locations": [],
"dates": []
}
```
### Structured Data Extraction
```
Extract structured information from the job posting.
Job Posting:
{posting}
Extracted Information (JSON):
{
"title": "",
"company": "",
"location": "",
"salary_range": "",
"requirements": [],
"responsibilities": []
}
```
## Generation Templates
### Email Generation
```
Write a professional {email_type} email.
To: {recipient}
Context: {context}
Key points to include:
{key_points}
Email:
Subject:
Body:
```
### Code Generation
```
Generate {language} code for the following task:
Task: {task_description}
Requirements:
{requirements}
Include:
- Error handling
- Input validation
- Inline comments
Code:
```
### Creative Writing
```
Write a {length}-word {style} story about {topic}.
Include these elements:
- {element_1}
- {element_2}
- {element_3}
Story:
```
## Transformation Templates
### Summarization
```
Summarize the following text in {num_sentences} sentences.
Text:
{text}
Summary:
```
### Translation with Context
```
Translate the following {source_lang} text to {target_lang}.
Context: {context}
Tone: {tone}
Text: {text}
Translation:
```
### Format Conversion
```
Convert the following {source_format} to {target_format}.
Input:
{input_data}
Output ({target_format}):
```
## Analysis Templates
### Code Review
```
Review the following code for:
1. Bugs and errors
2. Performance issues
3. Security vulnerabilities
4. Best practice violations
Code:
{code}
Review:
```
### SWOT Analysis
```
Conduct a SWOT analysis for: {subject}
Context: {context}
Analysis:
Strengths:
-
Weaknesses:
-
Opportunities:
-
Threats:
-
```
## Question Answering Templates
### RAG Template
```
Answer the question based on the provided context. If the context doesn't contain enough information, say so.
Context:
{context}
Question: {question}
Answer:
```
### Multi-Turn Q&A
```
Previous conversation:
{conversation_history}
New question: {question}
Answer (continue naturally from conversation):
```
## Specialized Templates
### SQL Query Generation
```
Generate a SQL query for the following request.
Database schema:
{schema}
Request: {request}
SQL Query:
```
### Regex Pattern Creation
```
Create a regex pattern to match: {requirement}
Test cases that should match:
{positive_examples}
Test cases that should NOT match:
{negative_examples}
Regex pattern:
```
### API Documentation
```
Generate API documentation for this function:
Code:
{function_code}
Documentation (follow {doc_format} format):
```
## Use these templates by filling in the {variables}

399
plugins/llm-application-dev/skills/prompt-engineering-patterns/references/chain-of-thought.md Normal file
View File

@@ -0,0 +1,399 @@
# Chain-of-Thought Prompting
## Overview
Chain-of-Thought (CoT) prompting elicits step-by-step reasoning from LLMs, dramatically improving performance on complex reasoning, math, and logic tasks.
## Core Techniques
### Zero-Shot CoT
Add a simple trigger phrase to elicit reasoning:
```python
def zero_shot_cot(query):
return f"""{query}
Let's think step by step:"""
# Example
query = "If a train travels 60 mph for 2.5 hours, how far does it go?"
prompt = zero_shot_cot(query)
# Model output:
# "Let's think step by step:
# 1. Speed = 60 miles per hour
# 2. Time = 2.5 hours
# 3. Distance = Speed × Time
# 4. Distance = 60 × 2.5 = 150 miles
# Answer: 150 miles"
```
### Few-Shot CoT
Provide examples with explicit reasoning chains:
```python
few_shot_examples = """
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 balls. How many tennis balls does he have now?
A: Let's think step by step:
1. Roger starts with 5 balls
2. He buys 2 cans, each with 3 balls
3. Balls from cans: 2 × 3 = 6 balls
4. Total: 5 + 6 = 11 balls
Answer: 11
Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many do they have?
A: Let's think step by step:
1. Started with 23 apples
2. Used 20 for lunch: 23 - 20 = 3 apples left
3. Bought 6 more: 3 + 6 = 9 apples
Answer: 9
Q: {user_query}
A: Let's think step by step:"""
```
### Self-Consistency
Generate multiple reasoning paths and take the majority vote:
```python
import openai
from collections import Counter
def self_consistency_cot(query, n=5, temperature=0.7):
prompt = f"{query}\n\nLet's think step by step:"
responses = []
for _ in range(n):
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}],
temperature=temperature
)
responses.append(extract_final_answer(response))
# Take majority vote
answer_counts = Counter(responses)
final_answer = answer_counts.most_common(1)[0][0]
return {
'answer': final_answer,
'confidence': answer_counts[final_answer] / n,
'all_responses': responses
}
```
## Advanced Patterns
### Least-to-Most Prompting
Break complex problems into simpler subproblems:
```python
def least_to_most_prompt(complex_query):
# Stage 1: Decomposition
decomp_prompt = f"""Break down this complex problem into simpler subproblems:
Problem: {complex_query}
Subproblems:"""
subproblems = get_llm_response(decomp_prompt)
# Stage 2: Sequential solving
solutions = []
context = ""
for subproblem in subproblems:
solve_prompt = f"""{context}
Solve this subproblem:
{subproblem}
Solution:"""
solution = get_llm_response(solve_prompt)
solutions.append(solution)
context += f"\n\nPreviously solved: {subproblem}\nSolution: {solution}"
# Stage 3: Final integration
final_prompt = f"""Given these solutions to subproblems:
{context}
Provide the final answer to: {complex_query}
Final Answer:"""
return get_llm_response(final_prompt)
```
### Tree-of-Thought (ToT)
Explore multiple reasoning branches:
```python
class TreeOfThought:
def __init__(self, llm_client, max_depth=3, branches_per_step=3):
self.client = llm_client
self.max_depth = max_depth
self.branches_per_step = branches_per_step
def solve(self, problem):
# Generate initial thought branches
initial_thoughts = self.generate_thoughts(problem, depth=0)
# Evaluate each branch
best_path = None
best_score = -1
for thought in initial_thoughts:
path, score = self.explore_branch(problem, thought, depth=1)
if score > best_score:
best_score = score
best_path = path
return best_path
def generate_thoughts(self, problem, context="", depth=0):
prompt = f"""Problem: {problem}
{context}
Generate {self.branches_per_step} different next steps in solving this problem:
1."""
response = self.client.complete(prompt)
return self.parse_thoughts(response)
def evaluate_thought(self, problem, thought_path):
prompt = f"""Problem: {problem}
Reasoning path so far:
{thought_path}
Rate this reasoning path from 0-10 for:
- Correctness
- Likelihood of reaching solution
- Logical coherence
Score:"""
return float(self.client.complete(prompt))
```
### Verification Step
Add explicit verification to catch errors:
```python
def cot_with_verification(query):
# Step 1: Generate reasoning and answer
reasoning_prompt = f"""{query}
Let's solve this step by step:"""
reasoning_response = get_llm_response(reasoning_prompt)
# Step 2: Verify the reasoning
verification_prompt = f"""Original problem: {query}
Proposed solution:
{reasoning_response}
Verify this solution by:
1. Checking each step for logical errors
2. Verifying arithmetic calculations
3. Ensuring the final answer makes sense
Is this solution correct? If not, what's wrong?
Verification:"""
verification = get_llm_response(verification_prompt)
# Step 3: Revise if needed
if "incorrect" in verification.lower() or "error" in verification.lower():
revision_prompt = f"""The previous solution had errors:
{verification}
Please provide a corrected solution to: {query}
Corrected solution:"""
return get_llm_response(revision_prompt)
return reasoning_response
```
## Domain-Specific CoT
### Math Problems
```python
math_cot_template = """
Problem: {problem}
Solution:
Step 1: Identify what we know
- {list_known_values}
Step 2: Identify what we need to find
- {target_variable}
Step 3: Choose relevant formulas
- {formulas}
Step 4: Substitute values
- {substitution}
Step 5: Calculate
- {calculation}
Step 6: Verify and state answer
- {verification}
Answer: {final_answer}
"""
```
### Code Debugging
```python
debug_cot_template = """
Code with error:
{code}
Error message:
{error}
Debugging process:
Step 1: Understand the error message
- {interpret_error}
Step 2: Locate the problematic line
- {identify_line}
Step 3: Analyze why this line fails
- {root_cause}
Step 4: Determine the fix
- {proposed_fix}
Step 5: Verify the fix addresses the error
- {verification}
Fixed code:
{corrected_code}
"""
```
### Logical Reasoning
```python
logic_cot_template = """
Premises:
{premises}
Question: {question}
Reasoning:
Step 1: List all given facts
{facts}
Step 2: Identify logical relationships
{relationships}
Step 3: Apply deductive reasoning
{deductions}
Step 4: Draw conclusion
{conclusion}
Answer: {final_answer}
"""
```
## Performance Optimization
### Caching Reasoning Patterns
```python
class ReasoningCache:
def __init__(self):
self.cache = {}
def get_similar_reasoning(self, problem, threshold=0.85):
problem_embedding = embed(problem)
for cached_problem, reasoning in self.cache.items():
similarity = cosine_similarity(
problem_embedding,
embed(cached_problem)
)
if similarity > threshold:
return reasoning
return None
def add_reasoning(self, problem, reasoning):
self.cache[problem] = reasoning
```
### Adaptive Reasoning Depth
```python
def adaptive_cot(problem, initial_depth=3):
depth = initial_depth
while depth <= 10: # Max depth
response = generate_cot(problem, num_steps=depth)
# Check if solution seems complete
if is_solution_complete(response):
return response
depth += 2 # Increase reasoning depth
return response # Return best attempt
```
## Evaluation Metrics
```python
def evaluate_cot_quality(reasoning_chain):
metrics = {
'coherence': measure_logical_coherence(reasoning_chain),
'completeness': check_all_steps_present(reasoning_chain),
'correctness': verify_final_answer(reasoning_chain),
'efficiency': count_unnecessary_steps(reasoning_chain),
'clarity': rate_explanation_clarity(reasoning_chain)
}
return metrics
```
## Best Practices
1. **Clear Step Markers**: Use numbered steps or clear delimiters
2. **Show All Work**: Don't skip steps, even obvious ones
3. **Verify Calculations**: Add explicit verification steps
4. **State Assumptions**: Make implicit assumptions explicit
5. **Check Edge Cases**: Consider boundary conditions
6. **Use Examples**: Show the reasoning pattern with examples first
## Common Pitfalls
- **Premature Conclusions**: Jumping to answer without full reasoning
- **Circular Logic**: Using the conclusion to justify the reasoning
- **Missing Steps**: Skipping intermediate calculations
- **Overcomplicated**: Adding unnecessary steps that confuse
- **Inconsistent Format**: Changing step structure mid-reasoning
## When to Use CoT
**Use CoT for:**
- Math and arithmetic problems
- Logical reasoning tasks
- Multi-step planning
- Code generation and debugging
- Complex decision making
**Skip CoT for:**
- Simple factual queries
- Direct lookups
- Creative writing
- Tasks requiring conciseness
- Real-time, latency-sensitive applications
## Resources
- Benchmark datasets for CoT evaluation
- Pre-built CoT prompt templates
- Reasoning verification tools
- Step extraction and parsing utilities

369
plugins/llm-application-dev/skills/prompt-engineering-patterns/references/few-shot-learning.md Normal file
View File

@@ -0,0 +1,369 @@
# Few-Shot Learning Guide
## Overview
Few-shot learning enables LLMs to perform tasks by providing a small number of examples (typically 1-10) within the prompt. This technique is highly effective for tasks requiring specific formats, styles, or domain knowledge.
## Example Selection Strategies
### 1. Semantic Similarity
Select examples most similar to the input query using embedding-based retrieval.
```python
from sentence_transformers import SentenceTransformer
import numpy as np
class SemanticExampleSelector:
def __init__(self, examples, model_name='all-MiniLM-L6-v2'):
self.model = SentenceTransformer(model_name)
self.examples = examples
self.example_embeddings = self.model.encode([ex['input'] for ex in examples])
def select(self, query, k=3):
query_embedding = self.model.encode([query])
similarities = np.dot(self.example_embeddings, query_embedding.T).flatten()
top_indices = np.argsort(similarities)[-k:][::-1]
return [self.examples[i] for i in top_indices]
```
**Best For**: Question answering, text classification, extraction tasks
### 2. Diversity Sampling
Maximize coverage of different patterns and edge cases.
```python
from sklearn.cluster import KMeans
class DiversityExampleSelector:
def __init__(self, examples, model_name='all-MiniLM-L6-v2'):
self.model = SentenceTransformer(model_name)
self.examples = examples
self.embeddings = self.model.encode([ex['input'] for ex in examples])
def select(self, k=5):
# Use k-means to find diverse cluster centers
kmeans = KMeans(n_clusters=k, random_state=42)
kmeans.fit(self.embeddings)
# Select example closest to each cluster center
diverse_examples = []
for center in kmeans.cluster_centers_:
distances = np.linalg.norm(self.embeddings - center, axis=1)
closest_idx = np.argmin(distances)
diverse_examples.append(self.examples[closest_idx])
return diverse_examples
```
**Best For**: Demonstrating task variability, edge case handling
### 3. Difficulty-Based Selection
Gradually increase example complexity to scaffold learning.
```python
class ProgressiveExampleSelector:
def __init__(self, examples):
# Examples should have 'difficulty' scores (0-1)
self.examples = sorted(examples, key=lambda x: x['difficulty'])
def select(self, k=3):
# Select examples with linearly increasing difficulty
step = len(self.examples) // k
return [self.examples[i * step] for i in range(k)]
```
**Best For**: Complex reasoning tasks, code generation
### 4. Error-Based Selection
Include examples that address common failure modes.
```python
class ErrorGuidedSelector:
def __init__(self, examples, error_patterns):
self.examples = examples
self.error_patterns = error_patterns # Common mistakes to avoid
def select(self, query, k=3):
# Select examples demonstrating correct handling of error patterns
selected = []
for pattern in self.error_patterns[:k]:
matching = [ex for ex in self.examples if pattern in ex['demonstrates']]
if matching:
selected.append(matching[0])
return selected
```
**Best For**: Tasks with known failure patterns, safety-critical applications
## Example Construction Best Practices
### Format Consistency
All examples should follow identical formatting:
```python
# Good: Consistent format
examples = [
{
"input": "What is the capital of France?",
"output": "Paris"
},
{
"input": "What is the capital of Germany?",
"output": "Berlin"
}
]
# Bad: Inconsistent format
examples = [
"Q: What is the capital of France? A: Paris",
{"question": "What is the capital of Germany?", "answer": "Berlin"}
]
```
### Input-Output Alignment
Ensure examples demonstrate the exact task you want the model to perform:
```python
# Good: Clear input-output relationship
example = {
"input": "Sentiment: The movie was terrible and boring.",
"output": "Negative"
}
# Bad: Ambiguous relationship
example = {
"input": "The movie was terrible and boring.",
"output": "This review expresses negative sentiment toward the film."
}
```
### Complexity Balance
Include examples spanning the expected difficulty range:
```python
examples = [
# Simple case
{"input": "2 + 2", "output": "4"},
# Moderate case
{"input": "15 * 3 + 8", "output": "53"},
# Complex case
{"input": "(12 + 8) * 3 - 15 / 5", "output": "57"}
]
```
## Context Window Management
### Token Budget Allocation
Typical distribution for a 4K context window:
```
System Prompt: 500 tokens (12%)
Few-Shot Examples: 1500 tokens (38%)
User Input: 500 tokens (12%)
Response: 1500 tokens (38%)
```
### Dynamic Example Truncation
```python
class TokenAwareSelector:
def __init__(self, examples, tokenizer, max_tokens=1500):
self.examples = examples
self.tokenizer = tokenizer
self.max_tokens = max_tokens
def select(self, query, k=5):
selected = []
total_tokens = 0
# Start with most relevant examples
candidates = self.rank_by_relevance(query)
for example in candidates[:k]:
example_tokens = len(self.tokenizer.encode(
f"Input: {example['input']}\nOutput: {example['output']}\n\n"
))
if total_tokens + example_tokens <= self.max_tokens:
selected.append(example)
total_tokens += example_tokens
else:
break
return selected
```
## Edge Case Handling
### Include Boundary Examples
```python
edge_case_examples = [
# Empty input
{"input": "", "output": "Please provide input text."},
# Very long input (truncated in example)
{"input": "..." + "word " * 1000, "output": "Input exceeds maximum length."},
# Ambiguous input
{"input": "bank", "output": "Ambiguous: Could refer to financial institution or river bank."},
# Invalid input
{"input": "!@#$%", "output": "Invalid input format. Please provide valid text."}
]
```
## Few-Shot Prompt Templates
### Classification Template
```python
def build_classification_prompt(examples, query, labels):
prompt = f"Classify the text into one of these categories: {', '.join(labels)}\n\n"
for ex in examples:
prompt += f"Text: {ex['input']}\nCategory: {ex['output']}\n\n"
prompt += f"Text: {query}\nCategory:"
return prompt
```
### Extraction Template
```python
def build_extraction_prompt(examples, query):
prompt = "Extract structured information from the text.\n\n"
for ex in examples:
prompt += f"Text: {ex['input']}\nExtracted: {json.dumps(ex['output'])}\n\n"
prompt += f"Text: {query}\nExtracted:"
return prompt
```
### Transformation Template
```python
def build_transformation_prompt(examples, query):
prompt = "Transform the input according to the pattern shown in examples.\n\n"
for ex in examples:
prompt += f"Input: {ex['input']}\nOutput: {ex['output']}\n\n"
prompt += f"Input: {query}\nOutput:"
return prompt
```
## Evaluation and Optimization
### Example Quality Metrics
```python
def evaluate_example_quality(example, validation_set):
metrics = {
'clarity': rate_clarity(example), # 0-1 score
'representativeness': calculate_similarity_to_validation(example, validation_set),
'difficulty': estimate_difficulty(example),
'uniqueness': calculate_uniqueness(example, other_examples)
}
return metrics
```
### A/B Testing Example Sets
```python
class ExampleSetTester:
def __init__(self, llm_client):
self.client = llm_client
def compare_example_sets(self, set_a, set_b, test_queries):
results_a = self.evaluate_set(set_a, test_queries)
results_b = self.evaluate_set(set_b, test_queries)
return {
'set_a_accuracy': results_a['accuracy'],
'set_b_accuracy': results_b['accuracy'],
'winner': 'A' if results_a['accuracy'] > results_b['accuracy'] else 'B',
'improvement': abs(results_a['accuracy'] - results_b['accuracy'])
}
def evaluate_set(self, examples, test_queries):
correct = 0
for query in test_queries:
prompt = build_prompt(examples, query['input'])
response = self.client.complete(prompt)
if response == query['expected_output']:
correct += 1
return {'accuracy': correct / len(test_queries)}
```
## Advanced Techniques
### Meta-Learning (Learning to Select)
Train a small model to predict which examples will be most effective:
```python
from sklearn.ensemble import RandomForestClassifier
class LearnedExampleSelector:
def __init__(self):
self.selector_model = RandomForestClassifier()
def train(self, training_data):
# training_data: list of (query, example, success) tuples
features = []
labels = []
for query, example, success in training_data:
features.append(self.extract_features(query, example))
labels.append(1 if success else 0)
self.selector_model.fit(features, labels)
def extract_features(self, query, example):
return [
semantic_similarity(query, example['input']),
len(example['input']),
len(example['output']),
keyword_overlap(query, example['input'])
]
def select(self, query, candidates, k=3):
scores = []
for example in candidates:
features = self.extract_features(query, example)
score = self.selector_model.predict_proba([features])[0][1]
scores.append((score, example))
return [ex for _, ex in sorted(scores, reverse=True)[:k]]
```
### Adaptive Example Count
Dynamically adjust the number of examples based on task difficulty:
```python
class AdaptiveExampleSelector:
def __init__(self, examples):
self.examples = examples
def select(self, query, max_examples=5):
# Start with 1 example
for k in range(1, max_examples + 1):
selected = self.get_top_k(query, k)
# Quick confidence check (could use a lightweight model)
if self.estimated_confidence(query, selected) > 0.9:
return selected
return selected # Return max_examples if never confident enough
```
## Common Mistakes
1. **Too Many Examples**: More isn't always better; can dilute focus
2. **Irrelevant Examples**: Examples should match the target task closely
3. **Inconsistent Formatting**: Confuses the model about output format
4. **Overfitting to Examples**: Model copies example patterns too literally
5. **Ignoring Token Limits**: Running out of space for actual input/output
## Resources
- Example dataset repositories
- Pre-built example selectors for common tasks
- Evaluation frameworks for few-shot performance
- Token counting utilities for different models

414
plugins/llm-application-dev/skills/prompt-engineering-patterns/references/prompt-optimization.md Normal file
View File

@@ -0,0 +1,414 @@
# Prompt Optimization Guide
## Systematic Refinement Process
### 1. Baseline Establishment
```python
def establish_baseline(prompt, test_cases):
results = {
'accuracy': 0,
'avg_tokens': 0,
'avg_latency': 0,
'success_rate': 0
}
for test_case in test_cases:
response = llm.complete(prompt.format(**test_case['input']))
results['accuracy'] += evaluate_accuracy(response, test_case['expected'])
results['avg_tokens'] += count_tokens(response)
results['avg_latency'] += measure_latency(response)
results['success_rate'] += is_valid_response(response)
# Average across test cases
n = len(test_cases)
return {k: v/n for k, v in results.items()}
```
### 2. Iterative Refinement Workflow
```
Initial Prompt → Test → Analyze Failures → Refine → Test → Repeat
```
```python
class PromptOptimizer:
def __init__(self, initial_prompt, test_suite):
self.prompt = initial_prompt
self.test_suite = test_suite
self.history = []
def optimize(self, max_iterations=10):
for i in range(max_iterations):
# Test current prompt
results = self.evaluate_prompt(self.prompt)
self.history.append({
'iteration': i,
'prompt': self.prompt,
'results': results
})
# Stop if good enough
if results['accuracy'] > 0.95:
break
# Analyze failures
failures = self.analyze_failures(results)
# Generate refinement suggestions
refinements = self.generate_refinements(failures)
# Apply best refinement
self.prompt = self.select_best_refinement(refinements)
return self.get_best_prompt()
```
### 3. A/B Testing Framework
```python
class PromptABTest:
def __init__(self, variant_a, variant_b):
self.variant_a = variant_a
self.variant_b = variant_b
def run_test(self, test_queries, metrics=['accuracy', 'latency']):
results = {
'A': {m: [] for m in metrics},
'B': {m: [] for m in metrics}
}
for query in test_queries:
# Randomly assign variant (50/50 split)
variant = 'A' if random.random() < 0.5 else 'B'
prompt = self.variant_a if variant == 'A' else self.variant_b
response, metrics_data = self.execute_with_metrics(
prompt.format(query=query['input'])
)
for metric in metrics:
results[variant][metric].append(metrics_data[metric])
return self.analyze_results(results)
def analyze_results(self, results):
from scipy import stats
analysis = {}
for metric in results['A'].keys():
a_values = results['A'][metric]
b_values = results['B'][metric]
# Statistical significance test
t_stat, p_value = stats.ttest_ind(a_values, b_values)
analysis[metric] = {
'A_mean': np.mean(a_values),
'B_mean': np.mean(b_values),
'improvement': (np.mean(b_values) - np.mean(a_values)) / np.mean(a_values),
'statistically_significant': p_value < 0.05,
'p_value': p_value,
'winner': 'B' if np.mean(b_values) > np.mean(a_values) else 'A'
}
return analysis
```
## Optimization Strategies
### Token Reduction
```python
def optimize_for_tokens(prompt):
optimizations = [
# Remove redundant phrases
('in order to', 'to'),
('due to the fact that', 'because'),
('at this point in time', 'now'),
# Consolidate instructions
('First, ...\\nThen, ...\\nFinally, ...', 'Steps: 1) ... 2) ... 3) ...'),
# Use abbreviations (after first definition)
('Natural Language Processing (NLP)', 'NLP'),
# Remove filler words
(' actually ', ' '),
(' basically ', ' '),
(' really ', ' ')
]
optimized = prompt
for old, new in optimizations:
optimized = optimized.replace(old, new)
return optimized
```
### Latency Reduction
```python
def optimize_for_latency(prompt):
strategies = {
'shorter_prompt': reduce_token_count(prompt),
'streaming': enable_streaming_response(prompt),
'caching': add_cacheable_prefix(prompt),
'early_stopping': add_stop_sequences(prompt)
}
# Test each strategy
best_strategy = None
best_latency = float('inf')
for name, modified_prompt in strategies.items():
latency = measure_average_latency(modified_prompt)
if latency < best_latency:
best_latency = latency
best_strategy = modified_prompt
return best_strategy
```
### Accuracy Improvement
```python
def improve_accuracy(prompt, failure_cases):
improvements = []
# Add constraints for common failures
if has_format_errors(failure_cases):
improvements.append("Output must be valid JSON with no additional text.")
# Add examples for edge cases
edge_cases = identify_edge_cases(failure_cases)
if edge_cases:
improvements.append(f"Examples of edge cases:\\n{format_examples(edge_cases)}")
# Add verification step
if has_logical_errors(failure_cases):
improvements.append("Before responding, verify your answer is logically consistent.")
# Strengthen instructions
if has_ambiguity_errors(failure_cases):
improvements.append(clarify_ambiguous_instructions(prompt))
return integrate_improvements(prompt, improvements)
```
## Performance Metrics
### Core Metrics
```python
class PromptMetrics:
@staticmethod
def accuracy(responses, ground_truth):
return sum(r == gt for r, gt in zip(responses, ground_truth)) / len(responses)
@staticmethod
def consistency(responses):
# Measure how often identical inputs produce identical outputs
from collections import defaultdict
input_responses = defaultdict(list)
for inp, resp in responses:
input_responses[inp].append(resp)
consistency_scores = []
for inp, resps in input_responses.items():
if len(resps) > 1:
# Percentage of responses that match the most common response
most_common_count = Counter(resps).most_common(1)[0][1]
consistency_scores.append(most_common_count / len(resps))
return np.mean(consistency_scores) if consistency_scores else 1.0
@staticmethod
def token_efficiency(prompt, responses):
avg_prompt_tokens = np.mean([count_tokens(prompt.format(**r['input'])) for r in responses])
avg_response_tokens = np.mean([count_tokens(r['output']) for r in responses])
return avg_prompt_tokens + avg_response_tokens
@staticmethod
def latency_p95(latencies):
return np.percentile(latencies, 95)
```
### Automated Evaluation
```python
def evaluate_prompt_comprehensively(prompt, test_suite):
results = {
'accuracy': [],
'consistency': [],
'latency': [],
'tokens': [],
'success_rate': []
}
# Run each test case multiple times for consistency measurement
for test_case in test_suite:
runs = []
for _ in range(3): # 3 runs per test case
start = time.time()
response = llm.complete(prompt.format(**test_case['input']))
latency = time.time() - start
runs.append(response)
results['latency'].append(latency)
results['tokens'].append(count_tokens(prompt) + count_tokens(response))
# Accuracy (best of 3 runs)
accuracies = [evaluate_accuracy(r, test_case['expected']) for r in runs]
results['accuracy'].append(max(accuracies))
# Consistency (how similar are the 3 runs?)
results['consistency'].append(calculate_similarity(runs))
# Success rate (all runs successful?)
results['success_rate'].append(all(is_valid(r) for r in runs))
return {
'avg_accuracy': np.mean(results['accuracy']),
'avg_consistency': np.mean(results['consistency']),
'p95_latency': np.percentile(results['latency'], 95),
'avg_tokens': np.mean(results['tokens']),
'success_rate': np.mean(results['success_rate'])
}
```
## Failure Analysis
### Categorizing Failures
```python
class FailureAnalyzer:
def categorize_failures(self, test_results):
categories = {
'format_errors': [],
'factual_errors': [],
'logic_errors': [],
'incomplete_responses': [],
'hallucinations': [],
'off_topic': []
}
for result in test_results:
if not result['success']:
category = self.determine_failure_type(
result['response'],
result['expected']
)
categories[category].append(result)
return categories
def generate_fixes(self, categorized_failures):
fixes = []
if categorized_failures['format_errors']:
fixes.append({
'issue': 'Format errors',
'fix': 'Add explicit format examples and constraints',
'priority': 'high'
})
if categorized_failures['hallucinations']:
fixes.append({
'issue': 'Hallucinations',
'fix': 'Add grounding instruction: "Base your answer only on provided context"',
'priority': 'critical'
})
if categorized_failures['incomplete_responses']:
fixes.append({
'issue': 'Incomplete responses',
'fix': 'Add: "Ensure your response fully addresses all parts of the question"',
'priority': 'medium'
})
return fixes
```
## Versioning and Rollback
### Prompt Version Control
```python
class PromptVersionControl:
def __init__(self, storage_path):
self.storage = storage_path
self.versions = []
def save_version(self, prompt, metadata):
version = {
'id': len(self.versions),
'prompt': prompt,
'timestamp': datetime.now(),
'metrics': metadata.get('metrics', {}),
'description': metadata.get('description', ''),
'parent_id': metadata.get('parent_id')
}
self.versions.append(version)
self.persist()
return version['id']
def rollback(self, version_id):
if version_id < len(self.versions):
return self.versions[version_id]['prompt']
raise ValueError(f"Version {version_id} not found")
def compare_versions(self, v1_id, v2_id):
v1 = self.versions[v1_id]
v2 = self.versions[v2_id]
return {
'diff': generate_diff(v1['prompt'], v2['prompt']),
'metrics_comparison': {
metric: {
'v1': v1['metrics'].get(metric),
'v2': v2['metrics'].get(metric'),
'change': v2['metrics'].get(metric, 0) - v1['metrics'].get(metric, 0)
}
for metric in set(v1['metrics'].keys()) | set(v2['metrics'].keys())
}
}
```
## Best Practices
1. **Establish Baseline**: Always measure initial performance
2. **Change One Thing**: Isolate variables for clear attribution
3. **Test Thoroughly**: Use diverse, representative test cases
4. **Track Metrics**: Log all experiments and results
5. **Validate Significance**: Use statistical tests for A/B comparisons
6. **Document Changes**: Keep detailed notes on what and why
7. **Version Everything**: Enable rollback to previous versions
8. **Monitor Production**: Continuously evaluate deployed prompts
## Common Optimization Patterns
### Pattern 1: Add Structure
```
Before: "Analyze this text"
After: "Analyze this text for:\n1. Main topic\n2. Key arguments\n3. Conclusion"
```
### Pattern 2: Add Examples
```
Before: "Extract entities"
After: "Extract entities\\n\\nExample:\\nText: Apple released iPhone\\nEntities: {company: Apple, product: iPhone}"
```
### Pattern 3: Add Constraints
```
Before: "Summarize this"
After: "Summarize in exactly 3 bullet points, 15 words each"
```
### Pattern 4: Add Verification
```
Before: "Calculate..."
After: "Calculate... Then verify your calculation is correct before responding."
```
## Tools and Utilities
- Prompt diff tools for version comparison
- Automated test runners
- Metric dashboards
- A/B testing frameworks
- Token counting utilities
- Latency profilers

470
plugins/llm-application-dev/skills/prompt-engineering-patterns/references/prompt-templates.md Normal file
View File

@@ -0,0 +1,470 @@
# Prompt Template Systems
## Template Architecture
### Basic Template Structure
```python
class PromptTemplate:
def __init__(self, template_string, variables=None):
self.template = template_string
self.variables = variables or []
def render(self, **kwargs):
missing = set(self.variables) - set(kwargs.keys())
if missing:
raise ValueError(f"Missing required variables: {missing}")
return self.template.format(**kwargs)
# Usage
template = PromptTemplate(
template_string="Translate {text} from {source_lang} to {target_lang}",
variables=['text', 'source_lang', 'target_lang']
)
prompt = template.render(
text="Hello world",
source_lang="English",
target_lang="Spanish"
)
```
### Conditional Templates
```python
class ConditionalTemplate(PromptTemplate):
def render(self, **kwargs):
# Process conditional blocks
result = self.template
# Handle if-blocks: {{#if variable}}content{{/if}}
import re
if_pattern = r'\{\{#if (\w+)\}\}(.*?)\{\{/if\}\}'
def replace_if(match):
var_name = match.group(1)
content = match.group(2)
return content if kwargs.get(var_name) else ''
result = re.sub(if_pattern, replace_if, result, flags=re.DOTALL)
# Handle for-loops: {{#each items}}{{this}}{{/each}}
each_pattern = r'\{\{#each (\w+)\}\}(.*?)\{\{/each\}\}'
def replace_each(match):
var_name = match.group(1)
content = match.group(2)
items = kwargs.get(var_name, [])
return '\\n'.join(content.replace('{{this}}', str(item)) for item in items)
result = re.sub(each_pattern, replace_each, result, flags=re.DOTALL)
# Finally, render remaining variables
return result.format(**kwargs)
# Usage
template = ConditionalTemplate("""
Analyze the following text:
{text}
{{#if include_sentiment}}
Provide sentiment analysis.
{{/if}}
{{#if include_entities}}
Extract named entities.
{{/if}}
{{#if examples}}
Reference examples:
{{#each examples}}
- {{this}}
{{/each}}
{{/if}}
""")
```
### Modular Template Composition
```python
class ModularTemplate:
def __init__(self):
self.components = {}
def register_component(self, name, template):
self.components[name] = template
def render(self, structure, **kwargs):
parts = []
for component_name in structure:
if component_name in self.components:
component = self.components[component_name]
parts.append(component.format(**kwargs))
return '\\n\\n'.join(parts)
# Usage
builder = ModularTemplate()
builder.register_component('system', "You are a {role}.")
builder.register_component('context', "Context: {context}")
builder.register_component('instruction', "Task: {task}")
builder.register_component('examples', "Examples:\\n{examples}")
builder.register_component('input', "Input: {input}")
builder.register_component('format', "Output format: {format}")
# Compose different templates for different scenarios
basic_prompt = builder.render(
['system', 'instruction', 'input'],
role='helpful assistant',
instruction='Summarize the text',
input='...'
)
advanced_prompt = builder.render(
['system', 'context', 'examples', 'instruction', 'input', 'format'],
role='expert analyst',
context='Financial analysis',
examples='...',
instruction='Analyze sentiment',
input='...',
format='JSON'
)
```
## Common Template Patterns
### Classification Template
```python
CLASSIFICATION_TEMPLATE = """
Classify the following {content_type} into one of these categories: {categories}
{{#if description}}
Category descriptions:
{description}
{{/if}}
{{#if examples}}
Examples:
{examples}
{{/if}}
{content_type}: {input}
Category:"""
```
### Extraction Template
```python
EXTRACTION_TEMPLATE = """
Extract structured information from the {content_type}.
Required fields:
{field_definitions}
{{#if examples}}
Example extraction:
{examples}
{{/if}}
{content_type}: {input}
Extracted information (JSON):"""
```
### Generation Template
```python
GENERATION_TEMPLATE = """
Generate {output_type} based on the following {input_type}.
Requirements:
{requirements}
{{#if style}}
Style: {style}
{{/if}}
{{#if constraints}}
Constraints:
{constraints}
{{/if}}
{{#if examples}}
Examples:
{examples}
{{/if}}
{input_type}: {input}
{output_type}:"""
```
### Transformation Template
```python
TRANSFORMATION_TEMPLATE = """
Transform the input {source_format} to {target_format}.
Transformation rules:
{rules}
{{#if examples}}
Example transformations:
{examples}
{{/if}}
Input {source_format}:
{input}
Output {target_format}:"""
```
## Advanced Features
### Template Inheritance
```python
class TemplateRegistry:
def __init__(self):
self.templates = {}
def register(self, name, template, parent=None):
if parent and parent in self.templates:
# Inherit from parent
base = self.templates[parent]
template = self.merge_templates(base, template)
self.templates[name] = template
def merge_templates(self, parent, child):
# Child overwrites parent sections
return {**parent, **child}
# Usage
registry = TemplateRegistry()
registry.register('base_analysis', {
'system': 'You are an expert analyst.',
'format': 'Provide analysis in structured format.'
})
registry.register('sentiment_analysis', {
'instruction': 'Analyze sentiment',
'format': 'Provide sentiment score from -1 to 1.'
}, parent='base_analysis')
```
### Variable Validation
```python
class ValidatedTemplate:
def __init__(self, template, schema):
self.template = template
self.schema = schema
def validate_vars(self, **kwargs):
for var_name, var_schema in self.schema.items():
if var_name in kwargs:
value = kwargs[var_name]
# Type validation
if 'type' in var_schema:
expected_type = var_schema['type']
if not isinstance(value, expected_type):
raise TypeError(f"{var_name} must be {expected_type}")
# Range validation
if 'min' in var_schema and value < var_schema['min']:
raise ValueError(f"{var_name} must be >= {var_schema['min']}")
if 'max' in var_schema and value > var_schema['max']:
raise ValueError(f"{var_name} must be <= {var_schema['max']}")
# Enum validation
if 'choices' in var_schema and value not in var_schema['choices']:
raise ValueError(f"{var_name} must be one of {var_schema['choices']}")
def render(self, **kwargs):
self.validate_vars(**kwargs)
return self.template.format(**kwargs)
# Usage
template = ValidatedTemplate(
template="Summarize in {length} words with {tone} tone",
schema={
'length': {'type': int, 'min': 10, 'max': 500},
'tone': {'type': str, 'choices': ['formal', 'casual', 'technical']}
}
)
```
### Template Caching
```python
class CachedTemplate:
def __init__(self, template):
self.template = template
self.cache = {}
def render(self, use_cache=True, **kwargs):
if use_cache:
cache_key = self.get_cache_key(kwargs)
if cache_key in self.cache:
return self.cache[cache_key]
result = self.template.format(**kwargs)
if use_cache:
self.cache[cache_key] = result
return result
def get_cache_key(self, kwargs):
return hash(frozenset(kwargs.items()))
def clear_cache(self):
self.cache = {}
```
## Multi-Turn Templates
### Conversation Template
```python
class ConversationTemplate:
def __init__(self, system_prompt):
self.system_prompt = system_prompt
self.history = []
def add_user_message(self, message):
self.history.append({'role': 'user', 'content': message})
def add_assistant_message(self, message):
self.history.append({'role': 'assistant', 'content': message})
def render_for_api(self):
messages = [{'role': 'system', 'content': self.system_prompt}]
messages.extend(self.history)
return messages
def render_as_text(self):
result = f"System: {self.system_prompt}\\n\\n"
for msg in self.history:
role = msg['role'].capitalize()
result += f"{role}: {msg['content']}\\n\\n"
return result
```
### State-Based Templates
```python
class StatefulTemplate:
def __init__(self):
self.state = {}
self.templates = {}
def set_state(self, **kwargs):
self.state.update(kwargs)
def register_state_template(self, state_name, template):
self.templates[state_name] = template
def render(self):
current_state = self.state.get('current_state', 'default')
template = self.templates.get(current_state)
if not template:
raise ValueError(f"No template for state: {current_state}")
return template.format(**self.state)
# Usage for multi-step workflows
workflow = StatefulTemplate()
workflow.register_state_template('init', """
Welcome! Let's {task}.
What is your {first_input}?
""")
workflow.register_state_template('processing', """
Thanks! Processing {first_input}.
Now, what is your {second_input}?
""")
workflow.register_state_template('complete', """
Great! Based on:
- {first_input}
- {second_input}
Here's the result: {result}
""")
```
## Best Practices
1. **Keep It DRY**: Use templates to avoid repetition
2. **Validate Early**: Check variables before rendering
3. **Version Templates**: Track changes like code
4. **Test Variations**: Ensure templates work with diverse inputs
5. **Document Variables**: Clearly specify required/optional variables
6. **Use Type Hints**: Make variable types explicit
7. **Provide Defaults**: Set sensible default values where appropriate
8. **Cache Wisely**: Cache static templates, not dynamic ones
## Template Libraries
### Question Answering
```python
QA_TEMPLATES = {
'factual': """Answer the question based on the context.
Context: {context}
Question: {question}
Answer:""",
'multi_hop': """Answer the question by reasoning across multiple facts.
Facts: {facts}
Question: {question}
Reasoning:""",
'conversational': """Continue the conversation naturally.
Previous conversation:
{history}
User: {question}
Assistant:"""
}
```
### Content Generation
```python
GENERATION_TEMPLATES = {
'blog_post': """Write a blog post about {topic}.
Requirements:
- Length: {word_count} words
- Tone: {tone}
- Include: {key_points}
Blog post:""",
'product_description': """Write a product description for {product}.
Features: {features}
Benefits: {benefits}
Target audience: {audience}
Description:""",
'email': """Write a {type} email.
To: {recipient}
Context: {context}
Key points: {key_points}
Email:"""
}
```
## Performance Considerations
- Pre-compile templates for repeated use
- Cache rendered templates when variables are static
- Minimize string concatenation in loops
- Use efficient string formatting (f-strings, .format())
- Profile template rendering for bottlenecks

189
plugins/llm-application-dev/skills/prompt-engineering-patterns/references/system-prompts.md Normal file
View File

@@ -0,0 +1,189 @@
# System Prompt Design
## Core Principles
System prompts set the foundation for LLM behavior. They define role, expertise, constraints, and output expectations.
## Effective System Prompt Structure
```
[Role Definition] + [Expertise Areas] + [Behavioral Guidelines] + [Output Format] + [Constraints]
```
### Example: Code Assistant
```
You are an expert software engineer with deep knowledge of Python, JavaScript, and system design.
Your expertise includes:
- Writing clean, maintainable, production-ready code
- Debugging complex issues systematically
- Explaining technical concepts clearly
- Following best practices and design patterns
Guidelines:
- Always explain your reasoning
- Prioritize code readability and maintainability
- Consider edge cases and error handling
- Suggest tests for new code
- Ask clarifying questions when requirements are ambiguous
Output format:
- Provide code in markdown code blocks
- Include inline comments for complex logic
- Explain key decisions after code blocks
```
## Pattern Library
### 1. Customer Support Agent
```
You are a friendly, empathetic customer support representative for {company_name}.
Your goals:
- Resolve customer issues quickly and effectively
- Maintain a positive, professional tone
- Gather necessary information to solve problems
- Escalate to human agents when needed
Guidelines:
- Always acknowledge customer frustration
- Provide step-by-step solutions
- Confirm resolution before closing
- Never make promises you can't guarantee
- If uncertain, say "Let me connect you with a specialist"
Constraints:
- Don't discuss competitor products
- Don't share internal company information
- Don't process refunds over $100 (escalate instead)
```
### 2. Data Analyst
```
You are an experienced data analyst specializing in business intelligence.
Capabilities:
- Statistical analysis and hypothesis testing
- Data visualization recommendations
- SQL query generation and optimization
- Identifying trends and anomalies
- Communicating insights to non-technical stakeholders
Approach:
1. Understand the business question
2. Identify relevant data sources
3. Propose analysis methodology
4. Present findings with visualizations
5. Provide actionable recommendations
Output:
- Start with executive summary
- Show methodology and assumptions
- Present findings with supporting data
- Include confidence levels and limitations
- Suggest next steps
```
### 3. Content Editor
```
You are a professional editor with expertise in {content_type}.
Editing focus:
- Grammar and spelling accuracy
- Clarity and conciseness
- Tone consistency ({tone})
- Logical flow and structure
- {style_guide} compliance
Review process:
1. Note major structural issues
2. Identify clarity problems
3. Mark grammar/spelling errors
4. Suggest improvements
5. Preserve author's voice
Format your feedback as:
- Overall assessment (1-2 sentences)
- Specific issues with line references
- Suggested revisions
- Positive elements to preserve
```
## Advanced Techniques
### Dynamic Role Adaptation
```python
def build_adaptive_system_prompt(task_type, difficulty):
base = "You are an expert assistant"
roles = {
'code': 'software engineer',
'write': 'professional writer',
'analyze': 'data analyst'
}
expertise_levels = {
'beginner': 'Explain concepts simply with examples',
'intermediate': 'Balance detail with clarity',
'expert': 'Use technical terminology and advanced concepts'
}
return f"""{base} specializing as a {roles[task_type]}.
Expertise level: {difficulty}
{expertise_levels[difficulty]}
"""
```
### Constraint Specification
```
Hard constraints (MUST follow):
- Never generate harmful, biased, or illegal content
- Do not share personal information
- Stop if asked to ignore these instructions
Soft constraints (SHOULD follow):
- Responses under 500 words unless requested
- Cite sources when making factual claims
- Acknowledge uncertainty rather than guessing
```
## Best Practices
1. **Be Specific**: Vague roles produce inconsistent behavior
2. **Set Boundaries**: Clearly define what the model should/shouldn't do
3. **Provide Examples**: Show desired behavior in the system prompt
4. **Test Thoroughly**: Verify system prompt works across diverse inputs
5. **Iterate**: Refine based on actual usage patterns
6. **Version Control**: Track system prompt changes and performance
## Common Pitfalls
- **Too Long**: Excessive system prompts waste tokens and dilute focus
- **Too Vague**: Generic instructions don't shape behavior effectively
- **Conflicting Instructions**: Contradictory guidelines confuse the model
- **Over-Constraining**: Too many rules can make responses rigid
- **Under-Specifying Format**: Missing output structure leads to inconsistency
## Testing System Prompts
```python
def test_system_prompt(system_prompt, test_cases):
results = []
for test in test_cases:
response = llm.complete(
system=system_prompt,
user_message=test['input']
)
results.append({
'test': test['name'],
'follows_role': check_role_adherence(response, system_prompt),
'follows_format': check_format(response, system_prompt),
'meets_constraints': check_constraints(response, system_prompt),
'quality': rate_quality(response, test['expected'])
})
return results
```

249
plugins/llm-application-dev/skills/prompt-engineering-patterns/scripts/optimize-prompt.py Normal file
View File

@@ -0,0 +1,249 @@
#!/usr/bin/env python3
"""
Prompt Optimization Script
Automatically test and optimize prompts using A/B testing and metrics tracking.
"""
import json
import time
from typing import List, Dict, Any
from dataclasses import dataclass
import numpy as np
@dataclass
class TestCase:
input: Dict[str, Any]
expected_output: str
metadata: Dict[str, Any] = None
class PromptOptimizer:
def __init__(self, llm_client, test_suite: List[TestCase]):
self.client = llm_client
self.test_suite = test_suite
self.results_history = []
def evaluate_prompt(self, prompt_template: str, test_cases: List[TestCase] = None) -> Dict[str, float]:
"""Evaluate a prompt template against test cases."""
if test_cases is None:
test_cases = self.test_suite
metrics = {
'accuracy': [],
'latency': [],
'token_count': [],
'success_rate': []
}
for test_case in test_cases:
start_time = time.time()
# Render prompt with test case inputs
prompt = prompt_template.format(**test_case.input)
# Get LLM response
response = self.client.complete(prompt)
# Measure latency
latency = time.time() - start_time
# Calculate metrics
metrics['latency'].append(latency)
metrics['token_count'].append(len(prompt.split()) + len(response.split()))
metrics['success_rate'].append(1 if response else 0)
# Check accuracy
accuracy = self.calculate_accuracy(response, test_case.expected_output)
metrics['accuracy'].append(accuracy)
# Aggregate metrics
return {
'avg_accuracy': np.mean(metrics['accuracy']),
'avg_latency': np.mean(metrics['latency']),
'p95_latency': np.percentile(metrics['latency'], 95),
'avg_tokens': np.mean(metrics['token_count']),
'success_rate': np.mean(metrics['success_rate'])
}
def calculate_accuracy(self, response: str, expected: str) -> float:
"""Calculate accuracy score between response and expected output."""
# Simple exact match
if response.strip().lower() == expected.strip().lower():
return 1.0
# Partial match using word overlap
response_words = set(response.lower().split())
expected_words = set(expected.lower().split())
if not expected_words:
return 0.0
overlap = len(response_words & expected_words)
return overlap / len(expected_words)
def optimize(self, base_prompt: str, max_iterations: int = 5) -> Dict[str, Any]:
"""Iteratively optimize a prompt."""
current_prompt = base_prompt
best_prompt = base_prompt
best_score = 0
for iteration in range(max_iterations):
print(f"\nIteration {iteration + 1}/{max_iterations}")
# Evaluate current prompt
metrics = self.evaluate_prompt(current_prompt)
print(f"Accuracy: {metrics['avg_accuracy']:.2f}, Latency: {metrics['avg_latency']:.2f}s")
# Track results
self.results_history.append({
'iteration': iteration,
'prompt': current_prompt,
'metrics': metrics
})
# Update best if improved
if metrics['avg_accuracy'] > best_score:
best_score = metrics['avg_accuracy']
best_prompt = current_prompt
# Stop if good enough
if metrics['avg_accuracy'] > 0.95:
print("Achieved target accuracy!")
break
# Generate variations for next iteration
variations = self.generate_variations(current_prompt, metrics)
# Test variations and pick best
best_variation = current_prompt
best_variation_score = metrics['avg_accuracy']
for variation in variations:
var_metrics = self.evaluate_prompt(variation)
if var_metrics['avg_accuracy'] > best_variation_score:
best_variation_score = var_metrics['avg_accuracy']
best_variation = variation
current_prompt = best_variation
return {
'best_prompt': best_prompt,
'best_score': best_score,
'history': self.results_history
}
def generate_variations(self, prompt: str, current_metrics: Dict) -> List[str]:
"""Generate prompt variations to test."""
variations = []
# Variation 1: Add explicit format instruction
variations.append(prompt + "\n\nProvide your answer in a clear, concise format.")
# Variation 2: Add step-by-step instruction
variations.append("Let's solve this step by step.\n\n" + prompt)
# Variation 3: Add verification step
variations.append(prompt + "\n\nVerify your answer before responding.")
# Variation 4: Make more concise
concise = self.make_concise(prompt)
if concise != prompt:
variations.append(concise)
# Variation 5: Add examples (if none present)
if "example" not in prompt.lower():
variations.append(self.add_examples(prompt))
return variations[:3] # Return top 3 variations
def make_concise(self, prompt: str) -> str:
"""Remove redundant words to make prompt more concise."""
replacements = [
("in order to", "to"),
("due to the fact that", "because"),
("at this point in time", "now"),
("in the event that", "if"),
]
result = prompt
for old, new in replacements:
result = result.replace(old, new)
return result
def add_examples(self, prompt: str) -> str:
"""Add example section to prompt."""
return f"""{prompt}
Example:
Input: Sample input
Output: Sample output
"""
def compare_prompts(self, prompt_a: str, prompt_b: str) -> Dict[str, Any]:
"""A/B test two prompts."""
print("Testing Prompt A...")
metrics_a = self.evaluate_prompt(prompt_a)
print("Testing Prompt B...")
metrics_b = self.evaluate_prompt(prompt_b)
return {
'prompt_a_metrics': metrics_a,
'prompt_b_metrics': metrics_b,
'winner': 'A' if metrics_a['avg_accuracy'] > metrics_b['avg_accuracy'] else 'B',
'improvement': abs(metrics_a['avg_accuracy'] - metrics_b['avg_accuracy'])
}
def export_results(self, filename: str):
"""Export optimization results to JSON."""
with open(filename, 'w') as f:
json.dump(self.results_history, f, indent=2)
def main():
# Example usage
test_suite = [
TestCase(
input={'text': 'This movie was amazing!'},
expected_output='Positive'
),
TestCase(
input={'text': 'Worst purchase ever.'},
expected_output='Negative'
),
TestCase(
input={'text': 'It was okay, nothing special.'},
expected_output='Neutral'
)
]
# Mock LLM client for demonstration
class MockLLMClient:
def complete(self, prompt):
# Simulate LLM response
if 'amazing' in prompt:
return 'Positive'
elif 'worst' in prompt.lower():
return 'Negative'
else:
return 'Neutral'
optimizer = PromptOptimizer(MockLLMClient(), test_suite)
base_prompt = "Classify the sentiment of: {text}\nSentiment:"
results = optimizer.optimize(base_prompt)
print("\n" + "="*50)
print("Optimization Complete!")
print(f"Best Accuracy: {results['best_score']:.2f}")
print(f"Best Prompt:\n{results['best_prompt']}")
optimizer.export_results('optimization_results.json')
if __name__ == '__main__':
main()

403
plugins/llm-application-dev/skills/rag-implementation/SKILL.md Normal file
View File

@@ -0,0 +1,403 @@
---
name: rag-implementation
description: Build Retrieval-Augmented Generation (RAG) systems for LLM applications with vector databases and semantic search. Use when implementing knowledge-grounded AI, building document Q&A systems, or integrating LLMs with external knowledge bases.
---
# RAG Implementation
Master Retrieval-Augmented Generation (RAG) to build LLM applications that provide accurate, grounded responses using external knowledge sources.
## When to Use This Skill
- Building Q&A systems over proprietary documents
- Creating chatbots with current, factual information
- Implementing semantic search with natural language queries
- Reducing hallucinations with grounded responses
- Enabling LLMs to access domain-specific knowledge
- Building documentation assistants
- Creating research tools with source citation
## Core Components
### 1. Vector Databases
**Purpose**: Store and retrieve document embeddings efficiently
**Options:**
- **Pinecone**: Managed, scalable, fast queries
- **Weaviate**: Open-source, hybrid search
- **Milvus**: High performance, on-premise
- **Chroma**: Lightweight, easy to use
- **Qdrant**: Fast, filtered search
- **FAISS**: Meta's library, local deployment
### 2. Embeddings
**Purpose**: Convert text to numerical vectors for similarity search
**Models:**
- **text-embedding-ada-002** (OpenAI): General purpose, 1536 dims
- **all-MiniLM-L6-v2** (Sentence Transformers): Fast, lightweight
- **e5-large-v2**: High quality, multilingual
- **Instructor**: Task-specific instructions
- **bge-large-en-v1.5**: SOTA performance
### 3. Retrieval Strategies
**Approaches:**
- **Dense Retrieval**: Semantic similarity via embeddings
- **Sparse Retrieval**: Keyword matching (BM25, TF-IDF)
- **Hybrid Search**: Combine dense + sparse
- **Multi-Query**: Generate multiple query variations
- **HyDE**: Generate hypothetical documents
### 4. Reranking
**Purpose**: Improve retrieval quality by reordering results
**Methods:**
- **Cross-Encoders**: BERT-based reranking
- **Cohere Rerank**: API-based reranking
- **Maximal Marginal Relevance (MMR)**: Diversity + relevance
- **LLM-based**: Use LLM to score relevance
## Quick Start
```python
from langchain.document_loaders import DirectoryLoader
from langchain.text_splitters import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.chains import RetrievalQA
from langchain.llms import OpenAI
# 1. Load documents
loader = DirectoryLoader('./docs', glob="**/*.txt")
documents = loader.load()
# 2. Split into chunks
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
length_function=len
)
chunks = text_splitter.split_documents(documents)
# 3. Create embeddings and vector store
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(chunks, embeddings)
# 4. Create retrieval chain
qa_chain = RetrievalQA.from_chain_type(
llm=OpenAI(),
chain_type="stuff",
retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
return_source_documents=True
)
# 5. Query
result = qa_chain({"query": "What are the main features?"})
print(result['result'])
print(result['source_documents'])
```
## Advanced RAG Patterns
### Pattern 1: Hybrid Search
```python
from langchain.retrievers import BM25Retriever, EnsembleRetriever
# Sparse retriever (BM25)
bm25_retriever = BM25Retriever.from_documents(chunks)
bm25_retriever.k = 5
# Dense retriever (embeddings)
embedding_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
# Combine with weights
ensemble_retriever = EnsembleRetriever(
retrievers=[bm25_retriever, embedding_retriever],
weights=[0.3, 0.7]
)
```
### Pattern 2: Multi-Query Retrieval
```python
from langchain.retrievers.multi_query import MultiQueryRetriever
# Generate multiple query perspectives
retriever = MultiQueryRetriever.from_llm(
retriever=vectorstore.as_retriever(),
llm=OpenAI()
)
# Single query → multiple variations → combined results
results = retriever.get_relevant_documents("What is the main topic?")
```
### Pattern 3: Contextual Compression
```python
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor
compressor = LLMChainExtractor.from_llm(llm)
compression_retriever = ContextualCompressionRetriever(
base_compressor=compressor,
base_retriever=vectorstore.as_retriever()
)
# Returns only relevant parts of documents
compressed_docs = compression_retriever.get_relevant_documents("query")
```
### Pattern 4: Parent Document Retriever
```python
from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
# Store for parent documents
store = InMemoryStore()
# Small chunks for retrieval, large chunks for context
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
retriever = ParentDocumentRetriever(
vectorstore=vectorstore,
docstore=store,
child_splitter=child_splitter,
parent_splitter=parent_splitter
)
```
## Document Chunking Strategies
### Recursive Character Text Splitter
```python
from langchain.text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
length_function=len,
separators=["\n\n", "\n", " ", ""] # Try these in order
)
```
### Token-Based Splitting
```python
from langchain.text_splitters import TokenTextSplitter
splitter = TokenTextSplitter(
chunk_size=512,
chunk_overlap=50
)
```
### Semantic Chunking
```python
from langchain.text_splitters import SemanticChunker
splitter = SemanticChunker(
embeddings=OpenAIEmbeddings(),
breakpoint_threshold_type="percentile"
)
```
### Markdown Header Splitter
```python
from langchain.text_splitters import MarkdownHeaderTextSplitter
headers_to_split_on = [
("#", "Header 1"),
("##", "Header 2"),
("###", "Header 3"),
]
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
```
## Vector Store Configurations
### Pinecone
```python
import pinecone
from langchain.vectorstores import Pinecone
pinecone.init(api_key="your-api-key", environment="us-west1-gcp")
index = pinecone.Index("your-index-name")
vectorstore = Pinecone(index, embeddings.embed_query, "text")
```
### Weaviate
```python
import weaviate
from langchain.vectorstores import Weaviate
client = weaviate.Client("http://localhost:8080")
vectorstore = Weaviate(client, "Document", "content", embeddings)
```
### Chroma (Local)
```python
from langchain.vectorstores import Chroma
vectorstore = Chroma(
collection_name="my_collection",
embedding_function=embeddings,
persist_directory="./chroma_db"
)
```
## Retrieval Optimization
### 1. Metadata Filtering
```python
# Add metadata during indexing
chunks_with_metadata = []
for i, chunk in enumerate(chunks):
chunk.metadata = {
"source": chunk.metadata.get("source"),
"page": i,
"category": determine_category(chunk.page_content)
}
chunks_with_metadata.append(chunk)
# Filter during retrieval
results = vectorstore.similarity_search(
"query",
filter={"category": "technical"},
k=5
)
```
### 2. Maximal Marginal Relevance
```python
# Balance relevance with diversity
results = vectorstore.max_marginal_relevance_search(
"query",
k=5,
fetch_k=20, # Fetch 20, return top 5 diverse
lambda_mult=0.5 # 0=max diversity, 1=max relevance
)
```
### 3. Reranking with Cross-Encoder
```python
from sentence_transformers import CrossEncoder
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
# Get initial results
candidates = vectorstore.similarity_search("query", k=20)
# Rerank
pairs = [[query, doc.page_content] for doc in candidates]
scores = reranker.predict(pairs)
# Sort by score and take top k
reranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:5]
```
## Prompt Engineering for RAG
### Contextual Prompt
```python
prompt_template = """Use the following context to answer the question. If you cannot answer based on the context, say "I don't have enough information."
Context:
{context}
Question: {question}
Answer:"""
```
### With Citations
```python
prompt_template = """Answer the question based on the context below. Include citations using [1], [2], etc.
Context:
{context}
Question: {question}
Answer (with citations):"""
```
### With Confidence
```python
prompt_template = """Answer the question using the context. Provide a confidence score (0-100%) for your answer.
Context:
{context}
Question: {question}
Answer:
Confidence:"""
```
## Evaluation Metrics
```python
def evaluate_rag_system(qa_chain, test_cases):
metrics = {
'accuracy': [],
'retrieval_quality': [],
'groundedness': []
}
for test in test_cases:
result = qa_chain({"query": test['question']})
# Check if answer matches expected
accuracy = calculate_accuracy(result['result'], test['expected'])
metrics['accuracy'].append(accuracy)
# Check if relevant docs were retrieved
retrieval_quality = evaluate_retrieved_docs(
result['source_documents'],
test['relevant_docs']
)
metrics['retrieval_quality'].append(retrieval_quality)
# Check if answer is grounded in context
groundedness = check_groundedness(
result['result'],
result['source_documents']
)
metrics['groundedness'].append(groundedness)
return {k: sum(v)/len(v) for k, v in metrics.items()}
```
## Resources
- **references/vector-databases.md**: Detailed comparison of vector DBs
- **references/embeddings.md**: Embedding model selection guide
- **references/retrieval-strategies.md**: Advanced retrieval techniques
- **references/reranking.md**: Reranking methods and when to use them
- **references/context-window.md**: Managing context limits
- **assets/vector-store-config.yaml**: Configuration templates
- **assets/retriever-pipeline.py**: Complete RAG pipeline
- **assets/embedding-models.md**: Model comparison and benchmarks
## Best Practices
1. **Chunk Size**: Balance between context and specificity (500-1000 tokens)
2. **Overlap**: Use 10-20% overlap to preserve context at boundaries
3. **Metadata**: Include source, page, timestamp for filtering and debugging
4. **Hybrid Search**: Combine semantic and keyword search for best results
5. **Reranking**: Improve top results with cross-encoder
6. **Citations**: Always return source documents for transparency
7. **Evaluation**: Continuously test retrieval quality and answer accuracy
8. **Monitoring**: Track retrieval metrics in production
## Common Issues
- **Poor Retrieval**: Check embedding quality, chunk size, query formulation
- **Irrelevant Results**: Add metadata filtering, use hybrid search, rerank
- **Missing Information**: Ensure documents are properly indexed
- **Slow Queries**: Optimize vector store, use caching, reduce k
- **Hallucinations**: Improve grounding prompt, add verification step

245
plugins/machine-learning-ops/skills/ml-pipeline-workflow/SKILL.md Normal file
View File

@@ -0,0 +1,245 @@
---
name: ml-pipeline-workflow
description: Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps practices, or automating model training and deployment workflows.
---
# ML Pipeline Workflow
Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.
## Overview
This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.
## When to Use This Skill
- Building new ML pipelines from scratch
- Designing workflow orchestration for ML systems
- Implementing data → model → deployment automation
- Setting up reproducible training workflows
- Creating DAG-based ML orchestration
- Integrating ML components into production systems
## What This Skill Provides
### Core Capabilities
1. **Pipeline Architecture**
- End-to-end workflow design
- DAG orchestration patterns (Airflow, Dagster, Kubeflow)
- Component dependencies and data flow
- Error handling and retry strategies
2. **Data Preparation**
- Data validation and quality checks
- Feature engineering pipelines
- Data versioning and lineage
- Train/validation/test splitting strategies
3. **Model Training**
- Training job orchestration
- Hyperparameter management
- Experiment tracking integration
- Distributed training patterns
4. **Model Validation**
- Validation frameworks and metrics
- A/B testing infrastructure
- Performance regression detection
- Model comparison workflows
5. **Deployment Automation**
- Model serving patterns
- Canary deployments
- Blue-green deployment strategies
- Rollback mechanisms
### Reference Documentation
See the `references/` directory for detailed guides:
- **data-preparation.md** - Data cleaning, validation, and feature engineering
- **model-training.md** - Training workflows and best practices
- **model-validation.md** - Validation strategies and metrics
- **model-deployment.md** - Deployment patterns and serving architectures
### Assets and Templates
The `assets/` directory contains:
- **pipeline-dag.yaml.template** - DAG template for workflow orchestration
- **training-config.yaml** - Training configuration template
- **validation-checklist.md** - Pre-deployment validation checklist
## Usage Patterns
### Basic Pipeline Setup
```python
# 1. Define pipeline stages
stages = [
"data_ingestion",
"data_validation",
"feature_engineering",
"model_training",
"model_validation",
"model_deployment"
]
# 2. Configure dependencies
# See assets/pipeline-dag.yaml.template for full example
```
### Production Workflow
1. **Data Preparation Phase**
- Ingest raw data from sources
- Run data quality checks
- Apply feature transformations
- Version processed datasets
2. **Training Phase**
- Load versioned training data
- Execute training jobs
- Track experiments and metrics
- Save trained models
3. **Validation Phase**
- Run validation test suite
- Compare against baseline
- Generate performance reports
- Approve for deployment
4. **Deployment Phase**
- Package model artifacts
- Deploy to serving infrastructure
- Configure monitoring
- Validate production traffic
## Best Practices
### Pipeline Design
- **Modularity**: Each stage should be independently testable
- **Idempotency**: Re-running stages should be safe
- **Observability**: Log metrics at every stage
- **Versioning**: Track data, code, and model versions
- **Failure Handling**: Implement retry logic and alerting
### Data Management
- Use data validation libraries (Great Expectations, TFX)
- Version datasets with DVC or similar tools
- Document feature engineering transformations
- Maintain data lineage tracking
### Model Operations
- Separate training and serving infrastructure
- Use model registries (MLflow, Weights & Biases)
- Implement gradual rollouts for new models
- Monitor model performance drift
- Maintain rollback capabilities
### Deployment Strategies
- Start with shadow deployments
- Use canary releases for validation
- Implement A/B testing infrastructure
- Set up automated rollback triggers
- Monitor latency and throughput
## Integration Points
### Orchestration Tools
- **Apache Airflow**: DAG-based workflow orchestration
- **Dagster**: Asset-based pipeline orchestration
- **Kubeflow Pipelines**: Kubernetes-native ML workflows
- **Prefect**: Modern dataflow automation
### Experiment Tracking
- MLflow for experiment tracking and model registry
- Weights & Biases for visualization and collaboration
- TensorBoard for training metrics
### Deployment Platforms
- AWS SageMaker for managed ML infrastructure
- Google Vertex AI for GCP deployments
- Azure ML for Azure cloud
- Kubernetes + KServe for cloud-agnostic serving
## Progressive Disclosure
Start with the basics and gradually add complexity:
1. **Level 1**: Simple linear pipeline (data → train → deploy)
2. **Level 2**: Add validation and monitoring stages
3. **Level 3**: Implement hyperparameter tuning
4. **Level 4**: Add A/B testing and gradual rollouts
5. **Level 5**: Multi-model pipelines with ensemble strategies
## Common Patterns
### Batch Training Pipeline
```yaml
# See assets/pipeline-dag.yaml.template
stages:
- name: data_preparation
dependencies: []
- name: model_training
dependencies: [data_preparation]
- name: model_evaluation
dependencies: [model_training]
- name: model_deployment
dependencies: [model_evaluation]
```
### Real-time Feature Pipeline
```python
# Stream processing for real-time features
# Combined with batch training
# See references/data-preparation.md
```
### Continuous Training
```python
# Automated retraining on schedule
# Triggered by data drift detection
# See references/model-training.md
```
## Troubleshooting
### Common Issues
- **Pipeline failures**: Check dependencies and data availability
- **Training instability**: Review hyperparameters and data quality
- **Deployment issues**: Validate model artifacts and serving config
- **Performance degradation**: Monitor data drift and model metrics
### Debugging Steps
1. Check pipeline logs for each stage
2. Validate input/output data at boundaries
3. Test components in isolation
4. Review experiment tracking metrics
5. Inspect model artifacts and metadata
## Next Steps
After setting up your pipeline:
1. Explore **hyperparameter-tuning** skill for optimization
2. Learn **experiment-tracking-setup** for MLflow/W&B
3. Review **model-deployment-patterns** for serving strategies
4. Implement monitoring with observability tools
## Related Skills
- **experiment-tracking-setup**: MLflow and Weights & Biases integration
- **hyperparameter-tuning**: Automated hyperparameter optimization
- **model-deployment-patterns**: Advanced deployment strategies

438
plugins/observability-monitoring/skills/distributed-tracing/SKILL.md Normal file
View File

@@ -0,0 +1,438 @@
---
name: distributed-tracing
description: Implement distributed tracing with Jaeger and Tempo to track requests across microservices and identify performance bottlenecks. Use when debugging microservices, analyzing request flows, or implementing observability for distributed systems.
---
# Distributed Tracing
Implement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.
## Purpose
Track requests across distributed systems to understand latency, dependencies, and failure points.
## When to Use
- Debug latency issues
- Understand service dependencies
- Identify bottlenecks
- Trace error propagation
- Analyze request paths
## Distributed Tracing Concepts
### Trace Structure
```
Trace (Request ID: abc123)
↓
Span (frontend) [100ms]
↓
Span (api-gateway) [80ms]
├→ Span (auth-service) [10ms]
└→ Span (user-service) [60ms]
└→ Span (database) [40ms]
```
### Key Components
- **Trace** - End-to-end request journey
- **Span** - Single operation within a trace
- **Context** - Metadata propagated between services
- **Tags** - Key-value pairs for filtering
- **Logs** - Timestamped events within a span
## Jaeger Setup
### Kubernetes Deployment
```bash
# Deploy Jaeger Operator
kubectl create namespace observability
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.51.0/jaeger-operator.yaml -n observability
# Deploy Jaeger instance
kubectl apply -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: jaeger
namespace: observability
spec:
strategy: production
storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
ingress:
enabled: true
EOF
```
### Docker Compose
```yaml
version: '3.8'
services:
jaeger:
image: jaegertracing/all-in-one:latest
ports:
- "5775:5775/udp"
- "6831:6831/udp"
- "6832:6832/udp"
- "5778:5778"
- "16686:16686" # UI
- "14268:14268" # Collector
- "14250:14250" # gRPC
- "9411:9411" # Zipkin
environment:
- COLLECTOR_ZIPKIN_HOST_PORT=:9411
```
**Reference:** See `references/jaeger-setup.md`
## Application Instrumentation
### OpenTelemetry (Recommended)
#### Python (Flask)
```python
from opentelemetry import trace
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from flask import Flask
# Initialize tracer
resource = Resource(attributes={SERVICE_NAME: "my-service"})
provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(JaegerExporter(
agent_host_name="jaeger",
agent_port=6831,
))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
# Instrument Flask
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
@app.route('/api/users')
def get_users():
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("get_users") as span:
span.set_attribute("user.count", 100)
# Business logic
users = fetch_users_from_db()
return {"users": users}
def fetch_users_from_db():
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("database_query") as span:
span.set_attribute("db.system", "postgresql")
span.set_attribute("db.statement", "SELECT * FROM users")
# Database query
return query_database()
```
#### Node.js (Express)
```javascript
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
// Initialize tracer
const provider = new NodeTracerProvider({
resource: { attributes: { 'service.name': 'my-service' } }
});
const exporter = new JaegerExporter({
endpoint: 'http://jaeger:14268/api/traces'
});
provider.addSpanProcessor(new BatchSpanProcessor(exporter));
provider.register();
// Instrument libraries
registerInstrumentations({
instrumentations: [
new HttpInstrumentation(),
new ExpressInstrumentation(),
],
});
const express = require('express');
const app = express();
app.get('/api/users', async (req, res) => {
const tracer = trace.getTracer('my-service');
const span = tracer.startSpan('get_users');
try {
const users = await fetchUsers();
span.setAttributes({ 'user.count': users.length });
res.json({ users });
} finally {
span.end();
}
});
```
#### Go
```go
package main
import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/jaeger"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)
func initTracer() (*sdktrace.TracerProvider, error) {
exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(
jaeger.WithEndpoint("http://jaeger:14268/api/traces"),
))
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceNameKey.String("my-service"),
)),
)
otel.SetTracerProvider(tp)
return tp, nil
}
func getUsers(ctx context.Context) ([]User, error) {
tracer := otel.Tracer("my-service")
ctx, span := tracer.Start(ctx, "get_users")
defer span.End()
span.SetAttributes(attribute.String("user.filter", "active"))
users, err := fetchUsersFromDB(ctx)
if err != nil {
span.RecordError(err)
return nil, err
}
span.SetAttributes(attribute.Int("user.count", len(users)))
return users, nil
}
```
**Reference:** See `references/instrumentation.md`
## Context Propagation
### HTTP Headers
```
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: congo=t61rcWkgMzE
```
### Propagation in HTTP Requests
#### Python
```python
from opentelemetry.propagate import inject
headers = {}
inject(headers) # Injects trace context
response = requests.get('http://downstream-service/api', headers=headers)
```
#### Node.js
```javascript
const { propagation } = require('@opentelemetry/api');
const headers = {};
propagation.inject(context.active(), headers);
axios.get('http://downstream-service/api', { headers });
```
## Tempo Setup (Grafana)
### Kubernetes Deployment
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: tempo-config
data:
tempo.yaml: |
server:
http_listen_port: 3200
distributor:
receivers:
jaeger:
protocols:
thrift_http:
grpc:
otlp:
protocols:
http:
grpc:
storage:
trace:
backend: s3
s3:
bucket: tempo-traces
endpoint: s3.amazonaws.com
querier:
frontend_worker:
frontend_address: tempo-query-frontend:9095
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tempo
spec:
replicas: 1
template:
spec:
containers:
- name: tempo
image: grafana/tempo:latest
args:
- -config.file=/etc/tempo/tempo.yaml
volumeMounts:
- name: config
mountPath: /etc/tempo
volumes:
- name: config
configMap:
name: tempo-config
```
**Reference:** See `assets/jaeger-config.yaml.template`
## Sampling Strategies
### Probabilistic Sampling
```yaml
# Sample 1% of traces
sampler:
type: probabilistic
param: 0.01
```
### Rate Limiting Sampling
```yaml
# Sample max 100 traces per second
sampler:
type: ratelimiting
param: 100
```
### Adaptive Sampling
```python
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
# Sample based on trace ID (deterministic)
sampler = ParentBased(root=TraceIdRatioBased(0.01))
```
## Trace Analysis
### Finding Slow Requests
**Jaeger Query:**
```
service=my-service
duration > 1s
```
### Finding Errors
**Jaeger Query:**
```
service=my-service
error=true
tags.http.status_code >= 500
```
### Service Dependency Graph
Jaeger automatically generates service dependency graphs showing:
- Service relationships
- Request rates
- Error rates
- Average latencies
## Best Practices
1. **Sample appropriately** (1-10% in production)
2. **Add meaningful tags** (user_id, request_id)
3. **Propagate context** across all service boundaries
4. **Log exceptions** in spans
5. **Use consistent naming** for operations
6. **Monitor tracing overhead** (<1% CPU impact)
7. **Set up alerts** for trace errors
8. **Implement distributed context** (baggage)
9. **Use span events** for important milestones
10. **Document instrumentation** standards
## Integration with Logging
### Correlated Logs
```python
import logging
from opentelemetry import trace
logger = logging.getLogger(__name__)
def process_request():
span = trace.get_current_span()
trace_id = span.get_span_context().trace_id
logger.info(
"Processing request",
extra={"trace_id": format(trace_id, '032x')}
)
```
## Troubleshooting
**No traces appearing:**
- Check collector endpoint
- Verify network connectivity
- Check sampling configuration
- Review application logs
**High latency overhead:**
- Reduce sampling rate
- Use batch span processor
- Check exporter configuration
## Reference Files
- `references/jaeger-setup.md` - Jaeger installation
- `references/instrumentation.md` - Instrumentation patterns
- `assets/jaeger-config.yaml.template` - Jaeger configuration
## Related Skills
- `prometheus-configuration` - For metrics
- `grafana-dashboards` - For visualization
- `slo-implementation` - For latency SLOs

369
plugins/observability-monitoring/skills/grafana-dashboards/SKILL.md Normal file
View File

@@ -0,0 +1,369 @@
---
name: grafana-dashboards
description: Create and manage production Grafana dashboards for real-time visualization of system and application metrics. Use when building monitoring dashboards, visualizing metrics, or creating operational observability interfaces.
---
# Grafana Dashboards
Create and manage production-ready Grafana dashboards for comprehensive system observability.
## Purpose
Design effective Grafana dashboards for monitoring applications, infrastructure, and business metrics.
## When to Use
- Visualize Prometheus metrics
- Create custom dashboards
- Implement SLO dashboards
- Monitor infrastructure
- Track business KPIs
## Dashboard Design Principles
### 1. Hierarchy of Information
```
┌─────────────────────────────────────┐
│ Critical Metrics (Big Numbers) │
├─────────────────────────────────────┤
│ Key Trends (Time Series) │
├─────────────────────────────────────┤
│ Detailed Metrics (Tables/Heatmaps) │
└─────────────────────────────────────┘
```
### 2. RED Method (Services)
- **Rate** - Requests per second
- **Errors** - Error rate
- **Duration** - Latency/response time
### 3. USE Method (Resources)
- **Utilization** - % time resource is busy
- **Saturation** - Queue length/wait time
- **Errors** - Error count
## Dashboard Structure
### API Monitoring Dashboard
```json
{
"dashboard": {
"title": "API Monitoring",
"tags": ["api", "production"],
"timezone": "browser",
"refresh": "30s",
"panels": [
{
"title": "Request Rate",
"type": "graph",
"targets": [
{
"expr": "sum(rate(http_requests_total[5m])) by (service)",
"legendFormat": "{{service}}"
}
],
"gridPos": {"x": 0, "y": 0, "w": 12, "h": 8}
},
{
"title": "Error Rate %",
"type": "graph",
"targets": [
{
"expr": "(sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m]))) * 100",
"legendFormat": "Error Rate"
}
],
"alert": {
"conditions": [
{
"evaluator": {"params": [5], "type": "gt"},
"operator": {"type": "and"},
"query": {"params": ["A", "5m", "now"]},
"type": "query"
}
]
},
"gridPos": {"x": 12, "y": 0, "w": 12, "h": 8}
},
{
"title": "P95 Latency",
"type": "graph",
"targets": [
{
"expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))",
"legendFormat": "{{service}}"
}
],
"gridPos": {"x": 0, "y": 8, "w": 24, "h": 8}
}
]
}
}
```
**Reference:** See `assets/api-dashboard.json`
## Panel Types
### 1. Stat Panel (Single Value)
```json
{
"type": "stat",
"title": "Total Requests",
"targets": [{
"expr": "sum(http_requests_total)"
}],
"options": {
"reduceOptions": {
"values": false,
"calcs": ["lastNotNull"]
},
"orientation": "auto",
"textMode": "auto",
"colorMode": "value"
},
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{"value": 0, "color": "green"},
{"value": 80, "color": "yellow"},
{"value": 90, "color": "red"}
]
}
}
}
}
```
### 2. Time Series Graph
```json
{
"type": "graph",
"title": "CPU Usage",
"targets": [{
"expr": "100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)"
}],
"yaxes": [
{"format": "percent", "max": 100, "min": 0},
{"format": "short"}
]
}
```
### 3. Table Panel
```json
{
"type": "table",
"title": "Service Status",
"targets": [{
"expr": "up",
"format": "table",
"instant": true
}],
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {"Time": true},
"indexByName": {},
"renameByName": {
"instance": "Instance",
"job": "Service",
"Value": "Status"
}
}
}
]
}
```
### 4. Heatmap
```json
{
"type": "heatmap",
"title": "Latency Heatmap",
"targets": [{
"expr": "sum(rate(http_request_duration_seconds_bucket[5m])) by (le)",
"format": "heatmap"
}],
"dataFormat": "tsbuckets",
"yAxis": {
"format": "s"
}
}
```
## Variables
### Query Variables
```json
{
"templating": {
"list": [
{
"name": "namespace",
"type": "query",
"datasource": "Prometheus",
"query": "label_values(kube_pod_info, namespace)",
"refresh": 1,
"multi": false
},
{
"name": "service",
"type": "query",
"datasource": "Prometheus",
"query": "label_values(kube_service_info{namespace=\"$namespace\"}, service)",
"refresh": 1,
"multi": true
}
]
}
}
```
### Use Variables in Queries
```
sum(rate(http_requests_total{namespace="$namespace", service=~"$service"}[5m]))
```
## Alerts in Dashboards
```json
{
"alert": {
"name": "High Error Rate",
"conditions": [
{
"evaluator": {
"params": [5],
"type": "gt"
},
"operator": {"type": "and"},
"query": {
"params": ["A", "5m", "now"]
},
"reducer": {"type": "avg"},
"type": "query"
}
],
"executionErrorState": "alerting",
"for": "5m",
"frequency": "1m",
"message": "Error rate is above 5%",
"noDataState": "no_data",
"notifications": [
{"uid": "slack-channel"}
]
}
}
```
## Dashboard Provisioning
**dashboards.yml:**
```yaml
apiVersion: 1
providers:
- name: 'default'
orgId: 1
folder: 'General'
type: file
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: true
options:
path: /etc/grafana/dashboards
```
## Common Dashboard Patterns
### Infrastructure Dashboard
**Key Panels:**
- CPU utilization per node
- Memory usage per node
- Disk I/O
- Network traffic
- Pod count by namespace
- Node status
**Reference:** See `assets/infrastructure-dashboard.json`
### Database Dashboard
**Key Panels:**
- Queries per second
- Connection pool usage
- Query latency (P50, P95, P99)
- Active connections
- Database size
- Replication lag
- Slow queries
**Reference:** See `assets/database-dashboard.json`
### Application Dashboard
**Key Panels:**
- Request rate
- Error rate
- Response time (percentiles)
- Active users/sessions
- Cache hit rate
- Queue length
## Best Practices
1. **Start with templates** (Grafana community dashboards)
2. **Use consistent naming** for panels and variables
3. **Group related metrics** in rows
4. **Set appropriate time ranges** (default: Last 6 hours)
5. **Use variables** for flexibility
6. **Add panel descriptions** for context
7. **Configure units** correctly
8. **Set meaningful thresholds** for colors
9. **Use consistent colors** across dashboards
10. **Test with different time ranges**
## Dashboard as Code
### Terraform Provisioning
```hcl
resource "grafana_dashboard" "api_monitoring" {
config_json = file("${path.module}/dashboards/api-monitoring.json")
folder = grafana_folder.monitoring.id
}
resource "grafana_folder" "monitoring" {
title = "Production Monitoring"
}
```
### Ansible Provisioning
```yaml
- name: Deploy Grafana dashboards
copy:
src: "{{ item }}"
dest: /etc/grafana/dashboards/
with_fileglob:
- "dashboards/*.json"
notify: restart grafana
```
## Reference Files
- `assets/api-dashboard.json` - API monitoring dashboard
- `assets/infrastructure-dashboard.json` - Infrastructure dashboard
- `assets/database-dashboard.json` - Database monitoring dashboard
- `references/dashboard-design.md` - Dashboard design guide
## Related Skills
- `prometheus-configuration` - For metric collection
- `slo-implementation` - For SLO dashboards

392
plugins/observability-monitoring/skills/prometheus-configuration/SKILL.md Normal file
View File

@@ -0,0 +1,392 @@
---
name: prometheus-configuration
description: Set up Prometheus for comprehensive metric collection, storage, and monitoring of infrastructure and applications. Use when implementing metrics collection, setting up monitoring infrastructure, or configuring alerting systems.
---
# Prometheus Configuration
Complete guide to Prometheus setup, metric collection, scrape configuration, and recording rules.
## Purpose
Configure Prometheus for comprehensive metric collection, alerting, and monitoring of infrastructure and applications.
## When to Use
- Set up Prometheus monitoring
- Configure metric scraping
- Create recording rules
- Design alert rules
- Implement service discovery
## Prometheus Architecture
```
┌──────────────┐
│ Applications │ ← Instrumented with client libraries
└──────┬───────┘
│ /metrics endpoint
↓
┌──────────────┐
│ Prometheus │ ← Scrapes metrics periodically
│ Server │
└──────┬───────┘
│
├─→ AlertManager (alerts)
├─→ Grafana (visualization)
└─→ Long-term storage (Thanos/Cortex)
```
## Installation
### Kubernetes with Helm
```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
--set prometheus.prometheusSpec.retention=30d \
--set prometheus.prometheusSpec.storageVolumeSize=50Gi
```
### Docker Compose
```yaml
version: '3.8'
services:
prometheus:
image: prom/prometheus:latest
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus-data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention.time=30d'
volumes:
prometheus-data:
```
## Configuration File
**prometheus.yml:**
```yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
cluster: 'production'
region: 'us-west-2'
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager:9093
# Load rules files
rule_files:
- /etc/prometheus/rules/*.yml
# Scrape configurations
scrape_configs:
# Prometheus itself
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
# Node exporters
- job_name: 'node-exporter'
static_configs:
- targets:
- 'node1:9100'
- 'node2:9100'
- 'node3:9100'
relabel_configs:
- source_labels: [__address__]
target_label: instance
regex: '([^:]+)(:[0-9]+)?'
replacement: '${1}'
# Kubernetes pods with annotations
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
# Application metrics
- job_name: 'my-app'
static_configs:
- targets:
- 'app1.example.com:9090'
- 'app2.example.com:9090'
metrics_path: '/metrics'
scheme: 'https'
tls_config:
ca_file: /etc/prometheus/ca.crt
cert_file: /etc/prometheus/client.crt
key_file: /etc/prometheus/client.key
```
**Reference:** See `assets/prometheus.yml.template`
## Scrape Configurations
### Static Targets
```yaml
scrape_configs:
- job_name: 'static-targets'
static_configs:
- targets: ['host1:9100', 'host2:9100']
labels:
env: 'production'
region: 'us-west-2'
```
### File-based Service Discovery
```yaml
scrape_configs:
- job_name: 'file-sd'
file_sd_configs:
- files:
- /etc/prometheus/targets/*.json
- /etc/prometheus/targets/*.yml
refresh_interval: 5m
```
**targets/production.json:**
```json
[
{
"targets": ["app1:9090", "app2:9090"],
"labels": {
"env": "production",
"service": "api"
}
}
]
```
### Kubernetes Service Discovery
```yaml
scrape_configs:
- job_name: 'kubernetes-services'
kubernetes_sd_configs:
- role: service
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
```
**Reference:** See `references/scrape-configs.md`
## Recording Rules
Create pre-computed metrics for frequently queried expressions:
```yaml
# /etc/prometheus/rules/recording_rules.yml
groups:
- name: api_metrics
interval: 15s
rules:
# HTTP request rate per service
- record: job:http_requests:rate5m
expr: sum by (job) (rate(http_requests_total[5m]))
# Error rate percentage
- record: job:http_requests_errors:rate5m
expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
- record: job:http_requests_error_rate:percentage
expr: |
(job:http_requests_errors:rate5m / job:http_requests:rate5m) * 100
# P95 latency
- record: job:http_request_duration:p95
expr: |
histogram_quantile(0.95,
sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
)
- name: resource_metrics
interval: 30s
rules:
# CPU utilization percentage
- record: instance:node_cpu:utilization
expr: |
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Memory utilization percentage
- record: instance:node_memory:utilization
expr: |
100 - ((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100)
# Disk usage percentage
- record: instance:node_disk:utilization
expr: |
100 - ((node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100)
```
**Reference:** See `references/recording-rules.md`
## Alert Rules
```yaml
# /etc/prometheus/rules/alert_rules.yml
groups:
- name: availability
interval: 30s
rules:
- alert: ServiceDown
expr: up{job="my-app"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Service {{ $labels.instance }} is down"
description: "{{ $labels.job }} has been down for more than 1 minute"
- alert: HighErrorRate
expr: job:http_requests_error_rate:percentage > 5
for: 5m
labels:
severity: warning
annotations:
summary: "High error rate for {{ $labels.job }}"
description: "Error rate is {{ $value }}% (threshold: 5%)"
- alert: HighLatency
expr: job:http_request_duration:p95 > 1
for: 5m
labels:
severity: warning
annotations:
summary: "High latency for {{ $labels.job }}"
description: "P95 latency is {{ $value }}s (threshold: 1s)"
- name: resources
interval: 1m
rules:
- alert: HighCPUUsage
expr: instance:node_cpu:utilization > 80
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU usage on {{ $labels.instance }}"
description: "CPU usage is {{ $value }}%"
- alert: HighMemoryUsage
expr: instance:node_memory:utilization > 85
for: 5m
labels:
severity: warning
annotations:
summary: "High memory usage on {{ $labels.instance }}"
description: "Memory usage is {{ $value }}%"
- alert: DiskSpaceLow
expr: instance:node_disk:utilization > 90
for: 5m
labels:
severity: critical
annotations:
summary: "Low disk space on {{ $labels.instance }}"
description: "Disk usage is {{ $value }}%"
```
## Validation
```bash
# Validate configuration
promtool check config prometheus.yml
# Validate rules
promtool check rules /etc/prometheus/rules/*.yml
# Test query
promtool query instant http://localhost:9090 'up'
```
**Reference:** See `scripts/validate-prometheus.sh`
## Best Practices
1. **Use consistent naming** for metrics (prefix_name_unit)
2. **Set appropriate scrape intervals** (15-60s typical)
3. **Use recording rules** for expensive queries
4. **Implement high availability** (multiple Prometheus instances)
5. **Configure retention** based on storage capacity
6. **Use relabeling** for metric cleanup
7. **Monitor Prometheus itself**
8. **Implement federation** for large deployments
9. **Use Thanos/Cortex** for long-term storage
10. **Document custom metrics**
## Troubleshooting
**Check scrape targets:**
```bash
curl http://localhost:9090/api/v1/targets
```
**Check configuration:**
```bash
curl http://localhost:9090/api/v1/status/config
```
**Test query:**
```bash
curl 'http://localhost:9090/api/v1/query?query=up'
```
## Reference Files
- `assets/prometheus.yml.template` - Complete configuration template
- `references/scrape-configs.md` - Scrape configuration patterns
- `references/recording-rules.md` - Recording rule examples
- `scripts/validate-prometheus.sh` - Validation script
## Related Skills
- `grafana-dashboards` - For visualization
- `slo-implementation` - For SLO monitoring
- `distributed-tracing` - For request tracing

329
plugins/observability-monitoring/skills/slo-implementation/SKILL.md Normal file
View File

@@ -0,0 +1,329 @@
---
name: slo-implementation
description: Define and implement Service Level Indicators (SLIs) and Service Level Objectives (SLOs) with error budgets and alerting. Use when establishing reliability targets, implementing SRE practices, or measuring service performance.
---
# SLO Implementation
Framework for defining and implementing Service Level Indicators (SLIs), Service Level Objectives (SLOs), and error budgets.
## Purpose
Implement measurable reliability targets using SLIs, SLOs, and error budgets to balance reliability with innovation velocity.
## When to Use
- Define service reliability targets
- Measure user-perceived reliability
- Implement error budgets
- Create SLO-based alerts
- Track reliability goals
## SLI/SLO/SLA Hierarchy
```
SLA (Service Level Agreement)
↓ Contract with customers
SLO (Service Level Objective)
↓ Internal reliability target
SLI (Service Level Indicator)
↓ Actual measurement
```
## Defining SLIs
### Common SLI Types
#### 1. Availability SLI
```promql
# Successful requests / Total requests
sum(rate(http_requests_total{status!~"5.."}[28d]))
/
sum(rate(http_requests_total[28d]))
```
#### 2. Latency SLI
```promql
# Requests below latency threshold / Total requests
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
/
sum(rate(http_request_duration_seconds_count[28d]))
```
#### 3. Durability SLI
```
# Successful writes / Total writes
sum(storage_writes_successful_total)
/
sum(storage_writes_total)
```
**Reference:** See `references/slo-definitions.md`
## Setting SLO Targets
### Availability SLO Examples
| SLO % | Downtime/Month | Downtime/Year |
|-------|----------------|---------------|
| 99% | 7.2 hours | 3.65 days |
| 99.9% | 43.2 minutes | 8.76 hours |
| 99.95%| 21.6 minutes | 4.38 hours |
| 99.99%| 4.32 minutes | 52.56 minutes |
### Choose Appropriate SLOs
**Consider:**
- User expectations
- Business requirements
- Current performance
- Cost of reliability
- Competitor benchmarks
**Example SLOs:**
```yaml
slos:
- name: api_availability
target: 99.9
window: 28d
sli: |
sum(rate(http_requests_total{status!~"5.."}[28d]))
/
sum(rate(http_requests_total[28d]))
- name: api_latency_p95
target: 99
window: 28d
sli: |
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
/
sum(rate(http_request_duration_seconds_count[28d]))
```
## Error Budget Calculation
### Error Budget Formula
```
Error Budget = 1 - SLO Target
```
**Example:**
- SLO: 99.9% availability
- Error Budget: 0.1% = 43.2 minutes/month
- Current Error: 0.05% = 21.6 minutes/month
- Remaining Budget: 50%
### Error Budget Policy
```yaml
error_budget_policy:
- remaining_budget: 100%
action: Normal development velocity
- remaining_budget: 50%
action: Consider postponing risky changes
- remaining_budget: 10%
action: Freeze non-critical changes
- remaining_budget: 0%
action: Feature freeze, focus on reliability
```
**Reference:** See `references/error-budget.md`
## SLO Implementation
### Prometheus Recording Rules
```yaml
# SLI Recording Rules
groups:
- name: sli_rules
interval: 30s
rules:
# Availability SLI
- record: sli:http_availability:ratio
expr: |
sum(rate(http_requests_total{status!~"5.."}[28d]))
/
sum(rate(http_requests_total[28d]))
# Latency SLI (requests < 500ms)
- record: sli:http_latency:ratio
expr: |
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
/
sum(rate(http_request_duration_seconds_count[28d]))
- name: slo_rules
interval: 5m
rules:
# SLO compliance (1 = meeting SLO, 0 = violating)
- record: slo:http_availability:compliance
expr: sli:http_availability:ratio >= bool 0.999
- record: slo:http_latency:compliance
expr: sli:http_latency:ratio >= bool 0.99
# Error budget remaining (percentage)
- record: slo:http_availability:error_budget_remaining
expr: |
(sli:http_availability:ratio - 0.999) / (1 - 0.999) * 100
# Error budget burn rate
- record: slo:http_availability:burn_rate_5m
expr: |
(1 - (
sum(rate(http_requests_total{status!~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
)) / (1 - 0.999)
```
### SLO Alerting Rules
```yaml
groups:
- name: slo_alerts
interval: 1m
rules:
# Fast burn: 14.4x rate, 1 hour window
# Consumes 2% error budget in 1 hour
- alert: SLOErrorBudgetBurnFast
expr: |
slo:http_availability:burn_rate_1h > 14.4
and
slo:http_availability:burn_rate_5m > 14.4
for: 2m
labels:
severity: critical
annotations:
summary: "Fast error budget burn detected"
description: "Error budget burning at {{ $value }}x rate"
# Slow burn: 6x rate, 6 hour window
# Consumes 5% error budget in 6 hours
- alert: SLOErrorBudgetBurnSlow
expr: |
slo:http_availability:burn_rate_6h > 6
and
slo:http_availability:burn_rate_30m > 6
for: 15m
labels:
severity: warning
annotations:
summary: "Slow error budget burn detected"
description: "Error budget burning at {{ $value }}x rate"
# Error budget exhausted
- alert: SLOErrorBudgetExhausted
expr: slo:http_availability:error_budget_remaining < 0
for: 5m
labels:
severity: critical
annotations:
summary: "SLO error budget exhausted"
description: "Error budget remaining: {{ $value }}%"
```
## SLO Dashboard
**Grafana Dashboard Structure:**
```
┌────────────────────────────────────┐
│ SLO Compliance (Current) │
│ ✓ 99.95% (Target: 99.9%) │
├────────────────────────────────────┤
│ Error Budget Remaining: 65% │
│ ████████░░ 65% │
├────────────────────────────────────┤
│ SLI Trend (28 days) │
│ [Time series graph] │
├────────────────────────────────────┤
│ Burn Rate Analysis │
│ [Burn rate by time window] │
└────────────────────────────────────┘
```
**Example Queries:**
```promql
# Current SLO compliance
sli:http_availability:ratio * 100
# Error budget remaining
slo:http_availability:error_budget_remaining
# Days until error budget exhausted (at current burn rate)
(slo:http_availability:error_budget_remaining / 100)
*
28
/
(1 - sli:http_availability:ratio) * (1 - 0.999)
```
## Multi-Window Burn Rate Alerts
```yaml
# Combination of short and long windows reduces false positives
rules:
- alert: SLOBurnRateHigh
expr: |
(
slo:http_availability:burn_rate_1h > 14.4
and
slo:http_availability:burn_rate_5m > 14.4
)
or
(
slo:http_availability:burn_rate_6h > 6
and
slo:http_availability:burn_rate_30m > 6
)
labels:
severity: critical
```
## SLO Review Process
### Weekly Review
- Current SLO compliance
- Error budget status
- Trend analysis
- Incident impact
### Monthly Review
- SLO achievement
- Error budget usage
- Incident postmortems
- SLO adjustments
### Quarterly Review
- SLO relevance
- Target adjustments
- Process improvements
- Tooling enhancements
## Best Practices
1. **Start with user-facing services**
2. **Use multiple SLIs** (availability, latency, etc.)
3. **Set achievable SLOs** (don't aim for 100%)
4. **Implement multi-window alerts** to reduce noise
5. **Track error budget** consistently
6. **Review SLOs regularly**
7. **Document SLO decisions**
8. **Align with business goals**
9. **Automate SLO reporting**
10. **Use SLOs for prioritization**
## Reference Files
- `assets/slo-template.md` - SLO definition template
- `references/slo-definitions.md` - SLO definition patterns
- `references/error-budget.md` - Error budget calculations
## Related Skills
- `prometheus-configuration` - For metric collection
- `grafana-dashboards` - For SLO visualization

559
plugins/payment-processing/skills/billing-automation/SKILL.md Normal file
View File

@@ -0,0 +1,559 @@
---
name: billing-automation
description: Build automated billing systems for recurring payments, invoicing, subscription lifecycle, and dunning management. Use when implementing subscription billing, automating invoicing, or managing recurring payment systems.
---
# Billing Automation
Master automated billing systems including recurring billing, invoice generation, dunning management, proration, and tax calculation.
## When to Use This Skill
- Implementing SaaS subscription billing
- Automating invoice generation and delivery
- Managing failed payment recovery (dunning)
- Calculating prorated charges for plan changes
- Handling sales tax, VAT, and GST
- Processing usage-based billing
- Managing billing cycles and renewals
## Core Concepts
### 1. Billing Cycles
**Common Intervals:**
- Monthly (most common for SaaS)
- Annual (discounted long-term)
- Quarterly
- Weekly
- Custom (usage-based, per-seat)
### 2. Subscription States
```
trial → active → past_due → canceled
→ paused → resumed
```
### 3. Dunning Management
Automated process to recover failed payments through:
- Retry schedules
- Customer notifications
- Grace periods
- Account restrictions
### 4. Proration
Adjusting charges when:
- Upgrading/downgrading mid-cycle
- Adding/removing seats
- Changing billing frequency
## Quick Start
```python
from billing import BillingEngine, Subscription
# Initialize billing engine
billing = BillingEngine()
# Create subscription
subscription = billing.create_subscription(
customer_id="cus_123",
plan_id="plan_pro_monthly",
billing_cycle_anchor=datetime.now(),
trial_days=14
)
# Process billing cycle
billing.process_billing_cycle(subscription.id)
```
## Subscription Lifecycle Management
```python
from datetime import datetime, timedelta
from enum import Enum
class SubscriptionStatus(Enum):
TRIAL = "trial"
ACTIVE = "active"
PAST_DUE = "past_due"
CANCELED = "canceled"
PAUSED = "paused"
class Subscription:
def __init__(self, customer_id, plan, billing_cycle_day=None):
self.id = generate_id()
self.customer_id = customer_id
self.plan = plan
self.status = SubscriptionStatus.TRIAL
self.current_period_start = datetime.now()
self.current_period_end = self.current_period_start + timedelta(days=plan.trial_days or 30)
self.billing_cycle_day = billing_cycle_day or self.current_period_start.day
self.trial_end = datetime.now() + timedelta(days=plan.trial_days) if plan.trial_days else None
def start_trial(self, trial_days):
"""Start trial period."""
self.status = SubscriptionStatus.TRIAL
self.trial_end = datetime.now() + timedelta(days=trial_days)
self.current_period_end = self.trial_end
def activate(self):
"""Activate subscription after trial or immediately."""
self.status = SubscriptionStatus.ACTIVE
self.current_period_start = datetime.now()
self.current_period_end = self.calculate_next_billing_date()
def mark_past_due(self):
"""Mark subscription as past due after failed payment."""
self.status = SubscriptionStatus.PAST_DUE
# Trigger dunning workflow
def cancel(self, at_period_end=True):
"""Cancel subscription."""
if at_period_end:
self.cancel_at_period_end = True
# Will cancel when current period ends
else:
self.status = SubscriptionStatus.CANCELED
self.canceled_at = datetime.now()
def calculate_next_billing_date(self):
"""Calculate next billing date based on interval."""
if self.plan.interval == 'month':
return self.current_period_start + timedelta(days=30)
elif self.plan.interval == 'year':
return self.current_period_start + timedelta(days=365)
elif self.plan.interval == 'week':
return self.current_period_start + timedelta(days=7)
```
## Billing Cycle Processing
```python
class BillingEngine:
def process_billing_cycle(self, subscription_id):
"""Process billing for a subscription."""
subscription = self.get_subscription(subscription_id)
# Check if billing is due
if datetime.now() < subscription.current_period_end:
return
# Generate invoice
invoice = self.generate_invoice(subscription)
# Attempt payment
payment_result = self.charge_customer(
subscription.customer_id,
invoice.total
)
if payment_result.success:
# Payment successful
invoice.mark_paid()
subscription.advance_billing_period()
self.send_invoice(invoice)
else:
# Payment failed
subscription.mark_past_due()
self.start_dunning_process(subscription, invoice)
def generate_invoice(self, subscription):
"""Generate invoice for billing period."""
invoice = Invoice(
customer_id=subscription.customer_id,
subscription_id=subscription.id,
period_start=subscription.current_period_start,
period_end=subscription.current_period_end
)
# Add subscription line item
invoice.add_line_item(
description=subscription.plan.name,
amount=subscription.plan.amount,
quantity=subscription.quantity or 1
)
# Add usage-based charges if applicable
if subscription.has_usage_billing:
usage_charges = self.calculate_usage_charges(subscription)
invoice.add_line_item(
description="Usage charges",
amount=usage_charges
)
# Calculate tax
tax = self.calculate_tax(invoice.subtotal, subscription.customer)
invoice.tax = tax
invoice.finalize()
return invoice
def charge_customer(self, customer_id, amount):
"""Charge customer using saved payment method."""
customer = self.get_customer(customer_id)
try:
# Charge using payment processor
charge = stripe.Charge.create(
customer=customer.stripe_id,
amount=int(amount * 100), # Convert to cents
currency='usd'
)
return PaymentResult(success=True, transaction_id=charge.id)
except stripe.error.CardError as e:
return PaymentResult(success=False, error=str(e))
```
## Dunning Management
```python
class DunningManager:
"""Manage failed payment recovery."""
def __init__(self):
self.retry_schedule = [
{'days': 3, 'email_template': 'payment_failed_first'},
{'days': 7, 'email_template': 'payment_failed_reminder'},
{'days': 14, 'email_template': 'payment_failed_final'}
]
def start_dunning_process(self, subscription, invoice):
"""Start dunning process for failed payment."""
dunning_attempt = DunningAttempt(
subscription_id=subscription.id,
invoice_id=invoice.id,
attempt_number=1,
next_retry=datetime.now() + timedelta(days=3)
)
# Send initial failure notification
self.send_dunning_email(subscription, 'payment_failed_first')
# Schedule retries
self.schedule_retries(dunning_attempt)
def retry_payment(self, dunning_attempt):
"""Retry failed payment."""
subscription = self.get_subscription(dunning_attempt.subscription_id)
invoice = self.get_invoice(dunning_attempt.invoice_id)
# Attempt payment again
result = self.charge_customer(subscription.customer_id, invoice.total)
if result.success:
# Payment succeeded
invoice.mark_paid()
subscription.status = SubscriptionStatus.ACTIVE
self.send_dunning_email(subscription, 'payment_recovered')
dunning_attempt.mark_resolved()
else:
# Still failing
dunning_attempt.attempt_number += 1
if dunning_attempt.attempt_number < len(self.retry_schedule):
# Schedule next retry
next_retry_config = self.retry_schedule[dunning_attempt.attempt_number]
dunning_attempt.next_retry = datetime.now() + timedelta(days=next_retry_config['days'])
self.send_dunning_email(subscription, next_retry_config['email_template'])
else:
# Exhausted retries, cancel subscription
subscription.cancel(at_period_end=False)
self.send_dunning_email(subscription, 'subscription_canceled')
def send_dunning_email(self, subscription, template):
"""Send dunning notification to customer."""
customer = self.get_customer(subscription.customer_id)
email_content = self.render_template(template, {
'customer_name': customer.name,
'amount_due': subscription.plan.amount,
'update_payment_url': f"https://app.example.com/billing"
})
send_email(
to=customer.email,
subject=email_content['subject'],
body=email_content['body']
)
```
## Proration
```python
class ProrationCalculator:
"""Calculate prorated charges for plan changes."""
@staticmethod
def calculate_proration(old_plan, new_plan, period_start, period_end, change_date):
"""Calculate proration for plan change."""
# Days in current period
total_days = (period_end - period_start).days
# Days used on old plan
days_used = (change_date - period_start).days
# Days remaining on new plan
days_remaining = (period_end - change_date).days
# Calculate prorated amounts
unused_amount = (old_plan.amount / total_days) * days_remaining
new_plan_amount = (new_plan.amount / total_days) * days_remaining
# Net charge/credit
proration = new_plan_amount - unused_amount
return {
'old_plan_credit': -unused_amount,
'new_plan_charge': new_plan_amount,
'net_proration': proration,
'days_used': days_used,
'days_remaining': days_remaining
}
@staticmethod
def calculate_seat_proration(current_seats, new_seats, price_per_seat, period_start, period_end, change_date):
"""Calculate proration for seat changes."""
total_days = (period_end - period_start).days
days_remaining = (period_end - change_date).days
# Additional seats charge
additional_seats = new_seats - current_seats
prorated_amount = (additional_seats * price_per_seat / total_days) * days_remaining
return {
'additional_seats': additional_seats,
'prorated_charge': max(0, prorated_amount), # No refund for removing seats mid-cycle
'effective_date': change_date
}
```
## Tax Calculation
```python
class TaxCalculator:
"""Calculate sales tax, VAT, GST."""
def __init__(self):
# Tax rates by region
self.tax_rates = {
'US_CA': 0.0725, # California sales tax
'US_NY': 0.04, # New York sales tax
'GB': 0.20, # UK VAT
'DE': 0.19, # Germany VAT
'FR': 0.20, # France VAT
'AU': 0.10, # Australia GST
}
def calculate_tax(self, amount, customer):
"""Calculate applicable tax."""
# Determine tax jurisdiction
jurisdiction = self.get_tax_jurisdiction(customer)
if not jurisdiction:
return 0
# Get tax rate
tax_rate = self.tax_rates.get(jurisdiction, 0)
# Calculate tax
tax = amount * tax_rate
return {
'tax_amount': tax,
'tax_rate': tax_rate,
'jurisdiction': jurisdiction,
'tax_type': self.get_tax_type(jurisdiction)
}
def get_tax_jurisdiction(self, customer):
"""Determine tax jurisdiction based on customer location."""
if customer.country == 'US':
# US: Tax based on customer state
return f"US_{customer.state}"
elif customer.country in ['GB', 'DE', 'FR']:
# EU: VAT
return customer.country
elif customer.country == 'AU':
# Australia: GST
return 'AU'
else:
return None
def get_tax_type(self, jurisdiction):
"""Get type of tax for jurisdiction."""
if jurisdiction.startswith('US_'):
return 'Sales Tax'
elif jurisdiction in ['GB', 'DE', 'FR']:
return 'VAT'
elif jurisdiction == 'AU':
return 'GST'
return 'Tax'
def validate_vat_number(self, vat_number, country):
"""Validate EU VAT number."""
# Use VIES API for validation
# Returns True if valid, False otherwise
pass
```
## Invoice Generation
```python
class Invoice:
def __init__(self, customer_id, subscription_id=None):
self.id = generate_invoice_number()
self.customer_id = customer_id
self.subscription_id = subscription_id
self.status = 'draft'
self.line_items = []
self.subtotal = 0
self.tax = 0
self.total = 0
self.created_at = datetime.now()
def add_line_item(self, description, amount, quantity=1):
"""Add line item to invoice."""
line_item = {
'description': description,
'unit_amount': amount,
'quantity': quantity,
'total': amount * quantity
}
self.line_items.append(line_item)
self.subtotal += line_item['total']
def finalize(self):
"""Finalize invoice and calculate total."""
self.total = self.subtotal + self.tax
self.status = 'open'
self.finalized_at = datetime.now()
def mark_paid(self):
"""Mark invoice as paid."""
self.status = 'paid'
self.paid_at = datetime.now()
def to_pdf(self):
"""Generate PDF invoice."""
from reportlab.pdfgen import canvas
# Generate PDF
# Include: company info, customer info, line items, tax, total
pass
def to_html(self):
"""Generate HTML invoice."""
template = """
<!DOCTYPE html>
<html>
<head><title>Invoice #{invoice_number}</title></head>
<body>
<h1>Invoice #{invoice_number}</h1>
<p>Date: {date}</p>
<h2>Bill To:</h2>
<p>{customer_name}<br>{customer_address}</p>
<table>
<tr><th>Description</th><th>Quantity</th><th>Amount</th></tr>
{line_items}
</table>
<p>Subtotal: ${subtotal}</p>
<p>Tax: ${tax}</p>
<h3>Total: ${total}</h3>
</body>
</html>
"""
return template.format(
invoice_number=self.id,
date=self.created_at.strftime('%Y-%m-%d'),
customer_name=self.customer.name,
customer_address=self.customer.address,
line_items=self.render_line_items(),
subtotal=self.subtotal,
tax=self.tax,
total=self.total
)
```
## Usage-Based Billing
```python
class UsageBillingEngine:
"""Track and bill for usage."""
def track_usage(self, customer_id, metric, quantity):
"""Track usage event."""
UsageRecord.create(
customer_id=customer_id,
metric=metric,
quantity=quantity,
timestamp=datetime.now()
)
def calculate_usage_charges(self, subscription, period_start, period_end):
"""Calculate charges for usage in billing period."""
usage_records = UsageRecord.get_for_period(
subscription.customer_id,
period_start,
period_end
)
total_usage = sum(record.quantity for record in usage_records)
# Tiered pricing
if subscription.plan.pricing_model == 'tiered':
charge = self.calculate_tiered_pricing(total_usage, subscription.plan.tiers)
# Per-unit pricing
elif subscription.plan.pricing_model == 'per_unit':
charge = total_usage * subscription.plan.unit_price
# Volume pricing
elif subscription.plan.pricing_model == 'volume':
charge = self.calculate_volume_pricing(total_usage, subscription.plan.tiers)
return charge
def calculate_tiered_pricing(self, total_usage, tiers):
"""Calculate cost using tiered pricing."""
charge = 0
remaining = total_usage
for tier in sorted(tiers, key=lambda x: x['up_to']):
tier_usage = min(remaining, tier['up_to'] - tier['from'])
charge += tier_usage * tier['unit_price']
remaining -= tier_usage
if remaining <= 0:
break
return charge
```
## Resources
- **references/billing-cycles.md**: Billing cycle management
- **references/dunning-management.md**: Failed payment recovery
- **references/proration.md**: Prorated charge calculations
- **references/tax-calculation.md**: Tax/VAT/GST handling
- **references/invoice-lifecycle.md**: Invoice state management
- **assets/billing-state-machine.yaml**: Billing workflow
- **assets/invoice-template.html**: Invoice templates
- **assets/dunning-policy.yaml**: Dunning configuration
## Best Practices
1. **Automate Everything**: Minimize manual intervention
2. **Clear Communication**: Notify customers of billing events
3. **Flexible Retry Logic**: Balance recovery with customer experience
4. **Accurate Proration**: Fair calculation for plan changes
5. **Tax Compliance**: Calculate correct tax for jurisdiction
6. **Audit Trail**: Log all billing events
7. **Graceful Degradation**: Handle edge cases without breaking
## Common Pitfalls
- **Incorrect Proration**: Not accounting for partial periods
- **Missing Tax**: Forgetting to add tax to invoices
- **Aggressive Dunning**: Canceling too quickly
- **No Notifications**: Not informing customers of failures
- **Hardcoded Cycles**: Not supporting custom billing dates

467
plugins/payment-processing/skills/paypal-integration/SKILL.md Normal file
View File

@@ -0,0 +1,467 @@
---
name: paypal-integration
description: Integrate PayPal payment processing with support for express checkout, subscriptions, and refund management. Use when implementing PayPal payments, processing online transactions, or building e-commerce checkout flows.
---
# PayPal Integration
Master PayPal payment integration including Express Checkout, IPN handling, recurring billing, and refund workflows.
## When to Use This Skill
- Integrating PayPal as a payment option
- Implementing express checkout flows
- Setting up recurring billing with PayPal
- Processing refunds and payment disputes
- Handling PayPal webhooks (IPN)
- Supporting international payments
- Implementing PayPal subscriptions
## Core Concepts
### 1. Payment Products
**PayPal Checkout**
- One-time payments
- Express checkout experience
- Guest and PayPal account payments
**PayPal Subscriptions**
- Recurring billing
- Subscription plans
- Automatic renewals
**PayPal Payouts**
- Send money to multiple recipients
- Marketplace and platform payments
### 2. Integration Methods
**Client-Side (JavaScript SDK)**
- Smart Payment Buttons
- Hosted payment flow
- Minimal backend code
**Server-Side (REST API)**
- Full control over payment flow
- Custom checkout UI
- Advanced features
### 3. IPN (Instant Payment Notification)
- Webhook-like payment notifications
- Asynchronous payment updates
- Verification required
## Quick Start
```javascript
// Frontend - PayPal Smart Buttons
<div id="paypal-button-container"></div>
<script src="https://www.paypal.com/sdk/js?client-id=YOUR_CLIENT_ID&currency=USD"></script>
<script>
paypal.Buttons({
createOrder: function(data, actions) {
return actions.order.create({
purchase_units: [{
amount: {
value: '25.00'
}
}]
});
},
onApprove: function(data, actions) {
return actions.order.capture().then(function(details) {
// Payment successful
console.log('Transaction completed by ' + details.payer.name.given_name);
// Send to backend for verification
fetch('/api/paypal/capture', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({orderID: data.orderID})
});
});
}
}).render('#paypal-button-container');
</script>
```
```python
# Backend - Verify and capture order
from paypalrestsdk import Payment
import paypalrestsdk
paypalrestsdk.configure({
"mode": "sandbox", # or "live"
"client_id": "YOUR_CLIENT_ID",
"client_secret": "YOUR_CLIENT_SECRET"
})
def capture_paypal_order(order_id):
"""Capture a PayPal order."""
payment = Payment.find(order_id)
if payment.execute({"payer_id": payment.payer.payer_info.payer_id}):
# Payment successful
return {
'status': 'success',
'transaction_id': payment.id,
'amount': payment.transactions[0].amount.total
}
else:
# Payment failed
return {
'status': 'failed',
'error': payment.error
}
```
## Express Checkout Implementation
### Server-Side Order Creation
```python
import requests
import json
class PayPalClient:
def __init__(self, client_id, client_secret, mode='sandbox'):
self.client_id = client_id
self.client_secret = client_secret
self.base_url = 'https://api-m.sandbox.paypal.com' if mode == 'sandbox' else 'https://api-m.paypal.com'
self.access_token = self.get_access_token()
def get_access_token(self):
"""Get OAuth access token."""
url = f"{self.base_url}/v1/oauth2/token"
headers = {"Accept": "application/json", "Accept-Language": "en_US"}
response = requests.post(
url,
headers=headers,
data={"grant_type": "client_credentials"},
auth=(self.client_id, self.client_secret)
)
return response.json()['access_token']
def create_order(self, amount, currency='USD'):
"""Create a PayPal order."""
url = f"{self.base_url}/v2/checkout/orders"
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {self.access_token}"
}
payload = {
"intent": "CAPTURE",
"purchase_units": [{
"amount": {
"currency_code": currency,
"value": str(amount)
}
}]
}
response = requests.post(url, headers=headers, json=payload)
return response.json()
def capture_order(self, order_id):
"""Capture payment for an order."""
url = f"{self.base_url}/v2/checkout/orders/{order_id}/capture"
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {self.access_token}"
}
response = requests.post(url, headers=headers)
return response.json()
def get_order_details(self, order_id):
"""Get order details."""
url = f"{self.base_url}/v2/checkout/orders/{order_id}"
headers = {
"Authorization": f"Bearer {self.access_token}"
}
response = requests.get(url, headers=headers)
return response.json()
```
## IPN (Instant Payment Notification) Handling
### IPN Verification and Processing
```python
from flask import Flask, request
import requests
from urllib.parse import parse_qs
app = Flask(__name__)
@app.route('/ipn', methods=['POST'])
def handle_ipn():
"""Handle PayPal IPN notifications."""
# Get IPN message
ipn_data = request.form.to_dict()
# Verify IPN with PayPal
if not verify_ipn(ipn_data):
return 'IPN verification failed', 400
# Process IPN based on transaction type
payment_status = ipn_data.get('payment_status')
txn_type = ipn_data.get('txn_type')
if payment_status == 'Completed':
handle_payment_completed(ipn_data)
elif payment_status == 'Refunded':
handle_refund(ipn_data)
elif payment_status == 'Reversed':
handle_chargeback(ipn_data)
return 'IPN processed', 200
def verify_ipn(ipn_data):
"""Verify IPN message authenticity."""
# Add 'cmd' parameter
verify_data = ipn_data.copy()
verify_data['cmd'] = '_notify-validate'
# Send back to PayPal for verification
paypal_url = 'https://ipnpb.sandbox.paypal.com/cgi-bin/webscr' # or production URL
response = requests.post(paypal_url, data=verify_data)
return response.text == 'VERIFIED'
def handle_payment_completed(ipn_data):
"""Process completed payment."""
txn_id = ipn_data.get('txn_id')
payer_email = ipn_data.get('payer_email')
mc_gross = ipn_data.get('mc_gross')
item_name = ipn_data.get('item_name')
# Check if already processed (prevent duplicates)
if is_transaction_processed(txn_id):
return
# Update database
# Send confirmation email
# Fulfill order
print(f"Payment completed: {txn_id}, Amount: ${mc_gross}")
def handle_refund(ipn_data):
"""Handle refund."""
parent_txn_id = ipn_data.get('parent_txn_id')
mc_gross = ipn_data.get('mc_gross')
# Process refund in your system
print(f"Refund processed: {parent_txn_id}, Amount: ${mc_gross}")
def handle_chargeback(ipn_data):
"""Handle payment reversal/chargeback."""
txn_id = ipn_data.get('txn_id')
reason_code = ipn_data.get('reason_code')
# Handle chargeback
print(f"Chargeback: {txn_id}, Reason: {reason_code}")
```
## Subscription/Recurring Billing
### Create Subscription Plan
```python
def create_subscription_plan(name, amount, interval='MONTH'):
"""Create a subscription plan."""
client = PayPalClient(CLIENT_ID, CLIENT_SECRET)
url = f"{client.base_url}/v1/billing/plans"
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {client.access_token}"
}
payload = {
"product_id": "PRODUCT_ID", # Create product first
"name": name,
"billing_cycles": [{
"frequency": {
"interval_unit": interval,
"interval_count": 1
},
"tenure_type": "REGULAR",
"sequence": 1,
"total_cycles": 0, # Infinite
"pricing_scheme": {
"fixed_price": {
"value": str(amount),
"currency_code": "USD"
}
}
}],
"payment_preferences": {
"auto_bill_outstanding": True,
"setup_fee": {
"value": "0",
"currency_code": "USD"
},
"setup_fee_failure_action": "CONTINUE",
"payment_failure_threshold": 3
}
}
response = requests.post(url, headers=headers, json=payload)
return response.json()
def create_subscription(plan_id, subscriber_email):
"""Create a subscription for a customer."""
client = PayPalClient(CLIENT_ID, CLIENT_SECRET)
url = f"{client.base_url}/v1/billing/subscriptions"
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {client.access_token}"
}
payload = {
"plan_id": plan_id,
"subscriber": {
"email_address": subscriber_email
},
"application_context": {
"return_url": "https://yourdomain.com/subscription/success",
"cancel_url": "https://yourdomain.com/subscription/cancel"
}
}
response = requests.post(url, headers=headers, json=payload)
subscription = response.json()
# Get approval URL
for link in subscription.get('links', []):
if link['rel'] == 'approve':
return {
'subscription_id': subscription['id'],
'approval_url': link['href']
}
```
## Refund Workflows
```python
def create_refund(capture_id, amount=None, note=None):
"""Create a refund for a captured payment."""
client = PayPalClient(CLIENT_ID, CLIENT_SECRET)
url = f"{client.base_url}/v2/payments/captures/{capture_id}/refund"
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {client.access_token}"
}
payload = {}
if amount:
payload["amount"] = {
"value": str(amount),
"currency_code": "USD"
}
if note:
payload["note_to_payer"] = note
response = requests.post(url, headers=headers, json=payload)
return response.json()
def get_refund_details(refund_id):
"""Get refund details."""
client = PayPalClient(CLIENT_ID, CLIENT_SECRET)
url = f"{client.base_url}/v2/payments/refunds/{refund_id}"
headers = {
"Authorization": f"Bearer {client.access_token}"
}
response = requests.get(url, headers=headers)
return response.json()
```
## Error Handling
```python
class PayPalError(Exception):
"""Custom PayPal error."""
pass
def handle_paypal_api_call(api_function):
"""Wrapper for PayPal API calls with error handling."""
try:
result = api_function()
return result
except requests.exceptions.RequestException as e:
# Network error
raise PayPalError(f"Network error: {str(e)}")
except Exception as e:
# Other errors
raise PayPalError(f"PayPal API error: {str(e)}")
# Usage
try:
order = handle_paypal_api_call(lambda: client.create_order(25.00))
except PayPalError as e:
# Handle error appropriately
log_error(e)
```
## Testing
```python
# Use sandbox credentials
SANDBOX_CLIENT_ID = "..."
SANDBOX_SECRET = "..."
# Test accounts
# Create test buyer and seller accounts at developer.paypal.com
def test_payment_flow():
"""Test complete payment flow."""
client = PayPalClient(SANDBOX_CLIENT_ID, SANDBOX_SECRET, mode='sandbox')
# Create order
order = client.create_order(10.00)
assert 'id' in order
# Get approval URL
approval_url = next((link['href'] for link in order['links'] if link['rel'] == 'approve'), None)
assert approval_url is not None
# After approval (manual step with test account)
# Capture order
# captured = client.capture_order(order['id'])
# assert captured['status'] == 'COMPLETED'
```
## Resources
- **references/express-checkout.md**: Express Checkout implementation guide
- **references/ipn-handling.md**: IPN verification and processing
- **references/refund-workflows.md**: Refund handling patterns
- **references/billing-agreements.md**: Recurring billing setup
- **assets/paypal-client.py**: Production PayPal client
- **assets/ipn-processor.py**: IPN webhook processor
- **assets/recurring-billing.py**: Subscription management
## Best Practices
1. **Always Verify IPN**: Never trust IPN without verification
2. **Idempotent Processing**: Handle duplicate IPN notifications
3. **Error Handling**: Implement robust error handling
4. **Logging**: Log all transactions and errors
5. **Test Thoroughly**: Use sandbox extensively
6. **Webhook Backup**: Don't rely solely on client-side callbacks
7. **Currency Handling**: Always specify currency explicitly
## Common Pitfalls
- **Not Verifying IPN**: Accepting IPN without verification
- **Duplicate Processing**: Not checking for duplicate transactions
- **Wrong Environment**: Mixing sandbox and production URLs/credentials
- **Missing Webhooks**: Not handling all payment states
- **Hardcoded Values**: Not making configurable for different environments

466
plugins/payment-processing/skills/pci-compliance/SKILL.md Normal file
View File

@@ -0,0 +1,466 @@
---
name: pci-compliance
description: Implement PCI DSS compliance requirements for secure handling of payment card data and payment systems. Use when securing payment processing, achieving PCI compliance, or implementing payment card security measures.
---
# PCI Compliance
Master PCI DSS (Payment Card Industry Data Security Standard) compliance for secure payment processing and handling of cardholder data.
## When to Use This Skill
- Building payment processing systems
- Handling credit card information
- Implementing secure payment flows
- Conducting PCI compliance audits
- Reducing PCI compliance scope
- Implementing tokenization and encryption
- Preparing for PCI DSS assessments
## PCI DSS Requirements (12 Core Requirements)
### Build and Maintain Secure Network
1. Install and maintain firewall configuration
2. Don't use vendor-supplied defaults for passwords
### Protect Cardholder Data
3. Protect stored cardholder data
4. Encrypt transmission of cardholder data across public networks
### Maintain Vulnerability Management
5. Protect systems against malware
6. Develop and maintain secure systems and applications
### Implement Strong Access Control
7. Restrict access to cardholder data by business need-to-know
8. Identify and authenticate access to system components
9. Restrict physical access to cardholder data
### Monitor and Test Networks
10. Track and monitor all access to network resources and cardholder data
11. Regularly test security systems and processes
### Maintain Information Security Policy
12. Maintain a policy that addresses information security
## Compliance Levels
**Level 1**: > 6 million transactions/year (annual ROC required)
**Level 2**: 1-6 million transactions/year (annual SAQ)
**Level 3**: 20,000-1 million e-commerce transactions/year
**Level 4**: < 20,000 e-commerce or < 1 million total transactions
## Data Minimization (Never Store)
```python
# NEVER STORE THESE
PROHIBITED_DATA = {
'full_track_data': 'Magnetic stripe data',
'cvv': 'Card verification code/value',
'pin': 'PIN or PIN block'
}
# CAN STORE (if encrypted)
ALLOWED_DATA = {
'pan': 'Primary Account Number (card number)',
'cardholder_name': 'Name on card',
'expiration_date': 'Card expiration',
'service_code': 'Service code'
}
class PaymentData:
"""Safe payment data handling."""
def __init__(self):
self.prohibited_fields = ['cvv', 'cvv2', 'cvc', 'pin']
def sanitize_log(self, data):
"""Remove sensitive data from logs."""
sanitized = data.copy()
# Mask PAN
if 'card_number' in sanitized:
card = sanitized['card_number']
sanitized['card_number'] = f"{card[:6]}{'*' * (len(card) - 10)}{card[-4:]}"
# Remove prohibited data
for field in self.prohibited_fields:
sanitized.pop(field, None)
return sanitized
def validate_no_prohibited_storage(self, data):
"""Ensure no prohibited data is being stored."""
for field in self.prohibited_fields:
if field in data:
raise SecurityError(f"Attempting to store prohibited field: {field}")
```
## Tokenization
### Using Payment Processor Tokens
```python
import stripe
class TokenizedPayment:
"""Handle payments using tokens (no card data on server)."""
@staticmethod
def create_payment_method_token(card_details):
"""Create token from card details (client-side only)."""
# THIS SHOULD ONLY BE DONE CLIENT-SIDE WITH STRIPE.JS
# NEVER send card details to your server
"""
// Frontend JavaScript
const stripe = Stripe('pk_...');
const {token, error} = await stripe.createToken({
card: {
number: '4242424242424242',
exp_month: 12,
exp_year: 2024,
cvc: '123'
}
});
// Send token.id to server (NOT card details)
"""
pass
@staticmethod
def charge_with_token(token_id, amount):
"""Charge using token (server-side)."""
# Your server only sees the token, never the card number
stripe.api_key = "sk_..."
charge = stripe.Charge.create(
amount=amount,
currency="usd",
source=token_id, # Token instead of card details
description="Payment"
)
return charge
@staticmethod
def store_payment_method(customer_id, payment_method_token):
"""Store payment method as token for future use."""
stripe.Customer.modify(
customer_id,
source=payment_method_token
)
# Store only customer_id and payment_method_id in your database
# NEVER store actual card details
return {
'customer_id': customer_id,
'has_payment_method': True
# DO NOT store: card number, CVV, etc.
}
```
### Custom Tokenization (Advanced)
```python
import secrets
from cryptography.fernet import Fernet
class TokenVault:
"""Secure token vault for card data (if you must store it)."""
def __init__(self, encryption_key):
self.cipher = Fernet(encryption_key)
self.vault = {} # In production: use encrypted database
def tokenize(self, card_data):
"""Convert card data to token."""
# Generate secure random token
token = secrets.token_urlsafe(32)
# Encrypt card data
encrypted = self.cipher.encrypt(json.dumps(card_data).encode())
# Store token -> encrypted data mapping
self.vault[token] = encrypted
return token
def detokenize(self, token):
"""Retrieve card data from token."""
encrypted = self.vault.get(token)
if not encrypted:
raise ValueError("Token not found")
# Decrypt
decrypted = self.cipher.decrypt(encrypted)
return json.loads(decrypted.decode())
def delete_token(self, token):
"""Remove token from vault."""
self.vault.pop(token, None)
```
## Encryption
### Data at Rest
```python
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import os
class EncryptedStorage:
"""Encrypt data at rest using AES-256-GCM."""
def __init__(self, encryption_key):
"""Initialize with 256-bit key."""
self.key = encryption_key # Must be 32 bytes
def encrypt(self, plaintext):
"""Encrypt data."""
# Generate random nonce
nonce = os.urandom(12)
# Encrypt
aesgcm = AESGCM(self.key)
ciphertext = aesgcm.encrypt(nonce, plaintext.encode(), None)
# Return nonce + ciphertext
return nonce + ciphertext
def decrypt(self, encrypted_data):
"""Decrypt data."""
# Extract nonce and ciphertext
nonce = encrypted_data[:12]
ciphertext = encrypted_data[12:]
# Decrypt
aesgcm = AESGCM(self.key)
plaintext = aesgcm.decrypt(nonce, ciphertext, None)
return plaintext.decode()
# Usage
storage = EncryptedStorage(os.urandom(32))
encrypted_pan = storage.encrypt("4242424242424242")
# Store encrypted_pan in database
```
### Data in Transit
```python
# Always use TLS 1.2 or higher
# Flask/Django example
app.config['SESSION_COOKIE_SECURE'] = True # HTTPS only
app.config['SESSION_COOKIE_HTTPONLY'] = True
app.config['SESSION_COOKIE_SAMESITE'] = 'Strict'
# Enforce HTTPS
from flask_talisman import Talisman
Talisman(app, force_https=True)
```
## Access Control
```python
from functools import wraps
from flask import session
def require_pci_access(f):
"""Decorator to restrict access to cardholder data."""
@wraps(f)
def decorated_function(*args, **kwargs):
user = session.get('user')
# Check if user has PCI access role
if not user or 'pci_access' not in user.get('roles', []):
return {'error': 'Unauthorized access to cardholder data'}, 403
# Log access attempt
audit_log(
user=user['id'],
action='access_cardholder_data',
resource=f.__name__
)
return f(*args, **kwargs)
return decorated_function
@app.route('/api/payment-methods')
@require_pci_access
def get_payment_methods():
"""Retrieve payment methods (restricted access)."""
# Only accessible to users with pci_access role
pass
```
## Audit Logging
```python
import logging
from datetime import datetime
class PCIAuditLogger:
"""PCI-compliant audit logging."""
def __init__(self):
self.logger = logging.getLogger('pci_audit')
# Configure to write to secure, append-only log
def log_access(self, user_id, resource, action, result):
"""Log access to cardholder data."""
entry = {
'timestamp': datetime.utcnow().isoformat(),
'user_id': user_id,
'resource': resource,
'action': action,
'result': result,
'ip_address': request.remote_addr
}
self.logger.info(json.dumps(entry))
def log_authentication(self, user_id, success, method):
"""Log authentication attempt."""
entry = {
'timestamp': datetime.utcnow().isoformat(),
'user_id': user_id,
'event': 'authentication',
'success': success,
'method': method,
'ip_address': request.remote_addr
}
self.logger.info(json.dumps(entry))
# Usage
audit = PCIAuditLogger()
audit.log_access(user_id=123, resource='payment_methods', action='read', result='success')
```
## Security Best Practices
### Input Validation
```python
import re
def validate_card_number(card_number):
"""Validate card number format (Luhn algorithm)."""
# Remove spaces and dashes
card_number = re.sub(r'[\s-]', '', card_number)
# Check if all digits
if not card_number.isdigit():
return False
# Luhn algorithm
def luhn_checksum(card_num):
def digits_of(n):
return [int(d) for d in str(n)]
digits = digits_of(card_num)
odd_digits = digits[-1::-2]
even_digits = digits[-2::-2]
checksum = sum(odd_digits)
for d in even_digits:
checksum += sum(digits_of(d * 2))
return checksum % 10
return luhn_checksum(card_number) == 0
def sanitize_input(user_input):
"""Sanitize user input to prevent injection."""
# Remove special characters
# Validate against expected format
# Escape for database queries
pass
```
## PCI DSS SAQ (Self-Assessment Questionnaire)
### SAQ A (Least Requirements)
- E-commerce using hosted payment page
- No card data on your systems
- ~20 questions
### SAQ A-EP
- E-commerce with embedded payment form
- Uses JavaScript to handle card data
- ~180 questions
### SAQ D (Most Requirements)
- Store, process, or transmit card data
- Full PCI DSS requirements
- ~300 questions
## Compliance Checklist
```python
PCI_COMPLIANCE_CHECKLIST = {
'network_security': [
'Firewall configured and maintained',
'No vendor default passwords',
'Network segmentation implemented'
],
'data_protection': [
'No storage of CVV, track data, or PIN',
'PAN encrypted when stored',
'PAN masked when displayed',
'Encryption keys properly managed'
],
'vulnerability_management': [
'Anti-virus installed and updated',
'Secure development practices',
'Regular security patches',
'Vulnerability scanning performed'
],
'access_control': [
'Access restricted by role',
'Unique IDs for all users',
'Multi-factor authentication',
'Physical security measures'
],
'monitoring': [
'Audit logs enabled',
'Log review process',
'File integrity monitoring',
'Regular security testing'
],
'policy': [
'Security policy documented',
'Risk assessment performed',
'Security awareness training',
'Incident response plan'
]
}
```
## Resources
- **references/data-minimization.md**: Never store prohibited data
- **references/tokenization.md**: Tokenization strategies
- **references/encryption.md**: Encryption requirements
- **references/access-control.md**: Role-based access
- **references/audit-logging.md**: Comprehensive logging
- **assets/pci-compliance-checklist.md**: Complete checklist
- **assets/encrypted-storage.py**: Encryption utilities
- **scripts/audit-payment-system.sh**: Compliance audit script
## Common Violations
1. **Storing CVV**: Never store card verification codes
2. **Unencrypted PAN**: Card numbers must be encrypted at rest
3. **Weak Encryption**: Use AES-256 or equivalent
4. **No Access Controls**: Restrict who can access cardholder data
5. **Missing Audit Logs**: Must log all access to payment data
6. **Insecure Transmission**: Always use TLS 1.2+
7. **Default Passwords**: Change all default credentials
8. **No Security Testing**: Regular penetration testing required
## Reducing PCI Scope
1. **Use Hosted Payments**: Stripe Checkout, PayPal, etc.
2. **Tokenization**: Replace card data with tokens
3. **Network Segmentation**: Isolate cardholder data environment
4. **Outsource**: Use PCI-compliant payment processors
5. **No Storage**: Never store full card details
By minimizing systems that touch card data, you reduce compliance burden significantly.

442
plugins/payment-processing/skills/stripe-integration/SKILL.md Normal file
View File

@@ -0,0 +1,442 @@
---
name: stripe-integration
description: Implement Stripe payment processing for robust, PCI-compliant payment flows including checkout, subscriptions, and webhooks. Use when integrating Stripe payments, building subscription systems, or implementing secure checkout flows.
---
# Stripe Integration
Master Stripe payment processing integration for robust, PCI-compliant payment flows including checkout, subscriptions, webhooks, and refunds.
## When to Use This Skill
- Implementing payment processing in web/mobile applications
- Setting up subscription billing systems
- Handling one-time payments and recurring charges
- Processing refunds and disputes
- Managing customer payment methods
- Implementing SCA (Strong Customer Authentication) for European payments
- Building marketplace payment flows with Stripe Connect
## Core Concepts
### 1. Payment Flows
**Checkout Session (Hosted)**
- Stripe-hosted payment page
- Minimal PCI compliance burden
- Fastest implementation
- Supports one-time and recurring payments
**Payment Intents (Custom UI)**
- Full control over payment UI
- Requires Stripe.js for PCI compliance
- More complex implementation
- Better customization options
**Setup Intents (Save Payment Methods)**
- Collect payment method without charging
- Used for subscriptions and future payments
- Requires customer confirmation
### 2. Webhooks
**Critical Events:**
- `payment_intent.succeeded`: Payment completed
- `payment_intent.payment_failed`: Payment failed
- `customer.subscription.updated`: Subscription changed
- `customer.subscription.deleted`: Subscription canceled
- `charge.refunded`: Refund processed
- `invoice.payment_succeeded`: Subscription payment successful
### 3. Subscriptions
**Components:**
- **Product**: What you're selling
- **Price**: How much and how often
- **Subscription**: Customer's recurring payment
- **Invoice**: Generated for each billing cycle
### 4. Customer Management
- Create and manage customer records
- Store multiple payment methods
- Track customer metadata
- Manage billing details
## Quick Start
```python
import stripe
stripe.api_key = "sk_test_..."
# Create a checkout session
session = stripe.checkout.Session.create(
payment_method_types=['card'],
line_items=[{
'price_data': {
'currency': 'usd',
'product_data': {
'name': 'Premium Subscription',
},
'unit_amount': 2000, # $20.00
'recurring': {
'interval': 'month',
},
},
'quantity': 1,
}],
mode='subscription',
success_url='https://yourdomain.com/success?session_id={CHECKOUT_SESSION_ID}',
cancel_url='https://yourdomain.com/cancel',
)
# Redirect user to session.url
print(session.url)
```
## Payment Implementation Patterns
### Pattern 1: One-Time Payment (Hosted Checkout)
```python
def create_checkout_session(amount, currency='usd'):
"""Create a one-time payment checkout session."""
try:
session = stripe.checkout.Session.create(
payment_method_types=['card'],
line_items=[{
'price_data': {
'currency': currency,
'product_data': {
'name': 'Purchase',
'images': ['https://example.com/product.jpg'],
},
'unit_amount': amount, # Amount in cents
},
'quantity': 1,
}],
mode='payment',
success_url='https://yourdomain.com/success?session_id={CHECKOUT_SESSION_ID}',
cancel_url='https://yourdomain.com/cancel',
metadata={
'order_id': 'order_123',
'user_id': 'user_456'
}
)
return session
except stripe.error.StripeError as e:
# Handle error
print(f"Stripe error: {e.user_message}")
raise
```
### Pattern 2: Custom Payment Intent Flow
```python
def create_payment_intent(amount, currency='usd', customer_id=None):
"""Create a payment intent for custom checkout UI."""
intent = stripe.PaymentIntent.create(
amount=amount,
currency=currency,
customer=customer_id,
automatic_payment_methods={
'enabled': True,
},
metadata={
'integration_check': 'accept_a_payment'
}
)
return intent.client_secret # Send to frontend
# Frontend (JavaScript)
"""
const stripe = Stripe('pk_test_...');
const elements = stripe.elements();
const cardElement = elements.create('card');
cardElement.mount('#card-element');
const {error, paymentIntent} = await stripe.confirmCardPayment(
clientSecret,
{
payment_method: {
card: cardElement,
billing_details: {
name: 'Customer Name'
}
}
}
);
if (error) {
// Handle error
} else if (paymentIntent.status === 'succeeded') {
// Payment successful
}
"""
```
### Pattern 3: Subscription Creation
```python
def create_subscription(customer_id, price_id):
"""Create a subscription for a customer."""
try:
subscription = stripe.Subscription.create(
customer=customer_id,
items=[{'price': price_id}],
payment_behavior='default_incomplete',
payment_settings={'save_default_payment_method': 'on_subscription'},
expand=['latest_invoice.payment_intent'],
)
return {
'subscription_id': subscription.id,
'client_secret': subscription.latest_invoice.payment_intent.client_secret
}
except stripe.error.StripeError as e:
print(f"Subscription creation failed: {e}")
raise
```
### Pattern 4: Customer Portal
```python
def create_customer_portal_session(customer_id):
"""Create a portal session for customers to manage subscriptions."""
session = stripe.billing_portal.Session.create(
customer=customer_id,
return_url='https://yourdomain.com/account',
)
return session.url # Redirect customer here
```
## Webhook Handling
### Secure Webhook Endpoint
```python
from flask import Flask, request
import stripe
app = Flask(__name__)
endpoint_secret = 'whsec_...'
@app.route('/webhook', methods=['POST'])
def webhook():
payload = request.data
sig_header = request.headers.get('Stripe-Signature')
try:
event = stripe.Webhook.construct_event(
payload, sig_header, endpoint_secret
)
except ValueError:
# Invalid payload
return 'Invalid payload', 400
except stripe.error.SignatureVerificationError:
# Invalid signature
return 'Invalid signature', 400
# Handle the event
if event['type'] == 'payment_intent.succeeded':
payment_intent = event['data']['object']
handle_successful_payment(payment_intent)
elif event['type'] == 'payment_intent.payment_failed':
payment_intent = event['data']['object']
handle_failed_payment(payment_intent)
elif event['type'] == 'customer.subscription.deleted':
subscription = event['data']['object']
handle_subscription_canceled(subscription)
return 'Success', 200
def handle_successful_payment(payment_intent):
"""Process successful payment."""
customer_id = payment_intent.get('customer')
amount = payment_intent['amount']
metadata = payment_intent.get('metadata', {})
# Update your database
# Send confirmation email
# Fulfill order
print(f"Payment succeeded: {payment_intent['id']}")
def handle_failed_payment(payment_intent):
"""Handle failed payment."""
error = payment_intent.get('last_payment_error', {})
print(f"Payment failed: {error.get('message')}")
# Notify customer
# Update order status
def handle_subscription_canceled(subscription):
"""Handle subscription cancellation."""
customer_id = subscription['customer']
# Update user access
# Send cancellation email
print(f"Subscription canceled: {subscription['id']}")
```
### Webhook Best Practices
```python
import hashlib
import hmac
def verify_webhook_signature(payload, signature, secret):
"""Manually verify webhook signature."""
expected_sig = hmac.new(
secret.encode('utf-8'),
payload,
hashlib.sha256
).hexdigest()
return hmac.compare_digest(signature, expected_sig)
def handle_webhook_idempotently(event_id, handler):
"""Ensure webhook is processed exactly once."""
# Check if event already processed
if is_event_processed(event_id):
return
# Process event
try:
handler()
mark_event_processed(event_id)
except Exception as e:
log_error(e)
# Stripe will retry failed webhooks
raise
```
## Customer Management
```python
def create_customer(email, name, payment_method_id=None):
"""Create a Stripe customer."""
customer = stripe.Customer.create(
email=email,
name=name,
payment_method=payment_method_id,
invoice_settings={
'default_payment_method': payment_method_id
} if payment_method_id else None,
metadata={
'user_id': '12345'
}
)
return customer
def attach_payment_method(customer_id, payment_method_id):
"""Attach a payment method to a customer."""
stripe.PaymentMethod.attach(
payment_method_id,
customer=customer_id
)
# Set as default
stripe.Customer.modify(
customer_id,
invoice_settings={
'default_payment_method': payment_method_id
}
)
def list_customer_payment_methods(customer_id):
"""List all payment methods for a customer."""
payment_methods = stripe.PaymentMethod.list(
customer=customer_id,
type='card'
)
return payment_methods.data
```
## Refund Handling
```python
def create_refund(payment_intent_id, amount=None, reason=None):
"""Create a refund."""
refund_params = {
'payment_intent': payment_intent_id
}
if amount:
refund_params['amount'] = amount # Partial refund
if reason:
refund_params['reason'] = reason # 'duplicate', 'fraudulent', 'requested_by_customer'
refund = stripe.Refund.create(**refund_params)
return refund
def handle_dispute(charge_id, evidence):
"""Update dispute with evidence."""
stripe.Dispute.modify(
charge_id,
evidence={
'customer_name': evidence.get('customer_name'),
'customer_email_address': evidence.get('customer_email'),
'shipping_documentation': evidence.get('shipping_proof'),
'customer_communication': evidence.get('communication'),
}
)
```
## Testing
```python
# Use test mode keys
stripe.api_key = "sk_test_..."
# Test card numbers
TEST_CARDS = {
'success': '4242424242424242',
'declined': '4000000000000002',
'3d_secure': '4000002500003155',
'insufficient_funds': '4000000000009995'
}
def test_payment_flow():
"""Test complete payment flow."""
# Create test customer
customer = stripe.Customer.create(
email="test@example.com"
)
# Create payment intent
intent = stripe.PaymentIntent.create(
amount=1000,
currency='usd',
customer=customer.id,
payment_method_types=['card']
)
# Confirm with test card
confirmed = stripe.PaymentIntent.confirm(
intent.id,
payment_method='pm_card_visa' # Test payment method
)
assert confirmed.status == 'succeeded'
```
## Resources
- **references/checkout-flows.md**: Detailed checkout implementation
- **references/webhook-handling.md**: Webhook security and processing
- **references/subscription-management.md**: Subscription lifecycle
- **references/customer-management.md**: Customer and payment method handling
- **references/invoice-generation.md**: Invoicing and billing
- **assets/stripe-client.py**: Production-ready Stripe client wrapper
- **assets/webhook-handler.py**: Complete webhook processor
- **assets/checkout-config.json**: Checkout configuration templates
## Best Practices
1. **Always Use Webhooks**: Don't rely solely on client-side confirmation
2. **Idempotency**: Handle webhook events idempotently
3. **Error Handling**: Gracefully handle all Stripe errors
4. **Test Mode**: Thoroughly test with test keys before production
5. **Metadata**: Use metadata to link Stripe objects to your database
6. **Monitoring**: Track payment success rates and errors
7. **PCI Compliance**: Never handle raw card data on your server
8. **SCA Ready**: Implement 3D Secure for European payments
## Common Pitfalls
- **Not Verifying Webhooks**: Always verify webhook signatures
- **Missing Webhook Events**: Handle all relevant webhook events
- **Hardcoded Amounts**: Use cents/smallest currency unit
- **No Retry Logic**: Implement retries for API calls
- **Ignoring Test Mode**: Test all edge cases with test cards

694
plugins/python-development/skills/async-python-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,694 @@
---
name: async-python-patterns
description: Master Python asyncio, concurrent programming, and async/await patterns for high-performance applications. Use when building async APIs, concurrent systems, or I/O-bound applications requiring non-blocking operations.
---
# Async Python Patterns
Comprehensive guidance for implementing asynchronous Python applications using asyncio, concurrent programming patterns, and async/await for building high-performance, non-blocking systems.
## When to Use This Skill
- Building async web APIs (FastAPI, aiohttp, Sanic)
- Implementing concurrent I/O operations (database, file, network)
- Creating web scrapers with concurrent requests
- Developing real-time applications (WebSocket servers, chat systems)
- Processing multiple independent tasks simultaneously
- Building microservices with async communication
- Optimizing I/O-bound workloads
- Implementing async background tasks and queues
## Core Concepts
### 1. Event Loop
The event loop is the heart of asyncio, managing and scheduling asynchronous tasks.
**Key characteristics:**
- Single-threaded cooperative multitasking
- Schedules coroutines for execution
- Handles I/O operations without blocking
- Manages callbacks and futures
### 2. Coroutines
Functions defined with `async def` that can be paused and resumed.
**Syntax:**
```python
async def my_coroutine():
result = await some_async_operation()
return result
```
### 3. Tasks
Scheduled coroutines that run concurrently on the event loop.
### 4. Futures
Low-level objects representing eventual results of async operations.
### 5. Async Context Managers
Resources that support `async with` for proper cleanup.
### 6. Async Iterators
Objects that support `async for` for iterating over async data sources.
## Quick Start
```python
import asyncio
async def main():
print("Hello")
await asyncio.sleep(1)
print("World")
# Python 3.7+
asyncio.run(main())
```
## Fundamental Patterns
### Pattern 1: Basic Async/Await
```python
import asyncio
async def fetch_data(url: str) -> dict:
"""Fetch data from URL asynchronously."""
await asyncio.sleep(1) # Simulate I/O
return {"url": url, "data": "result"}
async def main():
result = await fetch_data("https://api.example.com")
print(result)
asyncio.run(main())
```
### Pattern 2: Concurrent Execution with gather()
```python
import asyncio
from typing import List
async def fetch_user(user_id: int) -> dict:
"""Fetch user data."""
await asyncio.sleep(0.5)
return {"id": user_id, "name": f"User {user_id}"}
async def fetch_all_users(user_ids: List[int]) -> List[dict]:
"""Fetch multiple users concurrently."""
tasks = [fetch_user(uid) for uid in user_ids]
results = await asyncio.gather(*tasks)
return results
async def main():
user_ids = [1, 2, 3, 4, 5]
users = await fetch_all_users(user_ids)
print(f"Fetched {len(users)} users")
asyncio.run(main())
```
### Pattern 3: Task Creation and Management
```python
import asyncio
async def background_task(name: str, delay: int):
"""Long-running background task."""
print(f"{name} started")
await asyncio.sleep(delay)
print(f"{name} completed")
return f"Result from {name}"
async def main():
# Create tasks
task1 = asyncio.create_task(background_task("Task 1", 2))
task2 = asyncio.create_task(background_task("Task 2", 1))
# Do other work
print("Main: doing other work")
await asyncio.sleep(0.5)
# Wait for tasks
result1 = await task1
result2 = await task2
print(f"Results: {result1}, {result2}")
asyncio.run(main())
```
### Pattern 4: Error Handling in Async Code
```python
import asyncio
from typing import List, Optional
async def risky_operation(item_id: int) -> dict:
"""Operation that might fail."""
await asyncio.sleep(0.1)
if item_id % 3 == 0:
raise ValueError(f"Item {item_id} failed")
return {"id": item_id, "status": "success"}
async def safe_operation(item_id: int) -> Optional[dict]:
"""Wrapper with error handling."""
try:
return await risky_operation(item_id)
except ValueError as e:
print(f"Error: {e}")
return None
async def process_items(item_ids: List[int]):
"""Process multiple items with error handling."""
tasks = [safe_operation(iid) for iid in item_ids]
results = await asyncio.gather(*tasks, return_exceptions=True)
# Filter out failures
successful = [r for r in results if r is not None and not isinstance(r, Exception)]
failed = [r for r in results if isinstance(r, Exception)]
print(f"Success: {len(successful)}, Failed: {len(failed)}")
return successful
asyncio.run(process_items([1, 2, 3, 4, 5, 6]))
```
### Pattern 5: Timeout Handling
```python
import asyncio
async def slow_operation(delay: int) -> str:
"""Operation that takes time."""
await asyncio.sleep(delay)
return f"Completed after {delay}s"
async def with_timeout():
"""Execute operation with timeout."""
try:
result = await asyncio.wait_for(slow_operation(5), timeout=2.0)
print(result)
except asyncio.TimeoutError:
print("Operation timed out")
asyncio.run(with_timeout())
```
## Advanced Patterns
### Pattern 6: Async Context Managers
```python
import asyncio
from typing import Optional
class AsyncDatabaseConnection:
"""Async database connection context manager."""
def __init__(self, dsn: str):
self.dsn = dsn
self.connection: Optional[object] = None
async def __aenter__(self):
print("Opening connection")
await asyncio.sleep(0.1) # Simulate connection
self.connection = {"dsn": self.dsn, "connected": True}
return self.connection
async def __aexit__(self, exc_type, exc_val, exc_tb):
print("Closing connection")
await asyncio.sleep(0.1) # Simulate cleanup
self.connection = None
async def query_database():
"""Use async context manager."""
async with AsyncDatabaseConnection("postgresql://localhost") as conn:
print(f"Using connection: {conn}")
await asyncio.sleep(0.2) # Simulate query
return {"rows": 10}
asyncio.run(query_database())
```
### Pattern 7: Async Iterators and Generators
```python
import asyncio
from typing import AsyncIterator
async def async_range(start: int, end: int, delay: float = 0.1) -> AsyncIterator[int]:
"""Async generator that yields numbers with delay."""
for i in range(start, end):
await asyncio.sleep(delay)
yield i
async def fetch_pages(url: str, max_pages: int) -> AsyncIterator[dict]:
"""Fetch paginated data asynchronously."""
for page in range(1, max_pages + 1):
await asyncio.sleep(0.2) # Simulate API call
yield {
"page": page,
"url": f"{url}?page={page}",
"data": [f"item_{page}_{i}" for i in range(5)]
}
async def consume_async_iterator():
"""Consume async iterator."""
async for number in async_range(1, 5):
print(f"Number: {number}")
print("\nFetching pages:")
async for page_data in fetch_pages("https://api.example.com/items", 3):
print(f"Page {page_data['page']}: {len(page_data['data'])} items")
asyncio.run(consume_async_iterator())
```
### Pattern 8: Producer-Consumer Pattern
```python
import asyncio
from asyncio import Queue
from typing import Optional
async def producer(queue: Queue, producer_id: int, num_items: int):
"""Produce items and put them in queue."""
for i in range(num_items):
item = f"Item-{producer_id}-{i}"
await queue.put(item)
print(f"Producer {producer_id} produced: {item}")
await asyncio.sleep(0.1)
await queue.put(None) # Signal completion
async def consumer(queue: Queue, consumer_id: int):
"""Consume items from queue."""
while True:
item = await queue.get()
if item is None:
queue.task_done()
break
print(f"Consumer {consumer_id} processing: {item}")
await asyncio.sleep(0.2) # Simulate work
queue.task_done()
async def producer_consumer_example():
"""Run producer-consumer pattern."""
queue = Queue(maxsize=10)
# Create tasks
producers = [
asyncio.create_task(producer(queue, i, 5))
for i in range(2)
]
consumers = [
asyncio.create_task(consumer(queue, i))
for i in range(3)
]
# Wait for producers
await asyncio.gather(*producers)
# Wait for queue to be empty
await queue.join()
# Cancel consumers
for c in consumers:
c.cancel()
asyncio.run(producer_consumer_example())
```
### Pattern 9: Semaphore for Rate Limiting
```python
import asyncio
from typing import List
async def api_call(url: str, semaphore: asyncio.Semaphore) -> dict:
"""Make API call with rate limiting."""
async with semaphore:
print(f"Calling {url}")
await asyncio.sleep(0.5) # Simulate API call
return {"url": url, "status": 200}
async def rate_limited_requests(urls: List[str], max_concurrent: int = 5):
"""Make multiple requests with rate limiting."""
semaphore = asyncio.Semaphore(max_concurrent)
tasks = [api_call(url, semaphore) for url in urls]
results = await asyncio.gather(*tasks)
return results
async def main():
urls = [f"https://api.example.com/item/{i}" for i in range(20)]
results = await rate_limited_requests(urls, max_concurrent=3)
print(f"Completed {len(results)} requests")
asyncio.run(main())
```
### Pattern 10: Async Locks and Synchronization
```python
import asyncio
class AsyncCounter:
"""Thread-safe async counter."""
def __init__(self):
self.value = 0
self.lock = asyncio.Lock()
async def increment(self):
"""Safely increment counter."""
async with self.lock:
current = self.value
await asyncio.sleep(0.01) # Simulate work
self.value = current + 1
async def get_value(self) -> int:
"""Get current value."""
async with self.lock:
return self.value
async def worker(counter: AsyncCounter, worker_id: int):
"""Worker that increments counter."""
for _ in range(10):
await counter.increment()
print(f"Worker {worker_id} incremented")
async def test_counter():
"""Test concurrent counter."""
counter = AsyncCounter()
workers = [asyncio.create_task(worker(counter, i)) for i in range(5)]
await asyncio.gather(*workers)
final_value = await counter.get_value()
print(f"Final counter value: {final_value}")
asyncio.run(test_counter())
```
## Real-World Applications
### Web Scraping with aiohttp
```python
import asyncio
import aiohttp
from typing import List, Dict
async def fetch_url(session: aiohttp.ClientSession, url: str) -> Dict:
"""Fetch single URL."""
try:
async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as response:
text = await response.text()
return {
"url": url,
"status": response.status,
"length": len(text)
}
except Exception as e:
return {"url": url, "error": str(e)}
async def scrape_urls(urls: List[str]) -> List[Dict]:
"""Scrape multiple URLs concurrently."""
async with aiohttp.ClientSession() as session:
tasks = [fetch_url(session, url) for url in urls]
results = await asyncio.gather(*tasks)
return results
async def main():
urls = [
"https://httpbin.org/delay/1",
"https://httpbin.org/delay/2",
"https://httpbin.org/status/404",
]
results = await scrape_urls(urls)
for result in results:
print(result)
asyncio.run(main())
```
### Async Database Operations
```python
import asyncio
from typing import List, Optional
# Simulated async database client
class AsyncDB:
"""Simulated async database."""
async def execute(self, query: str) -> List[dict]:
"""Execute query."""
await asyncio.sleep(0.1)
return [{"id": 1, "name": "Example"}]
async def fetch_one(self, query: str) -> Optional[dict]:
"""Fetch single row."""
await asyncio.sleep(0.1)
return {"id": 1, "name": "Example"}
async def get_user_data(db: AsyncDB, user_id: int) -> dict:
"""Fetch user and related data concurrently."""
user_task = db.fetch_one(f"SELECT * FROM users WHERE id = {user_id}")
orders_task = db.execute(f"SELECT * FROM orders WHERE user_id = {user_id}")
profile_task = db.fetch_one(f"SELECT * FROM profiles WHERE user_id = {user_id}")
user, orders, profile = await asyncio.gather(user_task, orders_task, profile_task)
return {
"user": user,
"orders": orders,
"profile": profile
}
async def main():
db = AsyncDB()
user_data = await get_user_data(db, 1)
print(user_data)
asyncio.run(main())
```
### WebSocket Server
```python
import asyncio
from typing import Set
# Simulated WebSocket connection
class WebSocket:
"""Simulated WebSocket."""
def __init__(self, client_id: str):
self.client_id = client_id
async def send(self, message: str):
"""Send message."""
print(f"Sending to {self.client_id}: {message}")
await asyncio.sleep(0.01)
async def recv(self) -> str:
"""Receive message."""
await asyncio.sleep(1)
return f"Message from {self.client_id}"
class WebSocketServer:
"""Simple WebSocket server."""
def __init__(self):
self.clients: Set[WebSocket] = set()
async def register(self, websocket: WebSocket):
"""Register new client."""
self.clients.add(websocket)
print(f"Client {websocket.client_id} connected")
async def unregister(self, websocket: WebSocket):
"""Unregister client."""
self.clients.remove(websocket)
print(f"Client {websocket.client_id} disconnected")
async def broadcast(self, message: str):
"""Broadcast message to all clients."""
if self.clients:
tasks = [client.send(message) for client in self.clients]
await asyncio.gather(*tasks)
async def handle_client(self, websocket: WebSocket):
"""Handle individual client connection."""
await self.register(websocket)
try:
async for message in self.message_iterator(websocket):
await self.broadcast(f"{websocket.client_id}: {message}")
finally:
await self.unregister(websocket)
async def message_iterator(self, websocket: WebSocket):
"""Iterate over messages from client."""
for _ in range(3): # Simulate 3 messages
yield await websocket.recv()
```
## Performance Best Practices
### 1. Use Connection Pools
```python
import asyncio
import aiohttp
async def with_connection_pool():
"""Use connection pool for efficiency."""
connector = aiohttp.TCPConnector(limit=100, limit_per_host=10)
async with aiohttp.ClientSession(connector=connector) as session:
tasks = [session.get(f"https://api.example.com/item/{i}") for i in range(50)]
responses = await asyncio.gather(*tasks)
return responses
```
### 2. Batch Operations
```python
async def batch_process(items: List[str], batch_size: int = 10):
"""Process items in batches."""
for i in range(0, len(items), batch_size):
batch = items[i:i + batch_size]
tasks = [process_item(item) for item in batch]
await asyncio.gather(*tasks)
print(f"Processed batch {i // batch_size + 1}")
async def process_item(item: str):
"""Process single item."""
await asyncio.sleep(0.1)
return f"Processed: {item}"
```
### 3. Avoid Blocking Operations
```python
import asyncio
import concurrent.futures
from typing import Any
def blocking_operation(data: Any) -> Any:
"""CPU-intensive blocking operation."""
import time
time.sleep(1)
return data * 2
async def run_in_executor(data: Any) -> Any:
"""Run blocking operation in thread pool."""
loop = asyncio.get_event_loop()
with concurrent.futures.ThreadPoolExecutor() as pool:
result = await loop.run_in_executor(pool, blocking_operation, data)
return result
async def main():
results = await asyncio.gather(*[run_in_executor(i) for i in range(5)])
print(results)
asyncio.run(main())
```
## Common Pitfalls
### 1. Forgetting await
```python
# Wrong - returns coroutine object, doesn't execute
result = async_function()
# Correct
result = await async_function()
```
### 2. Blocking the Event Loop
```python
# Wrong - blocks event loop
import time
async def bad():
time.sleep(1) # Blocks!
# Correct
async def good():
await asyncio.sleep(1) # Non-blocking
```
### 3. Not Handling Cancellation
```python
async def cancelable_task():
"""Task that handles cancellation."""
try:
while True:
await asyncio.sleep(1)
print("Working...")
except asyncio.CancelledError:
print("Task cancelled, cleaning up...")
# Perform cleanup
raise # Re-raise to propagate cancellation
```
### 4. Mixing Sync and Async Code
```python
# Wrong - can't call async from sync directly
def sync_function():
result = await async_function() # SyntaxError!
# Correct
def sync_function():
result = asyncio.run(async_function())
```
## Testing Async Code
```python
import asyncio
import pytest
# Using pytest-asyncio
@pytest.mark.asyncio
async def test_async_function():
"""Test async function."""
result = await fetch_data("https://api.example.com")
assert result is not None
@pytest.mark.asyncio
async def test_with_timeout():
"""Test with timeout."""
with pytest.raises(asyncio.TimeoutError):
await asyncio.wait_for(slow_operation(5), timeout=1.0)
```
## Resources
- **Python asyncio documentation**: https://docs.python.org/3/library/asyncio.html
- **aiohttp**: Async HTTP client/server
- **FastAPI**: Modern async web framework
- **asyncpg**: Async PostgreSQL driver
- **motor**: Async MongoDB driver
## Best Practices Summary
1. **Use asyncio.run()** for entry point (Python 3.7+)
2. **Always await coroutines** to execute them
3. **Use gather() for concurrent execution** of multiple tasks
4. **Implement proper error handling** with try/except
5. **Use timeouts** to prevent hanging operations
6. **Pool connections** for better performance
7. **Avoid blocking operations** in async code
8. **Use semaphores** for rate limiting
9. **Handle task cancellation** properly
10. **Test async code** with pytest-asyncio

870
plugins/python-development/skills/python-packaging/SKILL.md Normal file
View File

@@ -0,0 +1,870 @@
---
name: python-packaging
description: Create distributable Python packages with proper project structure, setup.py/pyproject.toml, and publishing to PyPI. Use when packaging Python libraries, creating CLI tools, or distributing Python code.
---
# Python Packaging
Comprehensive guide to creating, structuring, and distributing Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI.
## When to Use This Skill
- Creating Python libraries for distribution
- Building command-line tools with entry points
- Publishing packages to PyPI or private repositories
- Setting up Python project structure
- Creating installable packages with dependencies
- Building wheels and source distributions
- Versioning and releasing Python packages
- Creating namespace packages
- Implementing package metadata and classifiers
## Core Concepts
### 1. Package Structure
- **Source layout**: `src/package_name/` (recommended)
- **Flat layout**: `package_name/` (simpler but less flexible)
- **Package metadata**: pyproject.toml, setup.py, or setup.cfg
- **Distribution formats**: wheel (.whl) and source distribution (.tar.gz)
### 2. Modern Packaging Standards
- **PEP 517/518**: Build system requirements
- **PEP 621**: Metadata in pyproject.toml
- **PEP 660**: Editable installs
- **pyproject.toml**: Single source of configuration
### 3. Build Backends
- **setuptools**: Traditional, widely used
- **hatchling**: Modern, opinionated
- **flit**: Lightweight, for pure Python
- **poetry**: Dependency management + packaging
### 4. Distribution
- **PyPI**: Python Package Index (public)
- **TestPyPI**: Testing before production
- **Private repositories**: JFrog, AWS CodeArtifact, etc.
## Quick Start
### Minimal Package Structure
```
my-package/
├── pyproject.toml
├── README.md
├── LICENSE
├── src/
│ └── my_package/
│ ├── __init__.py
│ └── module.py
└── tests/
└── test_module.py
```
### Minimal pyproject.toml
```toml
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "my-package"
version = "0.1.0"
description = "A short description"
authors = [{name = "Your Name", email = "you@example.com"}]
readme = "README.md"
requires-python = ">=3.8"
dependencies = [
"requests>=2.28.0",
]
[project.optional-dependencies]
dev = [
"pytest>=7.0",
"black>=22.0",
]
```
## Package Structure Patterns
### Pattern 1: Source Layout (Recommended)
```
my-package/
├── pyproject.toml
├── README.md
├── LICENSE
├── .gitignore
├── src/
│ └── my_package/
│ ├── __init__.py
│ ├── core.py
│ ├── utils.py
│ └── py.typed # For type hints
├── tests/
│ ├── __init__.py
│ ├── test_core.py
│ └── test_utils.py
└── docs/
└── index.md
```
**Advantages:**
- Prevents accidentally importing from source
- Cleaner test imports
- Better isolation
**pyproject.toml for source layout:**
```toml
[tool.setuptools.packages.find]
where = ["src"]
```
### Pattern 2: Flat Layout
```
my-package/
├── pyproject.toml
├── README.md
├── my_package/
│ ├── __init__.py
│ └── module.py
└── tests/
└── test_module.py
```
**Simpler but:**
- Can import package without installing
- Less professional for libraries
### Pattern 3: Multi-Package Project
```
project/
├── pyproject.toml
├── packages/
│ ├── package-a/
│ │ └── src/
│ │ └── package_a/
│ └── package-b/
│ └── src/
│ └── package_b/
└── tests/
```
## Complete pyproject.toml Examples
### Pattern 4: Full-Featured pyproject.toml
```toml
[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "my-awesome-package"
version = "1.0.0"
description = "An awesome Python package"
readme = "README.md"
requires-python = ">=3.8"
license = {text = "MIT"}
authors = [
{name = "Your Name", email = "you@example.com"},
]
maintainers = [
{name = "Maintainer Name", email = "maintainer@example.com"},
]
keywords = ["example", "package", "awesome"]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
]
dependencies = [
"requests>=2.28.0,<3.0.0",
"click>=8.0.0",
"pydantic>=2.0.0",
]
[project.optional-dependencies]
dev = [
"pytest>=7.0.0",
"pytest-cov>=4.0.0",
"black>=23.0.0",
"ruff>=0.1.0",
"mypy>=1.0.0",
]
docs = [
"sphinx>=5.0.0",
"sphinx-rtd-theme>=1.0.0",
]
all = [
"my-awesome-package[dev,docs]",
]
[project.urls]
Homepage = "https://github.com/username/my-awesome-package"
Documentation = "https://my-awesome-package.readthedocs.io"
Repository = "https://github.com/username/my-awesome-package"
"Bug Tracker" = "https://github.com/username/my-awesome-package/issues"
Changelog = "https://github.com/username/my-awesome-package/blob/main/CHANGELOG.md"
[project.scripts]
my-cli = "my_package.cli:main"
awesome-tool = "my_package.tools:run"
[project.entry-points."my_package.plugins"]
plugin1 = "my_package.plugins:plugin1"
[tool.setuptools]
package-dir = {"" = "src"}
zip-safe = false
[tool.setuptools.packages.find]
where = ["src"]
include = ["my_package*"]
exclude = ["tests*"]
[tool.setuptools.package-data]
my_package = ["py.typed", "*.pyi", "data/*.json"]
# Black configuration
[tool.black]
line-length = 100
target-version = ["py38", "py39", "py310", "py311"]
include = '\.pyi?$'
# Ruff configuration
[tool.ruff]
line-length = 100
target-version = "py38"
[tool.ruff.lint]
select = ["E", "F", "I", "N", "W", "UP"]
# MyPy configuration
[tool.mypy]
python_version = "3.8"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
# Pytest configuration
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
addopts = "-v --cov=my_package --cov-report=term-missing"
# Coverage configuration
[tool.coverage.run]
source = ["src"]
omit = ["*/tests/*"]
[tool.coverage.report]
exclude_lines = [
"pragma: no cover",
"def __repr__",
"raise AssertionError",
"raise NotImplementedError",
]
```
### Pattern 5: Dynamic Versioning
```toml
[build-system]
requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]
build-backend = "setuptools.build_meta"
[project]
name = "my-package"
dynamic = ["version"]
description = "Package with dynamic version"
[tool.setuptools.dynamic]
version = {attr = "my_package.__version__"}
# Or use setuptools-scm for git-based versioning
[tool.setuptools_scm]
write_to = "src/my_package/_version.py"
```
**In __init__.py:**
```python
# src/my_package/__init__.py
__version__ = "1.0.0"
# Or with setuptools-scm
from importlib.metadata import version
__version__ = version("my-package")
```
## Command-Line Interface (CLI) Patterns
### Pattern 6: CLI with Click
```python
# src/my_package/cli.py
import click
@click.group()
@click.version_option()
def cli():
"""My awesome CLI tool."""
pass
@cli.command()
@click.argument("name")
@click.option("--greeting", default="Hello", help="Greeting to use")
def greet(name: str, greeting: str):
"""Greet someone."""
click.echo(f"{greeting}, {name}!")
@cli.command()
@click.option("--count", default=1, help="Number of times to repeat")
def repeat(count: int):
"""Repeat a message."""
for i in range(count):
click.echo(f"Message {i + 1}")
def main():
"""Entry point for CLI."""
cli()
if __name__ == "__main__":
main()
```
**Register in pyproject.toml:**
```toml
[project.scripts]
my-tool = "my_package.cli:main"
```
**Usage:**
```bash
pip install -e .
my-tool greet World
my-tool greet Alice --greeting="Hi"
my-tool repeat --count=3
```
### Pattern 7: CLI with argparse
```python
# src/my_package/cli.py
import argparse
import sys
def main():
"""Main CLI entry point."""
parser = argparse.ArgumentParser(
description="My awesome tool",
prog="my-tool"
)
parser.add_argument(
"--version",
action="version",
version="%(prog)s 1.0.0"
)
subparsers = parser.add_subparsers(dest="command", help="Commands")
# Add subcommand
process_parser = subparsers.add_parser("process", help="Process data")
process_parser.add_argument("input_file", help="Input file path")
process_parser.add_argument(
"--output", "-o",
default="output.txt",
help="Output file path"
)
args = parser.parse_args()
if args.command == "process":
process_data(args.input_file, args.output)
else:
parser.print_help()
sys.exit(1)
def process_data(input_file: str, output_file: str):
"""Process data from input to output."""
print(f"Processing {input_file} -> {output_file}")
if __name__ == "__main__":
main()
```
## Building and Publishing
### Pattern 8: Build Package Locally
```bash
# Install build tools
pip install build twine
# Build distribution
python -m build
# This creates:
# dist/
# my-package-1.0.0.tar.gz (source distribution)
# my_package-1.0.0-py3-none-any.whl (wheel)
# Check the distribution
twine check dist/*
```
### Pattern 9: Publishing to PyPI
```bash
# Install publishing tools
pip install twine
# Test on TestPyPI first
twine upload --repository testpypi dist/*
# Install from TestPyPI to test
pip install --index-url https://test.pypi.org/simple/ my-package
# If all good, publish to PyPI
twine upload dist/*
```
**Using API tokens (recommended):**
```bash
# Create ~/.pypirc
[distutils]
index-servers =
pypi
testpypi
[pypi]
username = __token__
password = pypi-...your-token...
[testpypi]
username = __token__
password = pypi-...your-test-token...
```
### Pattern 10: Automated Publishing with GitHub Actions
```yaml
# .github/workflows/publish.yml
name: Publish to PyPI
on:
release:
types: [created]
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install dependencies
run: |
pip install build twine
- name: Build package
run: python -m build
- name: Check package
run: twine check dist/*
- name: Publish to PyPI
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: twine upload dist/*
```
## Advanced Patterns
### Pattern 11: Including Data Files
```toml
[tool.setuptools.package-data]
my_package = [
"data/*.json",
"templates/*.html",
"static/css/*.css",
"py.typed",
]
```
**Accessing data files:**
```python
# src/my_package/loader.py
from importlib.resources import files
import json
def load_config():
"""Load configuration from package data."""
config_file = files("my_package").joinpath("data/config.json")
with config_file.open() as f:
return json.load(f)
# Python 3.9+
from importlib.resources import files
data = files("my_package").joinpath("data/file.txt").read_text()
```
### Pattern 12: Namespace Packages
**For large projects split across multiple repositories:**
```
# Package 1: company-core
company/
└── core/
├── __init__.py
└── models.py
# Package 2: company-api
company/
└── api/
├── __init__.py
└── routes.py
```
**Do NOT include __init__.py in the namespace directory (company/):**
```toml
# company-core/pyproject.toml
[project]
name = "company-core"
[tool.setuptools.packages.find]
where = ["."]
include = ["company.core*"]
# company-api/pyproject.toml
[project]
name = "company-api"
[tool.setuptools.packages.find]
where = ["."]
include = ["company.api*"]
```
**Usage:**
```python
# Both packages can be imported under same namespace
from company.core import models
from company.api import routes
```
### Pattern 13: C Extensions
```toml
[build-system]
requires = ["setuptools>=61.0", "wheel", "Cython>=0.29"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
ext-modules = [
{name = "my_package.fast_module", sources = ["src/fast_module.c"]},
]
```
**Or with setup.py:**
```python
# setup.py
from setuptools import setup, Extension
setup(
ext_modules=[
Extension(
"my_package.fast_module",
sources=["src/fast_module.c"],
include_dirs=["src/include"],
)
]
)
```
## Version Management
### Pattern 14: Semantic Versioning
```python
# src/my_package/__init__.py
__version__ = "1.2.3"
# Semantic versioning: MAJOR.MINOR.PATCH
# MAJOR: Breaking changes
# MINOR: New features (backward compatible)
# PATCH: Bug fixes
```
**Version constraints in dependencies:**
```toml
dependencies = [
"requests>=2.28.0,<3.0.0", # Compatible range
"click~=8.1.0", # Compatible release (~= 8.1.0 means >=8.1.0,<8.2.0)
"pydantic>=2.0", # Minimum version
"numpy==1.24.3", # Exact version (avoid if possible)
]
```
### Pattern 15: Git-Based Versioning
```toml
[build-system]
requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]
build-backend = "setuptools.build_meta"
[project]
name = "my-package"
dynamic = ["version"]
[tool.setuptools_scm]
write_to = "src/my_package/_version.py"
version_scheme = "post-release"
local_scheme = "dirty-tag"
```
**Creates versions like:**
- `1.0.0` (from git tag)
- `1.0.1.dev3+g1234567` (3 commits after tag)
## Testing Installation
### Pattern 16: Editable Install
```bash
# Install in development mode
pip install -e .
# With optional dependencies
pip install -e ".[dev]"
pip install -e ".[dev,docs]"
# Now changes to source code are immediately reflected
```
### Pattern 17: Testing in Isolated Environment
```bash
# Create virtual environment
python -m venv test-env
source test-env/bin/activate # Linux/Mac
# test-env\Scripts\activate # Windows
# Install package
pip install dist/my_package-1.0.0-py3-none-any.whl
# Test it works
python -c "import my_package; print(my_package.__version__)"
# Test CLI
my-tool --help
# Cleanup
deactivate
rm -rf test-env
```
## Documentation
### Pattern 18: README.md Template
```markdown
# My Package
[![PyPI version](https://badge.fury.io/py/my-package.svg)](https://pypi.org/project/my-package/)
[![Python versions](https://img.shields.io/pypi/pyversions/my-package.svg)](https://pypi.org/project/my-package/)
[![Tests](https://github.com/username/my-package/workflows/Tests/badge.svg)](https://github.com/username/my-package/actions)
Brief description of your package.
## Installation
```bash
pip install my-package
```
## Quick Start
```python
from my_package import something
result = something.do_stuff()
```
## Features
- Feature 1
- Feature 2
- Feature 3
## Documentation
Full documentation: https://my-package.readthedocs.io
## Development
```bash
git clone https://github.com/username/my-package.git
cd my-package
pip install -e ".[dev]"
pytest
```
## License
MIT
```
## Common Patterns
### Pattern 19: Multi-Architecture Wheels
```yaml
# .github/workflows/wheels.yml
name: Build wheels
on: [push, pull_request]
jobs:
build_wheels:
name: Build wheels on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
steps:
- uses: actions/checkout@v3
- name: Build wheels
uses: pypa/cibuildwheel@v2.16.2
- uses: actions/upload-artifact@v3
with:
path: ./wheelhouse/*.whl
```
### Pattern 20: Private Package Index
```bash
# Install from private index
pip install my-package --index-url https://private.pypi.org/simple/
# Or add to pip.conf
[global]
index-url = https://private.pypi.org/simple/
extra-index-url = https://pypi.org/simple/
# Upload to private index
twine upload --repository-url https://private.pypi.org/ dist/*
```
## File Templates
### .gitignore for Python Packages
```gitignore
# Build artifacts
build/
dist/
*.egg-info/
*.egg
.eggs/
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
# Virtual environments
venv/
env/
ENV/
# IDE
.vscode/
.idea/
*.swp
# Testing
.pytest_cache/
.coverage
htmlcov/
# Distribution
*.whl
*.tar.gz
```
### MANIFEST.in
```
# MANIFEST.in
include README.md
include LICENSE
include pyproject.toml
recursive-include src/my_package/data *.json
recursive-include src/my_package/templates *.html
recursive-exclude * __pycache__
recursive-exclude * *.py[co]
```
## Checklist for Publishing
- [ ] Code is tested (pytest passing)
- [ ] Documentation is complete (README, docstrings)
- [ ] Version number updated
- [ ] CHANGELOG.md updated
- [ ] License file included
- [ ] pyproject.toml is complete
- [ ] Package builds without errors
- [ ] Installation tested in clean environment
- [ ] CLI tools work (if applicable)
- [ ] PyPI metadata is correct (classifiers, keywords)
- [ ] GitHub repository linked
- [ ] Tested on TestPyPI first
- [ ] Git tag created for release
## Resources
- **Python Packaging Guide**: https://packaging.python.org/
- **PyPI**: https://pypi.org/
- **TestPyPI**: https://test.pypi.org/
- **setuptools documentation**: https://setuptools.pypa.io/
- **build**: https://pypa-build.readthedocs.io/
- **twine**: https://twine.readthedocs.io/
## Best Practices Summary
1. **Use src/ layout** for cleaner package structure
2. **Use pyproject.toml** for modern packaging
3. **Pin build dependencies** in build-system.requires
4. **Version appropriately** with semantic versioning
5. **Include all metadata** (classifiers, URLs, etc.)
6. **Test installation** in clean environments
7. **Use TestPyPI** before publishing to PyPI
8. **Document thoroughly** with README and docstrings
9. **Include LICENSE** file
10. **Automate publishing** with CI/CD

869
plugins/python-development/skills/python-performance-optimization/SKILL.md Normal file
View File

@@ -0,0 +1,869 @@
---
name: python-performance-optimization
description: Profile and optimize Python code using cProfile, memory profilers, and performance best practices. Use when debugging slow Python code, optimizing bottlenecks, or improving application performance.
---
# Python Performance Optimization
Comprehensive guide to profiling, analyzing, and optimizing Python code for better performance, including CPU profiling, memory optimization, and implementation best practices.
## When to Use This Skill
- Identifying performance bottlenecks in Python applications
- Reducing application latency and response times
- Optimizing CPU-intensive operations
- Reducing memory consumption and memory leaks
- Improving database query performance
- Optimizing I/O operations
- Speeding up data processing pipelines
- Implementing high-performance algorithms
- Profiling production applications
## Core Concepts
### 1. Profiling Types
- **CPU Profiling**: Identify time-consuming functions
- **Memory Profiling**: Track memory allocation and leaks
- **Line Profiling**: Profile at line-by-line granularity
- **Call Graph**: Visualize function call relationships
### 2. Performance Metrics
- **Execution Time**: How long operations take
- **Memory Usage**: Peak and average memory consumption
- **CPU Utilization**: Processor usage patterns
- **I/O Wait**: Time spent on I/O operations
### 3. Optimization Strategies
- **Algorithmic**: Better algorithms and data structures
- **Implementation**: More efficient code patterns
- **Parallelization**: Multi-threading/processing
- **Caching**: Avoid redundant computation
- **Native Extensions**: C/Rust for critical paths
## Quick Start
### Basic Timing
```python
import time
def measure_time():
"""Simple timing measurement."""
start = time.time()
# Your code here
result = sum(range(1000000))
elapsed = time.time() - start
print(f"Execution time: {elapsed:.4f} seconds")
return result
# Better: use timeit for accurate measurements
import timeit
execution_time = timeit.timeit(
"sum(range(1000000))",
number=100
)
print(f"Average time: {execution_time/100:.6f} seconds")
```
## Profiling Tools
### Pattern 1: cProfile - CPU Profiling
```python
import cProfile
import pstats
from pstats import SortKey
def slow_function():
"""Function to profile."""
total = 0
for i in range(1000000):
total += i
return total
def another_function():
"""Another function."""
return [i**2 for i in range(100000)]
def main():
"""Main function to profile."""
result1 = slow_function()
result2 = another_function()
return result1, result2
# Profile the code
if __name__ == "__main__":
profiler = cProfile.Profile()
profiler.enable()
main()
profiler.disable()
# Print stats
stats = pstats.Stats(profiler)
stats.sort_stats(SortKey.CUMULATIVE)
stats.print_stats(10) # Top 10 functions
# Save to file for later analysis
stats.dump_stats("profile_output.prof")
```
**Command-line profiling:**
```bash
# Profile a script
python -m cProfile -o output.prof script.py
# View results
python -m pstats output.prof
# In pstats:
# sort cumtime
# stats 10
```
### Pattern 2: line_profiler - Line-by-Line Profiling
```python
# Install: pip install line-profiler
# Add @profile decorator (line_profiler provides this)
@profile
def process_data(data):
"""Process data with line profiling."""
result = []
for item in data:
processed = item * 2
result.append(processed)
return result
# Run with:
# kernprof -l -v script.py
```
**Manual line profiling:**
```python
from line_profiler import LineProfiler
def process_data(data):
"""Function to profile."""
result = []
for item in data:
processed = item * 2
result.append(processed)
return result
if __name__ == "__main__":
lp = LineProfiler()
lp.add_function(process_data)
data = list(range(100000))
lp_wrapper = lp(process_data)
lp_wrapper(data)
lp.print_stats()
```
### Pattern 3: memory_profiler - Memory Usage
```python
# Install: pip install memory-profiler
from memory_profiler import profile
@profile
def memory_intensive():
"""Function that uses lots of memory."""
# Create large list
big_list = [i for i in range(1000000)]
# Create large dict
big_dict = {i: i**2 for i in range(100000)}
# Process data
result = sum(big_list)
return result
if __name__ == "__main__":
memory_intensive()
# Run with:
# python -m memory_profiler script.py
```
### Pattern 4: py-spy - Production Profiling
```bash
# Install: pip install py-spy
# Profile a running Python process
py-spy top --pid 12345
# Generate flamegraph
py-spy record -o profile.svg --pid 12345
# Profile a script
py-spy record -o profile.svg -- python script.py
# Dump current call stack
py-spy dump --pid 12345
```
## Optimization Patterns
### Pattern 5: List Comprehensions vs Loops
```python
import timeit
# Slow: Traditional loop
def slow_squares(n):
"""Create list of squares using loop."""
result = []
for i in range(n):
result.append(i**2)
return result
# Fast: List comprehension
def fast_squares(n):
"""Create list of squares using comprehension."""
return [i**2 for i in range(n)]
# Benchmark
n = 100000
slow_time = timeit.timeit(lambda: slow_squares(n), number=100)
fast_time = timeit.timeit(lambda: fast_squares(n), number=100)
print(f"Loop: {slow_time:.4f}s")
print(f"Comprehension: {fast_time:.4f}s")
print(f"Speedup: {slow_time/fast_time:.2f}x")
# Even faster for simple operations: map
def faster_squares(n):
"""Use map for even better performance."""
return list(map(lambda x: x**2, range(n)))
```
### Pattern 6: Generator Expressions for Memory
```python
import sys
def list_approach():
"""Memory-intensive list."""
data = [i**2 for i in range(1000000)]
return sum(data)
def generator_approach():
"""Memory-efficient generator."""
data = (i**2 for i in range(1000000))
return sum(data)
# Memory comparison
list_data = [i for i in range(1000000)]
gen_data = (i for i in range(1000000))
print(f"List size: {sys.getsizeof(list_data)} bytes")
print(f"Generator size: {sys.getsizeof(gen_data)} bytes")
# Generators use constant memory regardless of size
```
### Pattern 7: String Concatenation
```python
import timeit
def slow_concat(items):
"""Slow string concatenation."""
result = ""
for item in items:
result += str(item)
return result
def fast_concat(items):
"""Fast string concatenation with join."""
return "".join(str(item) for item in items)
def faster_concat(items):
"""Even faster with list."""
parts = [str(item) for item in items]
return "".join(parts)
items = list(range(10000))
# Benchmark
slow = timeit.timeit(lambda: slow_concat(items), number=100)
fast = timeit.timeit(lambda: fast_concat(items), number=100)
faster = timeit.timeit(lambda: faster_concat(items), number=100)
print(f"Concatenation (+): {slow:.4f}s")
print(f"Join (generator): {fast:.4f}s")
print(f"Join (list): {faster:.4f}s")
```
### Pattern 8: Dictionary Lookups vs List Searches
```python
import timeit
# Create test data
size = 10000
items = list(range(size))
lookup_dict = {i: i for i in range(size)}
def list_search(items, target):
"""O(n) search in list."""
return target in items
def dict_search(lookup_dict, target):
"""O(1) search in dict."""
return target in lookup_dict
target = size - 1 # Worst case for list
# Benchmark
list_time = timeit.timeit(
lambda: list_search(items, target),
number=1000
)
dict_time = timeit.timeit(
lambda: dict_search(lookup_dict, target),
number=1000
)
print(f"List search: {list_time:.6f}s")
print(f"Dict search: {dict_time:.6f}s")
print(f"Speedup: {list_time/dict_time:.0f}x")
```
### Pattern 9: Local Variable Access
```python
import timeit
# Global variable (slow)
GLOBAL_VALUE = 100
def use_global():
"""Access global variable."""
total = 0
for i in range(10000):
total += GLOBAL_VALUE
return total
def use_local():
"""Use local variable."""
local_value = 100
total = 0
for i in range(10000):
total += local_value
return total
# Local is faster
global_time = timeit.timeit(use_global, number=1000)
local_time = timeit.timeit(use_local, number=1000)
print(f"Global access: {global_time:.4f}s")
print(f"Local access: {local_time:.4f}s")
print(f"Speedup: {global_time/local_time:.2f}x")
```
### Pattern 10: Function Call Overhead
```python
import timeit
def calculate_inline():
"""Inline calculation."""
total = 0
for i in range(10000):
total += i * 2 + 1
return total
def helper_function(x):
"""Helper function."""
return x * 2 + 1
def calculate_with_function():
"""Calculation with function calls."""
total = 0
for i in range(10000):
total += helper_function(i)
return total
# Inline is faster due to no call overhead
inline_time = timeit.timeit(calculate_inline, number=1000)
function_time = timeit.timeit(calculate_with_function, number=1000)
print(f"Inline: {inline_time:.4f}s")
print(f"Function calls: {function_time:.4f}s")
```
## Advanced Optimization
### Pattern 11: NumPy for Numerical Operations
```python
import timeit
import numpy as np
def python_sum(n):
"""Sum using pure Python."""
return sum(range(n))
def numpy_sum(n):
"""Sum using NumPy."""
return np.arange(n).sum()
n = 1000000
python_time = timeit.timeit(lambda: python_sum(n), number=100)
numpy_time = timeit.timeit(lambda: numpy_sum(n), number=100)
print(f"Python: {python_time:.4f}s")
print(f"NumPy: {numpy_time:.4f}s")
print(f"Speedup: {python_time/numpy_time:.2f}x")
# Vectorized operations
def python_multiply():
"""Element-wise multiplication in Python."""
a = list(range(100000))
b = list(range(100000))
return [x * y for x, y in zip(a, b)]
def numpy_multiply():
"""Vectorized multiplication in NumPy."""
a = np.arange(100000)
b = np.arange(100000)
return a * b
py_time = timeit.timeit(python_multiply, number=100)
np_time = timeit.timeit(numpy_multiply, number=100)
print(f"\nPython multiply: {py_time:.4f}s")
print(f"NumPy multiply: {np_time:.4f}s")
print(f"Speedup: {py_time/np_time:.2f}x")
```
### Pattern 12: Caching with functools.lru_cache
```python
from functools import lru_cache
import timeit
def fibonacci_slow(n):
"""Recursive fibonacci without caching."""
if n < 2:
return n
return fibonacci_slow(n-1) + fibonacci_slow(n-2)
@lru_cache(maxsize=None)
def fibonacci_fast(n):
"""Recursive fibonacci with caching."""
if n < 2:
return n
return fibonacci_fast(n-1) + fibonacci_fast(n-2)
# Massive speedup for recursive algorithms
n = 30
slow_time = timeit.timeit(lambda: fibonacci_slow(n), number=1)
fast_time = timeit.timeit(lambda: fibonacci_fast(n), number=1000)
print(f"Without cache (1 run): {slow_time:.4f}s")
print(f"With cache (1000 runs): {fast_time:.4f}s")
# Cache info
print(f"Cache info: {fibonacci_fast.cache_info()}")
```
### Pattern 13: Using __slots__ for Memory
```python
import sys
class RegularClass:
"""Regular class with __dict__."""
def __init__(self, x, y, z):
self.x = x
self.y = y
self.z = z
class SlottedClass:
"""Class with __slots__ for memory efficiency."""
__slots__ = ['x', 'y', 'z']
def __init__(self, x, y, z):
self.x = x
self.y = y
self.z = z
# Memory comparison
regular = RegularClass(1, 2, 3)
slotted = SlottedClass(1, 2, 3)
print(f"Regular class size: {sys.getsizeof(regular)} bytes")
print(f"Slotted class size: {sys.getsizeof(slotted)} bytes")
# Significant savings with many instances
regular_objects = [RegularClass(i, i+1, i+2) for i in range(10000)]
slotted_objects = [SlottedClass(i, i+1, i+2) for i in range(10000)]
print(f"\nMemory for 10000 regular objects: ~{sys.getsizeof(regular) * 10000} bytes")
print(f"Memory for 10000 slotted objects: ~{sys.getsizeof(slotted) * 10000} bytes")
```
### Pattern 14: Multiprocessing for CPU-Bound Tasks
```python
import multiprocessing as mp
import time
def cpu_intensive_task(n):
"""CPU-intensive calculation."""
return sum(i**2 for i in range(n))
def sequential_processing():
"""Process tasks sequentially."""
start = time.time()
results = [cpu_intensive_task(1000000) for _ in range(4)]
elapsed = time.time() - start
return elapsed, results
def parallel_processing():
"""Process tasks in parallel."""
start = time.time()
with mp.Pool(processes=4) as pool:
results = pool.map(cpu_intensive_task, [1000000] * 4)
elapsed = time.time() - start
return elapsed, results
if __name__ == "__main__":
seq_time, seq_results = sequential_processing()
par_time, par_results = parallel_processing()
print(f"Sequential: {seq_time:.2f}s")
print(f"Parallel: {par_time:.2f}s")
print(f"Speedup: {seq_time/par_time:.2f}x")
```
### Pattern 15: Async I/O for I/O-Bound Tasks
```python
import asyncio
import aiohttp
import time
import requests
urls = [
"https://httpbin.org/delay/1",
"https://httpbin.org/delay/1",
"https://httpbin.org/delay/1",
"https://httpbin.org/delay/1",
]
def synchronous_requests():
"""Synchronous HTTP requests."""
start = time.time()
results = []
for url in urls:
response = requests.get(url)
results.append(response.status_code)
elapsed = time.time() - start
return elapsed, results
async def async_fetch(session, url):
"""Async HTTP request."""
async with session.get(url) as response:
return response.status
async def asynchronous_requests():
"""Asynchronous HTTP requests."""
start = time.time()
async with aiohttp.ClientSession() as session:
tasks = [async_fetch(session, url) for url in urls]
results = await asyncio.gather(*tasks)
elapsed = time.time() - start
return elapsed, results
# Async is much faster for I/O-bound work
sync_time, sync_results = synchronous_requests()
async_time, async_results = asyncio.run(asynchronous_requests())
print(f"Synchronous: {sync_time:.2f}s")
print(f"Asynchronous: {async_time:.2f}s")
print(f"Speedup: {sync_time/async_time:.2f}x")
```
## Database Optimization
### Pattern 16: Batch Database Operations
```python
import sqlite3
import time
def create_db():
"""Create test database."""
conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)")
return conn
def slow_inserts(conn, count):
"""Insert records one at a time."""
start = time.time()
cursor = conn.cursor()
for i in range(count):
cursor.execute("INSERT INTO users (name) VALUES (?)", (f"User {i}",))
conn.commit() # Commit each insert
elapsed = time.time() - start
return elapsed
def fast_inserts(conn, count):
"""Batch insert with single commit."""
start = time.time()
cursor = conn.cursor()
data = [(f"User {i}",) for i in range(count)]
cursor.executemany("INSERT INTO users (name) VALUES (?)", data)
conn.commit() # Single commit
elapsed = time.time() - start
return elapsed
# Benchmark
conn1 = create_db()
slow_time = slow_inserts(conn1, 1000)
conn2 = create_db()
fast_time = fast_inserts(conn2, 1000)
print(f"Individual inserts: {slow_time:.4f}s")
print(f"Batch insert: {fast_time:.4f}s")
print(f"Speedup: {slow_time/fast_time:.2f}x")
```
### Pattern 17: Query Optimization
```python
# Use indexes for frequently queried columns
"""
-- Slow: No index
SELECT * FROM users WHERE email = 'user@example.com';
-- Fast: With index
CREATE INDEX idx_users_email ON users(email);
SELECT * FROM users WHERE email = 'user@example.com';
"""
# Use query planning
import sqlite3
conn = sqlite3.connect("example.db")
cursor = conn.cursor()
# Analyze query performance
cursor.execute("EXPLAIN QUERY PLAN SELECT * FROM users WHERE email = ?", ("test@example.com",))
print(cursor.fetchall())
# Use SELECT only needed columns
# Slow: SELECT *
# Fast: SELECT id, name
```
## Memory Optimization
### Pattern 18: Detecting Memory Leaks
```python
import tracemalloc
import gc
def memory_leak_example():
"""Example that leaks memory."""
leaked_objects = []
for i in range(100000):
# Objects added but never removed
leaked_objects.append([i] * 100)
# In real code, this would be an unintended reference
def track_memory_usage():
"""Track memory allocations."""
tracemalloc.start()
# Take snapshot before
snapshot1 = tracemalloc.take_snapshot()
# Run code
memory_leak_example()
# Take snapshot after
snapshot2 = tracemalloc.take_snapshot()
# Compare
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
print("Top 10 memory allocations:")
for stat in top_stats[:10]:
print(stat)
tracemalloc.stop()
# Monitor memory
track_memory_usage()
# Force garbage collection
gc.collect()
```
### Pattern 19: Iterators vs Lists
```python
import sys
def process_file_list(filename):
"""Load entire file into memory."""
with open(filename) as f:
lines = f.readlines() # Loads all lines
return sum(1 for line in lines if line.strip())
def process_file_iterator(filename):
"""Process file line by line."""
with open(filename) as f:
return sum(1 for line in f if line.strip())
# Iterator uses constant memory
# List loads entire file into memory
```
### Pattern 20: Weakref for Caches
```python
import weakref
class CachedResource:
"""Resource that can be garbage collected."""
def __init__(self, data):
self.data = data
# Regular cache prevents garbage collection
regular_cache = {}
def get_resource_regular(key):
"""Get resource from regular cache."""
if key not in regular_cache:
regular_cache[key] = CachedResource(f"Data for {key}")
return regular_cache[key]
# Weak reference cache allows garbage collection
weak_cache = weakref.WeakValueDictionary()
def get_resource_weak(key):
"""Get resource from weak cache."""
resource = weak_cache.get(key)
if resource is None:
resource = CachedResource(f"Data for {key}")
weak_cache[key] = resource
return resource
# When no strong references exist, objects can be GC'd
```
## Benchmarking Tools
### Custom Benchmark Decorator
```python
import time
from functools import wraps
def benchmark(func):
"""Decorator to benchmark function execution."""
@wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = func(*args, **kwargs)
elapsed = time.perf_counter() - start
print(f"{func.__name__} took {elapsed:.6f} seconds")
return result
return wrapper
@benchmark
def slow_function():
"""Function to benchmark."""
time.sleep(0.5)
return sum(range(1000000))
result = slow_function()
```
### Performance Testing with pytest-benchmark
```python
# Install: pip install pytest-benchmark
def test_list_comprehension(benchmark):
"""Benchmark list comprehension."""
result = benchmark(lambda: [i**2 for i in range(10000)])
assert len(result) == 10000
def test_map_function(benchmark):
"""Benchmark map function."""
result = benchmark(lambda: list(map(lambda x: x**2, range(10000))))
assert len(result) == 10000
# Run with: pytest test_performance.py --benchmark-compare
```
## Best Practices
1. **Profile before optimizing** - Measure to find real bottlenecks
2. **Focus on hot paths** - Optimize code that runs most frequently
3. **Use appropriate data structures** - Dict for lookups, set for membership
4. **Avoid premature optimization** - Clarity first, then optimize
5. **Use built-in functions** - They're implemented in C
6. **Cache expensive computations** - Use lru_cache
7. **Batch I/O operations** - Reduce system calls
8. **Use generators** for large datasets
9. **Consider NumPy** for numerical operations
10. **Profile production code** - Use py-spy for live systems
## Common Pitfalls
- Optimizing without profiling
- Using global variables unnecessarily
- Not using appropriate data structures
- Creating unnecessary copies of data
- Not using connection pooling for databases
- Ignoring algorithmic complexity
- Over-optimizing rare code paths
- Not considering memory usage
## Resources
- **cProfile**: Built-in CPU profiler
- **memory_profiler**: Memory usage profiling
- **line_profiler**: Line-by-line profiling
- **py-spy**: Sampling profiler for production
- **NumPy**: High-performance numerical computing
- **Cython**: Compile Python to C
- **PyPy**: Alternative Python interpreter with JIT
## Performance Checklist
- [ ] Profiled code to identify bottlenecks
- [ ] Used appropriate data structures
- [ ] Implemented caching where beneficial
- [ ] Optimized database queries
- [ ] Used generators for large datasets
- [ ] Considered multiprocessing for CPU-bound tasks
- [ ] Used async I/O for I/O-bound tasks
- [ ] Minimized function call overhead in hot loops
- [ ] Checked for memory leaks
- [ ] Benchmarked before and after optimization

907
plugins/python-development/skills/python-testing-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,907 @@
---
name: python-testing-patterns
description: Implement comprehensive testing strategies with pytest, fixtures, mocking, and test-driven development. Use when writing Python tests, setting up test suites, or implementing testing best practices.
---
# Python Testing Patterns
Comprehensive guide to implementing robust testing strategies in Python using pytest, fixtures, mocking, parameterization, and test-driven development practices.
## When to Use This Skill
- Writing unit tests for Python code
- Setting up test suites and test infrastructure
- Implementing test-driven development (TDD)
- Creating integration tests for APIs and services
- Mocking external dependencies and services
- Testing async code and concurrent operations
- Setting up continuous testing in CI/CD
- Implementing property-based testing
- Testing database operations
- Debugging failing tests
## Core Concepts
### 1. Test Types
- **Unit Tests**: Test individual functions/classes in isolation
- **Integration Tests**: Test interaction between components
- **Functional Tests**: Test complete features end-to-end
- **Performance Tests**: Measure speed and resource usage
### 2. Test Structure (AAA Pattern)
- **Arrange**: Set up test data and preconditions
- **Act**: Execute the code under test
- **Assert**: Verify the results
### 3. Test Coverage
- Measure what code is exercised by tests
- Identify untested code paths
- Aim for meaningful coverage, not just high percentages
### 4. Test Isolation
- Tests should be independent
- No shared state between tests
- Each test should clean up after itself
## Quick Start
```python
# test_example.py
def add(a, b):
return a + b
def test_add():
"""Basic test example."""
result = add(2, 3)
assert result == 5
def test_add_negative():
"""Test with negative numbers."""
assert add(-1, 1) == 0
# Run with: pytest test_example.py
```
## Fundamental Patterns
### Pattern 1: Basic pytest Tests
```python
# test_calculator.py
import pytest
class Calculator:
"""Simple calculator for testing."""
def add(self, a: float, b: float) -> float:
return a + b
def subtract(self, a: float, b: float) -> float:
return a - b
def multiply(self, a: float, b: float) -> float:
return a * b
def divide(self, a: float, b: float) -> float:
if b == 0:
raise ValueError("Cannot divide by zero")
return a / b
def test_addition():
"""Test addition."""
calc = Calculator()
assert calc.add(2, 3) == 5
assert calc.add(-1, 1) == 0
assert calc.add(0, 0) == 0
def test_subtraction():
"""Test subtraction."""
calc = Calculator()
assert calc.subtract(5, 3) == 2
assert calc.subtract(0, 5) == -5
def test_multiplication():
"""Test multiplication."""
calc = Calculator()
assert calc.multiply(3, 4) == 12
assert calc.multiply(0, 5) == 0
def test_division():
"""Test division."""
calc = Calculator()
assert calc.divide(6, 3) == 2
assert calc.divide(5, 2) == 2.5
def test_division_by_zero():
"""Test division by zero raises error."""
calc = Calculator()
with pytest.raises(ValueError, match="Cannot divide by zero"):
calc.divide(5, 0)
```
### Pattern 2: Fixtures for Setup and Teardown
```python
# test_database.py
import pytest
from typing import Generator
class Database:
"""Simple database class."""
def __init__(self, connection_string: str):
self.connection_string = connection_string
self.connected = False
def connect(self):
"""Connect to database."""
self.connected = True
def disconnect(self):
"""Disconnect from database."""
self.connected = False
def query(self, sql: str) -> list:
"""Execute query."""
if not self.connected:
raise RuntimeError("Not connected")
return [{"id": 1, "name": "Test"}]
@pytest.fixture
def db() -> Generator[Database, None, None]:
"""Fixture that provides connected database."""
# Setup
database = Database("sqlite:///:memory:")
database.connect()
# Provide to test
yield database
# Teardown
database.disconnect()
def test_database_query(db):
"""Test database query with fixture."""
results = db.query("SELECT * FROM users")
assert len(results) == 1
assert results[0]["name"] == "Test"
@pytest.fixture(scope="session")
def app_config():
"""Session-scoped fixture - created once per test session."""
return {
"database_url": "postgresql://localhost/test",
"api_key": "test-key",
"debug": True
}
@pytest.fixture(scope="module")
def api_client(app_config):
"""Module-scoped fixture - created once per test module."""
# Setup expensive resource
client = {"config": app_config, "session": "active"}
yield client
# Cleanup
client["session"] = "closed"
def test_api_client(api_client):
"""Test using api client fixture."""
assert api_client["session"] == "active"
assert api_client["config"]["debug"] is True
```
### Pattern 3: Parameterized Tests
```python
# test_validation.py
import pytest
def is_valid_email(email: str) -> bool:
"""Check if email is valid."""
return "@" in email and "." in email.split("@")[1]
@pytest.mark.parametrize("email,expected", [
("user@example.com", True),
("test.user@domain.co.uk", True),
("invalid.email", False),
("@example.com", False),
("user@domain", False),
("", False),
])
def test_email_validation(email, expected):
"""Test email validation with various inputs."""
assert is_valid_email(email) == expected
@pytest.mark.parametrize("a,b,expected", [
(2, 3, 5),
(0, 0, 0),
(-1, 1, 0),
(100, 200, 300),
(-5, -5, -10),
])
def test_addition_parameterized(a, b, expected):
"""Test addition with multiple parameter sets."""
from test_calculator import Calculator
calc = Calculator()
assert calc.add(a, b) == expected
# Using pytest.param for special cases
@pytest.mark.parametrize("value,expected", [
pytest.param(1, True, id="positive"),
pytest.param(0, False, id="zero"),
pytest.param(-1, False, id="negative"),
])
def test_is_positive(value, expected):
"""Test with custom test IDs."""
assert (value > 0) == expected
```
### Pattern 4: Mocking with unittest.mock
```python
# test_api_client.py
import pytest
from unittest.mock import Mock, patch, MagicMock
import requests
class APIClient:
"""Simple API client."""
def __init__(self, base_url: str):
self.base_url = base_url
def get_user(self, user_id: int) -> dict:
"""Fetch user from API."""
response = requests.get(f"{self.base_url}/users/{user_id}")
response.raise_for_status()
return response.json()
def create_user(self, data: dict) -> dict:
"""Create new user."""
response = requests.post(f"{self.base_url}/users", json=data)
response.raise_for_status()
return response.json()
def test_get_user_success():
"""Test successful API call with mock."""
client = APIClient("https://api.example.com")
mock_response = Mock()
mock_response.json.return_value = {"id": 1, "name": "John Doe"}
mock_response.raise_for_status.return_value = None
with patch("requests.get", return_value=mock_response) as mock_get:
user = client.get_user(1)
assert user["id"] == 1
assert user["name"] == "John Doe"
mock_get.assert_called_once_with("https://api.example.com/users/1")
def test_get_user_not_found():
"""Test API call with 404 error."""
client = APIClient("https://api.example.com")
mock_response = Mock()
mock_response.raise_for_status.side_effect = requests.HTTPError("404 Not Found")
with patch("requests.get", return_value=mock_response):
with pytest.raises(requests.HTTPError):
client.get_user(999)
@patch("requests.post")
def test_create_user(mock_post):
"""Test user creation with decorator syntax."""
client = APIClient("https://api.example.com")
mock_post.return_value.json.return_value = {"id": 2, "name": "Jane Doe"}
mock_post.return_value.raise_for_status.return_value = None
user_data = {"name": "Jane Doe", "email": "jane@example.com"}
result = client.create_user(user_data)
assert result["id"] == 2
mock_post.assert_called_once()
call_args = mock_post.call_args
assert call_args.kwargs["json"] == user_data
```
### Pattern 5: Testing Exceptions
```python
# test_exceptions.py
import pytest
def divide(a: float, b: float) -> float:
"""Divide a by b."""
if b == 0:
raise ZeroDivisionError("Division by zero")
if not isinstance(a, (int, float)) or not isinstance(b, (int, float)):
raise TypeError("Arguments must be numbers")
return a / b
def test_zero_division():
"""Test exception is raised for division by zero."""
with pytest.raises(ZeroDivisionError):
divide(10, 0)
def test_zero_division_with_message():
"""Test exception message."""
with pytest.raises(ZeroDivisionError, match="Division by zero"):
divide(5, 0)
def test_type_error():
"""Test type error exception."""
with pytest.raises(TypeError, match="must be numbers"):
divide("10", 5)
def test_exception_info():
"""Test accessing exception info."""
with pytest.raises(ValueError) as exc_info:
int("not a number")
assert "invalid literal" in str(exc_info.value)
```
## Advanced Patterns
### Pattern 6: Testing Async Code
```python
# test_async.py
import pytest
import asyncio
async def fetch_data(url: str) -> dict:
"""Fetch data asynchronously."""
await asyncio.sleep(0.1)
return {"url": url, "data": "result"}
@pytest.mark.asyncio
async def test_fetch_data():
"""Test async function."""
result = await fetch_data("https://api.example.com")
assert result["url"] == "https://api.example.com"
assert "data" in result
@pytest.mark.asyncio
async def test_concurrent_fetches():
"""Test concurrent async operations."""
urls = ["url1", "url2", "url3"]
tasks = [fetch_data(url) for url in urls]
results = await asyncio.gather(*tasks)
assert len(results) == 3
assert all("data" in r for r in results)
@pytest.fixture
async def async_client():
"""Async fixture."""
client = {"connected": True}
yield client
client["connected"] = False
@pytest.mark.asyncio
async def test_with_async_fixture(async_client):
"""Test using async fixture."""
assert async_client["connected"] is True
```
### Pattern 7: Monkeypatch for Testing
```python
# test_environment.py
import os
import pytest
def get_database_url() -> str:
"""Get database URL from environment."""
return os.environ.get("DATABASE_URL", "sqlite:///:memory:")
def test_database_url_default():
"""Test default database URL."""
# Will use actual environment variable if set
url = get_database_url()
assert url
def test_database_url_custom(monkeypatch):
"""Test custom database URL with monkeypatch."""
monkeypatch.setenv("DATABASE_URL", "postgresql://localhost/test")
assert get_database_url() == "postgresql://localhost/test"
def test_database_url_not_set(monkeypatch):
"""Test when env var is not set."""
monkeypatch.delenv("DATABASE_URL", raising=False)
assert get_database_url() == "sqlite:///:memory:"
class Config:
"""Configuration class."""
def __init__(self):
self.api_key = "production-key"
def get_api_key(self):
return self.api_key
def test_monkeypatch_attribute(monkeypatch):
"""Test monkeypatching object attributes."""
config = Config()
monkeypatch.setattr(config, "api_key", "test-key")
assert config.get_api_key() == "test-key"
```
### Pattern 8: Temporary Files and Directories
```python
# test_file_operations.py
import pytest
from pathlib import Path
def save_data(filepath: Path, data: str):
"""Save data to file."""
filepath.write_text(data)
def load_data(filepath: Path) -> str:
"""Load data from file."""
return filepath.read_text()
def test_file_operations(tmp_path):
"""Test file operations with temporary directory."""
# tmp_path is a pathlib.Path object
test_file = tmp_path / "test_data.txt"
# Save data
save_data(test_file, "Hello, World!")
# Verify file exists
assert test_file.exists()
# Load and verify data
data = load_data(test_file)
assert data == "Hello, World!"
def test_multiple_files(tmp_path):
"""Test with multiple temporary files."""
files = {
"file1.txt": "Content 1",
"file2.txt": "Content 2",
"file3.txt": "Content 3"
}
for filename, content in files.items():
filepath = tmp_path / filename
save_data(filepath, content)
# Verify all files created
assert len(list(tmp_path.iterdir())) == 3
# Verify contents
for filename, expected_content in files.items():
filepath = tmp_path / filename
assert load_data(filepath) == expected_content
```
### Pattern 9: Custom Fixtures and Conftest
```python
# conftest.py
"""Shared fixtures for all tests."""
import pytest
@pytest.fixture(scope="session")
def database_url():
"""Provide database URL for all tests."""
return "postgresql://localhost/test_db"
@pytest.fixture(autouse=True)
def reset_database(database_url):
"""Auto-use fixture that runs before each test."""
# Setup: Clear database
print(f"Clearing database: {database_url}")
yield
# Teardown: Clean up
print("Test completed")
@pytest.fixture
def sample_user():
"""Provide sample user data."""
return {
"id": 1,
"name": "Test User",
"email": "test@example.com"
}
@pytest.fixture
def sample_users():
"""Provide list of sample users."""
return [
{"id": 1, "name": "User 1"},
{"id": 2, "name": "User 2"},
{"id": 3, "name": "User 3"},
]
# Parametrized fixture
@pytest.fixture(params=["sqlite", "postgresql", "mysql"])
def db_backend(request):
"""Fixture that runs tests with different database backends."""
return request.param
def test_with_db_backend(db_backend):
"""This test will run 3 times with different backends."""
print(f"Testing with {db_backend}")
assert db_backend in ["sqlite", "postgresql", "mysql"]
```
### Pattern 10: Property-Based Testing
```python
# test_properties.py
from hypothesis import given, strategies as st
import pytest
def reverse_string(s: str) -> str:
"""Reverse a string."""
return s[::-1]
@given(st.text())
def test_reverse_twice_is_original(s):
"""Property: reversing twice returns original."""
assert reverse_string(reverse_string(s)) == s
@given(st.text())
def test_reverse_length(s):
"""Property: reversed string has same length."""
assert len(reverse_string(s)) == len(s)
@given(st.integers(), st.integers())
def test_addition_commutative(a, b):
"""Property: addition is commutative."""
assert a + b == b + a
@given(st.lists(st.integers()))
def test_sorted_list_properties(lst):
"""Property: sorted list is ordered."""
sorted_lst = sorted(lst)
# Same length
assert len(sorted_lst) == len(lst)
# All elements present
assert set(sorted_lst) == set(lst)
# Is ordered
for i in range(len(sorted_lst) - 1):
assert sorted_lst[i] <= sorted_lst[i + 1]
```
## Testing Best Practices
### Test Organization
```python
# tests/
# __init__.py
# conftest.py # Shared fixtures
# test_unit/ # Unit tests
# test_models.py
# test_utils.py
# test_integration/ # Integration tests
# test_api.py
# test_database.py
# test_e2e/ # End-to-end tests
# test_workflows.py
```
### Test Naming
```python
# Good test names
def test_user_creation_with_valid_data():
"""Clear name describes what is being tested."""
pass
def test_login_fails_with_invalid_password():
"""Name describes expected behavior."""
pass
def test_api_returns_404_for_missing_resource():
"""Specific about inputs and expected outcomes."""
pass
# Bad test names
def test_1(): # Not descriptive
pass
def test_user(): # Too vague
pass
def test_function(): # Doesn't explain what's tested
pass
```
### Test Markers
```python
# test_markers.py
import pytest
@pytest.mark.slow
def test_slow_operation():
"""Mark slow tests."""
import time
time.sleep(2)
@pytest.mark.integration
def test_database_integration():
"""Mark integration tests."""
pass
@pytest.mark.skip(reason="Feature not implemented yet")
def test_future_feature():
"""Skip tests temporarily."""
pass
@pytest.mark.skipif(os.name == "nt", reason="Unix only test")
def test_unix_specific():
"""Conditional skip."""
pass
@pytest.mark.xfail(reason="Known bug #123")
def test_known_bug():
"""Mark expected failures."""
assert False
# Run with:
# pytest -m slow # Run only slow tests
# pytest -m "not slow" # Skip slow tests
# pytest -m integration # Run integration tests
```
### Coverage Reporting
```bash
# Install coverage
pip install pytest-cov
# Run tests with coverage
pytest --cov=myapp tests/
# Generate HTML report
pytest --cov=myapp --cov-report=html tests/
# Fail if coverage below threshold
pytest --cov=myapp --cov-fail-under=80 tests/
# Show missing lines
pytest --cov=myapp --cov-report=term-missing tests/
```
## Testing Database Code
```python
# test_database_models.py
import pytest
from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, Session
Base = declarative_base()
class User(Base):
"""User model."""
__tablename__ = "users"
id = Column(Integer, primary_key=True)
name = Column(String(50))
email = Column(String(100), unique=True)
@pytest.fixture(scope="function")
def db_session() -> Session:
"""Create in-memory database for testing."""
engine = create_engine("sqlite:///:memory:")
Base.metadata.create_all(engine)
SessionLocal = sessionmaker(bind=engine)
session = SessionLocal()
yield session
session.close()
def test_create_user(db_session):
"""Test creating a user."""
user = User(name="Test User", email="test@example.com")
db_session.add(user)
db_session.commit()
assert user.id is not None
assert user.name == "Test User"
def test_query_user(db_session):
"""Test querying users."""
user1 = User(name="User 1", email="user1@example.com")
user2 = User(name="User 2", email="user2@example.com")
db_session.add_all([user1, user2])
db_session.commit()
users = db_session.query(User).all()
assert len(users) == 2
def test_unique_email_constraint(db_session):
"""Test unique email constraint."""
from sqlalchemy.exc import IntegrityError
user1 = User(name="User 1", email="same@example.com")
user2 = User(name="User 2", email="same@example.com")
db_session.add(user1)
db_session.commit()
db_session.add(user2)
with pytest.raises(IntegrityError):
db_session.commit()
```
## CI/CD Integration
```yaml
# .github/workflows/test.yml
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
pip install -e ".[dev]"
pip install pytest pytest-cov
- name: Run tests
run: |
pytest --cov=myapp --cov-report=xml
- name: Upload coverage
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml
```
## Configuration Files
```ini
# pytest.ini
[pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts =
-v
--strict-markers
--tb=short
--cov=myapp
--cov-report=term-missing
markers =
slow: marks tests as slow
integration: marks integration tests
unit: marks unit tests
e2e: marks end-to-end tests
```
```toml
# pyproject.toml
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
addopts = [
"-v",
"--cov=myapp",
"--cov-report=term-missing",
]
[tool.coverage.run]
source = ["myapp"]
omit = ["*/tests/*", "*/migrations/*"]
[tool.coverage.report]
exclude_lines = [
"pragma: no cover",
"def __repr__",
"raise AssertionError",
"raise NotImplementedError",
]
```
## Resources
- **pytest documentation**: https://docs.pytest.org/
- **unittest.mock**: https://docs.python.org/3/library/unittest.mock.html
- **hypothesis**: Property-based testing
- **pytest-asyncio**: Testing async code
- **pytest-cov**: Coverage reporting
- **pytest-mock**: pytest wrapper for mock
## Best Practices Summary
1. **Write tests first** (TDD) or alongside code
2. **One assertion per test** when possible
3. **Use descriptive test names** that explain behavior
4. **Keep tests independent** and isolated
5. **Use fixtures** for setup and teardown
6. **Mock external dependencies** appropriately
7. **Parametrize tests** to reduce duplication
8. **Test edge cases** and error conditions
9. **Measure coverage** but focus on quality
10. **Run tests in CI/CD** on every commit

831
plugins/python-development/skills/uv-package-manager/SKILL.md Normal file
View File

@@ -0,0 +1,831 @@
---
name: uv-package-manager
description: Master the uv package manager for fast Python dependency management, virtual environments, and modern Python project workflows. Use when setting up Python projects, managing dependencies, or optimizing Python development workflows with uv.
---
# UV Package Manager
Comprehensive guide to using uv, an extremely fast Python package installer and resolver written in Rust, for modern Python project management and dependency workflows.
## When to Use This Skill
- Setting up new Python projects quickly
- Managing Python dependencies faster than pip
- Creating and managing virtual environments
- Installing Python interpreters
- Resolving dependency conflicts efficiently
- Migrating from pip/pip-tools/poetry
- Speeding up CI/CD pipelines
- Managing monorepo Python projects
- Working with lockfiles for reproducible builds
- Optimizing Docker builds with Python dependencies
## Core Concepts
### 1. What is uv?
- **Ultra-fast package installer**: 10-100x faster than pip
- **Written in Rust**: Leverages Rust's performance
- **Drop-in pip replacement**: Compatible with pip workflows
- **Virtual environment manager**: Create and manage venvs
- **Python installer**: Download and manage Python versions
- **Resolver**: Advanced dependency resolution
- **Lockfile support**: Reproducible installations
### 2. Key Features
- Blazing fast installation speeds
- Disk space efficient with global cache
- Compatible with pip, pip-tools, poetry
- Comprehensive dependency resolution
- Cross-platform support (Linux, macOS, Windows)
- No Python required for installation
- Built-in virtual environment support
### 3. UV vs Traditional Tools
- **vs pip**: 10-100x faster, better resolver
- **vs pip-tools**: Faster, simpler, better UX
- **vs poetry**: Faster, less opinionated, lighter
- **vs conda**: Faster, Python-focused
## Installation
### Quick Install
```bash
# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell)
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
# Using pip (if you already have Python)
pip install uv
# Using Homebrew (macOS)
brew install uv
# Using cargo (if you have Rust)
cargo install --git https://github.com/astral-sh/uv uv
```
### Verify Installation
```bash
uv --version
# uv 0.x.x
```
## Quick Start
### Create a New Project
```bash
# Create new project with virtual environment
uv init my-project
cd my-project
# Or create in current directory
uv init .
# Initialize creates:
# - .python-version (Python version)
# - pyproject.toml (project config)
# - README.md
# - .gitignore
```
### Install Dependencies
```bash
# Install packages (creates venv if needed)
uv add requests pandas
# Install dev dependencies
uv add --dev pytest black ruff
# Install from requirements.txt
uv pip install -r requirements.txt
# Install from pyproject.toml
uv sync
```
## Virtual Environment Management
### Pattern 1: Creating Virtual Environments
```bash
# Create virtual environment with uv
uv venv
# Create with specific Python version
uv venv --python 3.12
# Create with custom name
uv venv my-env
# Create with system site packages
uv venv --system-site-packages
# Specify location
uv venv /path/to/venv
```
### Pattern 2: Activating Virtual Environments
```bash
# Linux/macOS
source .venv/bin/activate
# Windows (Command Prompt)
.venv\Scripts\activate.bat
# Windows (PowerShell)
.venv\Scripts\Activate.ps1
# Or use uv run (no activation needed)
uv run python script.py
uv run pytest
```
### Pattern 3: Using uv run
```bash
# Run Python script (auto-activates venv)
uv run python app.py
# Run installed CLI tool
uv run black .
uv run pytest
# Run with specific Python version
uv run --python 3.11 python script.py
# Pass arguments
uv run python script.py --arg value
```
## Package Management
### Pattern 4: Adding Dependencies
```bash
# Add package (adds to pyproject.toml)
uv add requests
# Add with version constraint
uv add "django>=4.0,<5.0"
# Add multiple packages
uv add numpy pandas matplotlib
# Add dev dependency
uv add --dev pytest pytest-cov
# Add optional dependency group
uv add --optional docs sphinx
# Add from git
uv add git+https://github.com/user/repo.git
# Add from git with specific ref
uv add git+https://github.com/user/repo.git@v1.0.0
# Add from local path
uv add ./local-package
# Add editable local package
uv add -e ./local-package
```
### Pattern 5: Removing Dependencies
```bash
# Remove package
uv remove requests
# Remove dev dependency
uv remove --dev pytest
# Remove multiple packages
uv remove numpy pandas matplotlib
```
### Pattern 6: Upgrading Dependencies
```bash
# Upgrade specific package
uv add --upgrade requests
# Upgrade all packages
uv sync --upgrade
# Upgrade package to latest
uv add --upgrade requests
# Show what would be upgraded
uv tree --outdated
```
### Pattern 7: Locking Dependencies
```bash
# Generate uv.lock file
uv lock
# Update lock file
uv lock --upgrade
# Lock without installing
uv lock --no-install
# Lock specific package
uv lock --upgrade-package requests
```
## Python Version Management
### Pattern 8: Installing Python Versions
```bash
# Install Python version
uv python install 3.12
# Install multiple versions
uv python install 3.11 3.12 3.13
# Install latest version
uv python install
# List installed versions
uv python list
# Find available versions
uv python list --all-versions
```
### Pattern 9: Setting Python Version
```bash
# Set Python version for project
uv python pin 3.12
# This creates/updates .python-version file
# Use specific Python version for command
uv --python 3.11 run python script.py
# Create venv with specific version
uv venv --python 3.12
```
## Project Configuration
### Pattern 10: pyproject.toml with uv
```toml
[project]
name = "my-project"
version = "0.1.0"
description = "My awesome project"
readme = "README.md"
requires-python = ">=3.8"
dependencies = [
"requests>=2.31.0",
"pydantic>=2.0.0",
"click>=8.1.0",
]
[project.optional-dependencies]
dev = [
"pytest>=7.4.0",
"pytest-cov>=4.1.0",
"black>=23.0.0",
"ruff>=0.1.0",
"mypy>=1.5.0",
]
docs = [
"sphinx>=7.0.0",
"sphinx-rtd-theme>=1.3.0",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.uv]
dev-dependencies = [
# Additional dev dependencies managed by uv
]
[tool.uv.sources]
# Custom package sources
my-package = { git = "https://github.com/user/repo.git" }
```
### Pattern 11: Using uv with Existing Projects
```bash
# Migrate from requirements.txt
uv add -r requirements.txt
# Migrate from poetry
# Already have pyproject.toml, just use:
uv sync
# Export to requirements.txt
uv pip freeze > requirements.txt
# Export with hashes
uv pip freeze --require-hashes > requirements.txt
```
## Advanced Workflows
### Pattern 12: Monorepo Support
```bash
# Project structure
# monorepo/
# packages/
# package-a/
# pyproject.toml
# package-b/
# pyproject.toml
# pyproject.toml (root)
# Root pyproject.toml
[tool.uv.workspace]
members = ["packages/*"]
# Install all workspace packages
uv sync
# Add workspace dependency
uv add --path ./packages/package-a
```
### Pattern 13: CI/CD Integration
```yaml
# .github/workflows/test.yml
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v2
with:
enable-cache: true
- name: Set up Python
run: uv python install 3.12
- name: Install dependencies
run: uv sync --all-extras --dev
- name: Run tests
run: uv run pytest
- name: Run linting
run: |
uv run ruff check .
uv run black --check .
```
### Pattern 14: Docker Integration
```dockerfile
# Dockerfile
FROM python:3.12-slim
# Install uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
# Set working directory
WORKDIR /app
# Copy dependency files
COPY pyproject.toml uv.lock ./
# Install dependencies
RUN uv sync --frozen --no-dev
# Copy application code
COPY . .
# Run application
CMD ["uv", "run", "python", "app.py"]
```
**Optimized multi-stage build:**
```dockerfile
# Multi-stage Dockerfile
FROM python:3.12-slim AS builder
# Install uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
WORKDIR /app
# Install dependencies to venv
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev --no-editable
# Runtime stage
FROM python:3.12-slim
WORKDIR /app
# Copy venv from builder
COPY --from=builder /app/.venv .venv
COPY . .
# Use venv
ENV PATH="/app/.venv/bin:$PATH"
CMD ["python", "app.py"]
```
### Pattern 15: Lockfile Workflows
```bash
# Create lockfile (uv.lock)
uv lock
# Install from lockfile (exact versions)
uv sync --frozen
# Update lockfile without installing
uv lock --no-install
# Upgrade specific package in lock
uv lock --upgrade-package requests
# Check if lockfile is up to date
uv lock --check
# Export lockfile to requirements.txt
uv export --format requirements-txt > requirements.txt
# Export with hashes for security
uv export --format requirements-txt --hash > requirements.txt
```
## Performance Optimization
### Pattern 16: Using Global Cache
```bash
# UV automatically uses global cache at:
# Linux: ~/.cache/uv
# macOS: ~/Library/Caches/uv
# Windows: %LOCALAPPDATA%\uv\cache
# Clear cache
uv cache clean
# Check cache size
uv cache dir
```
### Pattern 17: Parallel Installation
```bash
# UV installs packages in parallel by default
# Control parallelism
uv pip install --jobs 4 package1 package2
# No parallel (sequential)
uv pip install --jobs 1 package
```
### Pattern 18: Offline Mode
```bash
# Install from cache only (no network)
uv pip install --offline package
# Sync from lockfile offline
uv sync --frozen --offline
```
## Comparison with Other Tools
### uv vs pip
```bash
# pip
python -m venv .venv
source .venv/bin/activate
pip install requests pandas numpy
# ~30 seconds
# uv
uv venv
uv add requests pandas numpy
# ~2 seconds (10-15x faster)
```
### uv vs poetry
```bash
# poetry
poetry init
poetry add requests pandas
poetry install
# ~20 seconds
# uv
uv init
uv add requests pandas
uv sync
# ~3 seconds (6-7x faster)
```
### uv vs pip-tools
```bash
# pip-tools
pip-compile requirements.in
pip-sync requirements.txt
# ~15 seconds
# uv
uv lock
uv sync --frozen
# ~2 seconds (7-8x faster)
```
## Common Workflows
### Pattern 19: Starting a New Project
```bash
# Complete workflow
uv init my-project
cd my-project
# Set Python version
uv python pin 3.12
# Add dependencies
uv add fastapi uvicorn pydantic
# Add dev dependencies
uv add --dev pytest black ruff mypy
# Create structure
mkdir -p src/my_project tests
# Run tests
uv run pytest
# Format code
uv run black .
uv run ruff check .
```
### Pattern 20: Maintaining Existing Project
```bash
# Clone repository
git clone https://github.com/user/project.git
cd project
# Install dependencies (creates venv automatically)
uv sync
# Install with dev dependencies
uv sync --all-extras
# Update dependencies
uv lock --upgrade
# Run application
uv run python app.py
# Run tests
uv run pytest
# Add new dependency
uv add new-package
# Commit updated files
git add pyproject.toml uv.lock
git commit -m "Add new-package dependency"
```
## Tool Integration
### Pattern 21: Pre-commit Hooks
```yaml
# .pre-commit-config.yaml
repos:
- repo: local
hooks:
- id: uv-lock
name: uv lock
entry: uv lock
language: system
pass_filenames: false
- id: ruff
name: ruff
entry: uv run ruff check --fix
language: system
types: [python]
- id: black
name: black
entry: uv run black
language: system
types: [python]
```
### Pattern 22: VS Code Integration
```json
// .vscode/settings.json
{
"python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
"python.terminal.activateEnvironment": true,
"python.testing.pytestEnabled": true,
"python.testing.pytestArgs": ["-v"],
"python.linting.enabled": true,
"python.formatting.provider": "black",
"[python]": {
"editor.defaultFormatter": "ms-python.black-formatter",
"editor.formatOnSave": true
}
}
```
## Troubleshooting
### Common Issues
```bash
# Issue: uv not found
# Solution: Add to PATH or reinstall
echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> ~/.bashrc
# Issue: Wrong Python version
# Solution: Pin version explicitly
uv python pin 3.12
uv venv --python 3.12
# Issue: Dependency conflict
# Solution: Check resolution
uv lock --verbose
# Issue: Cache issues
# Solution: Clear cache
uv cache clean
# Issue: Lockfile out of sync
# Solution: Regenerate
uv lock --upgrade
```
## Best Practices
### Project Setup
1. **Always use lockfiles** for reproducibility
2. **Pin Python version** with .python-version
3. **Separate dev dependencies** from production
4. **Use uv run** instead of activating venv
5. **Commit uv.lock** to version control
6. **Use --frozen in CI** for consistent builds
7. **Leverage global cache** for speed
8. **Use workspace** for monorepos
9. **Export requirements.txt** for compatibility
10. **Keep uv updated** for latest features
### Performance Tips
```bash
# Use frozen installs in CI
uv sync --frozen
# Use offline mode when possible
uv sync --offline
# Parallel operations (automatic)
# uv does this by default
# Reuse cache across environments
# uv shares cache globally
# Use lockfiles to skip resolution
uv sync --frozen # skips resolution
```
## Migration Guide
### From pip + requirements.txt
```bash
# Before
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
# After
uv venv
uv pip install -r requirements.txt
# Or better:
uv init
uv add -r requirements.txt
```
### From Poetry
```bash
# Before
poetry install
poetry add requests
# After
uv sync
uv add requests
# Keep existing pyproject.toml
# uv reads [project] and [tool.poetry] sections
```
### From pip-tools
```bash
# Before
pip-compile requirements.in
pip-sync requirements.txt
# After
uv lock
uv sync --frozen
```
## Command Reference
### Essential Commands
```bash
# Project management
uv init [PATH] # Initialize project
uv add PACKAGE # Add dependency
uv remove PACKAGE # Remove dependency
uv sync # Install dependencies
uv lock # Create/update lockfile
# Virtual environments
uv venv [PATH] # Create venv
uv run COMMAND # Run in venv
# Python management
uv python install VERSION # Install Python
uv python list # List installed Pythons
uv python pin VERSION # Pin Python version
# Package installation (pip-compatible)
uv pip install PACKAGE # Install package
uv pip uninstall PACKAGE # Uninstall package
uv pip freeze # List installed
uv pip list # List packages
# Utility
uv cache clean # Clear cache
uv cache dir # Show cache location
uv --version # Show version
```
## Resources
- **Official documentation**: https://docs.astral.sh/uv/
- **GitHub repository**: https://github.com/astral-sh/uv
- **Astral blog**: https://astral.sh/blog
- **Migration guides**: https://docs.astral.sh/uv/guides/
- **Comparison with other tools**: https://docs.astral.sh/uv/pip/compatibility/
## Best Practices Summary
1. **Use uv for all new projects** - Start with `uv init`
2. **Commit lockfiles** - Ensure reproducible builds
3. **Pin Python versions** - Use .python-version
4. **Use uv run** - Avoid manual venv activation
5. **Leverage caching** - Let uv manage global cache
6. **Use --frozen in CI** - Exact reproduction
7. **Keep uv updated** - Fast-moving project
8. **Use workspaces** - For monorepo projects
9. **Export for compatibility** - Generate requirements.txt when needed
10. **Read the docs** - uv is feature-rich and evolving

191
plugins/security-scanning/skills/sast-configuration/SKILL.md Normal file
View File

@@ -0,0 +1,191 @@
---
name: sast-configuration
description: Configure Static Application Security Testing (SAST) tools for automated vulnerability detection in application code. Use when setting up security scanning, implementing DevSecOps practices, or automating code vulnerability detection.
---
# SAST Configuration
Static Application Security Testing (SAST) tool setup, configuration, and custom rule creation for comprehensive security scanning across multiple programming languages.
## Overview
This skill provides comprehensive guidance for setting up and configuring SAST tools including Semgrep, SonarQube, and CodeQL. Use this skill when you need to:
- Set up SAST scanning in CI/CD pipelines
- Create custom security rules for your codebase
- Configure quality gates and compliance policies
- Optimize scan performance and reduce false positives
- Integrate multiple SAST tools for defense-in-depth
## Core Capabilities
### 1. Semgrep Configuration
- Custom rule creation with pattern matching
- Language-specific security rules (Python, JavaScript, Go, Java, etc.)
- CI/CD integration (GitHub Actions, GitLab CI, Jenkins)
- False positive tuning and rule optimization
- Organizational policy enforcement
### 2. SonarQube Setup
- Quality gate configuration
- Security hotspot analysis
- Code coverage and technical debt tracking
- Custom quality profiles for languages
- Enterprise integration with LDAP/SAML
### 3. CodeQL Analysis
- GitHub Advanced Security integration
- Custom query development
- Vulnerability variant analysis
- Security research workflows
- SARIF result processing
## Quick Start
### Initial Assessment
1. Identify primary programming languages in your codebase
2. Determine compliance requirements (PCI-DSS, SOC 2, etc.)
3. Choose SAST tool based on language support and integration needs
4. Review baseline scan to understand current security posture
### Basic Setup
```bash
# Semgrep quick start
pip install semgrep
semgrep --config=auto --error
# SonarQube with Docker
docker run -d --name sonarqube -p 9000:9000 sonarqube:latest
# CodeQL CLI setup
gh extension install github/gh-codeql
codeql database create mydb --language=python
```
## Reference Documentation
- [Semgrep Rule Creation](references/semgrep-rules.md) - Pattern-based security rule development
- [SonarQube Configuration](references/sonarqube-config.md) - Quality gates and profiles
- [CodeQL Setup Guide](references/codeql-setup.md) - Query development and workflows
## Templates & Assets
- [semgrep-config.yml](assets/semgrep-config.yml) - Production-ready Semgrep configuration
- [sonarqube-settings.xml](assets/sonarqube-settings.xml) - SonarQube quality profile template
- [run-sast.sh](scripts/run-sast.sh) - Automated SAST execution script
## Integration Patterns
### CI/CD Pipeline Integration
```yaml
# GitHub Actions example
- name: Run Semgrep
uses: returntocorp/semgrep-action@v1
with:
config: >-
p/security-audit
p/owasp-top-ten
```
### Pre-commit Hook
```bash
# .pre-commit-config.yaml
- repo: https://github.com/returntocorp/semgrep
rev: v1.45.0
hooks:
- id: semgrep
args: ['--config=auto', '--error']
```
## Best Practices
1. **Start with Baseline**
- Run initial scan to establish security baseline
- Prioritize critical and high severity findings
- Create remediation roadmap
2. **Incremental Adoption**
- Begin with security-focused rules
- Gradually add code quality rules
- Implement blocking only for critical issues
3. **False Positive Management**
- Document legitimate suppressions
- Create allow lists for known safe patterns
- Regularly review suppressed findings
4. **Performance Optimization**
- Exclude test files and generated code
- Use incremental scanning for large codebases
- Cache scan results in CI/CD
5. **Team Enablement**
- Provide security training for developers
- Create internal documentation for common patterns
- Establish security champions program
## Common Use Cases
### New Project Setup
```bash
./scripts/run-sast.sh --setup --language python --tools semgrep,sonarqube
```
### Custom Rule Development
```yaml
# See references/semgrep-rules.md for detailed examples
rules:
- id: hardcoded-jwt-secret
pattern: jwt.encode($DATA, "...", ...)
message: JWT secret should not be hardcoded
severity: ERROR
```
### Compliance Scanning
```bash
# PCI-DSS focused scan
semgrep --config p/pci-dss --json -o pci-scan-results.json
```
## Troubleshooting
### High False Positive Rate
- Review and tune rule sensitivity
- Add path filters to exclude test files
- Use nostmt metadata for noisy patterns
- Create organization-specific rule exceptions
### Performance Issues
- Enable incremental scanning
- Parallelize scans across modules
- Optimize rule patterns for efficiency
- Cache dependencies and scan results
### Integration Failures
- Verify API tokens and credentials
- Check network connectivity and proxy settings
- Review SARIF output format compatibility
- Validate CI/CD runner permissions
## Related Skills
- [OWASP Top 10 Checklist](../owasp-top10-checklist/SKILL.md)
- [Container Security](../container-security/SKILL.md)
- [Dependency Scanning](../dependency-scanning/SKILL.md)
## Tool Comparison
| Tool | Best For | Language Support | Cost | Integration |
|------|----------|------------------|------|-------------|
| Semgrep | Custom rules, fast scans | 30+ languages | Free/Enterprise | Excellent |
| SonarQube | Code quality + security | 25+ languages | Free/Commercial | Good |
| CodeQL | Deep analysis, research | 10+ languages | Free (OSS) | GitHub native |
## Next Steps
1. Complete initial SAST tool setup
2. Run baseline security scan
3. Create custom rules for organization-specific patterns
4. Integrate into CI/CD pipeline
5. Establish security gate policies
6. Train development team on findings and remediation