Code Review Lottery

Developer

Original Idea

Code Review Lottery A Slack bot that fairly distributes code review assignments based on workload, expertise, and recent review history.

Product Requirements Document (PRD): Code Review Lottery

1. Executive Summary

Code Review Lottery is an intelligent Slack-integrated orchestration platform designed to eliminate "reviewer fatigue" and knowledge silos. By leveraging a weighted lottery algorithm and AI-powered expertise detection, the system automates the assignment of pull requests (PRs) based on real-time workload, domain mastery, and historical fairness metrics. It transforms code review from a manual bottleneck into a balanced, transparent, and data-driven process.

2. Problem Statement

Manual reviewer selection in software teams is prone to cognitive bias, often resulting in "hero culture" where a few senior developers are overwhelmed while others become isolated from critical parts of the codebase. This leads to:

  • Burnout: High-performers are over-leveraged for reviews.
  • Knowledge Silos: Specific code modules are only understood by 1-2 people ("Bus Factor").
  • Delivery Bottlenecks: PRs stall when assigned to overloaded or unavailable reviewers.
  • Unfair Distribution: Lack of visibility into who is contributing most to the team's quality gate.

3. Goals & Success Metrics

  • Review Parity: Reduce the standard deviation of reviews assigned across the team by 40% within three months.
  • Reduced Cycle Time: Decrease "Time to First Review" by 25% through workload-aware routing.
  • Knowledge Spread: Increase the number of "known contributors" per repository module by 15% using the Knowledge-Transfer "Wildcard" feature.
  • SLA Compliance: Maintain a >90% success rate for first reviews within 24 hours.

4. User Personas

4.1. The Software Engineer (The Reviewer/Author)

  • Needs: To receive timely feedback and not be interrupted by too many review requests when deep in their own tasks.
  • Pain Point: Receiving 5 review requests in one morning while trying to hit a sprint goal.

4.2. The Tech Lead (The Facilitator)

  • Needs: High-quality technical oversight and ensuring no PR stays "stale."
  • Pain Point: Manually reassigning PRs when someone goes on vacation or becomes a bottleneck.

4.3. The Engineering Manager (The Optimizer)

  • Needs: Visibility into team health, distribution parity, and identifying training needs.
  • Pain Point: Lack of data to justify hiring or to identify developers who are "taking" but not "giving" back to the review pool.

5. User Stories

  • As an Author, I want my PR to be automatically assigned to the best-fit reviewers so that I don't have to "beg" for reviews in Slack.
  • As a Reviewer, I want the system to ignore me when I have more than 3 active reviews so that I can focus on my current assignments.
  • As a Lead, I want the bot to occasionally assign a junior to a complex module alongside a senior so that we improve our "Bus Factor."
  • As a Manager, I want a dashboard showing the "Parity Ratio" (Authored vs. Reviewed) so I can coach team members on their contributions.

6. Functional Requirements

6.1. Weighted Lottery Engine

The core assignment logic uses a Proportional-Share model:

  • Baseline Tickets ($T$): Every team member starts with a fixed number of tickets.
  • Workload Modifier ($W$): Tickets are divided by $(1 + \text{Active PRs})$.
  • Expertise Multiplier ($E$): Tickets are multiplied (1.5x - 2.0x) if the user has high semantic achievement in the changed files.
  • Selection Probability: $P_i = T_i / \sum T$.

6.2. Expertise & Knowledge Detection (2026 AI-Native)

  • Semantic Achievement Mapping: Uses DeepSeek R1 or Claude 4.5 via LangChain to analyze commit history and categorize expertise (e.g., "Auth Guru," "CSS Architect").
  • Knowledge-Transfer Wildcard: A configurable 10% chance to assign a "non-expert" to a PR to encourage learning.

6.3. Slack Integration

  • Slash Commands: /lottery status to see current load; /lottery ooo [date] to set out-of-office.
  • Interactive Notifications: Buttons for "Acknowledge," "Reassign (Busy)," and "View PR."

6.4. Management Dashboard

  • Parity Ratio Scatter Plot: Visualizes "Givers" vs. "Takers."
  • SLA Pulse: Real-time gauges for time-to-first-review and time-to-merge.
  • Risk Detection: Highlighting "Bottleneck Reviewers" using red/amber/green thresholds.

7. Technical Requirements

7.1. Tech Stack (2026 Standards)

  • Frontend: React 19.2.x (Vite build), Tailwind CSS 4.1.x (Oxide engine), Tremor for dashboard components.
  • Backend: Node.js 24.x (Active LTS), Bolt.js 4.6.x (Slack framework).
  • Database: PostgreSQL 17+ with TimescaleDB extension for PR transition logging.
  • Infrastructure: AWS Lambda (ARM64/Graviton) using the Controller-Worker Pattern.
    • Lambda A (Ack): AwsLambdaReceiver for immediate 3s Slack response.
    • Queue: SQS FIFO for event reliability.
    • Lambda B (Worker): Business logic and AI processing.
  • Secrets: AWS Secrets Manager with Lambda Extension (No env variables).

7.2. Integrations

  • VCS: GitHub/GitLab Webhooks with HMAC-SHA256 signature validation.
  • VCS API: GitHub GraphQL API for deep repository indexing.
  • AI: Model Context Protocol (MCP) to connect repo history to Claude 4.5/DeepSeek R1.

8. Data Model

8.1. User Entity

  • slack_user_id (PK), vcs_user_id, timezone, is_available (bool).
  • expertise_json (JSONB): Semantic tags and scores.

8.2. PullRequest Entity

  • pr_id (PK), external_id, author_id, repo_id.
  • current_status (Enum: OPEN, REVIEW_REQUESTED, MERGED, CLOSED).

8.3. PRStatusHistory (Event Sourcing)

  • event_id (PK), pr_id, status, actor_id, occurred_at.
  • Note: Used for window functions to calculate SLA cycle times.

8.4. ReviewAssignment

  • assignment_id, pr_id, reviewer_id, tickets_at_selection, assigned_at, acknowledged_at.

9. API Specification

9.1. Webhook: VCS Event (GitHub)

  • Endpoint: POST /webhooks/vcs/github
  • Header: X-Hub-Signature-256 (HMAC validation required).
  • Action: Triggers the Lottery logic if action is opened or ready_for_review.

9.2. Dashboard: Team Metrics

  • Endpoint: GET /api/metrics/parity
  • Response: 
    {
  "users": [
    {"name": "Dev A", "authored": 12, "reviewed": 45, "load_index": 0.8},
    {"name": "Dev B", "authored": 25, "reviewed": 5, "load_index": 0.1}
  ]
}

10. UI/UX Requirements

  • Dashboard Layout: Actionable Minimalism.
    • Hero Section: Three KPI cards (Avg SLA, Open PRs, Team Parity Score).
    • Main View: The "Parity Ratio" Scatter Plot.
    • Sidebar: Real-time "Active Reviewers" list with progress bars.
  • Slack Style: Block Kit UI with "Modern-Heavy" card layouts for PR assignments.

11. Non-Functional Requirements

  • Security: crypto.timingSafeEqual for all webhook comparisons. Encryption at rest via KMS CMK.
  • Performance: Lambda cold-starts under 500ms using esbuild bundling and 1024MB memory allocation.
  • Scalability: Support up to 500 developers per Slack workspace using Redis-based caching for availability states.

12. Out of Scope

  • Automated code fixing or linting (use existing CI tools).
  • Direct IDE integrations (VS Code/IntelliJ).
  • Project management features (Jira ticket creation).

13. Risks & Mitigations

| Risk | Mitigation | | :--- | :--- | | Slack 3s Timeout | Implement Controller-Worker pattern with SQS. | | Algorithm Bias | Implement a "Reviewer Audit Log" to transparently show ticket math. | | VCS API Rate Limits | Use GraphQL to minimize calls and implement Redis caching. | | Token Leakage | Use AWS Lambda Secret Extension to avoid env var exposure. |

14. Implementation Tasks

Phase 1: Project Setup & Infrastructure

  • [ ] Initialize Node.js 24.x project with TypeScript.
  • [ ] Set up AWS CDK/Serverless Framework for ARM64 Lambda deployment.
  • [ ] Configure AwsLambdaReceiver with Bolt.js 4.6.x.
  • [ ] Initialize PostgreSQL with TimescaleDB extension.

Phase 2: Core Lottery Engine

  • [ ] Implement crypto.timingSafeEqual HMAC validation for GitHub webhooks.
  • [ ] Build the Ticket-based Lottery Service ($P_i = T_i / \sum T$).
  • [ ] Implement Workload Tracking logic (counting REVIEW_REQUESTED vs MERGED).
  • [ ] Create SQS FIFO queue for reliable event processing.

Phase 3: AI & Expertise Logic

  • [ ] Integrate LangChain with DeepSeek R1 for commit message analysis.
  • [ ] Build the "Semantic Achievement" mapping service.
  • [ ] Implement the Knowledge-Transfer Wildcard (randomizer override).

Phase 4: Frontend Dashboard

  • [ ] Initialize React 19.2.x with Vite and Tailwind CSS 4.1.x.
  • [ ] Build "Parity Ratio" scatter plot using Tremor components.
  • [ ] Implement SLA Countdown timers with pulsing orange/red states.
  • [ ] Configure Slack OAuth 2.0 flow for admin setup.

Phase 5: Polish & Security

  • [ ] Implement AWS Secrets Manager Lambda Extension.
  • [ ] Add IP allow-listing for GitHub/GitLab webhook CIDRs.
  • [ ] Perform a load test on the state transition window functions.
  • [ ] Finalize GDPR-compliant data deletion (Right to be Forgotten) for developers.