Learn Loop

1. Problem Statement

Engineering teams make the same mistakes repeatedly, and the lessons that would prevent them are buried in pull request threads, review comments, and the heads of senior engineers. Every PR carries rich signal — what changed, what a reviewer flagged, which convention was followed or broken, what architecture shifted — but that signal evaporates the moment the PR merges.

The usual fixes don't scale. Manual training programs are expensive to author and go stale fast. One-off review comments teach one person, once. Onboarding leans on whoever has time to explain the codebase.

Learn Loop lets anyone paste a GitHub PR link and get back codebase-specific learning modules, short quizzes, and personalized + team-wide learning plans — generated from the actual change, with no manual curriculum authoring.

2. Goals

• Turn real PR activity into practical, evidence-backed learning instead of generic lessons
• Surface recurring mistakes, strong patterns, and architectural change or drift from a single PR
• Generate personalized (per-engineer) and team-wide learning plans grounded in the company's own codebase
• Educate the team about architectural changes via a short module and a confirmation quiz
• Improve engineering quality and onboarding without managers manually building training programs
• Expose learning context and recommendations to IDEs and AI agents through an MCP server

3. Non-Goals

• Not an automatic code-fixing engine — it never modifies code
• No automatic PR discovery, webhooks, or repo-wide scanning — analysis is triggered by a pasted link (v1)
• Not a performance-scoring tool — recommendations are coaching aids, not individual ratings
• No non-GitHub providers such as GitLab or Bitbucket (v1)
• No enterprise SSO, advanced RBAC, or multi-tenant administration (v1)
• No real-time streaming ingestion or production background-job queue (v1)

4. Users

Primary — Engineering Manager. Owns team quality, onboarding, and coaching. Wants a fast answer to "what mistakes keep recurring and what should the team learn next?" from a real PR.

Secondary — Individual Engineer / New Joiner. Wants short, relevant modules tied to their actual work and this codebase's conventions. New joiners want a safe-contribution path and the recurring review comments to know.

Tertiary — Staff Engineer / Tech Lead. Tracks architectural blind spots, drift, and decisions that are repeatedly misunderstood; decides which strong patterns to teach broadly.

5. Input Model

5.1 PR Input (required)

The core input is a single GitHub pull request, submitted manually.

5.2 GitHub Access (required configuration)

The backend reads PR data through the GitHub API using a configured classic personal access token (PAT).

5.3 Workspace Configuration

The POC runs a single demo workspace with no user login or application auth.

5.4 Optional Supplemental Inputs

• Architecture notes or ADRs (markdown / text) to ground architecture-aware analysis
• Incident notes related to the repository
• AI coding-chat excerpts for richer context
• An offline, GitHub-shaped fixture is supported only as a fallback demo mode when credentials are unavailable

6. Core Features

6.1 PR Analysis Engine

Manual, on-demand PR analysis that runs roughly synchronously after submission.

• Parses owner, repo, and PR number from the URL
• Validates GitHub access only when a PAT is configured or the repo is private
• Fetches PR metadata, commits, changed files, diffs, review and discussion comments, and CI/check status
• Fetches bounded base-branch context: the diff plus snippets around each changed hunk, the full base-branch file only when small, and related files (tests, schemas, route registrations, shared utilities, architecture docs) by path heuristics — with a cap on total context per PR
• Normalizes everything into evidence records and separates PR facts from AI interpretation
• Flags fetch failures (missing/expired/unauthorized PAT, missing permission, unsupported repo) instead of failing silently

6.2 Pattern Detection

Groups PR evidence and repository context into engineering patterns.

• Categories: recurring mistakes, strong patterns, organizational blind spots, onboarding friction, architecture gaps, incident-linked risk, architecture change/drift, missing team communication
• Each pattern carries severity, confidence, trend, and an architecture_impact value (none, localized, cross_cutting, potential_drift, intentional_architecture_change), plus evidence references

6.3 Learning Module Generation

Turns patterns into short, practical modules grounded in this codebase.

• 5–12 minute modules with a problem summary, why-it-matters, good and risky examples, a checklist, and a short exercise
• Persisted in SQLite; re-running analysis updates an existing module tied to the same pattern/evidence rather than duplicating it
• Substantial modules include 2–4 quiz questions (multiple choice, true/false, short scenario) with grounded answer explanations

6.4 Learning Plans

Generates a personalized plan per engineer and one team-wide plan.

• Individual plan: recommended modules, reason for each, supporting evidence, priority, status
• Team plan: top recurring mistakes, strong patterns to reinforce, blind spots, architectural changes/drift, recommended next steps
• Completion is tracked per engineer; team-level completion is derived from individual progress (e.g. "4 of 7 completed")
• Quiz completion is required to mark a substantial module complete; lightweight reference modules are exempt

6.5 Architectural-Change Education

When a PR introduces a meaningful architectural change, Learn Loop auto-generates a team education module explaining what changed, why it matters, and how future work should adapt, plus a short confirmation quiz. The team plan gains a "What changed" section for that change.

6.6 Vector Search

Semantic retrieval over PR diffs, code snippets, comments, notes, and generated modules.

• Uses SQLite for metadata plus a local in-process vector index — no external hosted vector service
• Every AI-generated answer returns source references back to evidence, repository paths, and PR numbers

6.7 Dashboard

An operational, evidence-first engineering surface (not a marketing site). Views: Overview, Analyze PR, Patterns, Learning Plans, Modules, Evidence Explorer, Reports, and MCP / IDE Integration. The Analyze PR view takes the PR URL and shows connection status, analysis progress, and clear permission/PR errors.

6.8 MCP Server

Exposes Learn Loop context to IDEs and AI agents. Tools: get_learning_recommendations, search_engineering_knowledge, get_module, get_quiz_for_module, get_patterns_for_file, get_patterns_for_pr, get_team_blind_spots. Recent MCP queries appear as a lightweight operational/debug log — not individual performance monitoring.

6.9 Reporting Engine

Generates a concise learning report (markdown in the dashboard, optional JSON via API) covering the PR analyzed, architecture impact, top recurring mistakes, strong patterns, blind spots, recommended team plan, onboarding updates, team quizzes, and an evidence appendix.

7. User Flow

1. An engineering manager pastes a GitHub PR link into Learn Loop
2. The backend fetches PR metadata, commits, changed files, diffs, review comments, and bounded base-branch context
3. The analysis engine compares the change against the current codebase and any optional architecture notes
4. The system detects mistakes, strong patterns, architectural change, possible drift, and team-education needs
5. The dashboard shows PR-specific insights with supporting evidence from commits, diffs, comments, and files
6. The manager opens the generated team learning plan for the PR
7. An individual engineer opens a personalized plan with short modules and quiz questions
8. A module explains one topic using real snippets from the PR and codebase
9. If the PR introduces an architectural change, a team education module and quiz are generated
10. The MCP server answers, from an IDE: "What should I learn before changing this part of the code?"
11. The reporting view summarizes PR learnings, team blind spots, strong patterns, and next actions

8. Re-Analysis & History

Learn Loop is manually triggered — there is no scheduled analysis in v1.

• Re-running analysis on the same PR updates existing findings instead of creating duplicates
• Generated modules, completion status, and quiz attempts remain stable across re-analysis
• Prior PR analyses are stored so a user can compare a PR's findings over time
• New modules are created only when materially new evidence or a new pattern is detected

9. Handling Edge Cases

10. Success Metrics

11. Out of Scope (v1)

• Automatic PR discovery, webhook processing, or background scanning of every PR
• Full OAuth installation flow or GitHub App tokens (the POC uses a configured classic PAT)
• Enterprise SSO and advanced RBAC
• Support for non-GitHub repository providers
• Automatic code modification
• High-stakes performance scoring of individual engineers
• LMS integration
• Multi-tenant billing or organization administration
• Real-time streaming ingestion and production job-queue infrastructure

12. Open Questions

1. What exact base-branch context limits should apply — lines fetched around each hunk, max file size for full-file fetch, max related files, and total context cap per PR?
2. Should the local vector index be rebuilt from SQLite chunks on startup, or persisted to disk between runs?
3. Which additional API endpoints are needed — resolving unmapped GitHub users, submitting quiz answers, and viewing quiz attempts?
4. What are the AI prompt privacy/redaction rules — is source code sent to an external provider, are secrets redacted, and how much code can a single prompt include?