From 12dc3f54cab5bdfe1b834d05d70415c8259f73f4 Mon Sep 17 00:00:00 2001
From: Victor Romero <romerosanchezv@gmail.com>
Date: Wed, 15 Apr 2026 16:10:37 -0700
Subject: [PATCH] [skill] Add skill to analyze CI failures (#51188)

---
 .github/skills/analyze-ci-failures/SKILL.md   | 448 ++++++++++++++++++
 .../references/azure-devops-api.md            | 271 +++++++++++
 .../references/report-template.md             | 176 +++++++
 .../references/vcpkg-failure-patterns.md      | 361 ++++++++++++++
 4 files changed, 1256 insertions(+)
 create mode 100644 .github/skills/analyze-ci-failures/SKILL.md
 create mode 100644 .github/skills/analyze-ci-failures/references/azure-devops-api.md
 create mode 100644 .github/skills/analyze-ci-failures/references/report-template.md
 create mode 100644 .github/skills/analyze-ci-failures/references/vcpkg-failure-patterns.md

diff --git a/.github/skills/analyze-ci-failures/SKILL.md b/.github/skills/analyze-ci-failures/SKILL.md
new file mode 100644
index 0000000000..d5b4bf5461
--- /dev/null
+++ b/.github/skills/analyze-ci-failures/SKILL.md
@@ -0,0 +1,448 @@
+---
+name: analyze-ci-failures
+description: 'Analyze vcpkg Azure DevOps CI build failures. Downloads failure logs, identifies regression root causes, and generates a structured report categorizing build errors by package, triplet, and failure type.'
+argument-hint: 'Azure DevOps build URL (e.g., "https://dev.azure.com/vcpkg/public/_build/results?buildId=129315")'
+---
+
+# vcpkg CI Failures Analyzer
+
+## When to Use
+
+- Investigating why a CI build failed on Azure DevOps
+- Identifying which packages regressed in a PR or scheduled build
+- Triaging root causes before assigning bugs or reverting changes
+- Getting a quick summary of failures across all triplets in one build
+
+## Overview
+
+This skill fetches build metadata and failure logs directly from the Azure DevOps REST API, cross-references them with `scripts/ci.baseline.txt`, and produces a structured regression report:
+
+- **Build Summary**: Overall build result, trigger type (PR vs scheduled), and which jobs (triplets) failed
+- **Artifact Download**: Fetches every "failure logs for *" artifact published by the CI pipeline
+- **Log Analysis**: Parses per-package failure logs to extract compiler errors, missing dependencies, linker errors, and other root causes
+- **Regression Classification**: Distinguishes new regressions from known failures already listed in `ci.baseline.txt`
+- **Structured Report**: Organized by triplet, then package, with concise root-cause summaries
+
+## Detailed Workflow
+
+### Step 1: Parse the Build URL
+
+Given a URL such as:
+```
+https://dev.azure.com/vcpkg/public/_build/results?buildId=129315&view=results
+```
+
+Extract:
+- **organization**: `vcpkg`
+- **project**: `public`
+- **buildId**: `129315`
+
+The REST API base is: `https://dev.azure.com/{organization}/{project}/_apis`
+
+### Step 2: Fetch Build Metadata
+
+**Get overall build info:**
+```
+GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}?api-version=7.0
+```
+Key fields in the response:
+- `status` — `completed`, `inProgress`, etc.
+- `result` — `succeeded`, `failed`, `partiallySucceeded`, `canceled`
+- `reason` — `pullRequest`, `schedule`, `manual`, `individualCI`
+- `requestedFor.displayName` — who triggered the build
+- `sourceBranch` — branch or PR reference
+- `sourceVersion` — commit SHA
+- `_links.web.href` — canonical link back to the build page
+- `finishTime` — when it completed
+
+**Get the build timeline (all jobs and tasks):**
+```
+GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/timeline?api-version=7.0
+```
+The response contains a `records` array. Each record has:
+- `type` — `"Job"`, `"Task"`, `"Stage"`, `"Phase"`
+- `name` — display name (e.g., `"x64_windows"`, `"*** Test Modified Ports"`)
+- `result` — `"succeeded"`, `"failed"`, `"skipped"`, `"canceled"`
+- `state` — `"completed"`, `"inProgress"`, `"pending"`
+- `log.url` — URL to the raw task log (for individual task logs)
+- `parentId` — parent record ID (tasks belong to jobs)
+
+Filter for `type == "Job"` and `result == "failed"` to get a list of failing triplets.
+
+### Step 2b: Scan "Test Modified Ports" Step Logs for REGRESSION Lines
+
+**This step is required** — some failure types (notably `FILE_CONFLICTS`) are only reported in the step console output and do **not** produce entries in the failure log artifacts. Skipping this step will miss those regressions entirely.
+
+For each failed job in the timeline, find the task named `"*** Test Modified Ports"` (type `"Task"`, child of that job) and fetch its log:
+
+```
+GET {record.log.url}?api-version=7.0
+```
+
+The log is plain text. Scan it for lines matching the pattern:
+```
+REGRESSION: {port}:{triplet} failed with {FAILURE_TYPE}.
+```
+
+**Example output from a real build:**
+```
+REGRESSION: kf6i18n:x64-windows failed with FILE_CONFLICTS. If expected, add kf6i18n:x64-windows=fail to D:\a\_work\1\s\scripts\azure-pipelines/../ci.baseline.txt.
+REGRESSION: kf6itemmodels:x64-windows failed with FILE_CONFLICTS. If expected, add kf6itemmodels:x64-windows=fail to D:\a\_work\1\s\scripts\azure-pipelines/../ci.baseline.txt.
+```
+
+**Known `FAILURE_TYPE` values found in this log:**
+| Failure type | Also has artifact log? | Notes |
+|---|---|---|
+| `BUILD_FAILED` | ✅ Yes | Full build logs in "failure logs for {triplet}" artifact |
+| `FILE_CONFLICTS` | ❌ No | Port installs files that conflict with another port; only in step log |
+| `MISSING_FROM_BASELINE` | ❌ No | Port passed but is listed as `fail` in baseline; no artifact |
+| `CASCADE_BUILD_FAILED` | Sometimes | Dependency failure; root port has artifact, downstream may not |
+
+**PowerShell to extract all REGRESSION lines from a job's "Test Modified Ports" step:**
+```powershell
+# Get the timeline to find log URLs
+$timeline = Invoke-RestMethod "https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/timeline?api-version=7.0"
+
+# For each failed job, find its "Test Modified Ports" task
+$failedJobs = $timeline.records | Where-Object { $_.type -eq 'Job' -and $_.result -eq 'failed' }
+foreach ($job in $failedJobs) {
+    $testTask = $timeline.records | Where-Object {
+        $_.type -eq 'Task' -and
+        $_.parentId -eq $job.id -and
+        $_.name -like '*Test Modified Ports*'
+    }
+    if ($testTask -and $testTask.log.url) {
+        $logText = Invoke-RestMethod "$($testTask.log.url)?api-version=7.0"
+        $regressions = $logText -split "`n" | Where-Object { $_ -match '^REGRESSION:' }
+        $regressions | ForEach-Object { Write-Host "$($job.name): $_" }
+    }
+}
+```
+
+Collect all `REGRESSION:` lines from all jobs into a master list before proceeding to artifact download. This list is your **ground truth** for what failed — the failure log artifacts are supplementary evidence for diagnosing *why*.
+
+### Step 3: List Published Artifacts
+
+```
+GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/artifacts?api-version=7.0
+```
+Each artifact in `value[]` has:
+- `name` — e.g., `"failure logs for x64-windows"`, `"file lists for x64-windows"`, `"format.diff"`
+- `resource.type` — **`"PipelineArtifact"`** (not `"Container"` — the Container listing API does **not** work for these)
+- `resource.downloadUrl` — direct ZIP download URL hosted on `artprodwus21.artifacts.visualstudio.com`; no auth required for public builds
+
+**Focus on artifacts whose name starts with `"failure logs for"`** — these contain one subdirectory per failing port.
+
+**Prioritize by compressed artifact size** (from `resource.properties.artifactsize`):
+
+| Compressed size | Interpretation | Analysis depth |
+|---|---|---|
+| < 10 MB | Few failures (< ~15 ports) | Analyze all ports fully |
+| 10–50 MB | Moderate failures | Sample all ports, deep-dive key ones |
+| 50–120 MB | Many failures | Focus on unique error categories only |
+| > 120 MB | Mass failures — likely a systemic issue | Find the common root, skip per-port deep-dive |
+
+Start with the smallest artifacts (fastest feedback). Android triplets typically produce 115–123 MB artifacts and share root causes with Linux.
+
+### Step 4: Download Failure Log Artifacts
+
+**`web_fetch` cannot download binary ZIP files.** Use PowerShell with `Invoke-WebRequest` instead:
+
+```powershell
+$tmpDir = "$env:TEMP\vcpkg-ci-{buildId}"
+New-Item -ItemType Directory -Path $tmpDir -Force | Out-Null
+
+# Download the artifact ZIP using the downloadUrl from the artifacts API response
+$downloadUrl = "{resource.downloadUrl from artifacts API}"
+Invoke-WebRequest -Uri $downloadUrl -OutFile "$tmpDir\{triplet}.zip" -UseBasicParsing
+
+# Extract
+Expand-Archive "$tmpDir\{triplet}.zip" -DestinationPath "$tmpDir\{triplet}" -Force
+
+# List failed ports — each is a subdirectory inside "failure logs for {triplet}/"
+$root = "$tmpDir\{triplet}\failure logs for {triplet}"
+Get-ChildItem $root -Directory | Select-Object -ExpandProperty Name
+```
+
+**Confirmed ZIP structure (from build #129315):**
+```
+failure logs for x64-windows.zip
+└── failure logs for x64-windows/        ← top-level folder = artifact name
+    ├── boost-filesystem/                 ← one subdirectory per failing port
+    │   ├── stdout-x64-windows.log        ← PRIMARY: full vcpkg output, final error
+    │   ├── config-x64-windows-out.log    ← CMake configure stdout
+    │   ├── config-x64-windows-err.log    ← CMake configure stderr
+    │   ├── config-x64-windows-dbg-CMakeConfigureLog.yaml.log
+    │   ├── config-x64-windows-dbg-CMakeCache.txt.log
+    │   ├── config-x64-windows-dbg-ninja.log
+    │   ├── build-x64-windows-dbg-out.log ← compiler stdout (can be very large)
+    │   ├── build-x64-windows-dbg-err.log ← compiler stderr
+    │   ├── install-x64-windows-dbg-out.log
+    │   ├── install-x64-windows-dbg-err.log
+    │   ├── patch-x64-windows-0-err.log   ← patch application errors (numbered)
+    │   ├── patch-x64-windows-0-out.log
+    │   ├── extract-out.log / extract-err.log
+    │   └── generate-x64-windows-dbg-err.log  ← GN builds (v8, crashpad) only
+    ├── curl/
+    ├── openssl/
+    └── ...
+```
+
+**For Meson ports (e.g., mesa):**
+```
+mesa/
+├── cmake-get-vars_C_CXX-x64-linux-dbg.cmake.log
+├── config-x64-linux-dbg-meson-log.txt.log
+├── meson-x64-linux-dbg.log
+├── pip-install-packages-x64-linux-err.log
+├── venv-setup-x64-linux-err.log
+└── stdout-x64-linux.log
+```
+
+**For autotools ports (e.g., libxt):**
+```
+libxt/
+├── autoconf-x64-windows-err.log
+├── build-x64-windows-dbg-err.log   ← compiler warnings/errors
+├── build-x64-windows-dbg-out.log
+└── stdout-x64-windows.log
+```
+
+### Step 5: Analyze Log Content — Two-Phase Approach
+
+**Phase 1: Classify all failures quickly (one pass)**
+
+For each failing port, read only `stdout-{triplet}.log` last 20 lines:
+```powershell
+$ports = Get-ChildItem $root -Directory | Sort-Object Name
+foreach ($port in $ports) {
+    $stdout = Get-ChildItem $port.FullName -Filter "stdout*" | Select-Object -First 1
+    if ($stdout) {
+        $lines = Get-Content $stdout.FullName -Encoding UTF8
+        # Extract final error line
+        $keyError = ($lines | Where-Object { $_ -match "error:|CMake Error|FAILED" } | Select-Object -Last 1) -replace '^\s+',''
+        Write-Host "$($port.Name): $keyError"
+    }
+}
+```
+
+This gives a one-line classification per port. Identify common patterns across ports (many ports → same error = systemic issue).
+
+**Phase 2: Deep-dive the specific stage log for each port**
+
+After classifying, choose the right log file based on failure type:
+
+| Failure type indicated by stdout | Log to read next |
+|---|---|
+| `Command failed: cmake --build` (build/install) | `install-{triplet}-dbg-out.log` — grep for `error:` |
+| `Command failed: ninja -v` (configure) | `config-{triplet}-out.log` — last 20 lines |
+| `Command failed: ninja -v` (configure) | `config-{triplet}-dbg-CMakeConfigureLog.yaml.log` |
+| `Command failed: gn gen` or `ninja` (GN) | `generate-{triplet}-dbg-err.log` |
+| `Command failed: meson setup` | `config-{triplet}-dbg-meson-log.txt.log` |
+| `Command failed: make` (autotools) | `build-{triplet}-dbg-err.log` |
+| `Command failed: autoreconf` | `autoconf-{triplet}-err.log` |
+| Patch error | `patch-{triplet}-N-err.log` |
+
+**Important**: `install-*-out.log` and `build-*-out.log` can be very large (100KB–2MB+). Always grep for errors rather than reading the full file:
+```powershell
+$lines = Get-Content "install-x64-windows-dbg-out.log" -Encoding UTF8
+$lines | Where-Object { $_ -match "error:" -and $_ -notmatch "^--" } | Select-Object -First 10
+```
+
+**Note on empty error logs**: Many `*-err.log` files are empty (0 bytes) — the actual error output goes to stdout for CMake/ninja builds. Always check `*-out.log` if `*-err.log` is empty.
+
+**Note on test results API**: `GET /_apis/test/runs?buildId=...` requires authentication even for public projects. Skip this API; the failure logs contain sufficient information.
+
+### Step 5 (continued): Pattern Catalogue
+
+For each failure log, identify the root cause by scanning for known error patterns.
+See `references/vcpkg-failure-patterns.md` for a comprehensive pattern catalogue and `references/report-template.md` for a table of the most common high-frequency patterns to check first.
+
+**Quick triage checklist (scan each log in order):**
+
+1. **Dependency failure** — the port didn't actually fail itself; a dependency did:
+   ```
+   error: package dependency-name:triplet is not installed
+   -- Building dependency-name[core]:triplet failed
+   ```
+   Root cause: the dependency listed in the log, not the current package.
+
+2. **CMake configure error** — dependency not found or incompatible version:
+   ```
+   CMake Error: Could not find a package configuration file provided by "XYZ"
+   CMake Error at CMakeLists.txt:NN: find_package(XYZ REQUIRED)
+   ```
+
+3. **Compiler error** — actual source code compile failure:
+   ```
+   error C2065:  (MSVC)
+   error: use of undeclared identifier  (Clang/GCC)
+   fatal error: 'header.h' file not found
+   ```
+
+4. **Linker error**:
+   ```
+   error LNK2019: unresolved external symbol  (MSVC)
+   undefined reference to  (GCC/Clang)
+   ```
+
+5. **SHA512 mismatch** — downloaded archive hash changed:
+   ```
+   error: File does not have the expected hash
+   Expected: ...
+   Actual: ...
+   ```
+
+6. **Download failure** — network or URL issue:
+   ```
+   error: Failed to download
+   error: curl: (22)
+   ```
+
+7. **Post-build check failure** — installed files don't pass vcpkg checks:
+   ```
+   error: The following files are not installed
+   error: Usage of deprecated function
+   ```
+
+8. **Test failure** — port tests failed (check xUnit results):
+   ```
+   Test failed: ...
+   FAILED - ...
+   ```
+
+### Step 6: Cross-Reference with CI Baseline
+
+Read `scripts/ci.baseline.txt` from the local repository to classify each failure:
+
+```
+# Format: port-name=fail|skip|pass [# comment]
+boost-filesystem=fail  # needs icu update
+```
+
+**Classification logic:**
+- If the failing port is listed as `fail` or `skip` in the baseline → **Known / Expected** (not a regression)
+- If the failing port is NOT in the baseline, or listed as `pass` → **Regression** (new failure)
+- If a port that is expected to `fail` is now succeeding → **Unexpected Pass** (may need baseline update)
+
+### Step 7: Generate the Regression Report
+
+Produce a structured markdown report with the following sections:
+
+---
+
+**Template:**
+
+```markdown
+# vcpkg CI Failure Report
+
+**Build:** [#{buildId}]({build_web_url})
+**Triggered by:** {reason} — {requestedFor} on `{sourceBranch}` @ `{sourceVersion[:8]}`
+**Result:** {result} | **Finished:** {finishTime}
+
+---
+
+## Summary
+
+| Triplet | Status | New Regressions | Known Failures |
+|---------|--------|-----------------|----------------|
+| x64-windows | ❌ Failed | 3 | 1 |
+| x64-linux | ✅ Passed | 0 | 0 |
+| ... | | | |
+
+---
+
+## 🔴 New Regressions (Action Required)
+
+### x64-windows
+
+#### `portname` — Root Cause Category
+- **Error:** `<first relevant error line>`
+- **File:** `path/to/file.cpp:42`
+- **Analysis:** Short description of why this likely broke.
+- **Suggested fix:** ...
+
+...
+
+---
+
+## 🟡 Known / Expected Failures (Baseline)
+
+| Port | Triplet | Baseline Entry |
+|------|---------|----------------|
+| portname | x64-windows | `fail # reason` |
+...
+
+---
+
+## ℹ️ Notes
+
+- Dependency chain failures: {n} ports failed due to a common upstream failure in `dep-name`
+- Download failures: {n} (may be transient network issues)
+```
+
+---
+
+### Step 8: Dependency Chain Deduplication
+
+Before listing regressions, collapse dependency chains:
+
+- If `libA` fails and `libB`, `libC`, `libD` all fail because `libA` is a dependency, list only `libA` as the regression with a note: "caused {n} downstream failures: libB, libC, libD"
+- Check dependency info from the log: lines containing `-- Building {dep}:{triplet} failed` indicate the root of the chain
+
+### Step 9: Transient vs. Persistent Failures
+
+Flag failures that are likely transient (not actual regressions):
+- Download failures (network errors, SHA mismatches from upstream file changes)
+- Timeout failures (`error: build timed out`)
+- Flaky test failures (same port fails in one triplet but passes in others for unrelated reasons)
+
+Suggest re-running the build for transient failures rather than filing issues.
+
+## Quick Start
+
+**Minimal 3-step process:**
+1. Provide the build URL: `analyze-ci-failure https://dev.azure.com/vcpkg/public/_build/results?buildId=129315`
+2. Skill fetches build metadata, downloads all failure log artifacts, reads baseline
+3. Receive structured regression report
+
+## Output Format
+
+When generating the report, follow the exact format defined in `references/report-template.md`.
+That file contains:
+- The full markdown template with placeholder variables
+- A real worked example (build #129315, 24 ports) to calibrate tone and depth
+- Formatting rules (section order, priority levels, error quoting style)
+- Artifact size heuristics to guide how deeply to analyze each triplet
+- A table of common high-frequency patterns to scan for first
+
+When asked to produce the report, default to the full structured markdown report. You can also produce:
+- **Summary only**: Just the summary table and count of regressions
+- **Regressions only**: Skip known failures, show only new regressions
+- **Single triplet**: Focus on one platform (e.g., "x64-windows only")
+- **Port-focused**: Group by port name instead of triplet
+
+### Saving the Report
+
+After generating the report, **always save it as a markdown file** in the repository root:
+
+```
+ci-report-{buildId}.md
+```
+
+Use the `create` tool to write the full report content to this path. This ensures the report is persisted and can be easily reviewed or shared.
+
+## Important Notes
+
+- The vcpkg Azure DevOps project (`vcpkg/public`) is **publicly accessible** — no authentication token is required for artifact downloads or build/artifact API calls
+- **`web_fetch` cannot download binary ZIP files** — use PowerShell `Invoke-WebRequest` for artifact downloads
+- **`resource.type` is `"PipelineArtifact"`**, not `"Container"` — the Container listing API (`/_apis/resources/Containers/{id}`) does not work; use `downloadUrl` directly
+- **The test results API (`/_apis/test/runs`) requires authentication** even for public projects — skip it; failure logs contain sufficient information
+- **Many `*-err.log` files are 0 bytes** — CMake, ninja, and make typically write errors to stdout; always check `*-out.log` when `*-err.log` is empty
+- If a build is still `inProgress`, only partial results will be available; note this in the report
+- Failure log artifacts are only published when there are actual failures; a missing artifact for a triplet means that triplet passed
+- **`FILE_CONFLICTS` failures are invisible in artifacts** — they only appear as `REGRESSION:` lines in the "Test Modified Ports" step log; always scan step logs (Step 2b) before concluding the artifact list is complete
+- The `"format.diff"` artifact (if present) indicates formatting or version file issues, not port build failures — report these separately
+- `"z azcopy logs"` artifacts are infrastructure logs; skip these unless diagnosing asset cache issues
+- **Android artifact sizes (~115–123 MB) typically share root causes with Linux** — if Linux analysis reveals a systemic issue (e.g., compiler header change, Python version), assume Android is affected too without full re-analysis
+- **Clean up the temp directory** (`$env:TEMP\vcpkg-ci-{buildId}`) after analysis to avoid disk clutter
diff --git a/.github/skills/analyze-ci-failures/references/azure-devops-api.md b/.github/skills/analyze-ci-failures/references/azure-devops-api.md
new file mode 100644
index 0000000000..e97daa2532
--- /dev/null
+++ b/.github/skills/analyze-ci-failures/references/azure-devops-api.md
@@ -0,0 +1,271 @@
+# Azure DevOps REST API Reference for vcpkg CI
+
+## Base URL
+
+```
+https://dev.azure.com/vcpkg/public/_apis
+```
+
+All endpoints below are appended to this base. The `vcpkg/public` project is **publicly readable** — no Authorization header is required.
+
+## API Version
+
+Always append `?api-version=7.0` (or include it in query strings).
+
+---
+
+## Build Endpoints
+
+### Get Build Details
+
+```
+GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}?api-version=7.0
+```
+
+**Response fields:**
+```json
+{
+  "id": 129315,
+  "buildNumber": "20240315.1",
+  "status": "completed",
+  "result": "failed",
+  "reason": "pullRequest",
+  "sourceBranch": "refs/pull/12345/merge",
+  "sourceVersion": "abc123def456...",
+  "requestedFor": { "displayName": "Jane Developer" },
+  "startTime": "2024-03-15T10:00:00Z",
+  "finishTime": "2024-03-15T14:30:00Z",
+  "definition": { "name": "vcpkg.CI" },
+  "_links": {
+    "web": { "href": "https://dev.azure.com/vcpkg/public/_build/results?buildId=129315" }
+  }
+}
+```
+
+**`result` values:** `succeeded` | `failed` | `partiallySucceeded` | `canceled`
+**`reason` values:** `pullRequest` | `schedule` | `manual` | `individualCI` | `batchedCI`
+
+---
+
+### Get Build Timeline
+
+```
+GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/timeline?api-version=7.0
+```
+
+Returns all stages, jobs, tasks and their results.
+
+**Response structure:**
+```json
+{
+  "records": [
+    {
+      "id": "guid",
+      "parentId": "parent-guid-or-null",
+      "type": "Job",
+      "name": "x64_windows",
+      "displayName": "x64_windows",
+      "state": "completed",
+      "result": "failed",
+      "startTime": "2024-03-15T10:05:00Z",
+      "finishTime": "2024-03-15T12:15:00Z",
+      "log": {
+        "id": 42,
+        "type": "Container",
+        "url": "https://dev.azure.com/vcpkg/public/_apis/build/builds/129315/logs/42"
+      },
+      "errorCount": 1,
+      "warningCount": 0
+    }
+  ]
+}
+```
+
+**`type` values:** `Stage` | `Phase` | `Job` | `Task`
+**`result` values:** `succeeded` | `failed` | `skipped` | `canceled` | `succeededWithIssues`
+
+**To find failed jobs:**
+```python
+failed_jobs = [r for r in records if r['type'] == 'Job' and r['result'] == 'failed']
+```
+
+**Job names map to triplets** (underscores in job names, hyphens in triplets):
+- `x64_windows` → `x64-windows`
+- `x64_windows_static` → `x64-windows-static`
+- `arm64_osx` → `arm64-osx`
+- `x64_linux` → `x64-linux`
+- `arm_neon_android` → `arm-neon-android`
+
+---
+
+### Get a Specific Log
+
+```
+GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/logs/{logId}?api-version=7.0
+```
+
+Returns plain text of the log. Use this to read the "*** Test Modified Ports" task log, which contains `REGRESSION:` lines for **all** failures including those without artifact logs (e.g., `FILE_CONFLICTS`).
+
+**Finding the right log ID:** Look up the task record in the timeline response — the `log.id` field on the `"*** Test Modified Ports"` task record gives the log ID. Alternatively, use `log.url` directly.
+
+**Scanning for REGRESSION lines:**
+```powershell
+$logText = Invoke-RestMethod "https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/logs/{logId}?api-version=7.0"
+$logText -split "`n" | Where-Object { $_ -match '^REGRESSION:' }
+```
+
+**Example output:**
+```
+REGRESSION: kf6i18n:x64-windows failed with FILE_CONFLICTS. If expected, add kf6i18n:x64-windows=fail to .../ci.baseline.txt.
+REGRESSION: kf6itemmodels:x64-windows failed with FILE_CONFLICTS. If expected, add kf6itemmodels:x64-windows=fail to .../ci.baseline.txt.
+REGRESSION: v8:x64-windows failed with BUILD_FAILED. If expected, add v8:x64-windows=fail to .../ci.baseline.txt.
+```
+
+> ⚠️ **`FILE_CONFLICTS` failures do NOT produce failure log artifacts.** The only record of them is in this step log. Always scan all "Test Modified Ports" task logs before relying on artifacts alone.
+
+### List All Logs
+
+```
+GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/logs?api-version=7.0
+```
+
+---
+
+## Artifact Endpoints
+
+### List All Artifacts
+
+```
+GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/artifacts?api-version=7.0
+```
+
+**Response:**
+```json
+{
+  "count": 39,
+  "value": [
+    {
+      "id": 1107682,
+      "name": "failure logs for x64-windows",
+      "source": "7922e5c4-...",
+      "resource": {
+        "type": "PipelineArtifact",
+        "data": "DCCD3D9C96CE79A758B255BD87397A07371B8394EE52F6BFF35346C144D63F3501",
+        "properties": {
+          "RootId": "196FDE3B...",
+          "artifactsize": "8586850",
+          "HashType": "DEDUP1024K",
+          "DomainId": "0"
+        },
+        "url": "https://dev.azure.com/vcpkg/.../_apis/build/builds/129315/artifacts?artifactName=failure%20logs%20for%20x64-windows&api-version=7.0",
+        "downloadUrl": "https://artprodwus21.artifacts.visualstudio.com/.../_apis/artifact/{base64hash}/content?format=zip"
+      }
+    }
+  ]
+}
+```
+
+> ⚠️ **`resource.type` is `"PipelineArtifact"`**, not `"Container"`. The Container listing API
+> (`GET /_apis/resources/Containers/{id}`) does **not** work for these artifacts.
+> Use `resource.downloadUrl` directly to download the ZIP.
+
+**Artifact naming conventions:**
+| Artifact Name Pattern | Contents | Priority |
+|---|---|---|
+| `failure logs for {triplet}` | Per-port build failure log directories | **High** — primary source |
+| `file lists for {triplet}` | Installed file manifests | Low — skip unless needed |
+| `format.diff` | Formatting/manifest diff | **Medium** — indicates PR format issues |
+| `z azcopy logs for {triplet}` | Binary cache transfer logs | Low — infrastructure only |
+
+---
+
+### Download Artifact ZIP (Confirmed Working)
+
+Use the `downloadUrl` from the artifact listing. It resolves to an `artprodwus21.artifacts.visualstudio.com` URL. **`web_fetch` cannot download binary ZIPs** — use PowerShell:
+
+```powershell
+# $downloadUrl = value from resource.downloadUrl in the artifacts list response
+Invoke-WebRequest -Uri $downloadUrl -OutFile "$tmpDir\{triplet}.zip" -UseBasicParsing
+Expand-Archive "$tmpDir\{triplet}.zip" -DestinationPath "$tmpDir\{triplet}" -Force
+
+# List failing ports — each is a subdirectory inside the artifact folder
+$root = "$tmpDir\{triplet}\failure logs for {triplet}"
+Get-ChildItem $root -Directory | Select-Object -ExpandProperty Name
+```
+
+The ZIP always contains one top-level folder named exactly `"failure logs for {triplet}"`.
+Inside that folder, each **failing port has its own subdirectory** named after the port.
+
+> ❌ The Container API (`GET /_apis/resources/Containers/{containerId}`) does **not** work
+> for `PipelineArtifact` type. Do not attempt to use it.
+
+---
+
+## Test Results Endpoints
+
+### List Test Runs for a Build
+
+```
+GET https://dev.azure.com/{org}/{project}/_apis/test/runs?buildId={buildId}&api-version=7.0
+```
+
+**Response:**
+```json
+{
+  "value": [
+    {
+      "id": 9876,
+      "name": "x64-windows",
+      "state": "Completed",
+      "totalTests": 500,
+      "passedTests": 497,
+      "failedTests": 3
+    }
+  ]
+}
+```
+
+### Get Test Run Results
+
+```
+GET https://dev.azure.com/{org}/{project}/_apis/test/runs/{runId}/results?api-version=7.0&outcomes=Failed
+```
+
+Filter by `outcomes=Failed` to get only failures.
+
+**Response includes:**
+- `testCaseName` — usually `portname:triplet`
+- `outcome` — `Failed`
+- `errorMessage` — short error description
+- `stackTrace` — full failure output
+
+---
+
+## URL Parsing
+
+Given `https://dev.azure.com/vcpkg/public/_build/results?buildId=129315&view=results`:
+
+| Component | Value | How to Extract |
+|-----------|-------|----------------|
+| Organization | `vcpkg` | 4th path segment |
+| Project | `public` | 5th path segment |
+| Build ID | `129315` | `buildId` query parameter |
+
+```python
+# Python example
+from urllib.parse import urlparse, parse_qs
+url = "https://dev.azure.com/vcpkg/public/_build/results?buildId=129315&view=results"
+parts = urlparse(url)
+segments = parts.path.strip('/').split('/')
+org = segments[0]       # "vcpkg"
+project = segments[1]   # "public"
+build_id = parse_qs(parts.query)['buildId'][0]  # "129315"
+```
+
+---
+
+## Rate Limits and Auth
+
+- **Public projects** (like `vcpkg/public`): No authentication needed; anonymous access is allowed
+- **Rate limit**: ~200 requests per minute for anonymous; increase by adding a PAT as Bearer token if needed
+- **PAT header**: `Authorization: Basic {base64(":pat_token")}`
diff --git a/.github/skills/analyze-ci-failures/references/report-template.md b/.github/skills/analyze-ci-failures/references/report-template.md
new file mode 100644
index 0000000000..ad6d759f8a
--- /dev/null
+++ b/.github/skills/analyze-ci-failures/references/report-template.md
@@ -0,0 +1,176 @@
+# vcpkg CI Failure Report Template
+
+This file provides the canonical output format for the `analyze-ci-failure` skill.
+Use it as the exact template when generating reports.
+
+---
+
+## Report Format
+
+```markdown
+# vcpkg CI Failure Report — Build #{buildId}
+
+**Build:** [{buildNumber}]({build_web_url}) · `{definition_name}`
+**Triggered by:** {reason} on `{sourceBranch}` @ `{sourceVersion[:8]}`
+**Duration:** {startTime} → {finishTime} UTC (~{duration})
+**Result:** ❌ **FAILED** — {baseline_summary}
+
+---
+
+## Summary
+
+| Triplet | Unique Failures |
+|---------|----------------|
+| x64-windows | N |
+| x64-windows-release | N (identical to x64-windows / N unique) |
+| x64-linux | N (M shared with Windows) |
+| {other triplets} | *artifacts present (not analyzed)* |
+
+**Total unique failing ports across all analyzed triplets: N**
+**Estimated cross-platform failures: ≥N ports fail on multiple triplets**
+
+---
+
+## 🔴 Cross-Platform Regressions (Windows + Linux)
+
+### N. `port-name` — Short Root Cause Title
+
+- **Triplets:** x64-windows, x64-windows-release, x64-linux
+- **Error:** `exact error message from log`
+  in `path/to/file.ext:line`
+- **Root cause:** Explanation of why this happened — what changed, what broke.
+- **Suggested fix:** Concrete actionable recommendation.
+
+  ```cmake
+  # Code snippet if applicable
+  ```
+
+---
+
+## 🔴 Windows-Only Regressions
+
+### N. `port-name` — Short Root Cause Title
+
+- **Triplets:** x64-windows, x64-windows-release
+- **Error:** `exact error message`
+- **Root cause:** ...
+- **Suggested fix:** ...
+
+---
+
+## 🔴 Linux-Only Regressions
+
+### N. `port-name` — Short Root Cause Title
+
+- **Triplet:** x64-linux
+- **Error:** `exact error message`
+- **Root cause:** ...
+- **Suggested fix:** ...
+
+---
+
+## 📋 Action Recommendations
+
+| Priority | Port(s) | Recommended Action |
+|---|---|---|
+| 🔴 Immediate | `port` | Short action description |
+| 🟠 High | `port` | Short action description |
+| 🟡 Medium | `port` | Short action description |
+| ℹ️ Baseline | `port` | Add `=fail/skip # reason` to `ci.baseline.txt` |
+| ℹ️ Investigate | `port` | What to look into |
+```
+
+---
+
+## Real-World Example (Build #129315)
+
+The following is a real report generated for the scheduled master build on 2026-03-30.
+Use this as a reference for tone, depth, and structure.
+
+### Build Metadata
+
+- **Build:** [20260330.1](https://dev.azure.com/vcpkg/public/_build/results?buildId=129315)
+- **Reason:** Scheduled run on `refs/heads/master` @ `c4de8d6f`
+- **Duration:** ~35 hours
+- **Artifacts with failure logs:** all 13 triplets had failures
+
+### Failure Inventory
+
+| Port | Triplets | Root Cause Category | Severity |
+|------|----------|---------------------|----------|
+| `v8` | win | Python 3.13 removed `pipes` module | 🔴 Immediate |
+| `onnxruntime` | win, linux | CMake `math()` on empty TensorRT variable | 🔴 Immediate |
+| `wpilib` | win, linux | `wpi::Logger::Log` API mismatch in `DataLog.cpp` | 🔴 Immediate |
+| `ntf-core` | win, linux | `bal` dep ordering (win); missing UFID cmake file (linux) | 🟠 High |
+| `rmqcpp` | win, linux | Transitive `libpcre2-8` not found via `bal→bdl` | 🟠 High |
+| `azure-storage-cpp` | linux | Boost.ASIO `deadline_timer` removed | 🟠 High |
+| `cppcoro` | linux | `<experimental/coroutine>` removed in GCC 14 | 🟠 High |
+| `zookeeper` | win, linux | Java not found at configure time | 🟡 Medium |
+| `moos-essential` | win | POSIX socket headers + Winsock2 conflict | 🟡 Medium |
+| `libxt` | win | Clang `-Wunsafe-buffer-usage` promoted to error | 🟡 Medium |
+| `saucer` | linux | Upstream compiler version check too strict | 🟡 Medium |
+| `libaiff` | linux | Missing `#include <stdint.h>` in public header | 🟡 Medium |
+| `libhdfs3` | linux | `mode` macro conflict + missing `<stdint.h>` | 🟡 Medium |
+| `yubico-piv-tool` | linux | Clang-only flag `-Wshorten-64-to-32` passed to GCC | 🟡 Medium |
+| `libnick` | linux | `sqlcipher` not available via pkg-config | 🟡 Medium |
+| `salome-medcoupling` | linux | Missing `metis` vcpkg dependency | 🟡 Medium |
+| `mesa` | linux | `glslangValidator` not found by Meson | 🟡 Medium |
+| `crashpad` | linux | Compiler not found (exit code 127, likely Docker image change) | ℹ️ Investigate |
+| `arrayfire` | linux | Build truncated at 117/832 — likely OOM kill | ℹ️ Investigate |
+| `xbitmaps` | linux | `xorg-macros < 1.20` in Docker image | ℹ️ Investigate |
+| `nana` | linux | CMake generate fails, no detail in logs | ℹ️ Investigate |
+| `openslide` | win | Portfile explicitly rejects MSVC (`use clang-cl`) | ℹ️ Baseline |
+| `openzl` | win | Portfile explicitly rejects MSVC (`use clang-cl`) | ℹ️ Baseline |
+| `ms-gdkx` | win | Requires Microsoft GDK Xbox Extensions (not in CI) | ℹ️ Baseline |
+
+---
+
+## Formatting Rules
+
+1. **Section order**: Cross-Platform → Windows-Only → Linux-Only → macOS-Only → Android-Only → Action Table
+2. **Numbering**: Sequential across all sections (1, 2, 3 … N)
+3. **Error quotes**: Always quote the exact first meaningful error line from the log verbatim
+4. **Triplet notation**: Use short forms in narrative (`win`, `linux`) but full names in bullet points (`x64-windows`)
+5. **Baseline ports**: Ports that explicitly reject a platform by design (`MSVC is not supported`, environment requirements) go to the Baseline section, not regressions
+6. **Dependency chain**: If port A fails because port B failed, list only port B as the regression with a note "causes N downstream failures: portA, portC, …"
+7. **Transient failures**: Download failures (curl errors, SHA mismatches) and exit-code-127 failures should be flagged as potentially transient
+8. **Priority levels**:
+   - 🔴 **Immediate** — Core/popular ports, failures on 3+ triplets, or CI-environment regressions affecting many ports
+   - 🟠 **High** — Failures on 2+ triplets, popular ports, or clear upstream API breaks
+   - 🟡 **Medium** — Single-triplet failures with clear fixes
+   - ℹ️ **Baseline/Investigate** — Expected failures, environment issues, or unclear root cause
+
+---
+
+## Artifact Size Heuristics
+
+Use artifact sizes (from the artifacts list API response) to prioritize analysis:
+
+| Artifact Size | Interpretation |
+|---|---|
+| < 10 MB | Very few failures (< ~15 ports), analyze fully |
+| 10–50 MB | Moderate failures (~15–80 ports), sample representative ones |
+| 50–120 MB | Many failures (~80–200 ports), focus on unique errors only |
+| > 120 MB | Mass failures — likely a systemic issue (toolchain, CI image, common dep) |
+
+In build #129315, x64-windows (8.5 MB) and x64-linux (4.6 MB compressed) were small enough for complete analysis.
+Android artifacts (~115–123 MB) indicate widespread failures likely sharing a common root cause.
+
+---
+
+## Common Root Cause Patterns to Look For First
+
+When analyzing a new build, scan for these high-frequency patterns before deep-diving individual ports:
+
+| Pattern | Indicator in logs | Likely cause |
+|---|---|---|
+| `No module named 'pipes'` or similar Python import error | Python stdlib change | Python version upgrade on CI |
+| `fatal error: experimental/coroutine` | Many Linux ports | GCC 14 / compiler image upgrade |
+| `deadline_timer` / `posix_time` | Boost-dependent ports | Boost.ASIO API break |
+| `error: unrecognized command-line option` | Configure-time test | Wrong compiler flags for toolchain |
+| `FAILED: [code=127]` | Many compilation units | Compiler not in PATH (Docker image change) |
+| `math cannot parse the expression: ""` | CMake configure | Empty variable in math() — missing guard |
+| `Cannot open include file: 'sys/socket.h'` | Win + POSIX code | Missing WIN32 socket porting |
+| `must install xorg-macros` | autotools ports | CI image missing xorg-dev package |
+| Build truncated, no error message | OOM or timeout | Use `DISABLE_PARALLEL` or check resources |
+| Many ports → same dependency fails | dep chain | Find root dep; rest are downstream casualties |
diff --git a/.github/skills/analyze-ci-failures/references/vcpkg-failure-patterns.md b/.github/skills/analyze-ci-failures/references/vcpkg-failure-patterns.md
new file mode 100644
index 0000000000..4fef6073ad
--- /dev/null
+++ b/.github/skills/analyze-ci-failures/references/vcpkg-failure-patterns.md
@@ -0,0 +1,361 @@
+# vcpkg CI Failure Patterns Reference
+
+This reference catalogues common failure patterns found in vcpkg CI failure log artifacts. Use these patterns to triage and classify failures when analyzing build output.
+
+---
+
+## Log File Structure
+
+Each failure log (`*.log`) in the `"failure logs for {triplet}"` artifact contains the full vcpkg install output for a single port. The log typically follows this structure:
+
+```
+-- Downloading <url>...
+-- Extracting source...
+-- Applying patches...
+-- Using cached binary package.   ← binary cache hit (may skip build)
+-- Building <port>:<triplet>...
+-- Configuring...
+-- Building...   ← compiler output here
+-- Installing...
+-- Running tests...   ← if test feature enabled
+-- Completed.
+
+error: building <port>:<triplet> failed with: BUILD_FAILED
+```
+
+---
+
+## Category 1: Dependency Chain Failures
+
+**Symptom**: The port itself did not fail — a dependency failed first.
+
+**Log indicators:**
+```
+-- Building dependency-name[core]:x64-windows... failed
+error: package dependency-name:x64-windows is not installed
+```
+
+or more explicitly:
+```
+error: building portname:x64-windows failed with: BUILD_FAILED
+  ... the dependency 'depname:x64-windows' failed to install
+```
+
+**Triage action:**
+- Identify the root failing dependency (the one that failed first in the log)
+- All other ports in this chain are "downstream casualties"
+- Fix the root dependency; downstream failures should resolve automatically
+- In the report, list only the root dependency as a regression; collapse others with a note
+
+**How to find the root:**
+- Sort failure logs by time or by name
+- A port whose own build commands (not just dependency resolution) fail is the root
+- If `portA.log` shows `error: package portB is not installed` and `portB.log` shows actual compiler output, portB is the root
+
+---
+
+## Category 2: CMake Configuration Failures
+
+**Symptom**: `cmake --build` stage fails before any source is compiled.
+
+**Log indicators:**
+```
+CMake Error at CMakeLists.txt:42 (find_package):
+  By not providing "FindXYZ.cmake" in CMAKE_MODULE_PATH this project has
+  requested CMake to find a package configuration file provided by "XYZ",
+  but CMake did not find one.
+
+CMake Error: Could not find a package configuration file provided by "XYZ"
+  with any of the following names:
+    XYZConfig.cmake
+    xyz-config.cmake
+```
+
+**Common root causes:**
+- A dependency is not exposing its CMake targets correctly
+- A port's `vcpkg.cmake` or `vcpkg-cmake-config` integration is broken
+- A `find_package()` call in the upstream CMakeLists.txt uses a non-standard config name
+
+**Other CMake configure patterns:**
+```
+-- The CXX compiler identification is unknown   ← toolchain issue (rare in CI)
+CMake Error: Generator: ... not found          ← MSBuild/Ninja not available
+CMake Error (cmake_policy): ...                ← policy violation
+```
+
+---
+
+## Category 3: Compiler Errors
+
+**Symptom**: Source code fails to compile.
+
+### MSVC (Windows)
+```
+error C2065: 'identifier': undeclared identifier
+error C2039: 'member': is not a member of 'class'
+error C3861: 'function': identifier not found
+error C4996: 'function': This function may be unsafe   ← treated as error with /WX
+fatal error C1083: Cannot open include file: 'header.h'
+```
+
+### Clang (macOS, Linux)
+```
+error: use of undeclared identifier 'name'
+error: no member named 'member' in 'class'
+error: 'header.h' file not found
+error: unknown type name 'type_name'
+```
+
+### GCC (Linux)
+```
+error: 'name' was not declared in this scope
+error: invalid use of incomplete type
+error: 'header.h': No such file or directory
+```
+
+**Root cause categories:**
+- **C++ standard incompatibility**: Code uses features not available in the configured standard (C++14/17/20)
+- **Missing Windows includes**: `#include <windows.h>` missing or wrong order
+- **API deprecation**: Uses a function/class removed in a newer version of a dependency
+- **Compiler flag conflict**: vcpkg sets `-Werror` or `/WX`; warning-as-error triggers on upstream code
+
+---
+
+## Category 4: Linker Errors
+
+**Symptom**: Compilation succeeds but linking fails.
+
+### MSVC
+```
+error LNK2019: unresolved external symbol "__declspec(dllimport) ..." referenced in function "..."
+error LNK2001: unresolved external symbol
+error LNK1120: N unresolved externals
+```
+
+### GCC/Clang
+```
+undefined reference to `function_name'
+ld: symbol(s) not found for architecture arm64
+```
+
+**Root cause categories:**
+- **Missing `__declspec(dllexport)`**: Library exports are missing for DLL builds
+- **Static vs. dynamic mismatch**: Port links against static lib but needs dynamic (or vice versa)
+- **Missing dependency in vcpkg.json**: Library calls into another library not declared as a dependency
+- **Symbol visibility**: On Linux/macOS, symbols not exported with `__attribute__((visibility("default")))`
+
+---
+
+## Category 5: SHA512 Hash Mismatch
+
+**Symptom**: Download succeeds but hash verification fails.
+
+```
+error: File does not have the expected hash:
+          url: https://github.com/owner/repo/archive/v1.2.3.tar.gz
+        expected: abc123...
+          actual: def456...
+```
+
+**Root causes:**
+- **Upstream archive changed**: The upstream author re-tagged or replaced the release archive (mutable tags, force-push)
+- **Mirror cache stale**: vcpkg asset cache has an old version of the file
+- **Port hash not updated**: A PR updated the version but forgot to recalculate SHA512
+
+**Triage action**: Recalculate hash with:
+```bash
+vcpkg install portname --no-binarycaching
+```
+The error message will contain the correct new hash.
+
+---
+
+## Category 6: Download / Network Failures
+
+**Symptom**: vcpkg cannot download the source archive.
+
+```
+error: Failed to download https://example.com/archive.tar.gz
+error: curl: (22) The requested URL returned error: 404
+error: curl: (6) Could not resolve host: example.com
+```
+
+**Root causes:**
+- **URL moved**: Upstream deleted or moved the release (common with GitHub releases being retracted)
+- **CI network issue**: Transient connectivity problem (likely if only 1–2 ports fail this way, or a retry succeeds)
+- **Domain changed**: Package moved to a new hosting location
+
+**Triage action**: Check if the URL is still valid. If transient, re-run the build.
+
+---
+
+## Category 7: Post-Build Check Failures
+
+**Symptom**: Build and install succeed, but vcpkg's post-install checks fail.
+
+```
+error: The following files are not installed:
+    share/portname/copyright
+
+error: Found forbidden pattern in installed file:
+    include/portname/config.h: #define PORTNAME_EXPORTS
+
+error: Usage of deprecated vcpkg function 'vcpkg_apply_patches'
+```
+
+**Common post-build checks:**
+- Copyright file must exist at `share/{portname}/copyright`
+- No `.pdb` files in release packages (debug-only)
+- Headers must be in `include/`, not `include/{portname}/include/`
+- CMake config files in `share/{portname}/` not `lib/cmake/`
+- No absolute paths in installed CMake files
+
+---
+
+## Category 8: Test Failures
+
+**Symptom**: The port itself builds and installs successfully, but its tests fail.
+
+```
+Test #1: my_test ...  FAILED
+The following tests FAILED:
+    1 - my_test (Failed)
+```
+
+or in xUnit format (from test results API):
+```xml
+<testcase name="portname:x64-windows" ...>
+  <failure message="Test failed: ..." />
+</testcase>
+```
+
+**Triage action:**
+- Check whether the test passes on other triplets (may be platform-specific)
+- Check if this is a flaky test (re-run to confirm)
+- Check if the test was previously failing (baseline)
+
+---
+
+## Category 9: Formatting / Manifest Check Failures
+
+**Symptom**: The `"format.diff"` artifact is present and non-empty.
+
+This is not a port build failure but a CI check failure. The `format.diff` artifact contains a unified diff showing:
+- `vcpkg.json` manifest files that need formatting via `vcpkg format-manifest`
+- Baseline files (`ci.baseline.txt`) that need formatting via `vcpkg format-feature-baseline`
+- Version database files that need updating via `vcpkg x-add-version`
+
+**Triage action**: Apply the diff, or run the formatting commands locally:
+```bash
+./vcpkg format-manifest --all
+./vcpkg format-feature-baseline scripts/ci.baseline.txt
+./vcpkg x-add-version --all
+```
+
+---
+
+## Category 10: Build Timeout
+
+**Symptom**: Job exceeds the 2-day timeout limit (rare).
+
+```
+##[error]The job running on agent ... has exceeded the maximum execution time of 2880 minutes.
+```
+
+**Triage action**: Almost always an infinite loop or hang in the build system. Check the last lines of the log for the hanging operation.
+
+---
+
+## Category 11: File Conflicts
+
+**Symptom**: The port installs files that conflict with files already installed by another port. This failure is **only visible in the "Test Modified Ports" step log** — it does NOT produce a failure log artifact.
+
+```
+REGRESSION: kf6i18n:x64-windows failed with FILE_CONFLICTS. If expected, add kf6i18n:x64-windows=fail to .../ci.baseline.txt.
+```
+
+The error in the step log reads (before the REGRESSION summary line):
+```
+error: File conflicts:
+  C:\path\to\installed\x64-windows\bin\some-tool.exe  →  installed by portA and portB
+  C:\path\to\installed\x64-windows\include\shared.h   →  installed by portA and portB
+```
+
+**Where to find it**: In the "*** Test Modified Ports" task log for the affected job (not in any artifact ZIP).
+
+**Triage action**:
+- Identify which two ports are claiming the same installed files
+- Usually caused by: a new port that duplicates files from an existing one, or a port that started installing an additional binary that conflicts
+- Fix: rename the conflicting file, split the port, or add `conflicts` to the port manifest
+
+---
+
+## Category 12: Missing From Baseline (Unexpected Pass)
+
+**Symptom**: A port that is listed as `fail` in `ci.baseline.txt` now passes. This is reported as a regression only in the step log:
+
+```
+REGRESSION: portname:x64-windows failed with MISSING_FROM_BASELINE.
+```
+
+**Triage action**: Remove the `portname=fail` entry from `scripts/ci.baseline.txt` — the port is fixed.
+
+The file `scripts/ci.baseline.txt` in the repository root controls which ports are expected to fail.
+
+**Format:**
+```
+# Comment lines start with #
+portname=fail       # will skip in PR CI, attempt in scheduled
+portname=skip       # never built in CI
+portname=pass       # (same as not listed) must succeed
+```
+
+**Triplets covered by CI baseline:**
+- `arm-neon-android`
+- `arm64-android`
+- `arm64-osx`
+- `arm64-windows`
+- `arm64-windows-static-md`
+- `x64-android`
+- `x64-linux`
+- `x64-osx`
+- `x64-uwp`
+- `x64-windows`
+- `x64-windows-static`
+- `x64-windows-static-md`
+- `x86-windows`
+
+**Important**: Baseline entries apply to all triplets unless a triplet-specific entry exists (not currently supported; all triplet failures for a port use the same baseline entry).
+
+---
+
+## Regression Severity Levels
+
+Use these levels when reporting failures:
+
+| Level | Criteria | Action |
+|---|---|---|
+| 🔴 **Critical** | Core port (boost, openssl, curl, zlib, cmake) or many dependents fail | Immediate fix required |
+| 🟠 **High** | Popular port fails on multiple triplets | Fix before merge |
+| 🟡 **Medium** | Single port fails on 1–2 triplets | Fix or add to baseline |
+| 🟢 **Low** | Port already in baseline, or transient failure | No immediate action |
+| ℹ️ **Info** | Formatting/version file issues | Easy to fix with automated tools |
+
+---
+
+## Common vcpkg CI Triplet Notes
+
+| Triplet | Notes |
+|---|---|
+| `x64-windows` | Most common; MSVC, dynamic linking |
+| `x64-windows-static` | MSVC, static linking (`/MT`) |
+| `x64-windows-static-md` | MSVC, static linking (`/MD`) |
+| `arm64-windows` | Cross-compiled on x64 Windows; run tests on separate ARM hardware |
+| `x64-linux` | Ubuntu in Docker; GCC or Clang |
+| `arm64-linux` | Ubuntu ARM64 in Docker |
+| `arm64-osx` | macOS ARM (Apple Silicon); Clang |
+| `x64-osx` | macOS x86_64; Clang |
+| `x64-android` | Cross-compiled; Android NDK Clang |
+| `arm64-android` | Cross-compiled; Android NDK Clang |
+| `arm-neon-android` | Cross-compiled; Android NDK Clang |
+| `x64-uwp` | Universal Windows Platform; WinRT restrictions |