From cfc0e856012a5cf608d4fe36ead71720faa505e6 Mon Sep 17 00:00:00 2001 From: Victor Romero Date: Thu, 16 Apr 2026 16:25:05 -0700 Subject: [PATCH] [skills][analyze-ci-failures] Work on PR checks and save failure logs (#51209) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/analyze-ci-failures/SKILL.md | 85 ++++++++++++++++--- .../references/report-template.md | 11 ++- .../references/vcpkg-failure-patterns.md | 54 ++++++++++-- .gitignore | 3 + 4 files changed, 132 insertions(+), 21 deletions(-) diff --git a/.github/skills/analyze-ci-failures/SKILL.md b/.github/skills/analyze-ci-failures/SKILL.md index d5b4bf5461..ccb2551f4c 100644 --- a/.github/skills/analyze-ci-failures/SKILL.md +++ b/.github/skills/analyze-ci-failures/SKILL.md @@ -1,7 +1,7 @@ --- name: analyze-ci-failures description: 'Analyze vcpkg Azure DevOps CI build failures. Downloads failure logs, identifies regression root causes, and generates a structured report categorizing build errors by package, triplet, and failure type.' -argument-hint: 'Azure DevOps build URL (e.g., "https://dev.azure.com/vcpkg/public/_build/results?buildId=129315")' +argument-hint: 'Azure DevOps build URL or GitHub PR URL (e.g., "https://dev.azure.com/vcpkg/public/_build/results?buildId=129315" or "https://github.com/microsoft/vcpkg/pull/51134")' --- # vcpkg CI Failures Analyzer @@ -27,7 +27,9 @@ This skill fetches build metadata and failure logs directly from the Azure DevOp ### Step 1: Parse the Build URL -Given a URL such as: +The skill accepts either an **Azure DevOps build URL** or a **GitHub PR URL** as input. + +**Azure DevOps build URL:** ``` https://dev.azure.com/vcpkg/public/_build/results?buildId=129315&view=results ``` @@ -37,6 +39,16 @@ Extract: - **project**: `public` - **buildId**: `129315` +**GitHub PR URL:** +``` +https://github.com/microsoft/vcpkg/pull/51134 +``` + +Extract the PR number, then use the GitHub API to find the associated Azure DevOps build: +1. Fetch the PR's check runs: `GET /repos/microsoft/vcpkg/pulls/{pr_number}/check-runs` (use `github-mcp-server-pull_request_read` with method `get_check_runs`) +2. Find the check run named `microsoft.vcpkg.pr` — its `details_url` contains the Azure DevOps build URL with `buildId` +3. Extract the `buildId` from that URL and proceed with the normal workflow + The REST API base is: `https://dev.azure.com/{organization}/{project}/_apis` ### Step 2: Fetch Build Metadata @@ -50,11 +62,13 @@ Key fields in the response: - `result` — `succeeded`, `failed`, `partiallySucceeded`, `canceled` - `reason` — `pullRequest`, `schedule`, `manual`, `individualCI` - `requestedFor.displayName` — who triggered the build -- `sourceBranch` — branch or PR reference +- `sourceBranch` — branch or PR reference (e.g., `refs/pull/51202/merge` for PR builds) - `sourceVersion` — commit SHA - `_links.web.href` — canonical link back to the build page - `finishTime` — when it completed +**For PR-triggered builds** (`reason == "pullRequest"`): extract the PR number from `sourceBranch` (e.g., `refs/pull/51202/merge` → PR #51202). Use the GitHub API to fetch the PR title and author — include these in the report header for context, and link to the PR with `https://github.com/microsoft/vcpkg/pull/{pr_number}`. The PR is on the `microsoft/vcpkg` repository (not `vicroms/vcpkg`). + **Get the build timeline (all jobs and tasks):** ``` GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/timeline?api-version=7.0 @@ -94,6 +108,7 @@ REGRESSION: kf6itemmodels:x64-windows failed with FILE_CONFLICTS. If expected, a | Failure type | Also has artifact log? | Notes | |---|---|---| | `BUILD_FAILED` | ✅ Yes | Full build logs in "failure logs for {triplet}" artifact | +| `POST_BUILD_CHECKS_FAILED` | ✅ Yes | Build succeeded but vcpkg post-install validation failed (misplaced CMake files, missing usage, etc.) | | `FILE_CONFLICTS` | ❌ No | Port installs files that conflict with another port; only in step log | | `MISSING_FROM_BASELINE` | ❌ No | Port passed but is listed as `fail` in baseline; no artifact | | `CASCADE_BUILD_FAILED` | Sometimes | Dependency failure; root port has artifact, downstream may not | @@ -146,21 +161,56 @@ Start with the smallest artifacts (fastest feedback). Android triplets typically ### Step 4: Download Failure Log Artifacts -**`web_fetch` cannot download binary ZIP files.** Use PowerShell with `Invoke-WebRequest` instead: +**`web_fetch` cannot download binary ZIP files.** Use PowerShell with `Invoke-WebRequest` instead. + +**Output directory:** All downloaded artifacts and the final report are saved in a persistent directory under the repository root for manual review: + +``` +ci-failure-analysis/ +├── pr-51196/ ← PR-triggered build (named by PR number) +│ ├── logs/ ← extracted failure log artifacts +│ │ ├── x64-linux/ +│ │ │ └── failure logs for x64-linux/ +│ │ │ ├── ffmpeg/ +│ │ │ └── ... +│ │ ├── x64-windows-static/ +│ │ └── ... +│ └── report.md ← the generated analysis report +├── ci-130500/ ← scheduled/manual build (named by build ID) +│ ├── logs/ +│ └── report.md +└── ... +``` + +**Naming convention:** +- **PR-triggered builds** (`reason == "pullRequest"`): `pr-{pr_number}` (e.g., `pr-51196`) +- **All other builds** (scheduled, manual, individualCI): `ci-{buildId}` (e.g., `ci-130500`) + +This directory is listed in `.gitignore` and will not be committed. It allows reviewers to manually inspect the raw logs alongside the report. ```powershell -$tmpDir = "$env:TEMP\vcpkg-ci-{buildId}" -New-Item -ItemType Directory -Path $tmpDir -Force | Out-Null +# Determine the output directory name +if ($build.reason -eq 'pullRequest') { + $prNumber = ($build.sourceBranch -replace 'refs/pull/(\d+)/merge','$1') + $analysisDir = "$repoRoot\ci-failure-analysis\pr-$prNumber" +} else { + $analysisDir = "$repoRoot\ci-failure-analysis\ci-$buildId" +} +$logsDir = "$analysisDir\logs" +New-Item -ItemType Directory -Path $logsDir -Force | Out-Null # Download the artifact ZIP using the downloadUrl from the artifacts API response $downloadUrl = "{resource.downloadUrl from artifacts API}" -Invoke-WebRequest -Uri $downloadUrl -OutFile "$tmpDir\{triplet}.zip" -UseBasicParsing +Invoke-WebRequest -Uri $downloadUrl -OutFile "$logsDir\{triplet}.zip" -UseBasicParsing # Extract -Expand-Archive "$tmpDir\{triplet}.zip" -DestinationPath "$tmpDir\{triplet}" -Force +Expand-Archive "$logsDir\{triplet}.zip" -DestinationPath "$logsDir\{triplet}" -Force + +# Remove the ZIP after extraction to save space +Remove-Item "$logsDir\{triplet}.zip" -Force # List failed ports — each is a subdirectory inside "failure logs for {triplet}/" -$root = "$tmpDir\{triplet}\failure logs for {triplet}" +$root = "$logsDir\{triplet}\failure logs for {triplet}" Get-ChildItem $root -Directory | Select-Object -ExpandProperty Name ``` @@ -424,16 +474,21 @@ When asked to produce the report, default to the full structured markdown report ### Saving the Report -After generating the report, **always save it as a markdown file** in the repository root: +After generating the report, **always save it as `report.md`** inside the analysis output directory: ``` -ci-report-{buildId}.md +ci-failure-analysis/{pr-NNN|ci-NNN}/report.md ``` -Use the `create` tool to write the full report content to this path. This ensures the report is persisted and can be easily reviewed or shared. +Use the `create` tool to write the full report content to this path. The report lives alongside the downloaded failure logs, making it easy to cross-reference the analysis with the raw log files. + +**Examples:** +- PR #51196 analysis → `ci-failure-analysis/pr-51196/report.md` +- Scheduled build #129315 → `ci-failure-analysis/ci-129315/report.md` ## Important Notes +- **vcpkg only supports `>=` version constraints** — there is no way to express `<=` or `<` constraints. Never suggest "add a version constraint to prevent upgrading to version X" as a fix; it is not possible. Instead, recommend updating the consuming port to be compatible with the new version, or patching the port. - The vcpkg Azure DevOps project (`vcpkg/public`) is **publicly accessible** — no authentication token is required for artifact downloads or build/artifact API calls - **`web_fetch` cannot download binary ZIP files** — use PowerShell `Invoke-WebRequest` for artifact downloads - **`resource.type` is `"PipelineArtifact"`**, not `"Container"` — the Container listing API (`/_apis/resources/Containers/{id}`) does not work; use `downloadUrl` directly @@ -445,4 +500,8 @@ Use the `create` tool to write the full report content to this path. This ensure - The `"format.diff"` artifact (if present) indicates formatting or version file issues, not port build failures — report these separately - `"z azcopy logs"` artifacts are infrastructure logs; skip these unless diagnosing asset cache issues - **Android artifact sizes (~115–123 MB) typically share root causes with Linux** — if Linux analysis reveals a systemic issue (e.g., compiler header change, Python version), assume Android is affected too without full re-analysis -- **Clean up the temp directory** (`$env:TEMP\vcpkg-ci-{buildId}`) after analysis to avoid disk clutter +- **Downloaded logs are kept for manual review** in `ci-failure-analysis/` (gitignored). ZIP files are removed after extraction to save space, but the extracted logs and report are preserved. If disk space is a concern, old analysis directories can be deleted manually. +- **Case-sensitive file paths**: Linux and Android file systems are case-sensitive; Windows and macOS are not. A portfile referencing `${SOURCE_PATH}/LICENSE` will succeed on Windows/macOS even if the upstream file is named `License` or `license`, but will fail with "No such file or directory" on Linux/Android. When a BUILD_FAILED occurs only on Linux/Android with a missing file error, always check for a case mismatch against the actual upstream filename. +- **Portfile anti-patterns — do NOT recommend these as fixes:** + - `set(VCPKG_BUILD_TYPE release)` — this is **only** appropriate for header-only libraries. Ports that produce binary output (`.lib`, `.a`, `.dll`, `.so`) must build both debug and release configurations. Never suggest adding this to fix mismatched-binary warnings. + - `set(VCPKG_POLICY_* enabled)` — policy overrides are escape hatches for exceptional cases, not standard practice. Most ports should not set any `VCPKG_POLICY_*` variable. When a post-build check fails, the correct fix is almost always to fix the underlying issue (e.g., correct `vcpkg_cmake_config_fixup()` arguments, install missing files), not to suppress the warning with a policy override. diff --git a/.github/skills/analyze-ci-failures/references/report-template.md b/.github/skills/analyze-ci-failures/references/report-template.md index ad6d759f8a..f68007cc82 100644 --- a/.github/skills/analyze-ci-failures/references/report-template.md +++ b/.github/skills/analyze-ci-failures/references/report-template.md @@ -15,6 +15,10 @@ Use it as the exact template when generating reports. **Duration:** {startTime} → {finishTime} UTC (~{duration}) **Result:** ❌ **FAILED** — {baseline_summary} + + --- ## Summary @@ -134,7 +138,12 @@ Use this as a reference for tone, depth, and structure. 5. **Baseline ports**: Ports that explicitly reject a platform by design (`MSVC is not supported`, environment requirements) go to the Baseline section, not regressions 6. **Dependency chain**: If port A fails because port B failed, list only port B as the regression with a note "causes N downstream failures: portA, portC, …" 7. **Transient failures**: Download failures (curl errors, SHA mismatches) and exit-code-127 failures should be flagged as potentially transient -8. **Priority levels**: +8. **Version constraints**: Never suggest adding a `<=` version constraint as a fix — vcpkg only supports `>=` constraints. For API-breaking dependency updates, suggest updating the consuming port or patching it instead. +9. **PR context**: For PR-triggered builds, always include a link to the PR, the PR title, and the author in the report header. +10. **Portfile anti-patterns**: Never recommend `set(VCPKG_BUILD_TYPE release)` for ports that produce binaries — it is only for header-only libraries. Never recommend adding `VCPKG_POLICY_*` overrides to suppress post-build check warnings — fix the underlying issue instead. Most ports should not set any policy variable. +11. **Case-sensitivity**: When a file-not-found error occurs only on Linux/Android but the port passes on Windows/macOS, always verify the exact filename casing in the upstream repository before concluding the file is missing. +12. **`vcpkg_cmake_config_fixup` package name**: The `PACKAGE_NAME` argument must match the upstream CMake export name (what consumers pass to `find_package()`), which may differ from the vcpkg port name. +13. **Priority levels**: - 🔴 **Immediate** — Core/popular ports, failures on 3+ triplets, or CI-environment regressions affecting many ports - 🟠 **High** — Failures on 2+ triplets, popular ports, or clear upstream API breaks - 🟡 **Medium** — Single-triplet failures with clear fixes diff --git a/.github/skills/analyze-ci-failures/references/vcpkg-failure-patterns.md b/.github/skills/analyze-ci-failures/references/vcpkg-failure-patterns.md index 4fef6073ad..9c3b107528 100644 --- a/.github/skills/analyze-ci-failures/references/vcpkg-failure-patterns.md +++ b/.github/skills/analyze-ci-failures/references/vcpkg-failure-patterns.md @@ -119,6 +119,12 @@ error: 'header.h': No such file or directory - **API deprecation**: Uses a function/class removed in a newer version of a dependency - **Compiler flag conflict**: vcpkg sets `-Werror` or `/WX`; warning-as-error triggers on upstream code +**Suggested fix guidance for API deprecation failures:** +When a dependency update removes or renames an API that downstream ports still use: +1. Update the downstream port to a version compatible with the new API +2. Patch the downstream port's source to adapt to the new API +- **Do NOT suggest adding a `<=` version constraint** — vcpkg only supports `>=` version constraints; upper-bound pinning is not possible + --- ## Category 4: Linker Errors @@ -191,7 +197,7 @@ error: curl: (6) Could not resolve host: example.com ## Category 7: Post-Build Check Failures -**Symptom**: Build and install succeed, but vcpkg's post-install checks fail. +**Symptom**: Build and install succeed, but vcpkg's post-install checks fail. Reported as `POST_BUILD_CHECKS_FAILED` in the REGRESSION line. ``` error: The following files are not installed: @@ -203,12 +209,23 @@ error: Found forbidden pattern in installed file: error: Usage of deprecated vcpkg function 'vcpkg_apply_patches' ``` -**Common post-build checks:** -- Copyright file must exist at `share/{portname}/copyright` -- No `.pdb` files in release packages (debug-only) -- Headers must be in `include/`, not `include/{portname}/include/` -- CMake config files in `share/{portname}/` not `lib/cmake/` -- No absolute paths in installed CMake files +**Common post-build checks and their fixes:** + +| Check | Warning message | Correct fix | +|---|---|---| +| **Misplaced CMake config files** | `This port installs CMake files in places CMake files are not expected` | Fix `vcpkg_cmake_config_fixup()` — use the correct `PACKAGE_NAME` matching the upstream CMake export name (not necessarily the vcpkg port name), and set `CONFIG_PATH` to where the upstream installs its config (e.g., `lib/cmake/pkgname`) | +| **`lib/cmake` not merged** | `should be merged and moved to share/${PORT}/cmake` | Fixed by correcting `vcpkg_cmake_config_fixup()` as above | +| **Usage file not installed** | `this port contains a file named "usage" but didn't install it` | Add `file(INSTALL "${CMAKE_CURRENT_LIST_DIR}/usage" DESTINATION "${CURRENT_PACKAGES_DIR}/share/${PORT}")` | +| **Mismatched debug/release binaries** | `mismatching number of debug and release binaries` | Ensure the port builds both debug and release. Do **not** fix by adding `VCPKG_POLICY_MISMATCHED_NUMBER_OF_BINARIES` or `VCPKG_BUILD_TYPE release` (see portfile anti-patterns below) | +| **Missing copyright** | `share/portname/copyright not found` | Add `vcpkg_install_copyright(FILE_LIST "${SOURCE_PATH}/LICENSE")` with the correct filename and case | +| **Absolute paths in CMake files** | `absolute path found in CMake config` | Use `vcpkg_cmake_config_fixup()` which handles path rewriting | +| **No `.pdb` in release** | `.pdb files in release packages` | Ensure PDB files only install for debug configuration | + +**Portfile anti-patterns (do NOT recommend as fixes):** +- `set(VCPKG_BUILD_TYPE release)` — only appropriate for **header-only** libraries. Ports producing binaries (`.lib`, `.a`, `.dll`, `.so`) must build both debug and release. +- `set(VCPKG_POLICY_* enabled)` — policy overrides are escape hatches for exceptional cases. The correct fix is to resolve the underlying issue, not suppress the warning. + +**Key insight for `vcpkg_cmake_config_fixup()`**: The `PACKAGE_NAME` argument must match the **upstream CMake export name** (what `find_package()` looks for), which is often different from the vcpkg port name. For example, a port named `minecraft-server-status` whose upstream exports `minecraft_api` needs `vcpkg_cmake_config_fixup(PACKAGE_NAME "minecraft_api" CONFIG_PATH "lib/cmake/minecraft_api")`. --- @@ -321,6 +338,7 @@ portname=pass # (same as not listed) must succeed - `x64-osx` - `x64-uwp` - `x64-windows` +- `x64-windows-release` - `x64-windows-static` - `x64-windows-static-md` - `x86-windows` @@ -329,6 +347,28 @@ portname=pass # (same as not listed) must succeed --- +## Category 13: Case-Sensitivity Path Failures (Linux/Android only) + +**Symptom**: A file-not-found error occurs on Linux and Android triplets, but the same port builds successfully on Windows and macOS. + +``` +CMake Error at scripts/cmake/vcpkg_install_copyright.cmake:21 (file): + file INSTALL cannot find ".../src/v1.0.0/LICENSE": No such file or directory. +``` + +**Root cause**: The portfile references a filename with different casing than the actual file in the upstream repository. Linux and Android use case-sensitive file systems (`LICENSE` ≠ `License` ≠ `license`), while Windows and macOS are case-insensitive and will resolve any casing variant. + +**Common occurrences:** +- License files: `LICENSE` vs `License` vs `license` vs `COPYING` +- README files: `README.md` vs `Readme.md` vs `readme.md` +- Source directories with inconsistent casing + +**Diagnostic clue**: If a port fails with BUILD_FAILED on all Linux/Android triplets but succeeds (or fails for a *different* reason like POST_BUILD_CHECKS_FAILED) on Windows/macOS, check for a case mismatch in file paths used in the portfile. + +**Fix**: Correct the filename in `portfile.cmake` to match the exact casing in the upstream repository. Verify by checking the upstream repo's file listing. + +--- + ## Regression Severity Levels Use these levels when reporting failures: diff --git a/.gitignore b/.gitignore index d6d4792912..cd8fd5be35 100644 --- a/.gitignore +++ b/.gitignore @@ -307,6 +307,9 @@ __pycache__/ *.exe *.zip +# CI failure analysis output (generated by analyze-ci-failures skill) +/ci-failure-analysis/ + ############################################################ # vcpkg - End ############################################################