mirror of https://github.com/microsoft/vcpkg.git
[skills][analyze-ci-failures] Work on PR checks and save failure logs (#51209)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Victor Romero
GitHub
parent
355b2207a5
commit
cfc0e85601
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
name: analyze-ci-failures
|
||||
description: 'Analyze vcpkg Azure DevOps CI build failures. Downloads failure logs, identifies regression root causes, and generates a structured report categorizing build errors by package, triplet, and failure type.'
|
||||
argument-hint: 'Azure DevOps build URL (e.g., "https://dev.azure.com/vcpkg/public/_build/results?buildId=129315")'
|
||||
argument-hint: 'Azure DevOps build URL or GitHub PR URL (e.g., "https://dev.azure.com/vcpkg/public/_build/results?buildId=129315" or "https://github.com/microsoft/vcpkg/pull/51134")'
|
||||
---
|
||||
|
||||
# vcpkg CI Failures Analyzer
|
||||
|
|
@ -27,7 +27,9 @@ This skill fetches build metadata and failure logs directly from the Azure DevOp
|
|||
|
||||
### Step 1: Parse the Build URL
|
||||
|
||||
Given a URL such as:
|
||||
The skill accepts either an **Azure DevOps build URL** or a **GitHub PR URL** as input.
|
||||
|
||||
**Azure DevOps build URL:**
|
||||
```
|
||||
https://dev.azure.com/vcpkg/public/_build/results?buildId=129315&view=results
|
||||
```
|
||||
|
|
@ -37,6 +39,16 @@ Extract:
|
|||
- **project**: `public`
|
||||
- **buildId**: `129315`
|
||||
|
||||
**GitHub PR URL:**
|
||||
```
|
||||
https://github.com/microsoft/vcpkg/pull/51134
|
||||
```
|
||||
|
||||
Extract the PR number, then use the GitHub API to find the associated Azure DevOps build:
|
||||
1. Fetch the PR's check runs: `GET /repos/microsoft/vcpkg/pulls/{pr_number}/check-runs` (use `github-mcp-server-pull_request_read` with method `get_check_runs`)
|
||||
2. Find the check run named `microsoft.vcpkg.pr` — its `details_url` contains the Azure DevOps build URL with `buildId`
|
||||
3. Extract the `buildId` from that URL and proceed with the normal workflow
|
||||
|
||||
The REST API base is: `https://dev.azure.com/{organization}/{project}/_apis`
|
||||
|
||||
### Step 2: Fetch Build Metadata
|
||||
|
|
@ -50,11 +62,13 @@ Key fields in the response:
|
|||
- `result` — `succeeded`, `failed`, `partiallySucceeded`, `canceled`
|
||||
- `reason` — `pullRequest`, `schedule`, `manual`, `individualCI`
|
||||
- `requestedFor.displayName` — who triggered the build
|
||||
- `sourceBranch` — branch or PR reference
|
||||
- `sourceBranch` — branch or PR reference (e.g., `refs/pull/51202/merge` for PR builds)
|
||||
- `sourceVersion` — commit SHA
|
||||
- `_links.web.href` — canonical link back to the build page
|
||||
- `finishTime` — when it completed
|
||||
|
||||
**For PR-triggered builds** (`reason == "pullRequest"`): extract the PR number from `sourceBranch` (e.g., `refs/pull/51202/merge` → PR #51202). Use the GitHub API to fetch the PR title and author — include these in the report header for context, and link to the PR with `https://github.com/microsoft/vcpkg/pull/{pr_number}`. The PR is on the `microsoft/vcpkg` repository (not `vicroms/vcpkg`).
|
||||
|
||||
**Get the build timeline (all jobs and tasks):**
|
||||
```
|
||||
GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/timeline?api-version=7.0
|
||||
|
|
@ -94,6 +108,7 @@ REGRESSION: kf6itemmodels:x64-windows failed with FILE_CONFLICTS. If expected, a
|
|||
| Failure type | Also has artifact log? | Notes |
|
||||
|---|---|---|
|
||||
| `BUILD_FAILED` | ✅ Yes | Full build logs in "failure logs for {triplet}" artifact |
|
||||
| `POST_BUILD_CHECKS_FAILED` | ✅ Yes | Build succeeded but vcpkg post-install validation failed (misplaced CMake files, missing usage, etc.) |
|
||||
| `FILE_CONFLICTS` | ❌ No | Port installs files that conflict with another port; only in step log |
|
||||
| `MISSING_FROM_BASELINE` | ❌ No | Port passed but is listed as `fail` in baseline; no artifact |
|
||||
| `CASCADE_BUILD_FAILED` | Sometimes | Dependency failure; root port has artifact, downstream may not |
|
||||
|
|
@ -146,21 +161,56 @@ Start with the smallest artifacts (fastest feedback). Android triplets typically
|
|||
|
||||
### Step 4: Download Failure Log Artifacts
|
||||
|
||||
**`web_fetch` cannot download binary ZIP files.** Use PowerShell with `Invoke-WebRequest` instead:
|
||||
**`web_fetch` cannot download binary ZIP files.** Use PowerShell with `Invoke-WebRequest` instead.
|
||||
|
||||
**Output directory:** All downloaded artifacts and the final report are saved in a persistent directory under the repository root for manual review:
|
||||
|
||||
```
|
||||
ci-failure-analysis/
|
||||
├── pr-51196/ ← PR-triggered build (named by PR number)
|
||||
│ ├── logs/ ← extracted failure log artifacts
|
||||
│ │ ├── x64-linux/
|
||||
│ │ │ └── failure logs for x64-linux/
|
||||
│ │ │ ├── ffmpeg/
|
||||
│ │ │ └── ...
|
||||
│ │ ├── x64-windows-static/
|
||||
│ │ └── ...
|
||||
│ └── report.md ← the generated analysis report
|
||||
├── ci-130500/ ← scheduled/manual build (named by build ID)
|
||||
│ ├── logs/
|
||||
│ └── report.md
|
||||
└── ...
|
||||
```
|
||||
|
||||
**Naming convention:**
|
||||
- **PR-triggered builds** (`reason == "pullRequest"`): `pr-{pr_number}` (e.g., `pr-51196`)
|
||||
- **All other builds** (scheduled, manual, individualCI): `ci-{buildId}` (e.g., `ci-130500`)
|
||||
|
||||
This directory is listed in `.gitignore` and will not be committed. It allows reviewers to manually inspect the raw logs alongside the report.
|
||||
|
||||
```powershell
|
||||
$tmpDir = "$env:TEMP\vcpkg-ci-{buildId}"
|
||||
New-Item -ItemType Directory -Path $tmpDir -Force | Out-Null
|
||||
# Determine the output directory name
|
||||
if ($build.reason -eq 'pullRequest') {
|
||||
$prNumber = ($build.sourceBranch -replace 'refs/pull/(\d+)/merge','$1')
|
||||
$analysisDir = "$repoRoot\ci-failure-analysis\pr-$prNumber"
|
||||
} else {
|
||||
$analysisDir = "$repoRoot\ci-failure-analysis\ci-$buildId"
|
||||
}
|
||||
$logsDir = "$analysisDir\logs"
|
||||
New-Item -ItemType Directory -Path $logsDir -Force | Out-Null
|
||||
|
||||
# Download the artifact ZIP using the downloadUrl from the artifacts API response
|
||||
$downloadUrl = "{resource.downloadUrl from artifacts API}"
|
||||
Invoke-WebRequest -Uri $downloadUrl -OutFile "$tmpDir\{triplet}.zip" -UseBasicParsing
|
||||
Invoke-WebRequest -Uri $downloadUrl -OutFile "$logsDir\{triplet}.zip" -UseBasicParsing
|
||||
|
||||
# Extract
|
||||
Expand-Archive "$tmpDir\{triplet}.zip" -DestinationPath "$tmpDir\{triplet}" -Force
|
||||
Expand-Archive "$logsDir\{triplet}.zip" -DestinationPath "$logsDir\{triplet}" -Force
|
||||
|
||||
# Remove the ZIP after extraction to save space
|
||||
Remove-Item "$logsDir\{triplet}.zip" -Force
|
||||
|
||||
# List failed ports — each is a subdirectory inside "failure logs for {triplet}/"
|
||||
$root = "$tmpDir\{triplet}\failure logs for {triplet}"
|
||||
$root = "$logsDir\{triplet}\failure logs for {triplet}"
|
||||
Get-ChildItem $root -Directory | Select-Object -ExpandProperty Name
|
||||
```
|
||||
|
||||
|
|
@ -424,16 +474,21 @@ When asked to produce the report, default to the full structured markdown report
|
|||
|
||||
### Saving the Report
|
||||
|
||||
After generating the report, **always save it as a markdown file** in the repository root:
|
||||
After generating the report, **always save it as `report.md`** inside the analysis output directory:
|
||||
|
||||
```
|
||||
ci-report-{buildId}.md
|
||||
ci-failure-analysis/{pr-NNN|ci-NNN}/report.md
|
||||
```
|
||||
|
||||
Use the `create` tool to write the full report content to this path. This ensures the report is persisted and can be easily reviewed or shared.
|
||||
Use the `create` tool to write the full report content to this path. The report lives alongside the downloaded failure logs, making it easy to cross-reference the analysis with the raw log files.
|
||||
|
||||
**Examples:**
|
||||
- PR #51196 analysis → `ci-failure-analysis/pr-51196/report.md`
|
||||
- Scheduled build #129315 → `ci-failure-analysis/ci-129315/report.md`
|
||||
|
||||
## Important Notes
|
||||
|
||||
- **vcpkg only supports `>=` version constraints** — there is no way to express `<=` or `<` constraints. Never suggest "add a version constraint to prevent upgrading to version X" as a fix; it is not possible. Instead, recommend updating the consuming port to be compatible with the new version, or patching the port.
|
||||
- The vcpkg Azure DevOps project (`vcpkg/public`) is **publicly accessible** — no authentication token is required for artifact downloads or build/artifact API calls
|
||||
- **`web_fetch` cannot download binary ZIP files** — use PowerShell `Invoke-WebRequest` for artifact downloads
|
||||
- **`resource.type` is `"PipelineArtifact"`**, not `"Container"` — the Container listing API (`/_apis/resources/Containers/{id}`) does not work; use `downloadUrl` directly
|
||||
|
|
@ -445,4 +500,8 @@ Use the `create` tool to write the full report content to this path. This ensure
|
|||
- The `"format.diff"` artifact (if present) indicates formatting or version file issues, not port build failures — report these separately
|
||||
- `"z azcopy logs"` artifacts are infrastructure logs; skip these unless diagnosing asset cache issues
|
||||
- **Android artifact sizes (~115–123 MB) typically share root causes with Linux** — if Linux analysis reveals a systemic issue (e.g., compiler header change, Python version), assume Android is affected too without full re-analysis
|
||||
- **Clean up the temp directory** (`$env:TEMP\vcpkg-ci-{buildId}`) after analysis to avoid disk clutter
|
||||
- **Downloaded logs are kept for manual review** in `ci-failure-analysis/` (gitignored). ZIP files are removed after extraction to save space, but the extracted logs and report are preserved. If disk space is a concern, old analysis directories can be deleted manually.
|
||||
- **Case-sensitive file paths**: Linux and Android file systems are case-sensitive; Windows and macOS are not. A portfile referencing `${SOURCE_PATH}/LICENSE` will succeed on Windows/macOS even if the upstream file is named `License` or `license`, but will fail with "No such file or directory" on Linux/Android. When a BUILD_FAILED occurs only on Linux/Android with a missing file error, always check for a case mismatch against the actual upstream filename.
|
||||
- **Portfile anti-patterns — do NOT recommend these as fixes:**
|
||||
- `set(VCPKG_BUILD_TYPE release)` — this is **only** appropriate for header-only libraries. Ports that produce binary output (`.lib`, `.a`, `.dll`, `.so`) must build both debug and release configurations. Never suggest adding this to fix mismatched-binary warnings.
|
||||
- `set(VCPKG_POLICY_* enabled)` — policy overrides are escape hatches for exceptional cases, not standard practice. Most ports should not set any `VCPKG_POLICY_*` variable. When a post-build check fails, the correct fix is almost always to fix the underlying issue (e.g., correct `vcpkg_cmake_config_fixup()` arguments, install missing files), not to suppress the warning with a policy override.
|
||||
|
|
|
|||
|
|
@ -15,6 +15,10 @@ Use it as the exact template when generating reports.
|
|||
**Duration:** {startTime} → {finishTime} UTC (~{duration})
|
||||
**Result:** ❌ **FAILED** — {baseline_summary}
|
||||
|
||||
<!-- For PR builds: include a line linking to the PR with its title and author, e.g.:
|
||||
**PR:** [#51202](https://github.com/microsoft/vcpkg/pull/51202) — `[ltla-cppirlba] Update to 3.1.0` by @xiaozhuai
|
||||
-->
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
|
@ -134,7 +138,12 @@ Use this as a reference for tone, depth, and structure.
|
|||
5. **Baseline ports**: Ports that explicitly reject a platform by design (`MSVC is not supported`, environment requirements) go to the Baseline section, not regressions
|
||||
6. **Dependency chain**: If port A fails because port B failed, list only port B as the regression with a note "causes N downstream failures: portA, portC, …"
|
||||
7. **Transient failures**: Download failures (curl errors, SHA mismatches) and exit-code-127 failures should be flagged as potentially transient
|
||||
8. **Priority levels**:
|
||||
8. **Version constraints**: Never suggest adding a `<=` version constraint as a fix — vcpkg only supports `>=` constraints. For API-breaking dependency updates, suggest updating the consuming port or patching it instead.
|
||||
9. **PR context**: For PR-triggered builds, always include a link to the PR, the PR title, and the author in the report header.
|
||||
10. **Portfile anti-patterns**: Never recommend `set(VCPKG_BUILD_TYPE release)` for ports that produce binaries — it is only for header-only libraries. Never recommend adding `VCPKG_POLICY_*` overrides to suppress post-build check warnings — fix the underlying issue instead. Most ports should not set any policy variable.
|
||||
11. **Case-sensitivity**: When a file-not-found error occurs only on Linux/Android but the port passes on Windows/macOS, always verify the exact filename casing in the upstream repository before concluding the file is missing.
|
||||
12. **`vcpkg_cmake_config_fixup` package name**: The `PACKAGE_NAME` argument must match the upstream CMake export name (what consumers pass to `find_package()`), which may differ from the vcpkg port name.
|
||||
13. **Priority levels**:
|
||||
- 🔴 **Immediate** — Core/popular ports, failures on 3+ triplets, or CI-environment regressions affecting many ports
|
||||
- 🟠 **High** — Failures on 2+ triplets, popular ports, or clear upstream API breaks
|
||||
- 🟡 **Medium** — Single-triplet failures with clear fixes
|
||||
|
|
|
|||
|
|
@ -119,6 +119,12 @@ error: 'header.h': No such file or directory
|
|||
- **API deprecation**: Uses a function/class removed in a newer version of a dependency
|
||||
- **Compiler flag conflict**: vcpkg sets `-Werror` or `/WX`; warning-as-error triggers on upstream code
|
||||
|
||||
**Suggested fix guidance for API deprecation failures:**
|
||||
When a dependency update removes or renames an API that downstream ports still use:
|
||||
1. Update the downstream port to a version compatible with the new API
|
||||
2. Patch the downstream port's source to adapt to the new API
|
||||
- **Do NOT suggest adding a `<=` version constraint** — vcpkg only supports `>=` version constraints; upper-bound pinning is not possible
|
||||
|
||||
---
|
||||
|
||||
## Category 4: Linker Errors
|
||||
|
|
@ -191,7 +197,7 @@ error: curl: (6) Could not resolve host: example.com
|
|||
|
||||
## Category 7: Post-Build Check Failures
|
||||
|
||||
**Symptom**: Build and install succeed, but vcpkg's post-install checks fail.
|
||||
**Symptom**: Build and install succeed, but vcpkg's post-install checks fail. Reported as `POST_BUILD_CHECKS_FAILED` in the REGRESSION line.
|
||||
|
||||
```
|
||||
error: The following files are not installed:
|
||||
|
|
@ -203,12 +209,23 @@ error: Found forbidden pattern in installed file:
|
|||
error: Usage of deprecated vcpkg function 'vcpkg_apply_patches'
|
||||
```
|
||||
|
||||
**Common post-build checks:**
|
||||
- Copyright file must exist at `share/{portname}/copyright`
|
||||
- No `.pdb` files in release packages (debug-only)
|
||||
- Headers must be in `include/`, not `include/{portname}/include/`
|
||||
- CMake config files in `share/{portname}/` not `lib/cmake/`
|
||||
- No absolute paths in installed CMake files
|
||||
**Common post-build checks and their fixes:**
|
||||
|
||||
| Check | Warning message | Correct fix |
|
||||
|---|---|---|
|
||||
| **Misplaced CMake config files** | `This port installs CMake files in places CMake files are not expected` | Fix `vcpkg_cmake_config_fixup()` — use the correct `PACKAGE_NAME` matching the upstream CMake export name (not necessarily the vcpkg port name), and set `CONFIG_PATH` to where the upstream installs its config (e.g., `lib/cmake/pkgname`) |
|
||||
| **`lib/cmake` not merged** | `should be merged and moved to share/${PORT}/cmake` | Fixed by correcting `vcpkg_cmake_config_fixup()` as above |
|
||||
| **Usage file not installed** | `this port contains a file named "usage" but didn't install it` | Add `file(INSTALL "${CMAKE_CURRENT_LIST_DIR}/usage" DESTINATION "${CURRENT_PACKAGES_DIR}/share/${PORT}")` |
|
||||
| **Mismatched debug/release binaries** | `mismatching number of debug and release binaries` | Ensure the port builds both debug and release. Do **not** fix by adding `VCPKG_POLICY_MISMATCHED_NUMBER_OF_BINARIES` or `VCPKG_BUILD_TYPE release` (see portfile anti-patterns below) |
|
||||
| **Missing copyright** | `share/portname/copyright not found` | Add `vcpkg_install_copyright(FILE_LIST "${SOURCE_PATH}/LICENSE")` with the correct filename and case |
|
||||
| **Absolute paths in CMake files** | `absolute path found in CMake config` | Use `vcpkg_cmake_config_fixup()` which handles path rewriting |
|
||||
| **No `.pdb` in release** | `.pdb files in release packages` | Ensure PDB files only install for debug configuration |
|
||||
|
||||
**Portfile anti-patterns (do NOT recommend as fixes):**
|
||||
- `set(VCPKG_BUILD_TYPE release)` — only appropriate for **header-only** libraries. Ports producing binaries (`.lib`, `.a`, `.dll`, `.so`) must build both debug and release.
|
||||
- `set(VCPKG_POLICY_* enabled)` — policy overrides are escape hatches for exceptional cases. The correct fix is to resolve the underlying issue, not suppress the warning.
|
||||
|
||||
**Key insight for `vcpkg_cmake_config_fixup()`**: The `PACKAGE_NAME` argument must match the **upstream CMake export name** (what `find_package()` looks for), which is often different from the vcpkg port name. For example, a port named `minecraft-server-status` whose upstream exports `minecraft_api` needs `vcpkg_cmake_config_fixup(PACKAGE_NAME "minecraft_api" CONFIG_PATH "lib/cmake/minecraft_api")`.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -321,6 +338,7 @@ portname=pass # (same as not listed) must succeed
|
|||
- `x64-osx`
|
||||
- `x64-uwp`
|
||||
- `x64-windows`
|
||||
- `x64-windows-release`
|
||||
- `x64-windows-static`
|
||||
- `x64-windows-static-md`
|
||||
- `x86-windows`
|
||||
|
|
@ -329,6 +347,28 @@ portname=pass # (same as not listed) must succeed
|
|||
|
||||
---
|
||||
|
||||
## Category 13: Case-Sensitivity Path Failures (Linux/Android only)
|
||||
|
||||
**Symptom**: A file-not-found error occurs on Linux and Android triplets, but the same port builds successfully on Windows and macOS.
|
||||
|
||||
```
|
||||
CMake Error at scripts/cmake/vcpkg_install_copyright.cmake:21 (file):
|
||||
file INSTALL cannot find ".../src/v1.0.0/LICENSE": No such file or directory.
|
||||
```
|
||||
|
||||
**Root cause**: The portfile references a filename with different casing than the actual file in the upstream repository. Linux and Android use case-sensitive file systems (`LICENSE` ≠ `License` ≠ `license`), while Windows and macOS are case-insensitive and will resolve any casing variant.
|
||||
|
||||
**Common occurrences:**
|
||||
- License files: `LICENSE` vs `License` vs `license` vs `COPYING`
|
||||
- README files: `README.md` vs `Readme.md` vs `readme.md`
|
||||
- Source directories with inconsistent casing
|
||||
|
||||
**Diagnostic clue**: If a port fails with BUILD_FAILED on all Linux/Android triplets but succeeds (or fails for a *different* reason like POST_BUILD_CHECKS_FAILED) on Windows/macOS, check for a case mismatch in file paths used in the portfile.
|
||||
|
||||
**Fix**: Correct the filename in `portfile.cmake` to match the exact casing in the upstream repository. Verify by checking the upstream repo's file listing.
|
||||
|
||||
---
|
||||
|
||||
## Regression Severity Levels
|
||||
|
||||
Use these levels when reporting failures:
|
||||
|
|
|
|||
|
|
@ -307,6 +307,9 @@ __pycache__/
|
|||
*.exe
|
||||
*.zip
|
||||
|
||||
# CI failure analysis output (generated by analyze-ci-failures skill)
|
||||
/ci-failure-analysis/
|
||||
|
||||
############################################################
|
||||
# vcpkg - End
|
||||
############################################################
|
||||
|
|
|
|||
Loading…
Reference in New Issue