mirror of https://github.com/microsoft/vcpkg.git
[skill] Add skill to analyze CI failures (#51188)
This commit is contained in:
Victor Romero
GitHub
parent
d034cf165b
commit
12dc3f54ca
|
|
@ -0,0 +1,448 @@
|
|||
---
|
||||
name: analyze-ci-failures
|
||||
description: 'Analyze vcpkg Azure DevOps CI build failures. Downloads failure logs, identifies regression root causes, and generates a structured report categorizing build errors by package, triplet, and failure type.'
|
||||
argument-hint: 'Azure DevOps build URL (e.g., "https://dev.azure.com/vcpkg/public/_build/results?buildId=129315")'
|
||||
---
|
||||
|
||||
# vcpkg CI Failures Analyzer
|
||||
|
||||
## When to Use
|
||||
|
||||
- Investigating why a CI build failed on Azure DevOps
|
||||
- Identifying which packages regressed in a PR or scheduled build
|
||||
- Triaging root causes before assigning bugs or reverting changes
|
||||
- Getting a quick summary of failures across all triplets in one build
|
||||
|
||||
## Overview
|
||||
|
||||
This skill fetches build metadata and failure logs directly from the Azure DevOps REST API, cross-references them with `scripts/ci.baseline.txt`, and produces a structured regression report:
|
||||
|
||||
- **Build Summary**: Overall build result, trigger type (PR vs scheduled), and which jobs (triplets) failed
|
||||
- **Artifact Download**: Fetches every "failure logs for *" artifact published by the CI pipeline
|
||||
- **Log Analysis**: Parses per-package failure logs to extract compiler errors, missing dependencies, linker errors, and other root causes
|
||||
- **Regression Classification**: Distinguishes new regressions from known failures already listed in `ci.baseline.txt`
|
||||
- **Structured Report**: Organized by triplet, then package, with concise root-cause summaries
|
||||
|
||||
## Detailed Workflow
|
||||
|
||||
### Step 1: Parse the Build URL
|
||||
|
||||
Given a URL such as:
|
||||
```
|
||||
https://dev.azure.com/vcpkg/public/_build/results?buildId=129315&view=results
|
||||
```
|
||||
|
||||
Extract:
|
||||
- **organization**: `vcpkg`
|
||||
- **project**: `public`
|
||||
- **buildId**: `129315`
|
||||
|
||||
The REST API base is: `https://dev.azure.com/{organization}/{project}/_apis`
|
||||
|
||||
### Step 2: Fetch Build Metadata
|
||||
|
||||
**Get overall build info:**
|
||||
```
|
||||
GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}?api-version=7.0
|
||||
```
|
||||
Key fields in the response:
|
||||
- `status` — `completed`, `inProgress`, etc.
|
||||
- `result` — `succeeded`, `failed`, `partiallySucceeded`, `canceled`
|
||||
- `reason` — `pullRequest`, `schedule`, `manual`, `individualCI`
|
||||
- `requestedFor.displayName` — who triggered the build
|
||||
- `sourceBranch` — branch or PR reference
|
||||
- `sourceVersion` — commit SHA
|
||||
- `_links.web.href` — canonical link back to the build page
|
||||
- `finishTime` — when it completed
|
||||
|
||||
**Get the build timeline (all jobs and tasks):**
|
||||
```
|
||||
GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/timeline?api-version=7.0
|
||||
```
|
||||
The response contains a `records` array. Each record has:
|
||||
- `type` — `"Job"`, `"Task"`, `"Stage"`, `"Phase"`
|
||||
- `name` — display name (e.g., `"x64_windows"`, `"*** Test Modified Ports"`)
|
||||
- `result` — `"succeeded"`, `"failed"`, `"skipped"`, `"canceled"`
|
||||
- `state` — `"completed"`, `"inProgress"`, `"pending"`
|
||||
- `log.url` — URL to the raw task log (for individual task logs)
|
||||
- `parentId` — parent record ID (tasks belong to jobs)
|
||||
|
||||
Filter for `type == "Job"` and `result == "failed"` to get a list of failing triplets.
|
||||
|
||||
### Step 2b: Scan "Test Modified Ports" Step Logs for REGRESSION Lines
|
||||
|
||||
**This step is required** — some failure types (notably `FILE_CONFLICTS`) are only reported in the step console output and do **not** produce entries in the failure log artifacts. Skipping this step will miss those regressions entirely.
|
||||
|
||||
For each failed job in the timeline, find the task named `"*** Test Modified Ports"` (type `"Task"`, child of that job) and fetch its log:
|
||||
|
||||
```
|
||||
GET {record.log.url}?api-version=7.0
|
||||
```
|
||||
|
||||
The log is plain text. Scan it for lines matching the pattern:
|
||||
```
|
||||
REGRESSION: {port}:{triplet} failed with {FAILURE_TYPE}.
|
||||
```
|
||||
|
||||
**Example output from a real build:**
|
||||
```
|
||||
REGRESSION: kf6i18n:x64-windows failed with FILE_CONFLICTS. If expected, add kf6i18n:x64-windows=fail to D:\a\_work\1\s\scripts\azure-pipelines/../ci.baseline.txt.
|
||||
REGRESSION: kf6itemmodels:x64-windows failed with FILE_CONFLICTS. If expected, add kf6itemmodels:x64-windows=fail to D:\a\_work\1\s\scripts\azure-pipelines/../ci.baseline.txt.
|
||||
```
|
||||
|
||||
**Known `FAILURE_TYPE` values found in this log:**
|
||||
| Failure type | Also has artifact log? | Notes |
|
||||
|---|---|---|
|
||||
| `BUILD_FAILED` | ✅ Yes | Full build logs in "failure logs for {triplet}" artifact |
|
||||
| `FILE_CONFLICTS` | ❌ No | Port installs files that conflict with another port; only in step log |
|
||||
| `MISSING_FROM_BASELINE` | ❌ No | Port passed but is listed as `fail` in baseline; no artifact |
|
||||
| `CASCADE_BUILD_FAILED` | Sometimes | Dependency failure; root port has artifact, downstream may not |
|
||||
|
||||
**PowerShell to extract all REGRESSION lines from a job's "Test Modified Ports" step:**
|
||||
```powershell
|
||||
# Get the timeline to find log URLs
|
||||
$timeline = Invoke-RestMethod "https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/timeline?api-version=7.0"
|
||||
|
||||
# For each failed job, find its "Test Modified Ports" task
|
||||
$failedJobs = $timeline.records | Where-Object { $_.type -eq 'Job' -and $_.result -eq 'failed' }
|
||||
foreach ($job in $failedJobs) {
|
||||
$testTask = $timeline.records | Where-Object {
|
||||
$_.type -eq 'Task' -and
|
||||
$_.parentId -eq $job.id -and
|
||||
$_.name -like '*Test Modified Ports*'
|
||||
}
|
||||
if ($testTask -and $testTask.log.url) {
|
||||
$logText = Invoke-RestMethod "$($testTask.log.url)?api-version=7.0"
|
||||
$regressions = $logText -split "`n" | Where-Object { $_ -match '^REGRESSION:' }
|
||||
$regressions | ForEach-Object { Write-Host "$($job.name): $_" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Collect all `REGRESSION:` lines from all jobs into a master list before proceeding to artifact download. This list is your **ground truth** for what failed — the failure log artifacts are supplementary evidence for diagnosing *why*.
|
||||
|
||||
### Step 3: List Published Artifacts
|
||||
|
||||
```
|
||||
GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/artifacts?api-version=7.0
|
||||
```
|
||||
Each artifact in `value[]` has:
|
||||
- `name` — e.g., `"failure logs for x64-windows"`, `"file lists for x64-windows"`, `"format.diff"`
|
||||
- `resource.type` — **`"PipelineArtifact"`** (not `"Container"` — the Container listing API does **not** work for these)
|
||||
- `resource.downloadUrl` — direct ZIP download URL hosted on `artprodwus21.artifacts.visualstudio.com`; no auth required for public builds
|
||||
|
||||
**Focus on artifacts whose name starts with `"failure logs for"`** — these contain one subdirectory per failing port.
|
||||
|
||||
**Prioritize by compressed artifact size** (from `resource.properties.artifactsize`):
|
||||
|
||||
| Compressed size | Interpretation | Analysis depth |
|
||||
|---|---|---|
|
||||
| < 10 MB | Few failures (< ~15 ports) | Analyze all ports fully |
|
||||
| 10–50 MB | Moderate failures | Sample all ports, deep-dive key ones |
|
||||
| 50–120 MB | Many failures | Focus on unique error categories only |
|
||||
| > 120 MB | Mass failures — likely a systemic issue | Find the common root, skip per-port deep-dive |
|
||||
|
||||
Start with the smallest artifacts (fastest feedback). Android triplets typically produce 115–123 MB artifacts and share root causes with Linux.
|
||||
|
||||
### Step 4: Download Failure Log Artifacts
|
||||
|
||||
**`web_fetch` cannot download binary ZIP files.** Use PowerShell with `Invoke-WebRequest` instead:
|
||||
|
||||
```powershell
|
||||
$tmpDir = "$env:TEMP\vcpkg-ci-{buildId}"
|
||||
New-Item -ItemType Directory -Path $tmpDir -Force | Out-Null
|
||||
|
||||
# Download the artifact ZIP using the downloadUrl from the artifacts API response
|
||||
$downloadUrl = "{resource.downloadUrl from artifacts API}"
|
||||
Invoke-WebRequest -Uri $downloadUrl -OutFile "$tmpDir\{triplet}.zip" -UseBasicParsing
|
||||
|
||||
# Extract
|
||||
Expand-Archive "$tmpDir\{triplet}.zip" -DestinationPath "$tmpDir\{triplet}" -Force
|
||||
|
||||
# List failed ports — each is a subdirectory inside "failure logs for {triplet}/"
|
||||
$root = "$tmpDir\{triplet}\failure logs for {triplet}"
|
||||
Get-ChildItem $root -Directory | Select-Object -ExpandProperty Name
|
||||
```
|
||||
|
||||
**Confirmed ZIP structure (from build #129315):**
|
||||
```
|
||||
failure logs for x64-windows.zip
|
||||
└── failure logs for x64-windows/ ← top-level folder = artifact name
|
||||
├── boost-filesystem/ ← one subdirectory per failing port
|
||||
│ ├── stdout-x64-windows.log ← PRIMARY: full vcpkg output, final error
|
||||
│ ├── config-x64-windows-out.log ← CMake configure stdout
|
||||
│ ├── config-x64-windows-err.log ← CMake configure stderr
|
||||
│ ├── config-x64-windows-dbg-CMakeConfigureLog.yaml.log
|
||||
│ ├── config-x64-windows-dbg-CMakeCache.txt.log
|
||||
│ ├── config-x64-windows-dbg-ninja.log
|
||||
│ ├── build-x64-windows-dbg-out.log ← compiler stdout (can be very large)
|
||||
│ ├── build-x64-windows-dbg-err.log ← compiler stderr
|
||||
│ ├── install-x64-windows-dbg-out.log
|
||||
│ ├── install-x64-windows-dbg-err.log
|
||||
│ ├── patch-x64-windows-0-err.log ← patch application errors (numbered)
|
||||
│ ├── patch-x64-windows-0-out.log
|
||||
│ ├── extract-out.log / extract-err.log
|
||||
│ └── generate-x64-windows-dbg-err.log ← GN builds (v8, crashpad) only
|
||||
├── curl/
|
||||
├── openssl/
|
||||
└── ...
|
||||
```
|
||||
|
||||
**For Meson ports (e.g., mesa):**
|
||||
```
|
||||
mesa/
|
||||
├── cmake-get-vars_C_CXX-x64-linux-dbg.cmake.log
|
||||
├── config-x64-linux-dbg-meson-log.txt.log
|
||||
├── meson-x64-linux-dbg.log
|
||||
├── pip-install-packages-x64-linux-err.log
|
||||
├── venv-setup-x64-linux-err.log
|
||||
└── stdout-x64-linux.log
|
||||
```
|
||||
|
||||
**For autotools ports (e.g., libxt):**
|
||||
```
|
||||
libxt/
|
||||
├── autoconf-x64-windows-err.log
|
||||
├── build-x64-windows-dbg-err.log ← compiler warnings/errors
|
||||
├── build-x64-windows-dbg-out.log
|
||||
└── stdout-x64-windows.log
|
||||
```
|
||||
|
||||
### Step 5: Analyze Log Content — Two-Phase Approach
|
||||
|
||||
**Phase 1: Classify all failures quickly (one pass)**
|
||||
|
||||
For each failing port, read only `stdout-{triplet}.log` last 20 lines:
|
||||
```powershell
|
||||
$ports = Get-ChildItem $root -Directory | Sort-Object Name
|
||||
foreach ($port in $ports) {
|
||||
$stdout = Get-ChildItem $port.FullName -Filter "stdout*" | Select-Object -First 1
|
||||
if ($stdout) {
|
||||
$lines = Get-Content $stdout.FullName -Encoding UTF8
|
||||
# Extract final error line
|
||||
$keyError = ($lines | Where-Object { $_ -match "error:|CMake Error|FAILED" } | Select-Object -Last 1) -replace '^\s+',''
|
||||
Write-Host "$($port.Name): $keyError"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This gives a one-line classification per port. Identify common patterns across ports (many ports → same error = systemic issue).
|
||||
|
||||
**Phase 2: Deep-dive the specific stage log for each port**
|
||||
|
||||
After classifying, choose the right log file based on failure type:
|
||||
|
||||
| Failure type indicated by stdout | Log to read next |
|
||||
|---|---|
|
||||
| `Command failed: cmake --build` (build/install) | `install-{triplet}-dbg-out.log` — grep for `error:` |
|
||||
| `Command failed: ninja -v` (configure) | `config-{triplet}-out.log` — last 20 lines |
|
||||
| `Command failed: ninja -v` (configure) | `config-{triplet}-dbg-CMakeConfigureLog.yaml.log` |
|
||||
| `Command failed: gn gen` or `ninja` (GN) | `generate-{triplet}-dbg-err.log` |
|
||||
| `Command failed: meson setup` | `config-{triplet}-dbg-meson-log.txt.log` |
|
||||
| `Command failed: make` (autotools) | `build-{triplet}-dbg-err.log` |
|
||||
| `Command failed: autoreconf` | `autoconf-{triplet}-err.log` |
|
||||
| Patch error | `patch-{triplet}-N-err.log` |
|
||||
|
||||
**Important**: `install-*-out.log` and `build-*-out.log` can be very large (100KB–2MB+). Always grep for errors rather than reading the full file:
|
||||
```powershell
|
||||
$lines = Get-Content "install-x64-windows-dbg-out.log" -Encoding UTF8
|
||||
$lines | Where-Object { $_ -match "error:" -and $_ -notmatch "^--" } | Select-Object -First 10
|
||||
```
|
||||
|
||||
**Note on empty error logs**: Many `*-err.log` files are empty (0 bytes) — the actual error output goes to stdout for CMake/ninja builds. Always check `*-out.log` if `*-err.log` is empty.
|
||||
|
||||
**Note on test results API**: `GET /_apis/test/runs?buildId=...` requires authentication even for public projects. Skip this API; the failure logs contain sufficient information.
|
||||
|
||||
### Step 5 (continued): Pattern Catalogue
|
||||
|
||||
For each failure log, identify the root cause by scanning for known error patterns.
|
||||
See `references/vcpkg-failure-patterns.md` for a comprehensive pattern catalogue and `references/report-template.md` for a table of the most common high-frequency patterns to check first.
|
||||
|
||||
**Quick triage checklist (scan each log in order):**
|
||||
|
||||
1. **Dependency failure** — the port didn't actually fail itself; a dependency did:
|
||||
```
|
||||
error: package dependency-name:triplet is not installed
|
||||
-- Building dependency-name[core]:triplet failed
|
||||
```
|
||||
Root cause: the dependency listed in the log, not the current package.
|
||||
|
||||
2. **CMake configure error** — dependency not found or incompatible version:
|
||||
```
|
||||
CMake Error: Could not find a package configuration file provided by "XYZ"
|
||||
CMake Error at CMakeLists.txt:NN: find_package(XYZ REQUIRED)
|
||||
```
|
||||
|
||||
3. **Compiler error** — actual source code compile failure:
|
||||
```
|
||||
error C2065: (MSVC)
|
||||
error: use of undeclared identifier (Clang/GCC)
|
||||
fatal error: 'header.h' file not found
|
||||
```
|
||||
|
||||
4. **Linker error**:
|
||||
```
|
||||
error LNK2019: unresolved external symbol (MSVC)
|
||||
undefined reference to (GCC/Clang)
|
||||
```
|
||||
|
||||
5. **SHA512 mismatch** — downloaded archive hash changed:
|
||||
```
|
||||
error: File does not have the expected hash
|
||||
Expected: ...
|
||||
Actual: ...
|
||||
```
|
||||
|
||||
6. **Download failure** — network or URL issue:
|
||||
```
|
||||
error: Failed to download
|
||||
error: curl: (22)
|
||||
```
|
||||
|
||||
7. **Post-build check failure** — installed files don't pass vcpkg checks:
|
||||
```
|
||||
error: The following files are not installed
|
||||
error: Usage of deprecated function
|
||||
```
|
||||
|
||||
8. **Test failure** — port tests failed (check xUnit results):
|
||||
```
|
||||
Test failed: ...
|
||||
FAILED - ...
|
||||
```
|
||||
|
||||
### Step 6: Cross-Reference with CI Baseline
|
||||
|
||||
Read `scripts/ci.baseline.txt` from the local repository to classify each failure:
|
||||
|
||||
```
|
||||
# Format: port-name=fail|skip|pass [# comment]
|
||||
boost-filesystem=fail # needs icu update
|
||||
```
|
||||
|
||||
**Classification logic:**
|
||||
- If the failing port is listed as `fail` or `skip` in the baseline → **Known / Expected** (not a regression)
|
||||
- If the failing port is NOT in the baseline, or listed as `pass` → **Regression** (new failure)
|
||||
- If a port that is expected to `fail` is now succeeding → **Unexpected Pass** (may need baseline update)
|
||||
|
||||
### Step 7: Generate the Regression Report
|
||||
|
||||
Produce a structured markdown report with the following sections:
|
||||
|
||||
---
|
||||
|
||||
**Template:**
|
||||
|
||||
```markdown
|
||||
# vcpkg CI Failure Report
|
||||
|
||||
**Build:** [#{buildId}]({build_web_url})
|
||||
**Triggered by:** {reason} — {requestedFor} on `{sourceBranch}` @ `{sourceVersion[:8]}`
|
||||
**Result:** {result} | **Finished:** {finishTime}
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Triplet | Status | New Regressions | Known Failures |
|
||||
|---------|--------|-----------------|----------------|
|
||||
| x64-windows | ❌ Failed | 3 | 1 |
|
||||
| x64-linux | ✅ Passed | 0 | 0 |
|
||||
| ... | | | |
|
||||
|
||||
---
|
||||
|
||||
## 🔴 New Regressions (Action Required)
|
||||
|
||||
### x64-windows
|
||||
|
||||
#### `portname` — Root Cause Category
|
||||
- **Error:** `<first relevant error line>`
|
||||
- **File:** `path/to/file.cpp:42`
|
||||
- **Analysis:** Short description of why this likely broke.
|
||||
- **Suggested fix:** ...
|
||||
|
||||
...
|
||||
|
||||
---
|
||||
|
||||
## 🟡 Known / Expected Failures (Baseline)
|
||||
|
||||
| Port | Triplet | Baseline Entry |
|
||||
|------|---------|----------------|
|
||||
| portname | x64-windows | `fail # reason` |
|
||||
...
|
||||
|
||||
---
|
||||
|
||||
## ℹ️ Notes
|
||||
|
||||
- Dependency chain failures: {n} ports failed due to a common upstream failure in `dep-name`
|
||||
- Download failures: {n} (may be transient network issues)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 8: Dependency Chain Deduplication
|
||||
|
||||
Before listing regressions, collapse dependency chains:
|
||||
|
||||
- If `libA` fails and `libB`, `libC`, `libD` all fail because `libA` is a dependency, list only `libA` as the regression with a note: "caused {n} downstream failures: libB, libC, libD"
|
||||
- Check dependency info from the log: lines containing `-- Building {dep}:{triplet} failed` indicate the root of the chain
|
||||
|
||||
### Step 9: Transient vs. Persistent Failures
|
||||
|
||||
Flag failures that are likely transient (not actual regressions):
|
||||
- Download failures (network errors, SHA mismatches from upstream file changes)
|
||||
- Timeout failures (`error: build timed out`)
|
||||
- Flaky test failures (same port fails in one triplet but passes in others for unrelated reasons)
|
||||
|
||||
Suggest re-running the build for transient failures rather than filing issues.
|
||||
|
||||
## Quick Start
|
||||
|
||||
**Minimal 3-step process:**
|
||||
1. Provide the build URL: `analyze-ci-failure https://dev.azure.com/vcpkg/public/_build/results?buildId=129315`
|
||||
2. Skill fetches build metadata, downloads all failure log artifacts, reads baseline
|
||||
3. Receive structured regression report
|
||||
|
||||
## Output Format
|
||||
|
||||
When generating the report, follow the exact format defined in `references/report-template.md`.
|
||||
That file contains:
|
||||
- The full markdown template with placeholder variables
|
||||
- A real worked example (build #129315, 24 ports) to calibrate tone and depth
|
||||
- Formatting rules (section order, priority levels, error quoting style)
|
||||
- Artifact size heuristics to guide how deeply to analyze each triplet
|
||||
- A table of common high-frequency patterns to scan for first
|
||||
|
||||
When asked to produce the report, default to the full structured markdown report. You can also produce:
|
||||
- **Summary only**: Just the summary table and count of regressions
|
||||
- **Regressions only**: Skip known failures, show only new regressions
|
||||
- **Single triplet**: Focus on one platform (e.g., "x64-windows only")
|
||||
- **Port-focused**: Group by port name instead of triplet
|
||||
|
||||
### Saving the Report
|
||||
|
||||
After generating the report, **always save it as a markdown file** in the repository root:
|
||||
|
||||
```
|
||||
ci-report-{buildId}.md
|
||||
```
|
||||
|
||||
Use the `create` tool to write the full report content to this path. This ensures the report is persisted and can be easily reviewed or shared.
|
||||
|
||||
## Important Notes
|
||||
|
||||
- The vcpkg Azure DevOps project (`vcpkg/public`) is **publicly accessible** — no authentication token is required for artifact downloads or build/artifact API calls
|
||||
- **`web_fetch` cannot download binary ZIP files** — use PowerShell `Invoke-WebRequest` for artifact downloads
|
||||
- **`resource.type` is `"PipelineArtifact"`**, not `"Container"` — the Container listing API (`/_apis/resources/Containers/{id}`) does not work; use `downloadUrl` directly
|
||||
- **The test results API (`/_apis/test/runs`) requires authentication** even for public projects — skip it; failure logs contain sufficient information
|
||||
- **Many `*-err.log` files are 0 bytes** — CMake, ninja, and make typically write errors to stdout; always check `*-out.log` when `*-err.log` is empty
|
||||
- If a build is still `inProgress`, only partial results will be available; note this in the report
|
||||
- Failure log artifacts are only published when there are actual failures; a missing artifact for a triplet means that triplet passed
|
||||
- **`FILE_CONFLICTS` failures are invisible in artifacts** — they only appear as `REGRESSION:` lines in the "Test Modified Ports" step log; always scan step logs (Step 2b) before concluding the artifact list is complete
|
||||
- The `"format.diff"` artifact (if present) indicates formatting or version file issues, not port build failures — report these separately
|
||||
- `"z azcopy logs"` artifacts are infrastructure logs; skip these unless diagnosing asset cache issues
|
||||
- **Android artifact sizes (~115–123 MB) typically share root causes with Linux** — if Linux analysis reveals a systemic issue (e.g., compiler header change, Python version), assume Android is affected too without full re-analysis
|
||||
- **Clean up the temp directory** (`$env:TEMP\vcpkg-ci-{buildId}`) after analysis to avoid disk clutter
|
||||
|
|
@ -0,0 +1,271 @@
|
|||
# Azure DevOps REST API Reference for vcpkg CI
|
||||
|
||||
## Base URL
|
||||
|
||||
```
|
||||
https://dev.azure.com/vcpkg/public/_apis
|
||||
```
|
||||
|
||||
All endpoints below are appended to this base. The `vcpkg/public` project is **publicly readable** — no Authorization header is required.
|
||||
|
||||
## API Version
|
||||
|
||||
Always append `?api-version=7.0` (or include it in query strings).
|
||||
|
||||
---
|
||||
|
||||
## Build Endpoints
|
||||
|
||||
### Get Build Details
|
||||
|
||||
```
|
||||
GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}?api-version=7.0
|
||||
```
|
||||
|
||||
**Response fields:**
|
||||
```json
|
||||
{
|
||||
"id": 129315,
|
||||
"buildNumber": "20240315.1",
|
||||
"status": "completed",
|
||||
"result": "failed",
|
||||
"reason": "pullRequest",
|
||||
"sourceBranch": "refs/pull/12345/merge",
|
||||
"sourceVersion": "abc123def456...",
|
||||
"requestedFor": { "displayName": "Jane Developer" },
|
||||
"startTime": "2024-03-15T10:00:00Z",
|
||||
"finishTime": "2024-03-15T14:30:00Z",
|
||||
"definition": { "name": "vcpkg.CI" },
|
||||
"_links": {
|
||||
"web": { "href": "https://dev.azure.com/vcpkg/public/_build/results?buildId=129315" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**`result` values:** `succeeded` | `failed` | `partiallySucceeded` | `canceled`
|
||||
**`reason` values:** `pullRequest` | `schedule` | `manual` | `individualCI` | `batchedCI`
|
||||
|
||||
---
|
||||
|
||||
### Get Build Timeline
|
||||
|
||||
```
|
||||
GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/timeline?api-version=7.0
|
||||
```
|
||||
|
||||
Returns all stages, jobs, tasks and their results.
|
||||
|
||||
**Response structure:**
|
||||
```json
|
||||
{
|
||||
"records": [
|
||||
{
|
||||
"id": "guid",
|
||||
"parentId": "parent-guid-or-null",
|
||||
"type": "Job",
|
||||
"name": "x64_windows",
|
||||
"displayName": "x64_windows",
|
||||
"state": "completed",
|
||||
"result": "failed",
|
||||
"startTime": "2024-03-15T10:05:00Z",
|
||||
"finishTime": "2024-03-15T12:15:00Z",
|
||||
"log": {
|
||||
"id": 42,
|
||||
"type": "Container",
|
||||
"url": "https://dev.azure.com/vcpkg/public/_apis/build/builds/129315/logs/42"
|
||||
},
|
||||
"errorCount": 1,
|
||||
"warningCount": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**`type` values:** `Stage` | `Phase` | `Job` | `Task`
|
||||
**`result` values:** `succeeded` | `failed` | `skipped` | `canceled` | `succeededWithIssues`
|
||||
|
||||
**To find failed jobs:**
|
||||
```python
|
||||
failed_jobs = [r for r in records if r['type'] == 'Job' and r['result'] == 'failed']
|
||||
```
|
||||
|
||||
**Job names map to triplets** (underscores in job names, hyphens in triplets):
|
||||
- `x64_windows` → `x64-windows`
|
||||
- `x64_windows_static` → `x64-windows-static`
|
||||
- `arm64_osx` → `arm64-osx`
|
||||
- `x64_linux` → `x64-linux`
|
||||
- `arm_neon_android` → `arm-neon-android`
|
||||
|
||||
---
|
||||
|
||||
### Get a Specific Log
|
||||
|
||||
```
|
||||
GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/logs/{logId}?api-version=7.0
|
||||
```
|
||||
|
||||
Returns plain text of the log. Use this to read the "*** Test Modified Ports" task log, which contains `REGRESSION:` lines for **all** failures including those without artifact logs (e.g., `FILE_CONFLICTS`).
|
||||
|
||||
**Finding the right log ID:** Look up the task record in the timeline response — the `log.id` field on the `"*** Test Modified Ports"` task record gives the log ID. Alternatively, use `log.url` directly.
|
||||
|
||||
**Scanning for REGRESSION lines:**
|
||||
```powershell
|
||||
$logText = Invoke-RestMethod "https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/logs/{logId}?api-version=7.0"
|
||||
$logText -split "`n" | Where-Object { $_ -match '^REGRESSION:' }
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
```
|
||||
REGRESSION: kf6i18n:x64-windows failed with FILE_CONFLICTS. If expected, add kf6i18n:x64-windows=fail to .../ci.baseline.txt.
|
||||
REGRESSION: kf6itemmodels:x64-windows failed with FILE_CONFLICTS. If expected, add kf6itemmodels:x64-windows=fail to .../ci.baseline.txt.
|
||||
REGRESSION: v8:x64-windows failed with BUILD_FAILED. If expected, add v8:x64-windows=fail to .../ci.baseline.txt.
|
||||
```
|
||||
|
||||
> ⚠️ **`FILE_CONFLICTS` failures do NOT produce failure log artifacts.** The only record of them is in this step log. Always scan all "Test Modified Ports" task logs before relying on artifacts alone.
|
||||
|
||||
### List All Logs
|
||||
|
||||
```
|
||||
GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/logs?api-version=7.0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Artifact Endpoints
|
||||
|
||||
### List All Artifacts
|
||||
|
||||
```
|
||||
GET https://dev.azure.com/{org}/{project}/_apis/build/builds/{buildId}/artifacts?api-version=7.0
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"count": 39,
|
||||
"value": [
|
||||
{
|
||||
"id": 1107682,
|
||||
"name": "failure logs for x64-windows",
|
||||
"source": "7922e5c4-...",
|
||||
"resource": {
|
||||
"type": "PipelineArtifact",
|
||||
"data": "DCCD3D9C96CE79A758B255BD87397A07371B8394EE52F6BFF35346C144D63F3501",
|
||||
"properties": {
|
||||
"RootId": "196FDE3B...",
|
||||
"artifactsize": "8586850",
|
||||
"HashType": "DEDUP1024K",
|
||||
"DomainId": "0"
|
||||
},
|
||||
"url": "https://dev.azure.com/vcpkg/.../_apis/build/builds/129315/artifacts?artifactName=failure%20logs%20for%20x64-windows&api-version=7.0",
|
||||
"downloadUrl": "https://artprodwus21.artifacts.visualstudio.com/.../_apis/artifact/{base64hash}/content?format=zip"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
> ⚠️ **`resource.type` is `"PipelineArtifact"`**, not `"Container"`. The Container listing API
|
||||
> (`GET /_apis/resources/Containers/{id}`) does **not** work for these artifacts.
|
||||
> Use `resource.downloadUrl` directly to download the ZIP.
|
||||
|
||||
**Artifact naming conventions:**
|
||||
| Artifact Name Pattern | Contents | Priority |
|
||||
|---|---|---|
|
||||
| `failure logs for {triplet}` | Per-port build failure log directories | **High** — primary source |
|
||||
| `file lists for {triplet}` | Installed file manifests | Low — skip unless needed |
|
||||
| `format.diff` | Formatting/manifest diff | **Medium** — indicates PR format issues |
|
||||
| `z azcopy logs for {triplet}` | Binary cache transfer logs | Low — infrastructure only |
|
||||
|
||||
---
|
||||
|
||||
### Download Artifact ZIP (Confirmed Working)
|
||||
|
||||
Use the `downloadUrl` from the artifact listing. It resolves to an `artprodwus21.artifacts.visualstudio.com` URL. **`web_fetch` cannot download binary ZIPs** — use PowerShell:
|
||||
|
||||
```powershell
|
||||
# $downloadUrl = value from resource.downloadUrl in the artifacts list response
|
||||
Invoke-WebRequest -Uri $downloadUrl -OutFile "$tmpDir\{triplet}.zip" -UseBasicParsing
|
||||
Expand-Archive "$tmpDir\{triplet}.zip" -DestinationPath "$tmpDir\{triplet}" -Force
|
||||
|
||||
# List failing ports — each is a subdirectory inside the artifact folder
|
||||
$root = "$tmpDir\{triplet}\failure logs for {triplet}"
|
||||
Get-ChildItem $root -Directory | Select-Object -ExpandProperty Name
|
||||
```
|
||||
|
||||
The ZIP always contains one top-level folder named exactly `"failure logs for {triplet}"`.
|
||||
Inside that folder, each **failing port has its own subdirectory** named after the port.
|
||||
|
||||
> ❌ The Container API (`GET /_apis/resources/Containers/{containerId}`) does **not** work
|
||||
> for `PipelineArtifact` type. Do not attempt to use it.
|
||||
|
||||
---
|
||||
|
||||
## Test Results Endpoints
|
||||
|
||||
### List Test Runs for a Build
|
||||
|
||||
```
|
||||
GET https://dev.azure.com/{org}/{project}/_apis/test/runs?buildId={buildId}&api-version=7.0
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"value": [
|
||||
{
|
||||
"id": 9876,
|
||||
"name": "x64-windows",
|
||||
"state": "Completed",
|
||||
"totalTests": 500,
|
||||
"passedTests": 497,
|
||||
"failedTests": 3
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Get Test Run Results
|
||||
|
||||
```
|
||||
GET https://dev.azure.com/{org}/{project}/_apis/test/runs/{runId}/results?api-version=7.0&outcomes=Failed
|
||||
```
|
||||
|
||||
Filter by `outcomes=Failed` to get only failures.
|
||||
|
||||
**Response includes:**
|
||||
- `testCaseName` — usually `portname:triplet`
|
||||
- `outcome` — `Failed`
|
||||
- `errorMessage` — short error description
|
||||
- `stackTrace` — full failure output
|
||||
|
||||
---
|
||||
|
||||
## URL Parsing
|
||||
|
||||
Given `https://dev.azure.com/vcpkg/public/_build/results?buildId=129315&view=results`:
|
||||
|
||||
| Component | Value | How to Extract |
|
||||
|-----------|-------|----------------|
|
||||
| Organization | `vcpkg` | 4th path segment |
|
||||
| Project | `public` | 5th path segment |
|
||||
| Build ID | `129315` | `buildId` query parameter |
|
||||
|
||||
```python
|
||||
# Python example
|
||||
from urllib.parse import urlparse, parse_qs
|
||||
url = "https://dev.azure.com/vcpkg/public/_build/results?buildId=129315&view=results"
|
||||
parts = urlparse(url)
|
||||
segments = parts.path.strip('/').split('/')
|
||||
org = segments[0] # "vcpkg"
|
||||
project = segments[1] # "public"
|
||||
build_id = parse_qs(parts.query)['buildId'][0] # "129315"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rate Limits and Auth
|
||||
|
||||
- **Public projects** (like `vcpkg/public`): No authentication needed; anonymous access is allowed
|
||||
- **Rate limit**: ~200 requests per minute for anonymous; increase by adding a PAT as Bearer token if needed
|
||||
- **PAT header**: `Authorization: Basic {base64(":pat_token")}`
|
||||
|
|
@ -0,0 +1,176 @@
|
|||
# vcpkg CI Failure Report Template
|
||||
|
||||
This file provides the canonical output format for the `analyze-ci-failure` skill.
|
||||
Use it as the exact template when generating reports.
|
||||
|
||||
---
|
||||
|
||||
## Report Format
|
||||
|
||||
```markdown
|
||||
# vcpkg CI Failure Report — Build #{buildId}
|
||||
|
||||
**Build:** [{buildNumber}]({build_web_url}) · `{definition_name}`
|
||||
**Triggered by:** {reason} on `{sourceBranch}` @ `{sourceVersion[:8]}`
|
||||
**Duration:** {startTime} → {finishTime} UTC (~{duration})
|
||||
**Result:** ❌ **FAILED** — {baseline_summary}
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Triplet | Unique Failures |
|
||||
|---------|----------------|
|
||||
| x64-windows | N |
|
||||
| x64-windows-release | N (identical to x64-windows / N unique) |
|
||||
| x64-linux | N (M shared with Windows) |
|
||||
| {other triplets} | *artifacts present (not analyzed)* |
|
||||
|
||||
**Total unique failing ports across all analyzed triplets: N**
|
||||
**Estimated cross-platform failures: ≥N ports fail on multiple triplets**
|
||||
|
||||
---
|
||||
|
||||
## 🔴 Cross-Platform Regressions (Windows + Linux)
|
||||
|
||||
### N. `port-name` — Short Root Cause Title
|
||||
|
||||
- **Triplets:** x64-windows, x64-windows-release, x64-linux
|
||||
- **Error:** `exact error message from log`
|
||||
in `path/to/file.ext:line`
|
||||
- **Root cause:** Explanation of why this happened — what changed, what broke.
|
||||
- **Suggested fix:** Concrete actionable recommendation.
|
||||
|
||||
```cmake
|
||||
# Code snippet if applicable
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔴 Windows-Only Regressions
|
||||
|
||||
### N. `port-name` — Short Root Cause Title
|
||||
|
||||
- **Triplets:** x64-windows, x64-windows-release
|
||||
- **Error:** `exact error message`
|
||||
- **Root cause:** ...
|
||||
- **Suggested fix:** ...
|
||||
|
||||
---
|
||||
|
||||
## 🔴 Linux-Only Regressions
|
||||
|
||||
### N. `port-name` — Short Root Cause Title
|
||||
|
||||
- **Triplet:** x64-linux
|
||||
- **Error:** `exact error message`
|
||||
- **Root cause:** ...
|
||||
- **Suggested fix:** ...
|
||||
|
||||
---
|
||||
|
||||
## 📋 Action Recommendations
|
||||
|
||||
| Priority | Port(s) | Recommended Action |
|
||||
|---|---|---|
|
||||
| 🔴 Immediate | `port` | Short action description |
|
||||
| 🟠 High | `port` | Short action description |
|
||||
| 🟡 Medium | `port` | Short action description |
|
||||
| ℹ️ Baseline | `port` | Add `=fail/skip # reason` to `ci.baseline.txt` |
|
||||
| ℹ️ Investigate | `port` | What to look into |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Real-World Example (Build #129315)
|
||||
|
||||
The following is a real report generated for the scheduled master build on 2026-03-30.
|
||||
Use this as a reference for tone, depth, and structure.
|
||||
|
||||
### Build Metadata
|
||||
|
||||
- **Build:** [20260330.1](https://dev.azure.com/vcpkg/public/_build/results?buildId=129315)
|
||||
- **Reason:** Scheduled run on `refs/heads/master` @ `c4de8d6f`
|
||||
- **Duration:** ~35 hours
|
||||
- **Artifacts with failure logs:** all 13 triplets had failures
|
||||
|
||||
### Failure Inventory
|
||||
|
||||
| Port | Triplets | Root Cause Category | Severity |
|
||||
|------|----------|---------------------|----------|
|
||||
| `v8` | win | Python 3.13 removed `pipes` module | 🔴 Immediate |
|
||||
| `onnxruntime` | win, linux | CMake `math()` on empty TensorRT variable | 🔴 Immediate |
|
||||
| `wpilib` | win, linux | `wpi::Logger::Log` API mismatch in `DataLog.cpp` | 🔴 Immediate |
|
||||
| `ntf-core` | win, linux | `bal` dep ordering (win); missing UFID cmake file (linux) | 🟠 High |
|
||||
| `rmqcpp` | win, linux | Transitive `libpcre2-8` not found via `bal→bdl` | 🟠 High |
|
||||
| `azure-storage-cpp` | linux | Boost.ASIO `deadline_timer` removed | 🟠 High |
|
||||
| `cppcoro` | linux | `<experimental/coroutine>` removed in GCC 14 | 🟠 High |
|
||||
| `zookeeper` | win, linux | Java not found at configure time | 🟡 Medium |
|
||||
| `moos-essential` | win | POSIX socket headers + Winsock2 conflict | 🟡 Medium |
|
||||
| `libxt` | win | Clang `-Wunsafe-buffer-usage` promoted to error | 🟡 Medium |
|
||||
| `saucer` | linux | Upstream compiler version check too strict | 🟡 Medium |
|
||||
| `libaiff` | linux | Missing `#include <stdint.h>` in public header | 🟡 Medium |
|
||||
| `libhdfs3` | linux | `mode` macro conflict + missing `<stdint.h>` | 🟡 Medium |
|
||||
| `yubico-piv-tool` | linux | Clang-only flag `-Wshorten-64-to-32` passed to GCC | 🟡 Medium |
|
||||
| `libnick` | linux | `sqlcipher` not available via pkg-config | 🟡 Medium |
|
||||
| `salome-medcoupling` | linux | Missing `metis` vcpkg dependency | 🟡 Medium |
|
||||
| `mesa` | linux | `glslangValidator` not found by Meson | 🟡 Medium |
|
||||
| `crashpad` | linux | Compiler not found (exit code 127, likely Docker image change) | ℹ️ Investigate |
|
||||
| `arrayfire` | linux | Build truncated at 117/832 — likely OOM kill | ℹ️ Investigate |
|
||||
| `xbitmaps` | linux | `xorg-macros < 1.20` in Docker image | ℹ️ Investigate |
|
||||
| `nana` | linux | CMake generate fails, no detail in logs | ℹ️ Investigate |
|
||||
| `openslide` | win | Portfile explicitly rejects MSVC (`use clang-cl`) | ℹ️ Baseline |
|
||||
| `openzl` | win | Portfile explicitly rejects MSVC (`use clang-cl`) | ℹ️ Baseline |
|
||||
| `ms-gdkx` | win | Requires Microsoft GDK Xbox Extensions (not in CI) | ℹ️ Baseline |
|
||||
|
||||
---
|
||||
|
||||
## Formatting Rules
|
||||
|
||||
1. **Section order**: Cross-Platform → Windows-Only → Linux-Only → macOS-Only → Android-Only → Action Table
|
||||
2. **Numbering**: Sequential across all sections (1, 2, 3 … N)
|
||||
3. **Error quotes**: Always quote the exact first meaningful error line from the log verbatim
|
||||
4. **Triplet notation**: Use short forms in narrative (`win`, `linux`) but full names in bullet points (`x64-windows`)
|
||||
5. **Baseline ports**: Ports that explicitly reject a platform by design (`MSVC is not supported`, environment requirements) go to the Baseline section, not regressions
|
||||
6. **Dependency chain**: If port A fails because port B failed, list only port B as the regression with a note "causes N downstream failures: portA, portC, …"
|
||||
7. **Transient failures**: Download failures (curl errors, SHA mismatches) and exit-code-127 failures should be flagged as potentially transient
|
||||
8. **Priority levels**:
|
||||
- 🔴 **Immediate** — Core/popular ports, failures on 3+ triplets, or CI-environment regressions affecting many ports
|
||||
- 🟠 **High** — Failures on 2+ triplets, popular ports, or clear upstream API breaks
|
||||
- 🟡 **Medium** — Single-triplet failures with clear fixes
|
||||
- ℹ️ **Baseline/Investigate** — Expected failures, environment issues, or unclear root cause
|
||||
|
||||
---
|
||||
|
||||
## Artifact Size Heuristics
|
||||
|
||||
Use artifact sizes (from the artifacts list API response) to prioritize analysis:
|
||||
|
||||
| Artifact Size | Interpretation |
|
||||
|---|---|
|
||||
| < 10 MB | Very few failures (< ~15 ports), analyze fully |
|
||||
| 10–50 MB | Moderate failures (~15–80 ports), sample representative ones |
|
||||
| 50–120 MB | Many failures (~80–200 ports), focus on unique errors only |
|
||||
| > 120 MB | Mass failures — likely a systemic issue (toolchain, CI image, common dep) |
|
||||
|
||||
In build #129315, x64-windows (8.5 MB) and x64-linux (4.6 MB compressed) were small enough for complete analysis.
|
||||
Android artifacts (~115–123 MB) indicate widespread failures likely sharing a common root cause.
|
||||
|
||||
---
|
||||
|
||||
## Common Root Cause Patterns to Look For First
|
||||
|
||||
When analyzing a new build, scan for these high-frequency patterns before deep-diving individual ports:
|
||||
|
||||
| Pattern | Indicator in logs | Likely cause |
|
||||
|---|---|---|
|
||||
| `No module named 'pipes'` or similar Python import error | Python stdlib change | Python version upgrade on CI |
|
||||
| `fatal error: experimental/coroutine` | Many Linux ports | GCC 14 / compiler image upgrade |
|
||||
| `deadline_timer` / `posix_time` | Boost-dependent ports | Boost.ASIO API break |
|
||||
| `error: unrecognized command-line option` | Configure-time test | Wrong compiler flags for toolchain |
|
||||
| `FAILED: [code=127]` | Many compilation units | Compiler not in PATH (Docker image change) |
|
||||
| `math cannot parse the expression: ""` | CMake configure | Empty variable in math() — missing guard |
|
||||
| `Cannot open include file: 'sys/socket.h'` | Win + POSIX code | Missing WIN32 socket porting |
|
||||
| `must install xorg-macros` | autotools ports | CI image missing xorg-dev package |
|
||||
| Build truncated, no error message | OOM or timeout | Use `DISABLE_PARALLEL` or check resources |
|
||||
| Many ports → same dependency fails | dep chain | Find root dep; rest are downstream casualties |
|
||||
|
|
@ -0,0 +1,361 @@
|
|||
# vcpkg CI Failure Patterns Reference
|
||||
|
||||
This reference catalogues common failure patterns found in vcpkg CI failure log artifacts. Use these patterns to triage and classify failures when analyzing build output.
|
||||
|
||||
---
|
||||
|
||||
## Log File Structure
|
||||
|
||||
Each failure log (`*.log`) in the `"failure logs for {triplet}"` artifact contains the full vcpkg install output for a single port. The log typically follows this structure:
|
||||
|
||||
```
|
||||
-- Downloading <url>...
|
||||
-- Extracting source...
|
||||
-- Applying patches...
|
||||
-- Using cached binary package. ← binary cache hit (may skip build)
|
||||
-- Building <port>:<triplet>...
|
||||
-- Configuring...
|
||||
-- Building... ← compiler output here
|
||||
-- Installing...
|
||||
-- Running tests... ← if test feature enabled
|
||||
-- Completed.
|
||||
|
||||
error: building <port>:<triplet> failed with: BUILD_FAILED
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Category 1: Dependency Chain Failures
|
||||
|
||||
**Symptom**: The port itself did not fail — a dependency failed first.
|
||||
|
||||
**Log indicators:**
|
||||
```
|
||||
-- Building dependency-name[core]:x64-windows... failed
|
||||
error: package dependency-name:x64-windows is not installed
|
||||
```
|
||||
|
||||
or more explicitly:
|
||||
```
|
||||
error: building portname:x64-windows failed with: BUILD_FAILED
|
||||
... the dependency 'depname:x64-windows' failed to install
|
||||
```
|
||||
|
||||
**Triage action:**
|
||||
- Identify the root failing dependency (the one that failed first in the log)
|
||||
- All other ports in this chain are "downstream casualties"
|
||||
- Fix the root dependency; downstream failures should resolve automatically
|
||||
- In the report, list only the root dependency as a regression; collapse others with a note
|
||||
|
||||
**How to find the root:**
|
||||
- Sort failure logs by time or by name
|
||||
- A port whose own build commands (not just dependency resolution) fail is the root
|
||||
- If `portA.log` shows `error: package portB is not installed` and `portB.log` shows actual compiler output, portB is the root
|
||||
|
||||
---
|
||||
|
||||
## Category 2: CMake Configuration Failures
|
||||
|
||||
**Symptom**: `cmake --build` stage fails before any source is compiled.
|
||||
|
||||
**Log indicators:**
|
||||
```
|
||||
CMake Error at CMakeLists.txt:42 (find_package):
|
||||
By not providing "FindXYZ.cmake" in CMAKE_MODULE_PATH this project has
|
||||
requested CMake to find a package configuration file provided by "XYZ",
|
||||
but CMake did not find one.
|
||||
|
||||
CMake Error: Could not find a package configuration file provided by "XYZ"
|
||||
with any of the following names:
|
||||
XYZConfig.cmake
|
||||
xyz-config.cmake
|
||||
```
|
||||
|
||||
**Common root causes:**
|
||||
- A dependency is not exposing its CMake targets correctly
|
||||
- A port's `vcpkg.cmake` or `vcpkg-cmake-config` integration is broken
|
||||
- A `find_package()` call in the upstream CMakeLists.txt uses a non-standard config name
|
||||
|
||||
**Other CMake configure patterns:**
|
||||
```
|
||||
-- The CXX compiler identification is unknown ← toolchain issue (rare in CI)
|
||||
CMake Error: Generator: ... not found ← MSBuild/Ninja not available
|
||||
CMake Error (cmake_policy): ... ← policy violation
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Category 3: Compiler Errors
|
||||
|
||||
**Symptom**: Source code fails to compile.
|
||||
|
||||
### MSVC (Windows)
|
||||
```
|
||||
error C2065: 'identifier': undeclared identifier
|
||||
error C2039: 'member': is not a member of 'class'
|
||||
error C3861: 'function': identifier not found
|
||||
error C4996: 'function': This function may be unsafe ← treated as error with /WX
|
||||
fatal error C1083: Cannot open include file: 'header.h'
|
||||
```
|
||||
|
||||
### Clang (macOS, Linux)
|
||||
```
|
||||
error: use of undeclared identifier 'name'
|
||||
error: no member named 'member' in 'class'
|
||||
error: 'header.h' file not found
|
||||
error: unknown type name 'type_name'
|
||||
```
|
||||
|
||||
### GCC (Linux)
|
||||
```
|
||||
error: 'name' was not declared in this scope
|
||||
error: invalid use of incomplete type
|
||||
error: 'header.h': No such file or directory
|
||||
```
|
||||
|
||||
**Root cause categories:**
|
||||
- **C++ standard incompatibility**: Code uses features not available in the configured standard (C++14/17/20)
|
||||
- **Missing Windows includes**: `#include <windows.h>` missing or wrong order
|
||||
- **API deprecation**: Uses a function/class removed in a newer version of a dependency
|
||||
- **Compiler flag conflict**: vcpkg sets `-Werror` or `/WX`; warning-as-error triggers on upstream code
|
||||
|
||||
---
|
||||
|
||||
## Category 4: Linker Errors
|
||||
|
||||
**Symptom**: Compilation succeeds but linking fails.
|
||||
|
||||
### MSVC
|
||||
```
|
||||
error LNK2019: unresolved external symbol "__declspec(dllimport) ..." referenced in function "..."
|
||||
error LNK2001: unresolved external symbol
|
||||
error LNK1120: N unresolved externals
|
||||
```
|
||||
|
||||
### GCC/Clang
|
||||
```
|
||||
undefined reference to `function_name'
|
||||
ld: symbol(s) not found for architecture arm64
|
||||
```
|
||||
|
||||
**Root cause categories:**
|
||||
- **Missing `__declspec(dllexport)`**: Library exports are missing for DLL builds
|
||||
- **Static vs. dynamic mismatch**: Port links against static lib but needs dynamic (or vice versa)
|
||||
- **Missing dependency in vcpkg.json**: Library calls into another library not declared as a dependency
|
||||
- **Symbol visibility**: On Linux/macOS, symbols not exported with `__attribute__((visibility("default")))`
|
||||
|
||||
---
|
||||
|
||||
## Category 5: SHA512 Hash Mismatch
|
||||
|
||||
**Symptom**: Download succeeds but hash verification fails.
|
||||
|
||||
```
|
||||
error: File does not have the expected hash:
|
||||
url: https://github.com/owner/repo/archive/v1.2.3.tar.gz
|
||||
expected: abc123...
|
||||
actual: def456...
|
||||
```
|
||||
|
||||
**Root causes:**
|
||||
- **Upstream archive changed**: The upstream author re-tagged or replaced the release archive (mutable tags, force-push)
|
||||
- **Mirror cache stale**: vcpkg asset cache has an old version of the file
|
||||
- **Port hash not updated**: A PR updated the version but forgot to recalculate SHA512
|
||||
|
||||
**Triage action**: Recalculate hash with:
|
||||
```bash
|
||||
vcpkg install portname --no-binarycaching
|
||||
```
|
||||
The error message will contain the correct new hash.
|
||||
|
||||
---
|
||||
|
||||
## Category 6: Download / Network Failures
|
||||
|
||||
**Symptom**: vcpkg cannot download the source archive.
|
||||
|
||||
```
|
||||
error: Failed to download https://example.com/archive.tar.gz
|
||||
error: curl: (22) The requested URL returned error: 404
|
||||
error: curl: (6) Could not resolve host: example.com
|
||||
```
|
||||
|
||||
**Root causes:**
|
||||
- **URL moved**: Upstream deleted or moved the release (common with GitHub releases being retracted)
|
||||
- **CI network issue**: Transient connectivity problem (likely if only 1–2 ports fail this way, or a retry succeeds)
|
||||
- **Domain changed**: Package moved to a new hosting location
|
||||
|
||||
**Triage action**: Check if the URL is still valid. If transient, re-run the build.
|
||||
|
||||
---
|
||||
|
||||
## Category 7: Post-Build Check Failures
|
||||
|
||||
**Symptom**: Build and install succeed, but vcpkg's post-install checks fail.
|
||||
|
||||
```
|
||||
error: The following files are not installed:
|
||||
share/portname/copyright
|
||||
|
||||
error: Found forbidden pattern in installed file:
|
||||
include/portname/config.h: #define PORTNAME_EXPORTS
|
||||
|
||||
error: Usage of deprecated vcpkg function 'vcpkg_apply_patches'
|
||||
```
|
||||
|
||||
**Common post-build checks:**
|
||||
- Copyright file must exist at `share/{portname}/copyright`
|
||||
- No `.pdb` files in release packages (debug-only)
|
||||
- Headers must be in `include/`, not `include/{portname}/include/`
|
||||
- CMake config files in `share/{portname}/` not `lib/cmake/`
|
||||
- No absolute paths in installed CMake files
|
||||
|
||||
---
|
||||
|
||||
## Category 8: Test Failures
|
||||
|
||||
**Symptom**: The port itself builds and installs successfully, but its tests fail.
|
||||
|
||||
```
|
||||
Test #1: my_test ... FAILED
|
||||
The following tests FAILED:
|
||||
1 - my_test (Failed)
|
||||
```
|
||||
|
||||
or in xUnit format (from test results API):
|
||||
```xml
|
||||
<testcase name="portname:x64-windows" ...>
|
||||
<failure message="Test failed: ..." />
|
||||
</testcase>
|
||||
```
|
||||
|
||||
**Triage action:**
|
||||
- Check whether the test passes on other triplets (may be platform-specific)
|
||||
- Check if this is a flaky test (re-run to confirm)
|
||||
- Check if the test was previously failing (baseline)
|
||||
|
||||
---
|
||||
|
||||
## Category 9: Formatting / Manifest Check Failures
|
||||
|
||||
**Symptom**: The `"format.diff"` artifact is present and non-empty.
|
||||
|
||||
This is not a port build failure but a CI check failure. The `format.diff` artifact contains a unified diff showing:
|
||||
- `vcpkg.json` manifest files that need formatting via `vcpkg format-manifest`
|
||||
- Baseline files (`ci.baseline.txt`) that need formatting via `vcpkg format-feature-baseline`
|
||||
- Version database files that need updating via `vcpkg x-add-version`
|
||||
|
||||
**Triage action**: Apply the diff, or run the formatting commands locally:
|
||||
```bash
|
||||
./vcpkg format-manifest --all
|
||||
./vcpkg format-feature-baseline scripts/ci.baseline.txt
|
||||
./vcpkg x-add-version --all
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Category 10: Build Timeout
|
||||
|
||||
**Symptom**: Job exceeds the 2-day timeout limit (rare).
|
||||
|
||||
```
|
||||
##[error]The job running on agent ... has exceeded the maximum execution time of 2880 minutes.
|
||||
```
|
||||
|
||||
**Triage action**: Almost always an infinite loop or hang in the build system. Check the last lines of the log for the hanging operation.
|
||||
|
||||
---
|
||||
|
||||
## Category 11: File Conflicts
|
||||
|
||||
**Symptom**: The port installs files that conflict with files already installed by another port. This failure is **only visible in the "Test Modified Ports" step log** — it does NOT produce a failure log artifact.
|
||||
|
||||
```
|
||||
REGRESSION: kf6i18n:x64-windows failed with FILE_CONFLICTS. If expected, add kf6i18n:x64-windows=fail to .../ci.baseline.txt.
|
||||
```
|
||||
|
||||
The error in the step log reads (before the REGRESSION summary line):
|
||||
```
|
||||
error: File conflicts:
|
||||
C:\path\to\installed\x64-windows\bin\some-tool.exe → installed by portA and portB
|
||||
C:\path\to\installed\x64-windows\include\shared.h → installed by portA and portB
|
||||
```
|
||||
|
||||
**Where to find it**: In the "*** Test Modified Ports" task log for the affected job (not in any artifact ZIP).
|
||||
|
||||
**Triage action**:
|
||||
- Identify which two ports are claiming the same installed files
|
||||
- Usually caused by: a new port that duplicates files from an existing one, or a port that started installing an additional binary that conflicts
|
||||
- Fix: rename the conflicting file, split the port, or add `conflicts` to the port manifest
|
||||
|
||||
---
|
||||
|
||||
## Category 12: Missing From Baseline (Unexpected Pass)
|
||||
|
||||
**Symptom**: A port that is listed as `fail` in `ci.baseline.txt` now passes. This is reported as a regression only in the step log:
|
||||
|
||||
```
|
||||
REGRESSION: portname:x64-windows failed with MISSING_FROM_BASELINE.
|
||||
```
|
||||
|
||||
**Triage action**: Remove the `portname=fail` entry from `scripts/ci.baseline.txt` — the port is fixed.
|
||||
|
||||
The file `scripts/ci.baseline.txt` in the repository root controls which ports are expected to fail.
|
||||
|
||||
**Format:**
|
||||
```
|
||||
# Comment lines start with #
|
||||
portname=fail # will skip in PR CI, attempt in scheduled
|
||||
portname=skip # never built in CI
|
||||
portname=pass # (same as not listed) must succeed
|
||||
```
|
||||
|
||||
**Triplets covered by CI baseline:**
|
||||
- `arm-neon-android`
|
||||
- `arm64-android`
|
||||
- `arm64-osx`
|
||||
- `arm64-windows`
|
||||
- `arm64-windows-static-md`
|
||||
- `x64-android`
|
||||
- `x64-linux`
|
||||
- `x64-osx`
|
||||
- `x64-uwp`
|
||||
- `x64-windows`
|
||||
- `x64-windows-static`
|
||||
- `x64-windows-static-md`
|
||||
- `x86-windows`
|
||||
|
||||
**Important**: Baseline entries apply to all triplets unless a triplet-specific entry exists (not currently supported; all triplet failures for a port use the same baseline entry).
|
||||
|
||||
---
|
||||
|
||||
## Regression Severity Levels
|
||||
|
||||
Use these levels when reporting failures:
|
||||
|
||||
| Level | Criteria | Action |
|
||||
|---|---|---|
|
||||
| 🔴 **Critical** | Core port (boost, openssl, curl, zlib, cmake) or many dependents fail | Immediate fix required |
|
||||
| 🟠 **High** | Popular port fails on multiple triplets | Fix before merge |
|
||||
| 🟡 **Medium** | Single port fails on 1–2 triplets | Fix or add to baseline |
|
||||
| 🟢 **Low** | Port already in baseline, or transient failure | No immediate action |
|
||||
| ℹ️ **Info** | Formatting/version file issues | Easy to fix with automated tools |
|
||||
|
||||
---
|
||||
|
||||
## Common vcpkg CI Triplet Notes
|
||||
|
||||
| Triplet | Notes |
|
||||
|---|---|
|
||||
| `x64-windows` | Most common; MSVC, dynamic linking |
|
||||
| `x64-windows-static` | MSVC, static linking (`/MT`) |
|
||||
| `x64-windows-static-md` | MSVC, static linking (`/MD`) |
|
||||
| `arm64-windows` | Cross-compiled on x64 Windows; run tests on separate ARM hardware |
|
||||
| `x64-linux` | Ubuntu in Docker; GCC or Clang |
|
||||
| `arm64-linux` | Ubuntu ARM64 in Docker |
|
||||
| `arm64-osx` | macOS ARM (Apple Silicon); Clang |
|
||||
| `x64-osx` | macOS x86_64; Clang |
|
||||
| `x64-android` | Cross-compiled; Android NDK Clang |
|
||||
| `arm64-android` | Cross-compiled; Android NDK Clang |
|
||||
| `arm-neon-android` | Cross-compiled; Android NDK Clang |
|
||||
| `x64-uwp` | Universal Windows Platform; WinRT restrictions |
|
||||
Loading…
Reference in New Issue