CI pipelines: JUnit and SARIF

Keep your string files in a repository and check them on every pull request. A CI step uploads the files to POST /qa-adapter/v1/runs, waits for the run, downloads its findings as JUnit XML (your CI's test tab lists each finding as a test) and as SARIF 2.1.0 (GitHub code scanning annotates the file) and fails when a finding you care about exists. Everything runs through the API with the key your workspace already has: a key with qa:write to start the run and qa:read to read it.

The two files carry what the JSON download of the same run carries and nothing else, in two standard shapes.

The two files

GET /qa-adapter/v1/reports/{runId}/download?format=junit and ?format=sarif (qa:read, the same rules as the other formats: your own runs and runs shared into your workspace; another workspace's run is a 404). The links are also in the run's links.download, in the answer of POST /runs, of GET /reports/{runId} and in the run.completed webhook.

JUnit XML (name.junit.xml): one test suite per file of the run, one test case per finding, named after the check and the segment (Number mismatch: segment app.title).

• A finding at or above fail_on is a failure, with its message and the source, the target and the suggested fix in its body.
• A finding below fail_on is skipped, with its severity and message.
• A finding marked Ignored or Fixed in the app is skipped as "Marked Ignored in the app", whatever its severity. A decision you made in the app never fails a build.
• A spelling finding is never a failure, as it never decides the report's result.
• The AI review's own status (partial or failed) is a line in the suite's system-err, not a test case.
• A file with no findings has one passing test case, no findings.
• The suite's properties carry the runId, the profile, the report's status, the score (when the run has one), the segments, the words and the report link.

SARIF 2.1.0 (name.sarif): one result per finding.

• level is error for a critical finding, warning for a major one and note for a minor one; properties.severity gives the word (critical, major, minor or preferential) the API uses everywhere.
• ruleId is the check's code; tool.driver.rules names each check as the report does.
• partialFingerprints.errorId is the finding's stable error_id, so GitHub carries a dismissal from one run of the same file to the next.
• A finding marked Ignored or Fixed in the app carries a suppressions entry.
• The location is the file name under %SRCROOT%, the segment (the key of a string file) as a logical location and, when the file records one, the line as region.startLine. A file without line numbers (an XLIFF) gets the logical location only.
• The AI review's status is a tool execution notification.

Reviewers' notes appear only to a signed-in member of the run's own workspace. A key or a share link gets the decision without the note.

fail_on

fail_on says which findings the JUnit file counts as failures: critical (the default, the same rule that fails the report), major (critical and major) or minor (every finding the report counts). Another value is a 400, "fail_on must be critical, major or minor". SARIF carries every finding at its own level and ignores fail_on.

curl -o tr.junit.xml \
  "https://app.stillneedsahuman.com/qa-adapter/v1/reports/a1b2c3d4e5f6/download?format=junit&fail_on=major" \
  -H "Authorization: Bearer YOUR_TOKEN"

How a step fails

A CI marks the step failed from your script's exit code. The recipes below read the JUnit file's failures count and exit 1 when it is above zero. A minimum score is your step's own check: read score from GET /qa-adapter/v1/reports/{runId} and compare it with your number. A file too small to score has no score (null); decide in your script what that means for your build.

Upload one file per step, or one zip of a source file and its translation, when you want annotations: GitHub annotates a result only when its file path is a path in your repository, so the recipes rewrite the file name to that path with jq. A zip of several pairs keeps its file names and its results show without a file position.

GitHub Actions

Save the key as a repository secret named WW_API_KEY. The job checks locales/tr.json against locales/en.json, shows the findings as tests and as code scanning alerts and fails on a critical finding.

name: Translation check
on: [pull_request]

jobs:
  translation-check:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      checks: write
      security-events: write
    env:
      BASE: https://app.stillneedsahuman.com
      WW_API_KEY: ${{ secrets.WW_API_KEY }}
    steps:
      - uses: actions/checkout@v4

      - name: Run the check
        run: |
          set -eu
          (cd locales && zip -q ../strings.zip en.json tr.json)
          RUN=$(curl -sSf -X POST "$BASE/qa-adapter/v1/runs" \
            -H "Authorization: Bearer $WW_API_KEY" \
            -F "[email protected]" -F "profile=generic" | jq -r .runId)
          for i in $(seq 1 120); do
            STATUS=$(curl -sSf "$BASE/qa-adapter/v1/reports/$RUN" \
              -H "Authorization: Bearer $WW_API_KEY" | jq -r .status)
            [ "$STATUS" = "ai_running" ] || break
            sleep 5
          done
          curl -sSf -o qa.junit.xml "$BASE/qa-adapter/v1/reports/$RUN/download?format=junit&fail_on=critical" \
            -H "Authorization: Bearer $WW_API_KEY"
          curl -sSf "$BASE/qa-adapter/v1/reports/$RUN/download?format=sarif" \
            -H "Authorization: Bearer $WW_API_KEY" \
            | jq '(.runs[].results[].locations[].physicalLocation.artifactLocation.uri) = "locales/tr.json"' \
            > qa.sarif
          echo "Report: $BASE/qa-adapter/v1/reports/$RUN"

      - name: Show the findings as tests
        if: always()
        uses: dorny/test-reporter@v1
        with:
          name: Translation findings
          path: qa.junit.xml
          reporter: java-junit
          fail-on-error: false

      - name: Annotate the file
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: qa.sarif
          category: translation-check

      - name: Fail on critical findings
        run: |
          FAILURES=$(grep -o 'failures="[0-9]*"' qa.junit.xml | head -n 1 | tr -dc '0-9')
          echo "Failing findings: $FAILURES"
          [ "$FAILURES" -eq 0 ]

To keep the files with the run instead, add actions/upload-artifact@v4 with path: qa.*. Code scanning needs GitHub Advanced Security on a private repository.

GitLab CI

Save the key as a masked CI/CD variable named WW_API_KEY. The merge request's test summary lists the findings from the JUnit file; the SARIF file is kept with the job for any tool that reads SARIF.

translation_check:
  image: alpine:3.20
  variables:
    BASE: https://app.stillneedsahuman.com
  before_script:
    - apk add --no-cache curl jq zip
  script:
    - (cd locales && zip -q ../strings.zip en.json tr.json)
    - >
      RUN=$(curl -sSf -X POST "$BASE/qa-adapter/v1/runs"
      -H "Authorization: Bearer $WW_API_KEY"
      -F "[email protected]" -F "profile=generic" | jq -r .runId)
    - >
      for i in $(seq 1 120); do
      STATUS=$(curl -sSf "$BASE/qa-adapter/v1/reports/$RUN" -H "Authorization: Bearer $WW_API_KEY" | jq -r .status);
      [ "$STATUS" = "ai_running" ] || break; sleep 5; done
    - curl -sSf -o qa.junit.xml "$BASE/qa-adapter/v1/reports/$RUN/download?format=junit&fail_on=critical" -H "Authorization: Bearer $WW_API_KEY"
    - curl -sSf -o qa.sarif "$BASE/qa-adapter/v1/reports/$RUN/download?format=sarif" -H "Authorization: Bearer $WW_API_KEY"
    - FAILURES=$(grep -o 'failures="[0-9]*"' qa.junit.xml | head -n 1 | tr -dc '0-9')
    - echo "Failing findings $FAILURES, report $BASE/qa-adapter/v1/reports/$RUN"
    - test "$FAILURES" -eq 0
  artifacts:
    when: always
    paths:
      - qa.sarif
    reports:
      junit: qa.junit.xml

GitLab's security dashboard reads GitLab's own report format, not SARIF, so the SARIF file is a plain artifact here.

A minimum score

Add this to either recipe after the poll to fail a build under a score of your choice (here 95):

SCORE=$(curl -sSf "$BASE/qa-adapter/v1/reports/$RUN" -H "Authorization: Bearer $WW_API_KEY" | jq -r '.score // empty')
[ -z "$SCORE" ] || awk -v s="$SCORE" 'BEGIN { exit (s < 95) }'

A ready-made client

A small client that runs the upload, the poll and both downloads in one command is on its way. Until it is published, the two recipes above do the same with curl and jq.

Limits

GitHub code scanning takes up to 25,000 results and 10 MB per SARIF upload. A report that large is a reason to narrow the profile; the file is never cut. A JUnit viewer shows every finding as a test, so a run with 3,000 minor findings is 3,000 skipped tests: fail_on decides the failures, not the number of rows. The polls count against the read rate and the run against the run rate of your workspace (Rates and ceilings).