5.1 CI Runner (`certiqa-run.sh`) & Headless API Execution

How to trigger and gate on CertiQA test suites from terminals, Docker containers, and CI pipelines using the SaaS API.

Running Outside the App

A person clicking Run on the canvas is not an option for a nightly cron job or a pull-request check – that needs an automated runner with a deterministic exit code a pipeline can gate on. CertiQA provides the certiqa-run.sh integration script: a portable, curl/jq-based runner that triggers execution on the CertiQA platform, streams progress, downloads JUnit reports, and exits with a strict gate status code without requiring any private engine dependencies on your CI runners.

CERTIQA_PROJECT_ID and optional CERTIQA_SUITE_ID identify what to run; CERTIQA_API_URL points to your control plane (defaulting to https://api.certiqa.app). CERTIQA_TOKEN is your scoped API token – pull it from your CI provider’s secret store rather than typing it into a script. Add CERTIQA_TAGS or CERTIQA_CASE_IDS to target specific test slices.

Post-deploy checks against production: set CERTIQA_ENV to a production environment and use a production token. An organization owner or admin creates one in Project → Settings & Tools → Secrets & API Tokens with “Allow runs against production environments” ticked; it works for that project only. Steps that create, update or delete data are still blocked in production unless the run explicitly allows production mutations.

# Run a test suite headlessly in CI via certiqa-run.sh
export CERTIQA_API_URL="https://api.certiqa.app"
export CERTIQA_PROJECT_ID="PROJ-100"
export CERTIQA_SUITE_ID="SUITE-200"
export CERTIQA_TOKEN="certiqa_tok_..."

./certiqa-run.sh

Getting the Script

certiqa-run.sh is a single portable shell script that needs only sh, curl and jq. Download it into your repository or your CI image, read it, and pin it in your own source control:

curl -fsSLO https://docs.certiqa.app/scripts/certiqa-run.sh
chmod +x certiqa-run.sh

It is a sample you can copy and adapt. It calls the documented REST API (POST /api/v1/projects/{projectId}/hooks/trigger, then GET /api/v1/runs/{runId}/report) and has no other dependency on CertiQA.

Reading the Exit Code

The exit code is the whole contract between the runner and your pipeline – and it’s a four-way signal, not just pass/fail: a real regression (1), a run that did not finish in time (2), and an infrastructure or auth problem (3) each call for a different response, so a naive ‘any non-zero exit fails the job’ script treats them all alike.

Exit Code	Meaning	What Your Pipeline Should Do
0	Every non-quarantined test passed (Quality Gate PASS). A failing quarantined test does not fail the gate, so a run whose only failures are quarantined also exits 0.	Pass the job.
1	An active, non-quarantined test failed (Quality Gate FAIL).	Fail the job – this is a real regression.
2	Execution timed out waiting for results.	Fail or retry the job – timeout exceeded.
3	Infrastructure or auth error (bad token, unreachable API).	Treat as an infrastructure failure distinct from a test failure.

Sharding a Suite Across Parallel Jobs

–shard “/” (or the CERTIQA_SHARD environment variable, which the GitHub Action, GitLab template, and Jenkins library all read so you don’t have to wire the CLI flag yourself) runs only the i-th deterministic slice of an n-way split – pair it with your CI system’s own fan-out mechanism (a GitHub Actions matrix, GitLab’s parallel: keyword, a Jenkins declarative matrix) to run N jobs in parallel, each covering a different slice, with no two jobs ever running the same case and no case ever skipped. The split is a stable hash of each case’s position, not a random shuffle, so a given case always lands in the same shard run over run – useful when you’re correlating a shard’s failures back to a specific slice of your suite over time. Omit –shard entirely to keep running the full suite in a single job, exactly as before this flag existed.

Parallel vs Serial Execution & Concurrency Architecture

Optimizing test execution speed and infrastructure cost requires striking the right balance between isolation, parallelization, and serial dependency. CertiQA provides fine-grained controls at the Suite, Scenario, Case, and CLI levels:

Concept	Configuration	Best Practice & Recommended Usage
Suite Execution Mode	executionMode: “PARALLEL”	“SERIAL”
Scenario Execution Mode	executionMode: “SERIAL”	“PARALLEL”
Bail On Error	bailOnError: true	false (–bail)
Idempotency & Retry Safety	rerunnable: true	false (@non-rerunnable)
Single-Node Chromium Density	–workers (or CERTIQA_WORKERS)	Runs N isolated browser contexts concurrently on a single runner node. Recommended 2-4 workers per 2 vCPU / 4GB RAM node to amortize container and OS overhead.

# Run a suite targeted to a specific staging environment and tag slice
export CERTIQA_ENV="staging"
export CERTIQA_TAGS="smoke,critical"
./certiqa-run.sh

Reusing an Authenticated Session

–auth-setup-case “” (or CERTIQA_AUTH_SETUP_CASE_ID) names one case in the suite – typically a login flow – whose authenticated browser session (cookies, local storage, the works) is captured on success and reused by every other case in the same run, instead of every case replaying the full login UI from scratch. This cuts real wall-clock time on a suite with dozens of cases behind the same auth wall, and reduces how often a flaky login step can fail an otherwise-unrelated test. It’s off by default – set it once, and note that the designated case still runs in its own normal position in the suite in addition to the pre-pass, so its case count in a report will show up twice for that one case.

Running Inside Your Own Network

A hosted runner can’t reach an internal staging app sitting behind your VPN, and opening an inbound port just so a SaaS product can reach it is a non-starter for most security teams. The BYOC (bring-your-own-compute) runner solves this the other way around: it’s a long-lived container you deploy inside your own network that initiates all connections outbound over HTTPS (port 443) to pick up work and report results – no public IP or inbound firewall rule required, and it can reach anything your own network can reach, VPN-gated internal apps included. Self-hosted runners are part of the BYOC add-on: CertiQA provides the runner image through a private registry together with a runner token (contact [email protected]).

docker run -d --name certiqa-runner \
  --restart unless-stopped \
  -e CERTIQA_API_URL="https://api.certiqa.app" \
  -e CERTIQA_RUNNER_TOKEN="<runner token>" \
  -e CERTIQA_RUNNER_NAME="aws-us-east-1-worker-01" \
  -e CERTIQA_RUNNER_TAGS="aws,staging,internal-vpc" \
  <registry-provided runner image>

Behind a Corporate TLS Proxy

• If your network terminates outbound HTTPS through a corporate inspection proxy with its own CA, set NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca.pem on the container – otherwise the runner’s own outbound calls to certiqa-api fail TLS verification the same way any other Node process behind that proxy would.

• 🩺 How to Quarantine Flaky Tests — How a test gets marked quarantined, so its failures stop failing the gate.