Skip to main content

Testing

Sukko ships with a built-in tester service and CLI commands for validating correctness, measuring performance, and catching regressions — locally or against remote deployments.

How the Testing Stack Works​

The sukko test CLI commands are the orchestration layer. They talk to the tester service (sukko-tester, port 8090), which is the execution layer. The tester service connects to your gateway and provisioning API, runs the selected suite or test type, and streams results back to the CLI.

The tester runs as part of the Sukko stack — started automatically by sukko up or via Helm. You never need to call it directly; the CLI wraps everything.

Output signal vocabulary​

Every suite produces a two-level result:

LevelFieldValuesMeaning
SuiteReport.Statuspass / fail / errorOverall outcome. error means the suite couldn't run (e.g., admin key missing). pass with [skip] checks is still a pass.
CheckCheckResult.Statuspass / fail / skipPer-check result. skip means the check was bypassed — typically because the current edition doesn't include the feature.

A suite returns pass even when edition-gated checks are skipped. Use fail or error to gate CI; skip is expected behavior on lower editions.

Test Types​

TypePurposeDefault Duration
SmokeQuick connectivity and health checkInstant
ValidateEnd-to-end correctness suitesPer suite
LoadSustained connections with configurable publish rate5 minutes
StressPush to maximum capacity and find limits10 minutes
SoakLong-running stability — memory leaks, connection drift2 hours

Contexts: Local vs Remote​

Tests run against whichever environment your active context points to. Switch environments with a single command — no config files to edit.

Setting Up Contexts​

After sukko init, a local context is created automatically:

sukko context current
Name:            local
Gateway:         ws://localhost:3000
Provisioning:    http://localhost:8080
Tester:          http://localhost:8090
Environment:     local
Tenant:          local

To test against a remote deployment, create a context:

sukko context create staging \
  --gateway-url wss://ws.staging.example.com \
  --provisioning-url https://api.staging.example.com \
  --tester-url https://tester.staging.example.com \
  --environment staging \
  --tenant my-tenant

Switching Contexts​

sukko context use staging
sukko --context staging test smoke
sukko context list

Local vs Remote Behavior​

  • Local context (localhost URLs) — the tester uses its own Docker-internal service addresses. No context is sent to the tester API.
  • Remote context — the CLI forwards gateway URL, provisioning URL, and environment to the tester so it targets the right deployment.

Before Running Suites​

Setup Order Matters​

Run these steps in order the first time you bring up a local deployment:

sukko auth keygen        # Generate admin keypair — MUST run before sukko up
sukko up                 # Start the stack
sukko health             # Wait until all services report healthy
sukko status             # Confirm all services are running
sukko edition            # Check your edition and license key path
sukko test smoke         # Final gate — run this before any suite battery
Admin keypair must exist before the stack starts

If you run sukko up before sukko auth keygen, the provisioning service starts without a bootstrap admin key. The provisioning, tenant-isolation, and token-revocation suites will fail at setup with no obvious error. If this happens, run sukko auth keygen, then sukko down && sukko up to restart with the key in place.

Admin Keypair Requirement​

Three suites require an admin keypair to sign provisioning API requests:

SuiteRequires Admin Key
provisioningyes
tenant-isolationyes
token-revocationyes
All other suitesno

The tester service looks for the key at TESTER_ADMIN_KEY_FILE. In local Docker, sukko auth keygen writes the key to ~/.sukko/admin.key and sukko up mounts it automatically — no manual config needed.

For remote deployments, set TESTER_ADMIN_KEY_FILE on the tester service to the path of the key file on that host.

TESTER_ADMIN_KEY_FILE Format​

The file must contain the raw 64-byte Ed25519 binary private key — not a PEM-encoded PKCS#8 key. The common failure mode is exporting a key from OpenSSL as PKCS#8 PEM, which causes a parse error with no helpful message.

The correct format (produced by sukko auth keygen) is:

64 raw bytes: 32-byte seed || 32-byte public key

To verify:

wc -c ~/.sukko/admin.key   # must print exactly 64

Edition Requirements​

Four suites require a specific edition:

SuiteMinimum Edition
ssePro PRO
token-revocationPro PRO
pushPro PRO (the FCM/APNs mobile legs are Enterprise ENTERPRISE)
webhooksPro PRO

Running a gated suite on the wrong edition returns a pass result with one [skip] check (the edition gate). No failures — the suite just skips the gated checks. To actually validate the feature, you need a valid license key for the required edition.

Validation Suites​

Run any suite with:

sukko test validate --suite <name> --follow

Suite run order​

Run all 18 suites in this order. Each suite is independent and can be run individually.

#SuiteMin EditionAdmin KeyDescription
1smokeCommunitynoBasic connectivity gate
2authCommunitynoJWT auth validation
3channelsCommunitynoChannel subscribe/unsubscribe lifecycle
4pubsubCommunitynoPub-sub delivery with channel scoping
5orderingCommunitynoFIFO message ordering
6reconnectCommunitynoDisconnect/reconnect recovery
7ratelimitCommunitynoRate limit enforcement
8ssePronoSSE transport validation (SSE transport is Pro-gated)
9rest-publishCommunitynoREST publish delivery (/api/v1/publish is open on all editions; the SSE receive leg skips below Pro)
10provisioningCommunityyesProvisioning API CRUD lifecycle
11tenant-isolationCommunityyesCross-tenant message isolation
12token-revocationProyesToken revocation force-disconnect
13edition-limitsCommunitynoEdition boundary limits
14pushPronoPush notification pipeline (Web Push is Pro; the FCM/APNs legs are Enterprise and skip on Pro)
15license-reloadCommunitynoLicense hot-reload propagation
16api-keyCommunitynoStatic API key auth in isolation
17upgradeCommunitynoAuth upgrade flow: API key → JWT
18webhooksPronoWebhook delivery: happy-path, retry, degraded

Operator-Relevant Suites​

These suites validate features you configure and operate. Run them as part of your deployment checklist.

provisioning​

sukko test validate --suite provisioning --follow

Validates the provisioning API end-to-end: create and delete tenants, register and revoke signing keys, create API keys, create and delete routing rules, and verify that provisioned tenants can connect and receive messages.

Requires admin keypair — the suite calls admin-only endpoints.
Minimum edition: Community

pubsub​

sukko test validate --suite pubsub --follow

Validates message delivery with channel scoping. Publishes to public, user-scoped (dm.{sub}), and group-scoped (room.{group}) channels. Verifies that each subscriber receives exactly the messages they should and no messages they shouldn't.

Minimum edition: Community

token-revocation PRO​

sukko test validate --suite token-revocation --follow

Validates the token revocation feature: revoke a JWT by token ID (jti) and by subject (sub), then verify that the revoked token is rejected on a new connection. Also verifies that valid tokens are unaffected.

On Community edition, the gated checks are skipped and the suite reports pass with one [skip]. On Pro+, all checks run.

Requires admin keypair (to provision the test tenant).
Minimum edition: Pro for full validation

sse​

sukko test validate --suite sse --follow

Validates the SSE transport: connects as an SSE client, subscribes to channels, verifies message delivery. Confirms SSE clients and WebSocket clients on the same channel both receive messages.

Minimum edition: Pro (the SSE transport is Pro-gated; on Community the gated checks skip and the suite reports pass with [skip])

rest-publish​

sukko test validate --suite rest-publish --follow

Validates the REST publish endpoint: publishes a message via HTTP and verifies it is delivered to WebSocket subscribers on the target channel. After the delivery check passes, the suite also asserts rest publish mid equality — the REST response's mid (the message's stable identity) must be present and equal to the mid on the delivered envelope. In CI this check is required to pass on all Pro and Enterprise cells (both backends).

Minimum edition: Community (/api/v1/publish is open on all editions; the SSE receive leg skips below Pro)

edition-limits​

sukko test validate --suite edition-limits --follow

Validates that edition limits are enforced: verifies tenant count limits, connection count limits, and shard limits match the active edition. Does not test MaxTopicsPerTenant.

Minimum edition: Community

push PRO​

sukko test validate --suite push --follow

Validates the push notification pipeline: device registration, subscription CRUD, credential management, and — when a mock receiver is configured — actual Web Push delivery. The delivery check registers a subscription against the tester's built-in mock WebPush receiver with real P-256 client keys, publishes a message, and asserts the encrypted notification arrives (RFC 8030). The delivery check runs only when TESTER_PUSH_RECEIVER_HOST points at the receiver (it skips on managed deployments where no receiver is reachable); the rest of the suite runs regardless. This delivery path is exercised continuously by the e2e-push-validate CI job against a kafka-mode, Enterprise-licensed stack.

Real delivery is Web Push only

End-to-end delivery is verified for Web Push (VAPID) only. The FCM (Android) and APNs (iOS) legs register a device with a fake provider token and assert the subscription is accepted — they do not deliver to Google/Apple (that requires live external services). A green suite means Web Push is delivered and verified; mobile push is registered but not delivered.

On Community, the suite is skipped. On Pro, the Web Push checks (including real delivery) run; the Enterprise-gated FCM/APNs legs skip.
Minimum edition: Pro (Web Push); Enterprise for the FCM/APNs mobile legs

license-reload​

sukko test validate --suite license-reload --follow

Validates hot license reload: loads a new license key via the webhook endpoint, verifies the edition changes, then restores the original license. Requires TESTER_SIGNING_KEY_FILE — a separate PEM PKCS#8 ECDSA P-256 private key for signing test license tokens.

Minimum edition: Community (tests the reload mechanism, not a specific edition)

api-key​

sukko test validate --suite api-key --follow

Validates static API key authentication in isolation — no JWT provisioning. Confirms that an API-key client is accepted, can subscribe to a public channel, is denied a private channel, and cannot REST-publish.

Prerequisites: none — --suite api-key is sufficient. The tester infers api-key mode from the suite and, when no key or tenant is supplied, self-provisions a throwaway tenant and a run-scoped API key, cleaning both up afterward. To pin a specific key, set TESTER_API_KEY (or pass api_key in the request); an explicitly supplied key or tenant_id is used as-is and never removed.

Minimum edition: Community

upgrade​

sukko test validate --suite upgrade --follow

Validates the auth upgrade flow: the client connects with a static API key, sends an auth message containing a JWT, and asserts that the server responds with auth_ack for a valid JWT and auth_error for an expired one. This suite is the canonical test for the dual-credential connection pattern.

Prerequisites: none — --suite upgrade is sufficient. The tester infers upgrade mode from the suite and, when no key or tenant is supplied, self-provisions a throwaway tenant, a JWT keypair (to mint the upgrade token), and a run-scoped API key (to make the initial connection), cleaning them up afterward. To pin a specific key, set TESTER_API_KEY (or pass api_key in the request); explicitly supplied credentials are used as-is and never removed. The upgrade timeout is controlled by TESTER_AUTH_UPGRADE_TIMEOUT (default 10s, max 60s).

Minimum edition: Community

webhooks​

sukko test validate --suite webhooks --follow

Validates the full webhook delivery pipeline across three scenarios:

ScenarioBehaviourExpected
Happy-pathEndpoint always returns 2001 delivery, HMAC valid
Retry-successFirst 2 attempts → 500, third → 2003 deliveries total
DegradedEndpoint always returns 500 (2 max retries)Webhook status transitions to degraded

Each scenario provisions a throw-away routing rule and webhook, publishes a message via REST, and polls the tester's built-in /webhook-receive/{runID} endpoint to assert delivery count, HMAC signature, and response status codes.

Prerequisites: Set TESTER_WEBHOOK_BASE_URL to the externally-reachable URL of this tester pod (e.g., http://tester.sukko-ci.svc.cluster.local:8085). The webhook-worker must be able to reach this URL to deliver payloads.

Minimum edition: Pro. Requires Kafka backend (MESSAGE_BACKEND=kafka).

Skip behavior: When TESTER_WEBHOOK_BASE_URL is not set, or the current edition is Community, all checks return [skip] and the suite still reports pass.

kafka-ingest​

sukko test validate --suite kafka-ingest --follow

Publishes a message directly to Kafka/Redpanda — bypassing the gateway and ws-server's producer — and verifies a gateway-subscribed client receives it. This exercises the ingestion pipeline (Kafka → ws-server consumer → broadcast → client) in isolation, matching how production data actually arrives (from an upstream producer, not a WebSocket client). It is the only tester suite that connects to the message backend itself; every other suite publishes through the gateway.

To reach a managed Kafka (Confluent Cloud, Redpanda Cloud, AWS MSK, Aiven), configure the tester's Kafka connection env vars: KAFKA_BROKERS, KAFKA_SASL_ENABLED, KAFKA_SASL_MECHANISM (plain for Confluent Cloud, scram-sha-256 for Redpanda Cloud/Aiven, or scram-sha-512 for AWS MSK), KAFKA_SASL_USERNAME, KAFKA_SASL_PASSWORD, KAFKA_TLS_ENABLED, and optionally KAFKA_TLS_CA_PATH. To override the broker addresses for a single run, pass --kafka-brokers. Credentials always come from the tester's environment — never from the CLI.

Prerequisites:

  • The server under test must run MESSAGE_BACKEND=kafka — otherwise records dropped into Kafka are never consumed and the suite times out (the failure message includes this hint).
  • The tester and the server under test must use the same KAFKA_TOPIC_NAMESPACE — set the identical explicit value on both (the namespace is no longer inferred from ENVIRONMENT). This is required whenever KAFKA_BROKERS is set; a per-run override is available via the kafka_topic_namespace request field.

Minimum edition: Community.

Skip behavior: When KAFKA_BROKERS is not set, the suite returns [skip] and the run still reports pass.

Developer-Internal Suites​

These suites validate core WebSocket infrastructure. They run without an admin keypair and pass on Community. Include them in automated CI but they rarely need manual attention:

SuiteWhat it validates
authJWT validation — expired tokens, wrong key, wrong tenant, revoked keys
channelsChannel subscribe and unsubscribe operations
orderingMessage FIFO ordering — messages arrive in publish order
reconnectDisconnect/reconnect recovery with message replay
ratelimitRate limit enforcement — clients throttled at configured limits
tenant-isolationCross-tenant isolation — tenant A cannot see tenant B's messages (requires admin key)

Load, Stress, and Soak Tests​

Load Test​

Sustained load with configurable connections and publish rate. Use this to verify your deployment handles expected traffic.

sukko test load \
  --connections 100 \
  --duration 5m \
  --publish-rate 10 \
  --ramp-rate 5 \
  --follow

Stress Test​

Push beyond expected capacity to find breaking points and verify graceful degradation.

sukko test stress \
  --connections 1000 \
  --duration 10m \
  --ramp-rate 100 \
  --follow

Soak Test​

Long-running stability test. Catches memory leaks, connection drift, and JWT refresh failures.

sukko test soak \
  --connections 100 \
  --duration 2h \
  --publish-rate 10 \
  --follow
FlagLoad/Stress/SoakDescription
--connectionsallNumber of WebSocket connections
--durationallTest duration (30s, 5m, 2h)
--publish-rateload, soakMessages per second
--ramp-rateload, stressConnections per second during ramp-up

Streaming Output​

Add --follow (or -f) to stream metrics in real time:

sukko test load --connections 100 --duration 5m --follow

For machine-parseable output:

sukko test load --connections 100 --duration 5m --follow --output json

Deployment Validation Checklist​

Run these steps in order after deploying to a new environment or after a significant configuration change.

Before starting: Confirm you have run sukko auth keygen before the stack was first started. If not, see the admin keypair setup below.

Step 1 — Environment cleanup​

If re-validating an existing environment, remove leftover tester tenants from a previous run:

# List all tenants
sukko tenants list

# Delete any test tenants (prefixed with tester- or test-)
sukko tenants deprovision tester-pubsub --force
sukko tenants deprovision tester-isolation-a --force
# ... repeat for each leftover test tenant

Step 2 — Health check​

sukko health
sukko status

All services must report healthy before proceeding. If any service is unhealthy, investigate logs before running tests.

Step 3 — Smoke test​

sukko test smoke --follow

The smoke test must pass before running any other suite. If it fails, fix the underlying connectivity issue first.

Step 4 — Admin keypair setup (local)​

For local Docker deployments managed by sukko up, the admin key is mounted automatically from ~/.sukko/admin.key. Verify it is present:

wc -c ~/.sukko/admin.key   # must print 64

If missing, run sukko auth keygen then restart: sukko down && sukko up.

Step 5 — Admin keypair setup (remote)​

For remote/staging deployments, the admin key must be registered in provisioning and the tester must be configured to use it:

# Register the admin public key (run once per environment)
sukko auth register --env staging

# Verify the tester service has TESTER_ADMIN_KEY_FILE set to the
# corresponding private key file on its host

Step 6 — Core suite battery​

Run the operator-relevant suites in order:

sukko test validate --suite provisioning --follow
sukko test validate --suite pubsub --follow
sukko test validate --suite tenant-isolation --follow
sukko test validate --suite rest-publish --follow
sukko test validate --suite edition-limits --follow

Step 7 — Edition-gated suites​

Run only the suites your edition supports:

# Pro+ only
sukko test validate --suite sse --follow
sukko test validate --suite token-revocation --follow

# Pro+ (the FCM/APNs mobile legs skip below Enterprise)
sukko test validate --suite push --follow

Step 8 — Auth mode suites (optional)​

If your deployment uses API key auth or the auth upgrade flow, run:

# No prerequisites — self-provisions a tenant + key when none is supplied
sukko test validate --suite api-key --follow

# No prerequisites — self-provisions a tenant, keypair, and key when none is supplied
sukko test validate --suite upgrade --follow

Step 9 — Webhooks suite (Pro+)​

# Requires TESTER_WEBHOOK_BASE_URL and MESSAGE_BACKEND=kafka
sukko test validate --suite webhooks --follow

All checks are automatically skipped (suite still passes) when the base URL is not set or the edition is Community.

Excluded from standard checklist (developer-internal, automated CI only): auth, channels, ordering, reconnect, ratelimit, license-reload.

Troubleshooting​

Tester service unreachable​

Symptom: sukko test smoke returns connect: connection refused or times out immediately.

Fix: The tester service is not running or not reachable at the configured URL.

  • Local: sukko status — check that the tester service shows healthy. If not, sukko down && sukko up.
  • Remote: verify TESTER_URL in the CLI context points to the correct host and port. The tester listens on port 8090 by default.

Suite fails with 403 EDITION_LIMIT​

Symptom: A suite reports a check failure with message This feature requires pro edition or higher (or enterprise).

Fix: You are running a gated suite on an edition that does not support it. Either:

  • Load a valid Pro/Enterprise license: sukko license push --file <license.key>
  • Accept that the suite will skip the gated checks and report pass with [skip]

Routing rules rejected with 400 TOPIC_NOT_PROVISIONED​

Symptom: The provisioning suite fails when creating a routing rule. Error: TOPIC_NOT_PROVISIONED.

Fix: In the current release, routing rules can only reference topics that already exist (default and dead-letter are always available). A routing rule referencing any other topic suffix will be rejected until that topic is explicitly provisioned. The suite should only reference default — if it references other topics, the tester version may not match the provisioning API version.

pubsub or tenant-isolation fails with 400 (version mismatch)​

Symptom: Suite fails immediately with HTTP 400 and no descriptive message.

Fix: The tester service version does not match the provisioning API version. Ensure the tester and provisioning images are from the same release. Check image tags: sukko status --verbose.

license-reload suite fails at signing step​

Symptom: license-reload suite fails with a key parsing or signature error.

Fix: TESTER_SIGNING_KEY_FILE on the tester service is not set, is set to the wrong file, or contains a key in the wrong format. It must be a PEM PKCS#8 ECDSA P-256 private key — a different format from TESTER_ADMIN_KEY_FILE (which is a raw 64-byte Ed25519 admin key). Verify the file begins with -----BEGIN PRIVATE KEY----- (head -1 "$TESTER_SIGNING_KEY_FILE").

license-reload suite fails to restore original license​

Symptom: The license-reload suite passes the reload check but fails the restore step. After the suite, your deployment may be running on a test edition.

Fix: The suite loads a test license then tries to reload the original. If the original is no longer accessible (e.g., SUKKO_LICENSE_KEY is not set), the restore fails. Set SUKKO_LICENSE_KEY in the tester service environment so it can restore. If the deployment is stuck on the wrong edition after a failed restore, reload the correct license manually: sukko license push --file <license.key>.

RequireAdminProvider: admin keypair not found​

Symptom: The provisioning, tenant-isolation, or token-revocation suite fails at the very first check with a message about missing admin credentials.

Fix: TESTER_ADMIN_KEY_FILE is not set on the tester service, or the file does not exist at the specified path.

  • Local: the key should be mounted automatically by sukko up. If missing, run sukko auth keygen then restart.
  • Remote: set TESTER_ADMIN_KEY_FILE to the path of the raw 64-byte admin private key on the tester host.

edition-limits suite leaves orphaned test tenants​

Symptom: After the edition-limits suite, sukko tenants list shows tenants prefixed with tester-limits-.

Fix: The suite creates test tenants and normally cleans them up. If it fails mid-run, tenants are left behind. Clean up manually:

sukko tenants deprovision tester-limits-1 --force
sukko tenants deprovision tester-limits-2 --force
# ... repeat as needed

Smoke test times out​

Symptom: sukko test smoke hangs for 30 seconds then returns a timeout error.

Fix: The gateway is not accepting WebSocket connections. Check:

  1. sukko health — all services healthy?
  2. Is the gateway port (default 3000) accessible from the tester service?
  3. Are there firewall rules blocking WebSocket upgrade requests?

Context points to wrong environment​

Symptom: Tests pass locally but fail against the remote context, or vice versa — in ways that suggest the wrong URL is being used.

Fix: Confirm the active context:

sukko context current

Switch if needed: sukko context use <name>. For a one-off test against a different environment without switching: sukko --context staging test smoke.

Admin key silently ignored (catch-22)​

Symptom: The stack is running but the provisioning suite fails at admin setup even though TESTER_ADMIN_KEY_FILE is set correctly.

Root cause: sukko up was run before sukko auth keygen. The provisioning service started without ADMIN_BOOTSTRAP_KEY and registered no admin key. The tester has a valid private key but it has no corresponding public key registered in provisioning, so admin JWTs are rejected.

Fix: Run sukko auth keygen to generate the keypair, then restart the stack so provisioning starts with the bootstrap key: sukko down && sukko up.

TESTER_ADMIN_KEY_ID mismatch​

Symptom: The provisioning, tenant-isolation, or token-revocation suite fails at admin setup with a JWT validation error (wrong_kid or invalid_key_id), even though TESTER_ADMIN_KEY_FILE is set and the file contains 64 bytes.

Root cause: The kid (key ID) in JWTs issued by the tester must match the ID under which the public key was registered in the provisioning service. TESTER_ADMIN_KEY_ID defaults to bootstrap-0. If the key was registered under a different ID, JWTs are rejected with wrong_kid.

Fix: Set TESTER_ADMIN_KEY_ID to the key ID used during sukko auth keygen (shown in the output). The default is bootstrap-0 when sukko auth keygen is run without flags.

TESTER_ADMIN_KEY_FILE missing at startup​

Symptom: The tester service fails to start entirely — it does not serve any HTTP requests and the container exits immediately with a startup error referencing the admin key file.

Root cause: TESTER_ADMIN_KEY_FILE is set to a path that does not exist on the tester container. The tester validates the key file path at startup and exits if the file cannot be opened. This is a startup failure, not a test failure — suites will not run at all.

Fix: Verify the file exists at the configured path on the tester container or pod. In local Docker, sukko up mounts ~/.sukko/admin.key automatically — run sukko auth keygen first, then sukko down && sukko up. On Kubernetes, confirm the secret volume or ConfigMap is mounted at the correct path.

Webhooks suite skips everything​

Symptom: sukko test validate --suite webhooks reports pass with all checks returning [skip].

Cause: Either (a) TESTER_WEBHOOK_BASE_URL is not set, or (b) the current edition is Community. Both conditions cause the suite to skip all checks rather than fail.

Fix: For (a), set TESTER_WEBHOOK_BASE_URL to the externally-reachable HTTP URL of the tester pod — the webhook-worker must be able to reach this URL to deliver payloads. For (b), the webhooks suite requires Pro edition or higher.

Orphaned tester tenants from previous runs​

Symptom: The pubsub, provisioning, or tenant-isolation suite fails immediately with 409 TENANT_ALREADY_EXISTS.

Fix: A previous test run was interrupted and left test tenants behind. Clean up:

sukko tenants list
# Delete any tenants prefixed with tester-
sukko tenants deprovision <tester-tenant-id> --force

Next Steps​