Running our OpenID server through the official conformance suite, and what it found in us

redb.Identity

We were sure our OpenID server returned profile claims correctly. Unit tests green, demo run green, a manual curl check showed name, family_name, email, all there. Then we ran the server through the OpenID Foundation conformance suite, and it showed the user's phone number and address sitting in the id_token. An id_token is forwarded to third parties and logged as proof of sign-in. So the user's phone number travelled far further than the client ever asked for. That is a PII leak, and none of our tests caught it, because we did not know what to look for.

That is what this article is about: how the OIDF conformance suite works, how to set it up and prepare your server for it, why you end up running it by hand through a real browser, and what it found in us. And why running such a suite is worth it for anyone writing their own identity server, not only those going for certification.

Why an external arbiter at all

OAuth 2.1 and OpenID Connect are not one document but a stack of RFCs and specs where it is easy to get things "almost right". A token is issued, sign-in works, the demo runs, and a detail that misses the spec lives on unnoticed until a specific client or auditor trips over it. The classic ones:

  • a claim ends up in the wrong token (id_token instead of userinfo) and leaks where it should not;
  • userinfo returns extra fields because it filters by a deny-list rather than an allow-list;
  • the discovery document promises a feature the endpoint does not have, or the other way round;
  • an authorization code is accepted twice because of a race in single-use enforcement.

Your own tests will not catch this, for a simple reason: a test checks what you know. The spec is large, nobody holds it all in their head, and the blind spots sit exactly where you are confident everything is fine. A conformance suite is an external arbiter that reads the spec for you and drives the server by its letter, not by your expectations.

The OpenID Foundation maintains the official conformance suite, the very one real certification runs on. You can stand it up locally and run your server for free, without any certification application. That is what we did.

Setting the suite up

The suite itself is a Java application in Docker: a test server, nginx, and MongoDB. It acts as the client (the Relying Party): it registers applications with your OP, drives the flows, inspects the responses, and diffs them against the spec. It comes up with a plain compose:

docker compose -f conformance/docker-compose.yml up -d   # server + nginx + mongodb

The UI opens at https://localhost.emobix.co.uk:8443 (the domain resolves to localhost, and the suite ships its own certificate). From there you need a plan, not individual tests. We generate it with a script that:

  1. checks that your OP's discovery responds and the issuer is the expected one;
  2. registers a few clients via DCR (count matters, see below);
  3. creates a plan for the oidcc-basic-certification-test-plan profile with the variant server_metadata=discovery, client_registration=static_client;
  4. prints the plan link in the UI.

Three things in this setup are non-obvious, and each cost us time.

Issuer and real HTTPS. The suite requires the issuer in discovery to match the address it reaches you at, over HTTPS. We ran the OP at https://host.docker.internal:5002 with a self-signed certificate: the suite inside Docker reaches the host by that name, and redirects come back to its own callback. If the issuer in your discovery is not host.docker.internal, the plan will not create, the script checks this first.

Three clients in the plan, not two. The oidcc-server-client-secret-post module internally overwrites client with the value under the client_secret_post key. If there is no such client in the plan config, the test fails before it issues a single request to the OP with a "no client" message, even though a client is right there. We dug this out by decompiling the suite jar: the plan needs a third client registered under that key.

A fully populated test user. The oidcc-scope-* tests sign in as a live user and check that userinfo returned every claim the scope implies. If the user is missing birthdate, zoneinfo, address, and the rest of OIDC §5.1, the suite raises a WARNING per missing claim. The run user must be filled in completely; a separate seed script does that for us.

Why by hand, not by script

The suite has a built-in "browser" (HtmlUnit) that walks the authorize link itself. On paper this lets you run a plan automatically. In practice it chokes on the Bootstrap 5.3.3 that the suite's own callback page pulls from a CDN: page processing dies, the callback never posts back, and tests hang in WAITING forever.

So we ran the plan by hand through a real Chrome, which is, incidentally, exactly how real OIDF certification is performed. The mechanics are simple: the plan carries no browser section, so the suite shows an authorization link for each test and waits for a live visit. You log in once, the session then lives in the browser, and subsequent tests redirect straight through. A few tests (prompt=login, max_age) deliberately ask for the password again, that is the point.

The run goes one test at a time: all tests in a plan share one callback alias, and starting the next before the previous reached FINISHED kills the previous one.

What the suite found in us

This is where it gets interesting. The first version of our write-up blamed several non-passing modules on "harness limitations" and "suite configuration". When we stopped waving them away and dug in, it turned out almost all of it was our real defects. It is worth listing them plainly.

PII in the id_token. The oidcc-server module showed that scope-derived claims (profile, email, phone, address) were baked into the id_token. In the authorization-code flow they belong in UserInfo (OIDC §5.4), not the id_token. The distinction matters: the id_token is forwarded to third parties and logged as proof of sign-in. The user's phone number and address travelled further than the client asked. The fix: scope-derived claims are now AccessToken-destination only, served from /connect/userinfo and absent from the id_token.

UserInfo returning extra fields. UserInfo copied claims through a deny-list and dragged along the token's plumbing: OpenIddict's oi_* internals, jti, exp, iat, at_hash. All of that describes the token, not the user (§5.3). A deny-list is leaky by definition: forget to forbid one thing and it escapes. We replaced it with an explicit allow-list.

An incomplete set of profile claims. oidcc-scope-address/phone/all raised WARNING, and it was not "the client wasn't registered with those scopes". The suite diffs userinfo against the exact OIDC §5.1 list and warns per missing claim. We were emitting an incomplete set. The fix: the full §5.1 set, with updated_at as a JSON number and *_verified as JSON booleans, the types get checked too.

The claims parameter was missing. oidcc-claims-essential requests name as an essential claim through the claims parameter (§5.5). We had not implemented it at all, and our own discovery honestly said claims_parameter_supported: false. We implemented §5.5; discovery now says true.

UserInfo did not accept the token in the body. oidcc-userinfo-post-body: POST to /connect/userinfo did not accept the access token in the form-encoded body (RFC 6750 §2.2). We fixed the form-to-body mapping.

Note the nature of these bugs. Not one of them breaks sign-in. Everything works, the demo is green, the user logs in. The defect is in where a claim went, and in discovery promising something the endpoint does not do. Only someone reading the spec line by line catches this. In our case the suite did the reading.

The result, and an honest read of the non-PASSED

Basic OP profile, 35 modules, 0 failures. The breakdown: 29 PASSED, 3 REVIEW, 1 WARNING, 2 SKIPPED. The last three categories are worth spelling out, because to an outsider SKIPPED and WARNING in a report look like failure, and they are not.

3 REVIEW are tests where the suite wants a screenshot uploaded as evidence (prompt-login, max-age, ensure-registered-redirect-uri). The server handles them correctly end to end; the suite simply also wants visual proof and therefore does not auto-finish them. Pass-equivalent.

1 WARNING is oidcc-server flagging two claims in the id_token that we put there deliberately (internal oi_tkn_id and redb:user_id for the server's own needs). That is our decision, not a defect.

2 SKIPPED, and here is the important part. Both skipped modules test the unsigned (alg:none) request object, a JWT with no signature. The suite skips them itself when the server does not advertise alg:none support in request_object_signing_alg_values_supported. We do not advertise it on purpose: alg:none throws away the exact integrity guarantee the JWT-Secured Authorization Request (RFC 9101) exists for, and FAPI 2.0 forbids it outright. SKIPPED here means "the server declined to advertise an unsafe mode", which is the correct answer, not a missing feature. Turning either skip into a pass would require accepting alg:none, a security regression.

So read the badge like this: two skips, zero failures, and both skips are the server refusing an unsafe mode that the security profiles tell it to refuse.

What sits underneath: our own checks

The conformance suite is an external arbiter, but running it by hand on every commit is not realistic. So underneath it sit two of our own layers that catch regressions automatically.

Unit and integration tests, 1824 of them, green across three providers (PostgreSQL, SQL Server, SQLite). This is not just "a token was issued": there are about 38 areas, from protocol flows to storage internals. TokenFlow (introspection, revocation, refresh rotation, device code, single-use code), FullStack (end-to-end HTTP runs of authorize→token→userinfo), Scim (user and group provisioning, bulk, ETag), Federation (external OIDC providers), Dpop, Consent, MFA, DataProtection, TxIntegrity (atomicity under concurrency). Every suite finding, once fixed, is pinned by a test so it does not come back.

A demo run, 62 probes against a live server. This is a separate layer with a different job: not a unit check in isolation, but RFC probes against a running OP over real HTTP. authcode_pkce, device_code, client_credentials, token_exchange, dpop, par, jar_request_object, private_key_jwt, mfa_totp, scim_enterprise, backchannel_logout, throttle_rfc6585, and so on. The principle here matters: a failing demo is a server bug, not a reason to widen the list of accepted response codes. A demo probe reproduces the behaviour the spec expects; if the server answers otherwise, the server gets fixed.

The layering is deliberate. Unit tests are fast and catch logic. Demos check what is only visible on a live server over HTTP: headers, codes, discovery shape. The suite is the external arbiter on the letter of the spec. Each layer catches what the others miss.

Why anyone should do this

If you are writing your own OpenID/OAuth server, run it through the OIDF conformance suite locally, even if you do not need certification. The reason is simple: a claim in the wrong token, or discovery that promises the wrong thing, is not theory. We were returning the user's phone number in the id_token and were sure everything was right, until an external arbiter showed otherwise. Our own tests did not find it, because we did not know it was a bug.

The suite is free, comes up in Docker, and a single Basic OP run takes an evening. In return you get a line-by-line diff of your server against the spec by someone who actually read it, the OpenID Foundation's machine. For an identity server that people trust with sign-in, that is far cheaper than learning about the leak from someone who found it with less friendly intent.

One more note, on reading the report honestly. SKIPPED and WARNING in a conformance output are not necessarily failure; often they are exactly the place where the server correctly refused to do something unsafe. Read the reason of a line, not its colour.

We do not carry the OpenID Certified™ mark, that is a trademark, granted by the Foundation through a separate process. This is specifically about a local run of the official suite: it honestly shows where the server diverges from the spec, and that alone is enormous value, independent of formal certification.

More of my writing: redbase.app/articles, and on dev.to.