Vidocq Documentation
Version

This page consolidates the status of the official Jakarta Servlet 6.1 TCK runs against Foy. It is regenerated on every release. The day-to-day source of truth remains TCK.md at the repo root.

Summary

This is a measurement, not a conformance claim. At the latest full run (jakarta.tck:servlet-tck-runtime:6.1.0, Phase 5 exit, 2026-10-10):

  • 1714 tests run: 1659 passing, 7 skipped, 0 failures / 48 errors — 96.8 % of the tests run pass (Phase 4 exit: 1649; Phase 3 exit: 1587). A skipped test is never counted as passing;

  • Phase 5 added response streaming (commit on buffer overflow and flushBuffer), response trailers, the async lifecycle (timeout routed to an error dispatch, AsyncListener events), non-blocking I/O (ReadListener, WriteListener) and HTTP Upgrade (HttpUpgradeHandler, WebConnection); no class regressed and 6 improved (10 tests);

  • the 48 remaining errors are security (37: FORM, BASIC, client certificate, deny-uncovered), multipart (8) and JSP-dependent tests (3);

  • the 7 skipped tests are HTTP/2 server push, an accepted gap (see What is skipped).

Per-family breakdown (passing / run):

Family Run Pass Skipped Score Reading

api.* (core engine)

859

848

0

98.7 %

✅ The Servlet engine itself is solid; the rest is multipart (8) and JSP (3)

pluggability.*

646

646

0

100 %

✅ Fragments, ordering and initializers

spec.* (behavioral, including security)

207

163

7

78.7 %

⚠️ Security enforcement (37); server push skipped (7)

compat.*

2

2

0

100 %

✅ Leading-slash dispatch compat (legacy 2.2/2.3 descriptors)

Exact figures live in target/surefire-reports/ after an --all run and in TCK.md (per-family tallies, root causes §3, roadmap §4). For historical detail, browse git log — TCK.md on the main branch.

What is skipped

The full run reports Skipped: 7, all in spec.serverpush.ServerPushTests:

  • HTTP/2 server push is an accepted gap: chappe has no h2c Upgrade nor PUSH_PROMISE writer, and push is deprecated in Servlet 6.1 in favour of 103 Early Hints. HttpServletRequest.newPushBuilder() returns null, and the TCK’s own switch servlet.tck.support.http2Push=false (a surefire system property of the tck profile in foy-tck/pom.xml) skips the 7 tests that need push. The eighth test of the class runs and passes.

No other test battery is excluded. Two more product decisions show up as errors rather than skips:

  • No JSP engine in Foy itself — the 3 JSP-dependent ServletContext40Tests fail. A JSP-engine SPI in Foy, plugged by Ibarra (Jakarta Pages 4.0), is the planned way to run them. For dynamic pages today, use Cassini (REST) or pure Servlets.

  • Hang safety net — some TCK clients read raw sockets without SO_TIMEOUT and would hang forever on an unanswered request; foy-tck sets a global JUnit 120 s timeout (SEPARATE_THREAD mode). No test hits it at the Phase 5 exit.

Reading the logs

The run log carries expected WARNING lines that are not failures. servlet.tck.api.jakarta_servlet.asynccontext.ACListener2 throws an IOException from onComplete on purpose, and Foy logs every listener failure at WARNING with its stack trace (AsyncListener.onComplete threw).

Per-class figures come from foy-tck/tck-tally.sh <log>, which prints <class> <run> <bad> <skipped>, where bad (the third column) is failures + errors; a class passes run - bad - skipped tests. Comparing two tallies is a LC_ALL=C join (both files sorted with LC_ALL=C); the script header gives the exact commands.

Runner architecture

foy-tck is an in-reactor module gated behind the tck Maven profile (TCK harmonisation, same pattern as the vidocq-runtime-tck-* runners): a plain mvn install neither downloads nor runs anything TCK-related, and the release reactor never sees the module. Invoke through ./run-official-tck-servlet6.1.sh, or directly via ./mvnw -Ptck,tck-official -pl foy-tck test.

Historical note. The module used to be excluded from the reactor as a standalone Model 4.0.0 POM: ShrinkWrap Maven Resolver 3.3 (transitive dependency of the official Jakarta TCK) could not parse the Model 4.1.0 POMs the workspace used at the time (Bad artifact coordinates …​ jar:). That constraint disappeared with the workspace-wide migration to Maven 3.9.16 / Model 4.0.0.

Reproduce locally

Prerequisites

Install the official Jakarta Servlet 6.1 TCK artefacts (non-public) into your local M2:

  • jakarta.tck:servlet-tck-runtime:6.1.0

  • jakarta.tck:servlet-tck-util:6.1.0

  • jakarta.tck:servlet-tck:6.1.0 (pom)

Detailed procedure in foy-tck/README.md.

Run the TCK

cd foy
./run-official-tck-servlet6.1.sh                     # smoke test (fast)
./run-official-tck-servlet6.1.sh --all               # full suite (long)
./run-official-tck-servlet6.1.sh -Dtest=ServletTests # one class
./run-official-tck-servlet6.1.sh --family compat     # one family: api | spec | pluggability | compat
./run-official-tck-servlet6.1.sh --failing           # only the classes failing in tck-baseline.txt
./run-official-tck-servlet6.1.sh --no-install --family api   # skip the reactor install
./run-official-tck-servlet6.1.sh --dry-run --failing # print the Maven commands only

--no-install skips the reactor build (use it when the code is unchanged), --family and --failing select tests (--failing reads the bad count — failures + errors, the third column — of foy-tck/tck-baseline.txt, or the tally file given as argument), --dry-run prints the Maven commands, and --help lists every mode.

CI integration

- name: Build foy reactor
  run: cd foy && ./mvnw -ntp install -DskipTests

- name: Run Jakarta Servlet 6.1 TCK
  run: cd foy && ./run-official-tck-servlet6.1.sh --all
  # Prereq: official TCK artefacts cached in ~/.m2

Surefire reports live in foy-tck/target/surefire-reports/.