>` placeholder, bolds table headers, and applies Courier New to code-like spans. Temporary artifacts (`*.temp.md`, `*.temp.docx`, `.temp-mermaid/`) are created during conversion and cleaned up automatically. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### `Missing script: "export-word"` [Section titled “Missing script: "export-word"”](#missing-script-export-word) You’re on an old checkout. The scripts `export-word`/`md-to-docx` are defined in `package.json` and run `scripts/md-to-docx.cjs`. Pull the latest and `npm install`. ### `require is not defined` when running the script directly [Section titled “require is not defined when running the script directly”](#require-is-not-defined-when-running-the-script-directly) Run it through npm (`npm run export-word`) or call the `.cjs` file (`node scripts/md-to-docx.cjs`). The script is CommonJS; the package is `"type": "module"`, so a plain `.js` extension would be treated as ESM and fail. ### Pandoc not found [Section titled “Pandoc not found”](#pandoc-not-found) Install Pandoc: `brew install pandoc` (macOS), then `pandoc --version`. ### `ModuleNotFoundError: No module named 'docx'` [Section titled “ModuleNotFoundError: No module named 'docx'”](#modulenotfounderror-no-module-named-docx) `pip3 install --break-system-packages python-docx` ### Mermaid diagram conversion fails (`could not determine executable to run`) [Section titled “Mermaid diagram conversion fails (could not determine executable to run)”](#mermaid-diagram-conversion-fails-could-not-determine-executable-to-run) mermaid-cli isn’t available. Either install it (`npm install --save-dev @mermaid-js/mermaid-cli`) or ignore it — the export still completes and leaves those diagrams as code blocks (non-fatal). ### Word shows “unknown content” / offers to repair on open [Section titled “Word shows “unknown content” / offers to repair on open”](#word-shows-unknown-content--offers-to-repair-on-open) Click **Yes**; Word auto-repairs without data loss. This is a minor DOCX XML quirk. ## Related files [Section titled “Related files”](#related-files) * `scripts/md-to-docx.cjs` — conversion orchestrator (template path, naming, Pandoc + merge) * `scripts/merge-docx-template-v3.py` — colophon/header merge and formatting * `word-template/20251111_DIL_CTN_file_name.docx` — the reference template ## Summary [Section titled “Summary”](#summary) ```bash npm run sync-arc42 # (optional) refresh arc42 from CNCL-BDI npm run export-word # convert docs/ -> Export/*.docx using the template open Export/ # view generated documents rm -rf Export/ # clean output ``` # Architecture Diagram **Last Updated:** October 20, 2025 **Status:** Living Document *** ## System Architecture Overview [Section titled “System Architecture Overview”](#system-architecture-overview) This interactive architecture diagram provides a comprehensive view of the CTN Association Register system components, their relationships, and data flows. The diagram is maintained in IcePanel and represents the current state of the system architecture. *** ## Interactive Diagram [Section titled “Interactive Diagram”](#interactive-diagram) *** ## How to Use [Section titled “How to Use”](#how-to-use) * **Zoom**: Use the zoom controls or mouse wheel to zoom in/out * **Pan**: Click and drag to navigate the diagram * **Interact**: Click on components to see details and relationships * **Fullscreen**: Use the fullscreen button for a larger view *** ## Related Documentation [Section titled “Related Documentation”](#related-documentation) * [arc42 Architecture Documentation](/arc42/ctn-arc42-index/) * [Building Blocks](/arc42/05-building-blocks/ctn-building-blocks/) * [Deployment View](/arc42/07-deployment/ctn-deployment/) * [System Context](/arc42/03-context/ctn-context/) # Query the arc42 Documentation with AI Tools > How team members can connect the CTN arc42 documentation (GitHub) to Claude, ChatGPT and Google Gemini to ask questions about the architecture This guide explains how anyone in the team can ask questions about the CTN arc42 architecture documentation using their own AI assistant — including **Claude**, **ChatGPT**, **Google Gemini**, **GitHub Copilot**, **Perplexity**, **Cursor** and **JetBrains Junie** — by connecting it to the GitHub repository where the documentation lives. The tools fall into three groups: assistants that connect to GitHub directly (Claude, ChatGPT, Copilot, Gemini), AI code editors that read the project after you open it (Cursor, JetBrains Junie), and tools that work from a public URL (Perplexity). Pick whichever you already use. ## Overview [Section titled “Overview”](#overview) The arc42 documentation is the single source of truth for the CTN architecture. The source markdown files live in the public GitHub repository: * **Repository:** [`Basic-Data-Infrastructure/CNCL-BDI`](https://github.com/Basic-Data-Infrastructure/CNCL-BDI) * **Folder:** [`docs/arc42`](https://github.com/Basic-Data-Infrastructure/CNCL-BDI/tree/main/docs/arc42) Because the repository is public, every team member can point their AI tool at it and ask questions such as *“How does association onboarding work?”* or *“Which ADR describes the token claim composition?”* — and get answers grounded in the actual documentation, with references back to the source files. > The AI provider interfaces below change regularly and most of these connectors are still in beta. If a menu item looks slightly different, search the tool’s help for “GitHub” — the overall flow is the same: connect GitHub, select this repository, then ask your question. ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * An account for the AI tool you want to use (Claude, ChatGPT or Gemini). * Access to the repository. It is public, so no special permissions are needed for reading. If it is ever made private, you will need to authorize the connector with a GitHub account that has access (and possibly GitHub-organization admin approval). *** ## Option 1 — Claude (claude.ai) [Section titled “Option 1 — Claude (claude.ai)”](#option-1--claude-claudeai) Claude has a built-in GitHub integration that lets you attach repository files directly to a conversation. 1. Open a new chat at [claude.ai](https://claude.ai). 2. Click the **+** button in the lower-left corner of the message box. 3. Choose **Add from GitHub**. 4. If this is your first time, you will be redirected to GitHub to **authenticate / authorize** Claude. Approve access. 5. In the file browser, navigate to `Basic-Data-Infrastructure/CNCL-BDI` and select the **`docs/arc42`** folder (or specific files you care about). 6. Send your question. Claude reads the selected content and answers based on it. **Tips** * You can add multiple folders or repositories as long as they fit in Claude’s context window. For broad questions, selecting the whole `docs/arc42` folder works well. * When the documentation is updated, click **Sync now** to pull the latest changes into your chat. *** ## Option 2 — ChatGPT (chatgpt.com) [Section titled “Option 2 — ChatGPT (chatgpt.com)”](#option-2--chatgpt-chatgptcom) ChatGPT connects to GitHub as an “app” / connector, after which it can read code, README and documentation files from the repositories you grant access to. 1. In ChatGPT, open **Settings → Apps** (also called Connectors). 2. Find **GitHub** in the app directory and click to connect. 3. You are sent to GitHub to **install and authorize** the ChatGPT app. Grant it access to the `CNCL-BDI` repository (or all repositories you are allowed to share). 4. Wait a few minutes — there is typically a **\~5-minute delay** before a repository shows up as available in ChatGPT. 5. Start a chat, enable the **GitHub** connector (or use **Deep research → GitHub** for thorough, cited answers), and ask your question about the arc42 docs. **Tips** * If the repository does not appear, open **Settings → Apps → GitHub → Choose repositories** and make sure `CNCL-BDI` is selected. * For a brand-new or private repository that will not show up, you can nudge GitHub’s index by running a search on GitHub itself in the form `repo:Basic-Data-Infrastructure/CNCL-BDI`. *** ## Option 3 — Google Gemini (gemini.google.com) [Section titled “Option 3 — Google Gemini (gemini.google.com)”](#option-3--google-gemini-geminigooglecom) Google’s general-purpose AI assistant is **Gemini**. The Gemini web app can import a GitHub repository directly into a prompt. 1. Open [gemini.google.com](https://gemini.google.com). 2. In the prompt box, click **Add file** (the attach/+ control). 3. Choose **Import code**. 4. Paste the repository or branch URL: `https://github.com/Basic-Data-Infrastructure/CNCL-BDI` (you can also point at the `docs/arc42` path or a specific branch). 5. Click **Import**, then ask your question about the architecture. ### Recommended alternative: NotebookLM [Section titled “Recommended alternative: NotebookLM”](#recommended-alternative-notebooklm) For document-style question-and-answer work, Google’s **[NotebookLM](https://notebooklm.google.com)** is often a better fit than the Gemini chat app. It gives source-grounded, citation-backed answers from exactly the documents you add. 1. Open NotebookLM and create a new notebook. 2. Add the arc42 documents as sources — the simplest approach is to paste the GitHub file/folder URLs, or download the documents and upload the markdown files. 3. Ask questions; NotebookLM answers only from the sources you provided and cites them. *** ## Option 4 — GitHub Copilot [Section titled “Option 4 — GitHub Copilot”](#option-4--github-copilot) If you already use **GitHub Copilot**, you can ask questions about the repository without any extra setup — Copilot Chat works directly on GitHub when you are in the context of the repository. 1. Open the repository on GitHub: [`Basic-Data-Infrastructure/CNCL-BDI`](https://github.com/Basic-Data-Infrastructure/CNCL-BDI). 2. Click the **Copilot icon** in the top navigation bar. A chat panel opens on the page you are viewing. 3. Ask your question. Because you are in the repository’s context, Copilot automatically indexes it and answers based on its contents — e.g. *“Summarise the arc42 deployment chapter in docs/arc42.”* 4. As you browse to specific files, pull requests or issues, they are attached as context, so you can narrow your questions to, for example, `docs/arc42/09-decisions`. **Tips** * The same Copilot Chat is available in your **IDE** (VS Code, Visual Studio, JetBrains) and in **GitHub Mobile** — handy if you prefer to work where the code is. * A GitHub Copilot license is required (individual or provided through your organization). *** ## Option 5 — Perplexity [Section titled “Option 5 — Perplexity”](#option-5--perplexity) Perplexity has no GitHub connector, but it reads public web pages — which is all you need, because the repository is public. 1. Open [perplexity.ai](https://www.perplexity.ai). 2. Paste a public URL into the prompt and ask your question. Two good choices: * A specific file or folder in the repo, e.g. a chapter under [`docs/arc42`](https://github.com/Basic-Data-Infrastructure/CNCL-BDI/tree/main/docs/arc42). * The published **`llms-full.txt`** (the entire documentation in one file — see *Alternative* below), which gives Perplexity the full context in a single link. 3. Perplexity answers with citations back to the source. **Tip:** For repeated use, create a **Space** in Perplexity and add the URLs as sources, so every question in that Space is grounded in the arc42 documentation. *** ## Option 6 — Cursor [Section titled “Option 6 — Cursor”](#option-6--cursor) Cursor is an AI code editor. It is a good fit for team members who already work in an IDE, because it answers from the project you have open. 1. Clone the repository (or open a copy you already have): ```bash git clone https://github.com/Basic-Data-Infrastructure/CNCL-BDI ``` 2. Open the folder in Cursor. 3. Open the chat (`Cmd/Ctrl + L`), reference the docs with `@` — for example `@docs/arc42` or `@Codebase` — and ask your question. Cursor indexes the project and answers from it. *** ## Option 7 — JetBrains Junie [Section titled “Option 7 — JetBrains Junie”](#option-7--jetbrains-junie) **Junie** is JetBrains’ AI coding agent (it creates the `.junie` folder in a project). This is the setup used at Conclusion. 1. Open the `CNCL-BDI` project in a JetBrains IDE (IntelliJ IDEA, PyCharm, etc.) with the Junie plugin — or run the Junie CLI from inside the project folder. 2. In the Junie chat, reference files or folders with `@` (e.g. `@docs/arc42`) and ask your question. Junie reads the referenced documents and answers from them. 3. Project context for Junie lives in **`.junie/guidelines.md`** (or `AGENTS.md`) in the repository root. Because it is committed to the repo, the whole team shares the same context and gets consistent answers. The arc42 documents themselves are in `docs/arc42`. *** ## What you can ask [Section titled “What you can ask”](#what-you-can-ask) Once connected, useful questions include: * “Summarise chapter 5 (Building Blocks) of the CTN arc42 documentation.” * “Which architecture decision (ADR) covers external IdP federation, and what was decided?” * “Explain the difference between users and members in the CTN context.” * “What are the main risks listed in the arc42 documentation?” * “Define the glossary terms used in the association onboarding flow.” ## Keeping answers up to date [Section titled “Keeping answers up to date”](#keeping-answers-up-to-date) These tools read a snapshot of the repository at the time you connect or sync. After the documentation is updated on GitHub, use the tool’s refresh/sync option (or re-import) so the AI sees the latest version. ## Alternative: the published documentation site [Section titled “Alternative: the published documentation site”](#alternative-the-published-documentation-site) If you would rather not connect GitHub at all, the CTN documentation portal automatically publishes machine-readable versions of the docs for AI tools: * **Compact index:** <https://docs.preview.inlandbase.com/llms.txt> * **Full documentation (single file):** <https://docs.preview.inlandbase.com/llms-full.txt> Paste either URL into any AI tool that accepts a link (Perplexity, Claude, ChatGPT, Gemini, …) and ask your questions from there. The `llms-full.txt` file contains the complete documentation, so it gives the AI the most context in a single link. ## References [Section titled “References”](#references) * Claude — [Use the GitHub integration](https://support.claude.com/en/articles/10167454-use-the-github-integration) * ChatGPT — [Connecting GitHub to ChatGPT](https://help.openai.com/en/articles/11145903-connecting-github-to-chatgpt) * Gemini — [Import a GitHub repository & ask about it in the Gemini web app](https://support.google.com/gemini/answer/16176929) * GitHub Copilot — [Asking GitHub Copilot questions in GitHub](https://docs.github.com/en/copilot/how-tos/chat-with-copilot/chat-in-github) * Perplexity — [perplexity.ai](https://www.perplexity.ai) * Cursor — [Cursor documentation](https://docs.cursor.com) * JetBrains Junie — [Getting started with Junie](https://junie.jetbrains.com/docs/) · [Guidelines and memory](https://junie.jetbrains.com/docs/guidelines-and-memory.html) # How To: Update Arc42 Documentation **Last Updated:** May 31, 2026 **Difficulty:** Beginner **Estimated Time:** 10–15 minutes *** ## Overview [Section titled “Overview”](#overview) This guide explains how to update or add arc42 architecture documentation **and** publish it to the CTN Documentation Portal. The most important thing to understand first: **there are two repositories.** | Repository | Where | What it is | | --------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------ | | **CNCL-BDI** (source) | GitHub: `Basic-Data-Infrastructure/CNCL-BDI` | The **single source of truth** for all arc42 docs. You edit content **here**. | | **CTN-Documentation** (publishing site) | Azure DevOps: `DeNoronhaConsulting/CTN Documentation` | The **Astro + Starlight** website that publishes the docs. It does **not** author content. | > ⚠️ **Do not edit `docs/arc42/` inside the CTN-Documentation repo.** That folder is generated by the sync step and will be **wiped and overwritten** the next time anyone runs `npm run sync-arc42`. Any edit you make there directly will be lost. Edit the docs in **CNCL-BDI** instead. How content reaches the live site: ``` graph LR A[Edit docs in CNCL-BDI/docs/arc42] --> B[Push to GitHub] B --> C[npm run sync-arc42 in CTN-Documentation] C --> D[npm run build -> dist/] D --> E[Commit docs/arc42 + push to Azure DevOps] E --> F[Azure Pipeline builds & deploys] F --> G[Live site] ``` > **Note:** This how-to guide itself (and everything under `docs/how-to/`) **is** authored directly in the CTN-Documentation repo — only `docs/arc42/` is synced from CNCL-BDI. *** ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * ✅ Git access to **both** repos: CNCL-BDI (GitHub) and CTN-Documentation (Azure DevOps) * ✅ The two repos checked out **as siblings** in the same parent folder: ```plaintext <parent>/ ├── CNCL-BDI/ # source content └── CTN-Documentation/ # publishing site ``` (The sync script looks for `../CNCL-BDI/docs/arc42` by default; override with the `ARC42_SOURCE` env var.) * ✅ **Node.js ≥ 22.12.0** — required by Astro 6. Node 20 will fail with *“not supported by Astro!”*. Use `nvm use 22` or a Node ≥ 22 install. * ✅ A text/markdown editor (VS Code, etc.) *** ## Document Status Workflow [Section titled “Document Status Workflow”](#document-status-workflow) All arc42 documents use status metadata in frontmatter to track their lifecycle: | Status | Description | When to use | | ------------- | ------------------ | ----------------------------------- | | **Draft** | Initial work | Being written, not ready for review | | **Review** | Ready for feedback | Complete enough for team review | | **Approved** | Finalized | Reviewed and approved | | **Published** | Active version | Live and in use | | **Archived** | Old version | Superseded or no longer relevant | ```yaml --- status: Published lastUpdate: 2026-05-31 --- ``` *** ## Quick Reference [Section titled “Quick Reference”](#quick-reference) **TL;DR — edit in the source repo, sync, build, publish:** ```bash # 1. Edit content in the SOURCE repo (CNCL-BDI) cd ../CNCL-BDI code docs/arc42/05-building-blocks/ctn-connector.md git add docs/arc42 && git commit -m "docs: update connector" && git push # to GitHub # 2. Sync + build + publish from the PUBLISHING repo (CTN-Documentation) cd ../CTN-Documentation nvm use 22 # ensure Node >= 22.12.0 npm run sync-arc42 # copy CNCL-BDI/docs/arc42 -> docs/arc42 (wipe + replace) npm run build # prepare-content + astro build -> dist/ (also validates links) git add docs/arc42 && git commit -m "docs: sync arc42 from CNCL-BDI" && git push # to Azure DevOps # 3. Done — the Azure pipeline builds and deploys automatically ``` *** ## Step-by-Step Process [Section titled “Step-by-Step Process”](#step-by-step-process) ### Step 1: Edit the content in CNCL-BDI (source of truth) [Section titled “Step 1: Edit the content in CNCL-BDI (source of truth)”](#step-1-edit-the-content-in-cncl-bdi-source-of-truth) All arc42 documents live in the **CNCL-BDI** repo under `docs/arc42/`: ```plaintext CNCL-BDI/docs/arc42/ ├── ctn-arc42-index.md ├── 00-guidelines/ctn-documentation-guidelines.md ├── 01-introduction/ │ ├── ctn-initiative-overview.md │ ├── bdi-framework-principles.md │ └── ctn-terminology-clarification.md ├── 02-constraints/02-constraints.md ├── 03-context/ctn-context.md ├── 04-solution-strategy/ctn-solution-strategy.md ├── 05-building-blocks/ │ ├── ctn-building-blocks.md │ ├── ctn-connector.md │ ├── ctn-association-register.md │ └── ... ├── 06-runtime/ ... 07-deployment/ ... 08-crosscutting/ ... ├── 09-decisions/ # ADRs (00001-…), ADR-TEMPLATE.md, README.md ├── 10-quality/ ... 11-risks/ ... 12-glossary/ ... 13-roadmap/ ``` **File naming:** `descriptive-title.md`, lowercase-with-hyphens, **no date prefixes**. Place each file in the correct numbered chapter directory — the website sidebar is **autogenerated** from these directories. **Editing an existing document:** 1. Open the file in CNCL-BDI, e.g. `docs/arc42/05-building-blocks/ctn-connector.md`. 2. Make your changes in Markdown. 3. Update the frontmatter `status` and `lastUpdate` as appropriate. 4. Save. **Creating a new document:** ```markdown --- status: Draft lastUpdate: 2026-05-31 --- # New Component Architecture ## Overview Your content here... ``` Drop it in the relevant chapter directory — it appears in the sidebar automatically after the next build. ### Step 2: Commit & push the source repo (GitHub) [Section titled “Step 2: Commit & push the source repo (GitHub)”](#step-2-commit--push-the-source-repo-github) ```bash cd ../CNCL-BDI git add docs/arc42 git commit -m "docs: add connector retry semantics" git push # pushes to GitHub (Basic-Data-Infrastructure/CNCL-BDI) ``` This makes your edit the canonical version. (It is also what feeds GitHub Discussions–based comments on the site, via giscus.) ### Step 3: Sync into the publishing repo [Section titled “Step 3: Sync into the publishing repo”](#step-3-sync-into-the-publishing-repo) From the **CTN-Documentation** repo, copy the latest arc42 content in: ```bash cd ../CTN-Documentation nvm use 22 # Astro requires Node >= 22.12.0 npm install # first time only / after dependency changes npm run sync-arc42 ``` `sync-arc42` runs `scripts/sync-arc42.cjs`, which: * copies `../CNCL-BDI/docs/arc42` → `docs/arc42`, **wiping the destination first** so deleted/renamed upstream files don’t leave stale copies; * can be pointed elsewhere via the `ARC42_SOURCE` env var; * **auto-skips** when the source isn’t present (e.g. in CI) and uses the already-committed `docs/arc42`. You should see `✅ Synced N markdown file(s) into docs/arc42`. ### Step 4: Build and preview locally [Section titled “Step 4: Build and preview locally”](#step-4-build-and-preview-locally) ```bash # One-off production build (also validates all internal links) npm run build # …or run a live dev server with hot reload npm run dev # then open the printed http://localhost:4321 URL # …or preview the built output npm run preview ``` `npm run build` runs `prepare-content` (stages `docs/arc42` + `docs/how-to` into `src/content/docs/`, builds the splash homepage and “recently updated” cards) and then `astro build`, producing static HTML in `dist/`. A healthy build ends with something like: ```plaintext [build] N page(s) built in … · All internal links are valid. · [build] Complete! ``` If the link validator reports broken links, fix them in **CNCL-BDI**, then re-run `sync-arc42` and `build`. ### Step 5: Commit the synced docs & push (publish) [Section titled “Step 5: Commit the synced docs & push (publish)”](#step-5-commit-the-synced-docs--push-publish) The Azure pipeline only checks out the CTN-Documentation repo, so the synced `docs/arc42` content **must be committed here** for the live site to update. ```bash git add docs/arc42 git commit -m "docs: sync arc42 from CNCL-BDI" git push # pushes to Azure DevOps (DeNoronhaConsulting/CTN Documentation) ``` ### Step 6: Automatic deployment [Section titled “Step 6: Automatic deployment”](#step-6-automatic-deployment) After the push to `main`, the Azure DevOps pipeline (`azure-pipelines.yml`) automatically: 1. ✅ Installs Node `22.x` and dependencies 2. ✅ Runs `npm run build` (in CI the sync is skipped — it builds from the committed `docs/arc42`) 3. ✅ Copies `staticwebapp.config.json` into `dist/` 4. ✅ Deploys `dist/` to the Azure Static Web App **Verify deployment (\~4–5 minutes):** 1. Check the build at the pipeline (link below). 2. Visit the portal: <https://docs.preview.inlandbase.com> 3. Confirm your page appears in the sidebar and at its URL. Starlight URLs are lowercase, directory-style, with **no `.html`**, e.g. `https://docs.preview.inlandbase.com/arc42/05-building-blocks/ctn-connector/` *** ## Markdown Best Practices [Section titled “Markdown Best Practices”](#markdown-best-practices) ### Headings [Section titled “Headings”](#headings) ```markdown # H1 — Page title (use once per document) ## H2 — Major section ### H3 — Subsection ``` ### Links [Section titled “Links”](#links) Prefer relative links between docs; the build validates internal links. ```markdown [Link text](https://example.com) [Another arc42 page](../05-building-blocks/ctn-connector.md) ``` ### Code blocks [Section titled “Code blocks”](#code-blocks) ````markdown ```typescript interface User { id: string; name: string; } ``` ```` ### Tables [Section titled “Tables”](#tables) ```markdown | Feature | Status | Priority | |---------|--------|----------| | Auth | ✅ Done | High | ``` ### Mermaid diagrams [Section titled “Mermaid diagrams”](#mermaid-diagrams) Mermaid is rendered at **build time** via the `astro-mermaid` integration — just use a fenced `mermaid` block: ````markdown ```mermaid graph LR A[Client] --> B[API Gateway] --> C[Service] --> D[Database] ``` ```` *** ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Build fails with “Node.js … is not supported by Astro!” [Section titled “Build fails with “Node.js … is not supported by Astro!””](#build-fails-with-nodejs--is-not-supported-by-astro) Astro 6 requires **Node ≥ 22.12.0**. Run `node --version`; if it’s older, `nvm use 22` (or install Node 22+). ### `sync-arc42` reports “arc42 source not found” [Section titled “sync-arc42 reports “arc42 source not found””](#sync-arc42-reports-arc42-source-not-found) The CNCL-BDI repo isn’t a sibling of CTN-Documentation, or is checked out elsewhere. Either place it at `../CNCL-BDI`, or set the source explicitly: ```bash ARC42_SOURCE=/absolute/path/to/CNCL-BDI/docs/arc42 npm run sync-arc42 ``` ### My edit to `docs/arc42` in CTN-Documentation disappeared [Section titled “My edit to docs/arc42 in CTN-Documentation disappeared”](#my-edit-to-docsarc42-in-ctn-documentation-disappeared) Expected — that folder is overwritten by `sync-arc42`. Make the edit in **CNCL-BDI** and re-sync. ### Document doesn’t appear in the sidebar [Section titled “Document doesn’t appear in the sidebar”](#document-doesnt-appear-in-the-sidebar) 1. Confirm the file is in the correct numbered chapter directory under CNCL-BDI’s `docs/arc42/` (the sidebar autogenerates per directory). 2. Confirm you ran `npm run sync-arc42` and then `npm run build`. 3. Check the build output for errors. ### Build reports broken internal links [Section titled “Build reports broken internal links”](#build-reports-broken-internal-links) The `starlight-links-validator` plugin fails the build on broken internal links. Fix the link in **CNCL-BDI**, re-sync, rebuild. ### Build fails in Azure DevOps [Section titled “Build fails in Azure DevOps”](#build-fails-in-azure-devops) 1. Check the pipeline logs (link below). 2. Reproduce locally with `npm run build`. 3. Look for Markdown/link errors or missing dependencies. *** ## Quick Tips [Section titled “Quick Tips”](#quick-tips) ### DO ✅ [Section titled “DO ✅”](#do-) * Edit arc42 content in **CNCL-BDI**, never in CTN-Documentation’s `docs/arc42` * Keep the two repos as **siblings** * Use Node ≥ 22.12.0 * Run `sync-arc42` → `build` and check links before committing * Use `lowercase-with-hyphens.md` filenames in the right chapter directory * Keep `status` and `lastUpdate` frontmatter current ### DON’T ❌ [Section titled “DON’T ❌”](#dont-) * Don’t edit the synced `docs/arc42` in the publishing repo * Don’t use date prefixes in filenames * Don’t push to Azure DevOps without first syncing the latest source * Don’t expect content changes to publish from a CNCL-BDI push alone — you must sync + commit + push in CTN-Documentation *** ## Related Documentation [Section titled “Related Documentation”](#related-documentation) * **[arc42 Template](https://arc42.org/overview)** — the architecture documentation standard * **[Markdown Guide](https://www.markdownguide.org/)** — Markdown syntax reference * **[Mermaid Documentation](https://mermaid.js.org/)** — diagram syntax * **[Starlight](https://starlight.astro.build/)** — the docs framework this site is built on * **Azure DevOps build pipeline** — <https://dev.azure.com/DeNoronhaConsulting/CTN%20Documentation/_build> *** ## Summary [Section titled “Summary”](#summary) 1. ✏️ **Edit** arc42 docs in **CNCL-BDI** (`docs/arc42/`) and push to GitHub 2. 🔄 **Sync** into CTN-Documentation: `npm run sync-arc42` 3. 🧪 **Build & preview**: `npm run build` (or `npm run dev`) — fix any broken links 4. 💾 **Commit & push** the synced `docs/arc42` to Azure DevOps 5. ⏱️ **Wait \~4–5 min** for the pipeline to build and deploy 6. ✅ **Verify** on <https://docs.preview.inlandbase.com>

This is the full developer documentation for CTN Documentation # Connected Trade Network Documentation > Your comprehensive resource for understanding and working with the Connected Trade Network (CTN) — the Association Register, endpoint registration and local authorisation at the data provider. Welcome to the **CTN Documentation Portal** — technical documentation for the Connected Trade Network, a BDI-based trust and discovery infrastructure for secure container-transport data sharing. ## Portals [Section titled “Portals”](#portals) [Preview ](https://asr.preview.inlandbase.com/)Admin & member portal · ● Online ## Explore the documentation [Section titled “Explore the documentation”](#explore-the-documentation) [Overview ](/arc42/ctn-arc42-index/)The arc42 index and 2026 scope. [Introduction & Goals ](/arc42/01-introduction/ctn-initiative-overview/)Business context and what CTN delivers. [Solution Strategy ](/arc42/04-solution-strategy/ctn-solution-strategy/)Core architectural decisions and approach. [Building Blocks ](/arc42/05-building-blocks/ctn-building-blocks/)Component structure of the Association Register. [Deployment ](/arc42/07-deployment/ctn-deployment/)Infrastructure: Kubernetes (AKS), OpenTofu, GitOps. [Architecture Decisions ](/arc42/09-decisions/00001-record-architecture-decisions/)The ADR series (00001–00008). [Glossary ](/arc42/12-glossary/ctn-glossary/)Terms, acronyms and BDI/CTN concepts. [How-To guides ](/how-to/export-to-word/)Practical guides for working with the docs. ## Recently updated [Section titled “Recently updated”](#recently-updated) [Database Schema Management ](/arc42/05-building-blocks/ctn-database-schema-management-cloud-agnostic/)Updated 2026-07-30 [CTN Endpoint Lifecycle ](/arc42/05-building-blocks/ctn-endpoint-lifecycle/)Updated 2026-07-30 [REQ-20260625-COMP: Bulk Import as a Distinct, Admin-Owned Onboarding Workflow ](/arc42/09-decisions/req-20260625-comp-bulk-import-distinct-workflow/)Updated 2026-07-30 [REQ-20260625: General document store for MIME-typed uploads ](/arc42/09-decisions/req-20260625-document-store/)Updated 2026-07-30 [DIL / CTN Glossary ](/arc42/12-glossary/ctn-glossary/)Updated 2026-07-30 [ADR-00023: CTN Test Strategy — layers, ownership, and CI gating ](/arc42/09-decisions/00023-test-strategy/)Updated 2026-07-08 ## Project links [Section titled “Project links”](#project-links) [GitHub — CNCL-BDI ](https://github.com/Basic-Data-Infrastructure/CNCL-BDI)Documentation source & code. [Jira — BCSDC ](https://topsectorlogistiek.atlassian.net/jira/core/projects/BCSDC/list/?filter=allissues\&jql=project%20%3D%20%22BCSDC%22%20ORDER%20BY%20created%20DESC)Backlog & issue tracking. [IcePanel ](https://s.icepanel.io/NpRuqrse6C9RwI/REQL)Interactive architecture model. # CTN Documentation Guidelines ## 1. Where the documentation lives [Section titled “1. Where the documentation lives”](#1-where-the-documentation-lives) The CTN architecture documentation lives in the **CNCL-BDI** GitHub repository, alongside the source code that implements it: * **Repository:** * **Documentation root:** [`docs/`](../../) * **Architecture chapters (this set):** [`docs/arc42/`](../) * **Architecture Decision Records:** [`docs/arc42/09-decisions/`](../09-decisions/) Keeping the documentation in the same repository as the code has three concrete benefits: 1. **Versioning comes for free.** Every change is a git commit; every release tag captures both the code and the documentation that described it at that moment. 2. **Pull-requests are the change process.** Documentation changes go through the same review workflow as code, which means the architecture board reviews proposals before they land on `main`. 3. **History is queryable.** `git log docs/arc42/05-building-blocks/ctn-association-register.md` shows exactly when and why a building block changed — and `git blame` ties any sentence in the document back to a commit, author and date. ## 2. Format: Markdown [Section titled “2. Format: Markdown”](#2-format-markdown) All technical documentation is written in **Markdown** (CommonMark + GitHub Flavored Markdown). Markdown is the right format for this repository because: * It is **plain text**, so it diffs cleanly in pull requests. * It is **portable** — readable in any editor, on GitHub’s web view, in static-site generators, and in IDEs. * It supports **embedded diagrams** via Mermaid (see §5) without needing a separate binary format. * It is the de-facto standard for technical documentation in open-source projects, so anyone joining the project knows how to read and edit it. ### Conventions [Section titled “Conventions”](#conventions) * Use `kebab-case.md` for filenames (e.g. `ctn-association-register.md`). * Every document starts with a YAML front-matter block: ```yaml --- status: Review # Draft | Review | Published lastUpdate: 2026-05-29 --- ``` * The first heading after the front-matter is `# Title` (one H1 per document). * Cross-references use **relative paths** between Markdown files so that they keep working both on GitHub and in offline checkouts. ## 3. Editing — Typora for people who don’t live in IDEs [Section titled “3. Editing — Typora for people who don’t live in IDEs”](#3-editing--typora-for-people-who-dont-live-in-ides) For contributors who do not regularly work in a code editor, **[Typora](https://typora.io)** is the recommended desktop editor. Typora gives you a WYSIWYG view of the Markdown — what you type renders inline as headings, lists, tables, and so on — without hiding the underlying Markdown source. That makes it usable both for newcomers (it looks like a normal document editor) and for collaborators who want to see exactly what is going into the repository (the source is one click away). Practical notes: * **Open the folder, not a single file.** Open `docs/arc42/` (or the repository root) as a project in Typora’s file-tree panel; that gives you cross-document navigation and link-completion. * **Mermaid diagrams render automatically** in Typora’s preview. You can write and preview them without leaving the editor. * **Saving and committing.** Typora only writes the Markdown file; the commit-to-git step is done outside Typora — via the GitHub Desktop client, a terminal, or an IDE. For non-technical users this is usually the only friction point; the project maintainers can do the commit on a contributor’s behalf when needed. * **Licence.** Typora is a paid commercial product (modest one-off licence per user). Free alternatives that work for read-only viewing or light editing: VS Code with the built-in Markdown preview, Obsidian (free for non-commercial use), or GitHub’s web editor. ## 4. Way of Documenting [Section titled “4. Way of Documenting”](#4-way-of-documenting) The CTN architecture follows two complementary documentation frameworks: **arc42** for the chapter structure and **C4** for the way components are described at different abstraction levels. ### arc42 [Section titled “arc42”](#arc42) The thirteen-chapter arc42 template (`01-introduction`, `02-constraints`, …, `13-roadmap`) is the table of contents of `docs/arc42/`. Every architectural concern has a home in one of these chapters, which keeps the documentation discoverable. ### C4 Model [Section titled “C4 Model”](#c4-model) The [C4 model](https://c4model.com/) is a lightweight visual and verbal language used to bring consistency to explaining software systems. Its strength is the same vocabulary at multiple zoom levels, suitable for both business stakeholders and engineers. **Audiences:** * Business stakeholders * Product managers * Engineering teams * Operations teams **Benefits:** * Quick comprehension across all stakeholder levels * Consistent communication * Scalable from high-level to detailed views #### C4 Abstraction Levels [Section titled “C4 Abstraction Levels”](#c4-abstraction-levels) ``` graph TD L1[Level 1: System Context] L2[Level 2: Container] L3[Level 3: Component] L4[Level 4: Code] L1 -->|Zoom in| L2 L2 -->|Zoom in| L3 L3 -->|Zoom in| L4 style L1 fill:#e1f5ff style L2 fill:#b3e0ff style L3 fill:#80ccff style L4 fill:#4db8ff ``` ##### 1. Actors [Section titled “1. Actors”](#1-actors) **Definition:** Those who interact with our systems. **Types:** * Human users (end users, administrators) * Machine actors (external systems, APIs) ##### 2. Context (Level 1) [Section titled “2. Context (Level 1)”](#2-context-level-1) **Definition:** The highest level of abstraction. **Characteristics:** * Shows the system in its environment * Identifies external dependencies * Suitable for all stakeholders ##### 3. Containers (Level 2) [Section titled “3. Containers (Level 2)”](#3-containers-level-2) **Definition:** Individual runnable and/or deployable units that make up a system. **Examples:** * Web applications * Mobile apps * Databases * File systems * Microservices **Note:** “Container” in C4 refers to something that executes code or stores data — not specifically Docker containers. ##### 4. Components (Level 3) [Section titled “4. Components (Level 3)”](#4-components-level-3) **Definition:** Building blocks that resemble the modules that make up a container. **Characteristics:** * Logical groupings of functionality * Clear responsibilities * Well-defined interfaces ##### 5. Code (Level 4) [Section titled “5. Code (Level 4)”](#5-code-level-4) **Definition:** Detailed implementation level. **In most situations, this level includes:** * UML class diagrams * Entity-relationship diagrams * Sequence diagrams * State diagrams ## 5. Tooling [Section titled “5. Tooling”](#5-tooling) ### [IcePanel](https://icepanel.io) [Section titled “IcePanel”](#icepanel) **Features:** * Interactive C4 diagrams * Zoom in and out to the desired level * Support for “flows” to visualise user and machine interactions * Collaborative editing * Version-control integration **Use Cases:** * Context diagrams * Container diagrams * Component diagrams * Flow visualisations * Stakeholder presentations ### [Mermaid](https://www.mermaidchart.com/) [Section titled “Mermaid”](#mermaid) **Purpose:** UML class diagrams, sequence diagrams, flow diagrams, and any other diagram type Mermaid supports — embedded directly inside the Markdown. **Advantages:** * Text-based (code as documentation) * Version-control-friendly (diffs read like text) * Easy to maintain — edit the diagram in the same file as the prose * Renders inline on GitHub, in Typora, in most IDEs, and in static-site generators **Use Cases:** * Sequence diagrams for runtime behaviour * Class diagrams for detailed component structure * State diagrams for workflow visualisation ### [Typora](https://typora.io) [Section titled “Typora”](#typora) WYSIWYG Markdown editor for contributors who prefer not to work in an IDE. See §3 above. *** ## References [Section titled “References”](#references) * **Repository:** * **C4 Model:** * **arc42:** * **IcePanel:** * **Mermaid:** * **Typora:** * **CommonMark:** * **GitHub Flavored Markdown spec:** This document serves as a guideline for CTN architecture documentation. # BDI Framework Principles The Basic Data Infrastructure (BDI) framework provides a foundational set of principles, patterns, and protocols for secure, sovereign data exchange between organizations. This document serves as the reference for the [seven core BDI principles](https://bdi.gitbook.io/public/readme/introduction/core-principles) and how they guide architecture decisions. ## Purpose and Scope [Section titled “Purpose and Scope”](#purpose-and-scope) **Purpose**: Define the BDI framework principles that govern all BDI-compliant implementations **Scope**: * Applies to all BDI implementations, including CTN * Defines architectural constraints and requirements * Provides guidance for design decisions * Establishes baseline for compliance evaluation **Relationship to CTN**: The Connected Trade Network (CTN) is a specific implementation of the BDI framework tailored for container transport. While CTN adheres to these principles, it may adapt or extend them based on specific requirements. All such adaptations will be documented in Architecture Decision Records (ADRs). ## The Seven Core BDI Principles [Section titled “The Seven Core BDI Principles”](#the-seven-core-bdi-principles) ### Principle 1: Connecting Physical and Digital Processes [Section titled “Principle 1: Connecting Physical and Digital Processes”](#principle-1-connecting-physical-and-digital-processes) **Definition**: Digital data flows and representations must accurately mirror and support physical operational processes. **Rationale**: * Ensures digital systems reflect real-world operations * Maintains consistency between physical and digital states * Enables reliable decision-making based on digital data **Requirements**: 1. Digital identity must map to physical entities (organizations, vehicles, containers) 2. Data updates must reflect actual operational state changes 3. Digital processes must support (not constrain) physical operations 4. Timing of digital events must align with physical milestones **CTN Implementation (2026)**: * **Association Register**: Links legal entities to digital identities (VAD tokens). * **Endpoint Registration**: Maps registered API endpoints to the data products a provider exposes. * **Local authorisation at the data provider**: each provider verifies the VAD and applies its own policy before releasing operational data. **Verification**: * Digital identities verified against official registries (National Chamber(s) of Commerce, LEI, VIES) * Transport relationships validated before digital access granted * Real-time synchronization between operational state and digital records *** ### Principle 2: Event-Driven Architecture [Section titled “Principle 2: Event-Driven Architecture”](#principle-2-event-driven-architecture) **Definition**: Systems communicate through asynchronous events representing state changes rather than direct API calls. **Rationale**: * Loose coupling between components * Scalability through publish-subscribe patterns * Real-time notifications of relevant changes * Audit trail through event history **Requirements**: 1. State changes must trigger events 2. Events must be published to interested subscribers 3. Event format must follow standard specifications (CloudEvents) 4. Event ordering and delivery must be reliable **CTN Implementation (2026)**: * **Event broker**: Reference implementation uses a cloud-managed event broker (Azure Event Grid as reference; AWS EventBridge, Apache Pulsar or Kafka as equally valid alternatives — see [Solution Strategy](../04-solution-strategy/ctn-solution-strategy.md)). The 2026 delivery does not require a CTN-wide event bus: providers expose their own webhooks and consumers subscribe at the source. * **Event Categories**: * Participation events (participant.joined, participant.verified, participant.suspended) * Endpoint events (endpoint.registered, endpoint.suspended, endpoint.retired) * Data-product events (per provider — provider-defined types such as milestone.reached, status.changed) * **Subscribers**: data consumers subscribe to the provider’s events directly. * **Event Sourcing**: complete audit trail of all state changes inside the ASR. **Event Schema Example**: ```json { "specversion": "1.0", "type": "ctn.participant.verified", "source": "association-register", "id": "A234-1234-1234", "time": "2026-05-29T14:30:00Z", "datacontenttype": "application/json", "data": { "participant_id": "ORG-789012", "verification_source": "KvK", "verified_at": "2026-05-29T14:29:55Z" } } ``` *** ### Principle 3: Zero Trust [Section titled “Principle 3: Zero Trust”](#principle-3-zero-trust) **Definition**: No implicit trust exists between parties. Every interaction requires explicit verification and authorization. **Rationale**: * Enhanced security through continuous verification * Protection against compromised credentials * Clear accountability for all actions * Reduced impact of security breaches **Requirements**: 1. Every request must be authenticated 2. Every request must be authorized 3. Trust relationships must be explicitly established 4. Verification must occur at every boundary 5. Least privilege access principles **CTN Implementation (2026)**: * **No Implicit Trust**: every request from a consumer carries a VAD that the provider re-validates. * **VAD-based authorisation**: the VAD proves participation and trustworthiness. The data provider decides locally what that participation entitles the consumer to. * **Policy-Based Decisions**: the data provider evaluates whether the request can be authorised. * **Audit Trail**: complete logging of authentication and authorisation decisions. **Zero Trust Flow (2026)**: 1. Network Trust (is the request from an allowed network?) 2. Authentication (the OAuth client credential is verified by the IdP) 3. Business Assurance (the VAD signature, claims and expiry are validated against the ASR’s JWKS) 4. Data Appropriateness (does this consumer’s tier / participation status allow this data product?) 5. Final Decision (combine all factors for the access decision) *** ### Principle 4: Dynamic Data [Section titled “Principle 4: Dynamic Data”](#principle-4-dynamic-data) **Definition**: Data represents current operational state and is accessed in real-time rather than through static copies. **Rationale**: * Ensures decisions based on latest information * Eliminates data synchronization issues * Supports time-sensitive operations * Reduces storage and maintenance overhead **Requirements**: 1. Data must reflect current operational state 2. Stale data must be detectable (timestamps, versions) 3. Updates must propagate to interested parties 4. Cache invalidation strategies must exist 5. Real-time access patterns must be supported **CTN Implementation (2026)**: * **Real-Time Authorisation**: the data provider makes decisions based on the freshly-validated VAD and its own current policy. * **Cache with TTL**: providers cache the ASR’s JWKS with a short TTL; participation status is re-checked at token-refresh time. * **Event Notifications**: changes propagate immediately via the provider’s webhooks. * **API Design**: RESTful endpoints return current state, not snapshots. * **Temporal Data**: all records include timestamps for freshness verification. *** ### Principle 5: Data at the Source [Section titled “Principle 5: Data at the Source”](#principle-5-data-at-the-source) **Definition**: Data remains with its owner/custodian and is accessed through controlled interfaces rather than being centrally replicated. **Rationale**: * Data sovereignty maintained by data owners * Single source of truth * Reduced data governance complexity * Clear liability and responsibility * GDPR compliance through data minimization **Requirements**: 1. Data must remain with custodians 2. Access must be controlled by data custodians 3. No mandatory central data repositories 4. APIs must provide controlled access 5. Data ownership must be clear **CTN Implementation (2026)**: * **Distributed Architecture**: one shared register (the ASR) for identity and discovery; data stays with the providers. * Association Register: controls participation data. * Data Owner / Provider Systems: control operational data and policy. * **No Central Data Lake**: data never leaves the provider’s systems. * **API-Based Access**: standardised interfaces for data retrieval; OpenAPI for every registered endpoint. * **Policy Enforcement**: providers control access policies; CTN does not run a central PDP in 2026. * **Selective Disclosure**: providers can vary the data shape they release per requester tier, per data product, per jurisdiction. The exact access-tier model is policy-owned by each data provider in 2026; CTN does not impose a single FULL/LIMITED/BASIC scheme on the network. *** ### Principle 6: Local Decision-Making [Section titled “Principle 6: Local Decision-Making”](#principle-6-local-decision-making) **Definition**: Organizations maintain autonomy over their own authorization policies and business rules without central control. **Rationale**: * Business sovereignty * Flexibility for organizational requirements * Reduced coordination overhead * Resilient architecture (no central point of failure) * Faster decision-making **Requirements**: 1. Organizations control their own policies 2. No mandatory central policy authority 3. Interoperability through standards, not central control 4. Local policy enforcement 5. Transparent decision-making **CTN Implementation (2026)**: * **Single shared register**: the ASR controls participation and discovery; everything else is local. * **Local Authorisation**: each data provider runs its own access-control logic on validated VAD claims. * Policies defined by the provider. * No central policy repository. * **Federation, Not Centralisation**: * Standards enable interoperability. * Each organisation makes its own access decisions. * Voluntary participation. **Example: Custodian Autonomy**: ```plaintext Terminal A Policy: - Verified participants get real-time ETA updates - Non-participants get hourly updates - Weather data only for verified participants Terminal B Policy: - All parties get real-time updates - Premium-tier participants get predictive ETAs - Free tier has rate limits Both valid CTN implementations with different business rules ``` *** ### Principle 7: Coherent Security [Section titled “Principle 7: Coherent Security”](#principle-7-coherent-security) **Definition**: Security measures must work together across all layers to provide comprehensive protection without gaps or conflicts. **Rationale**: * Defense in depth * No single point of failure * Comprehensive audit trail * Regulatory compliance * Risk mitigation **Requirements**: 1. Security must span all architectural layers 2. Security measures must not conflict 3. Complete audit trails must exist 4. Compliance must be demonstrable 5. Security must not compromise usability **CTN Implementation**: **Network Layer**: * Private endpoints for databases * Network Security Groups (NSGs) * Web Application Firewall (WAF) * DDoS protection **Identity Layer**: * Keycloak (self-hosted) as the single IdP, with outward federation (Entra ID / eHerkenning / local) per [ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md). * Managed identities for cloud-provider integrations. * OAuth 2.0 / OIDC for API access (client credentials for M2M). * MFA enforced on admin surfaces. **Application Layer**: * VAD-based authorisation (signed JWT) * Provider-side policy-based access control * Rate limiting and throttling * Input validation **Data Layer**: * Encryption at rest (AES-256) * Encryption in transit (TLS 1.3) * Column-level encryption for PII * Key rotation policies **Monitoring Layer**: * Azure Sentinel for threat detection * Log Analytics for audit trails * Real-time alerting * Security Information and Event Management (SIEM) **Coherence Through**: ```plaintext Trust Chain (2026): 1. Network verification (allowed sources) 2. Identity verification (OAuth authentication via the IdP) 3. Business assurance (VAD signature + claims validated against JWKS) 4. Policy evaluation (provider's local policy) 5. Data access (shaped response) 6. Audit logging (complete trail) All layers work together, no gaps. ``` ## BDI Compliance Criteria [Section titled “BDI Compliance Criteria”](#bdi-compliance-criteria) An implementation is BDI-compliant when: 1. ✅ **Physical-Digital Connection**: Digital state reflects physical operations 2. ✅ **Event-Driven**: State changes propagate via events 3. ✅ **Zero Trust**: Every interaction explicitly verified 4. ✅ **Dynamic Data**: Current state accessible in real-time 5. ✅ **Data at Source**: Data stays with custodians 6. ✅ **Local Decisions**: Autonomous policy control 7. ✅ **Coherent Security**: Defense in depth across all layers ## Deviation Guidelines [Section titled “Deviation Guidelines”](#deviation-guidelines) When an implementation needs to deviate from BDI principles: 1. **Document the Deviation**: Create ADR explaining why 2. **Classify the Type**: * **BDI-Extended**: Adds capabilities (usually acceptable) * **BDI-Adapted**: Modifies approach (requires justification) * **Non-BDI**: Violates principles (rarely acceptable) 3. **Assess Impact**: How does it affect interoperability? 4. **Provide Justification**: Domain-specific requirements that necessitate deviation 5. **Get Approval**: Architecture review must approve deviations ## Relationship to Standards [Section titled “Relationship to Standards”](#relationship-to-standards) BDI principles align with and extend: * **GDPR**: Data sovereignty and minimization * **Zero Trust Architecture**: NIST 800-207 * **Event-Driven Architecture**: CloudEvents specification * **API Design**: OpenAPI, RESTful principles * **Data Governance**: ISO 27001, SOC 2 ## References [Section titled “References”](#references) * BDI Framework Official Documentation: *** # Connected Trade Network Initiative - Business Overview ## CTN as a BDI Implementation [Section titled “CTN as a BDI Implementation”](#ctn-as-a-bdi-implementation) The **Connected Trade Network (CTN)** is a digital infrastructure initiative that implements the **Basic Data Infrastructure (BDI) framework** for the **container transport industry**. CTN provides secure data-sharing capabilities for the physical process of moving containers between parties. ### Relationship to BDI [Section titled “Relationship to BDI”](#relationship-to-bdi) * **BDI Framework**: foundational principles, patterns and protocols for sovereign data exchange between organisations. * **CTN Initiative**: a digital infrastructure that applies BDI principles to the container-transport domain. * **Container Transport**: the physical logistics process that CTN digitally coordinates and secures. * **Industry-Specific Adaptations**: documented in Architecture Decision Records under [`09-decisions/`](../09-decisions/). ### Key Concepts from BDI [Section titled “Key Concepts from BDI”](#key-concepts-from-bdi) * **VAD (Verifiable Association Data)** — a standardised token proving organisation membership and trustworthiness in the Connected Trade Network. Issued by the Association Register; the basis for all 2026 authorisation. * **Seven Core BDI Principles** — physical-digital connection, event-driven architecture, zero trust, dynamic data, data at source, local decision-making, and coherent security. ## The Business Challenge in Container Transport [Section titled “The Business Challenge in Container Transport”](#the-business-challenge-in-container-transport) The **container transport process** involves an average of 20 different parties — from cargo owners and forwarders to terminals, carriers, and truck drivers. Each party needs timely access to relevant information (arrival times, container locations, customs status) to operate efficiently. Today, however, end-to-end visibility breaks down the moment a container leaves the deepsea terminal: hinterland parties miss arrival times, terminals miss pickup intentions, and the chain falls back on phone calls and ad-hoc Excels. The **Connected Trade Network (CTN) initiative** addresses this by providing the trust and discovery infrastructure that lets these parties share operational data — securely and on their own terms. ## The Solution for 2026: Association Register + Endpoint Registration [Section titled “The Solution for 2026: Association Register + Endpoint Registration”](#the-solution-for-2026-association-register--endpoint-registration) CTN 2026 delivers the **minimum infrastructure** needed to make the first practical use case (“Visibility Achterland Containerlogistiek”) run. Three building blocks: ### **Association Register (ASR)** — “Who is a verified participant?” [Section titled “Association Register (ASR) — “Who is a verified participant?””](#association-register-asr--who-is-a-verified-participant) The Association Register manages participation in the transport association and verifies the legitimacy of organisations against KvK, KBO, VIES and GLEIF. It issues **VAD tokens** that prove an organisation is a verified participant in good standing. **Business Value:** * Establishes trust between previously unknown parties. * Reduces due-diligence cost through centralised verification. * Provides a single source of truth for participant status and contact details. ### **Endpoint Registration** — “Where do I send my request, and how is it described?” [Section titled “Endpoint Registration — “Where do I send my request, and how is it described?””](#endpoint-registration--where-do-i-send-my-request-and-how-is-it-described) Data providers register their API endpoints in the ASR (label, base URL, supported BDI data products, OpenAPI specification). The ASR validates the spec, checks reachability, publishes API documentation, and lists the endpoint in the discovery service. Data consumers register their applications and receive M2M OAuth client credentials. **Business Value:** * Removes the bilateral-integration overhead that today makes onboarding slow. * Makes the available data products discoverable across the network. * Standardises authentication: one OAuth client-credentials flow, regardless of provider. ### **Local Authorisation at the Data Provider** — “Is this request allowed?” [Section titled “Local Authorisation at the Data Provider — “Is this request allowed?””](#local-authorisation-at-the-data-provider--is-this-request-allowed) Authorisation is **not centralised** in CTN. Each data provider decides locally — based on the VAD claims it has just validated — whether the requesting party is allowed to receive the requested data, and with what shape. CTN provides the verifiable identity; the data owner decides the policy. **Business Value:** * The data owner stays in control. CTN never sees or proxies the data itself. * Policies can vary per provider, per data product, per requester, without coordination. * No central rule engine to operate, version or audit on behalf of others. ## How It Works in 2026 [Section titled “How It Works in 2026”](#how-it-works-in-2026) 1. The consumer obtains a **VAD token** from the ASR (OAuth 2.0 client-credentials). 2. The consumer calls the provider’s endpoint with the VAD attached. 3. The provider validates the VAD signature and claims against the ASR’s JWKS endpoint. 4. The provider applies its local policy (participant status, jurisdiction, data product) and decides allow/deny. 5. The provider returns the data — shaped by its policy. In all cases: * Data providers retain complete control over their information. * Audit trails capture all access for compliance purposes. * Participants can receive richer data based on the provider’s policy. ## Alignment with BDI Core Principles [Section titled “Alignment with BDI Core Principles”](#alignment-with-bdi-core-principles) The 2026 architecture implements the seven BDI principles as follows: 1. **Connecting Physical and Digital Processes** — registered endpoints bridge real-world container movements with digital data sharing. 2. **Event-Driven Architecture** — providers publish status changes that consumers can subscribe to over standard protocols. 3. **Zero Trust** — every request is verified through VAD validation; no implicit trust between participants. 4. **Dynamic Data** — data is fetched on demand from the source; no central copy goes stale. 5. **Data at the Source** — data providers retain ownership and control; CTN never relays data content. 6. **Local Decision-Making** — each data provider runs its own authorisation policy. 7. **Coherent Security** — TLS 1.3 transport, JWS-signed tokens, OAuth client credentials, JWKS-based key discovery. ## Business Benefits [Section titled “Business Benefits”](#business-benefits) ### For Cargo Owners & Forwarders [Section titled “For Cargo Owners & Forwarders”](#for-cargo-owners--forwarders) * Better planning — real-time access to container locations and ETAs. * Fewer delays caused by information gaps. * Partnership flexibility — work with new parties without bespoke integration. ### For Terminals & Carriers [Section titled “For Terminals & Carriers”](#for-terminals--carriers) * Controlled sharing of operational data; full local policy control. * Reduced manual coordination and phone-call traffic. * A predictable API contract: register an endpoint once, every verified consumer can find it. ### For Transport Companies & Drivers [Section titled “For Transport Companies & Drivers”](#for-transport-companies--drivers) * Access to pickup times, container readiness, and location details. * Better planning based on real-time terminal information. * Reduced waiting time at terminals. ### For the Industry Overall [Section titled “For the Industry Overall”](#for-the-industry-overall) * Less integration friction across organisational boundaries. * Stronger protection against unauthorised data access. * A scalable foundation for additional use cases — once a party is in the ASR, the next data product they consume or expose costs only an endpoint registration. ## Implementation Approach [Section titled “Implementation Approach”](#implementation-approach) CTN is rolled out in phases — see the [Implementation Roadmap](../13-roadmap/ctn-roadmap.md) for details. The 2026 focus: 1. ASR live with self-service onboarding (eHerkenning) and periodic re-verification. 2. First wave of providers and consumers register endpoints; first OAuth M2M credentials issued. 3. Use case **Visibility Achterland Containerlogistiek** goes live: Portbase data combined with inland-terminal operational data in a single dashboard for import containers moving over barge into the hinterland. Each organisation can adopt at its own pace; the design avoids forcing a synchronised network-wide cutover. ## Getting Started [Section titled “Getting Started”](#getting-started) Organisations can participate by: 1. **Joining the Association** — apply for participation; pass legitimacy checks; designate an authorised representative via eHerkenning. 2. **Registering Endpoints** — providers upload their OpenAPI spec; consumers register their applications. 3. **Connecting** — the first use case is Visibility Achterland Containerlogistiek; further use cases reuse the same plumbing. The solution provides a foundation for modernising container-transport data sharing while keeping the control and security that data owners require. *** *This overview is a business view of the 2026 architecture. For detailed technical specifications, see the building-block documents under [`05-building-blocks/`](../05-building-blocks/).* # CTN Terminology Clarification ## Purpose [Section titled “Purpose”](#purpose) This document clarifies the distinction between two frequently used terms in our documentation: 1. **Connected Trade Network (CTN)** - the initiative/project/system 2. **Container Transport** - the business process being improved Maintaining this clarity is essential for stakeholder communication, especially when discussing the CTN initiative with industry partners who work in container transport. *** ## Key Distinctions [Section titled “Key Distinctions”](#key-distinctions) ### Connected Trade Network (CTN) [Section titled “Connected Trade Network (CTN)”](#connected-trade-network-ctn) **What it is:** * A **digital infrastructure initiative** implementing the BDI framework * For 2026: an **Association Register** plus **endpoint registration** that lets data providers and consumers discover and authenticate each other * A **technology platform** enabling secure data sharing * An **ecosystem** of tools, standards, and protocols **Use cases:** * “The **Connected Trade Network (CTN)** enables secure data sharing…” * “The **CTN initiative** addresses container transport challenges…” * “The **CTN system** implements BDI principles for…” * “Participants in the **CTN ecosystem**…” **Key attributes:** * Has a defined architecture (arc42 documentation) * Has a defined 2026 scope (Association Register + endpoint registration + use case Visibility Achterland Containerlogistiek) * Implements BDI framework principles * Provides digital infrastructure * Cloud-agnostic by design (Azure as the current reference implementation) *** ### Container Transport [Section titled “Container Transport”](#container-transport) **What it is:** * The **business process** of moving containers from origin to destination * The **physical operations** involving ships, trains, trucks, terminals * The **industry domain** that CTN serves * The **logistics activities** being digitally coordinated **Use cases:** * “**Container transport** involves an average of 20 different parties…” * “Challenges in the **container transport industry**…” * “**Container transport operations** require real-time data sharing…” * “**Container transport data** includes ETAs, locations, customs status…” * “Parties involved in **container transport**: carriers, terminals, truckers…” **Key attributes:** * Physical, operational process * Involves multiple modes (ship, rail, truck) * Requires coordination between many parties * Generates operational data (locations, times, status) * Existed before CTN and will continue regardless of CTN *** ## Relationship Between CTN and Container Transport [Section titled “Relationship Between CTN and Container Transport”](#relationship-between-ctn-and-container-transport) ```plaintext ┌─────────────────────────────────────────────┐ │ Container Transport (Business Process) │ │ │ │ Physical movement of containers involving: │ │ • Ocean shipping │ │ • Rail transport │ │ • Trucking │ │ • Terminal operations │ │ • Customs clearance │ │ │ │ ┌───────────────────────────────────────┐ │ │ │ Connected Trade Network (CTN) │ │ │ │ │ │ │ │ Digital infrastructure that: │ │ │ │ • Coordinates container transport │ │ │ │ • Enables secure data sharing │ │ │ │ • Manages party relationships │ │ │ │ • Controls data access │ │ │ │ • Issues digital credentials │ │ │ └───────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────┘ CTN is the digital layer that improves container transport ``` **The relationship:** * **Container transport** is the **problem domain** * **CTN** is the **solution** being built *** ## Common Terminology Patterns [Section titled “Common Terminology Patterns”](#common-terminology-patterns) ### ✅ Correct Usage [Section titled “✅ Correct Usage”](#-correct-usage) **When referring to the initiative/system:** * “The Connected Trade Network (CTN) initiative” * “CTN architecture” * “CTN implementation” * “CTN registers” * “CTN participants” * “CTN ecosystem” * “Deploying CTN on Azure” * “CTN’s Association Register” **When referring to the business process:** * “Container transport operations” * “Container transport industry” * “Container transport data” * “Container transport challenges” * “Parties in container transport” * “Container transport efficiency” * “Intermodal container transport” **When showing the relationship:** * “CTN addresses container transport challenges” * “CTN enables secure data sharing in container transport” * “CTN implements BDI for the container transport domain” * “Container transport parties use CTN to coordinate operations” ### ❌ Avoid Ambiguous Usage [Section titled “❌ Avoid Ambiguous Usage”](#-avoid-ambiguous-usage) **Ambiguous:** * “CTN involves 20 different parties” ❌ * Better: “Container transport involves 20 different parties that CTN coordinates” ✅ * “CTN moves containers from origin to destination” ❌ * Better: “CTN coordinates the data sharing for container transport from origin to destination” ✅ * “Container transport architecture” ❌ * Better: “CTN architecture for container transport” ✅ *** ## Writing Guidelines [Section titled “Writing Guidelines”](#writing-guidelines) ### When describing CTN [Section titled “When describing CTN”](#when-describing-ctn) **Focus on:** * Digital infrastructure * Data sharing capabilities * Register architecture * Token-based authorization * BDI implementation * Technology platform **Example sentences:** * “The **CTN initiative** provides a secure platform for data sharing in **container transport**.” * “The **CTN architecture** implements BDI principles to address **container transport** coordination challenges.” * “**Container transport** parties use the **CTN system** to share operational data securely.” ### When describing Container Transport [Section titled “When describing Container Transport”](#when-describing-container-transport) **Focus on:** * Physical operations * Party relationships * Operational data * Logistics processes * Industry challenges * Multi-modal coordination **Example sentences:** * “**Container transport** typically involves ocean carriers, terminals, rail operators, and trucking companies.” * “Challenges in **container transport** include fragmented data and lack of visibility.” * “**Container transport operations** generate large amounts of operational data at terminals, vessels, and warehouses.” *** ## Document Review Checklist [Section titled “Document Review Checklist”](#document-review-checklist) When writing or reviewing CTN documentation, verify: * [ ] “CTN” refers to the digital infrastructure/initiative, not the physical transport process * [ ] “Container transport” refers to the business process, not the IT system * [ ] The relationship between CTN and container transport is clear * [ ] Stakeholders can distinguish between the solution (CTN) and the problem domain (container transport) * [ ] Technical components are attributed to CTN (registers, tokens, APIs) * [ ] Operational elements are attributed to container transport (ships, terminals, cargo) *** ## Glossary References [Section titled “Glossary References”](#glossary-references) For complete term definitions, see: * [CTN Glossary](../12-glossary/ctn-glossary.md) - Full terminology reference * [BDI Framework Principles](bdi-framework-principles.md) - BDI and CTN relationship *** ## Summary [Section titled “Summary”](#summary) | Aspect | Connected Trade Network (CTN) | Container Transport | | ------------------ | -------------------------------------- | ---------------------------------------- | | **Nature** | Digital infrastructure initiative | Physical logistics process | | **Type** | IT system/platform | Business operations | | **Components** | Registers, tokens, APIs, policies | Ships, terminals, trucks, rail | | **Scope** | Data sharing and coordination | Moving physical containers | | **Deliverable** | Software platform on Azure | Transported cargo | | **Stakeholders** | IT architects, developers, data owners | Logistics operators, carriers, terminals | | **Documentation** | Arc42 architecture docs | Standard operating procedures | | **Success metric** | Secure data sharing enabled | Containers delivered on time | **Remember:** CTN is the **digital solution** that improves **container transport** operations through secure, coordinated data sharing. *** *This clarification guide should be referenced when writing or reviewing any CTN documentation to ensure consistent and clear terminology.* # 2. Architecture Constraints ## Technical Constraints [Section titled “Technical Constraints”](#technical-constraints) ### Platform Requirements [Section titled “Platform Requirements”](#platform-requirements) For now Azure is the reference cloud provider. Going forward a multi-cloud setup (including AWS) is anticipated. The architecture stays vendor-agnostic where possible: components such as Apache Pulsar and Keycloak are evaluated as portable alternatives, and managed services (PostgreSQL, secret stores) are preferred over self-managed VMs. | Constraint | Description | Impact | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | **Cloud Platform (Azure reference)** | Azure is the reference cloud; the architecture stays cloud-agnostic at container level (OCI image) so the same artefact runs on AWS or self-hosted Kubernetes | Avoid Azure-only constructs that cannot be expressed elsewhere | | **Event-Driven Architecture** | Asynchronous communication; in 2026 providers expose their own webhooks (a CTN-wide event broker is deferred — see [Roadmap](../13-roadmap/ctn-roadmap.md)) | Event-based integration patterns | | **Backend: Java + Quarkus** | Backend services run on Java 25 LTS with Quarkus (see [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md)) | Standard OCI image; cloud-agnostic deployment | | **PostgreSQL Database** | Database must be PostgreSQL (managed; Azure Flexible Server in the reference deployment) | SQL Server and NoSQL options excluded | | **Frontend: not yet decided** | The ASR portal frontend stack is an open decision (see [Coding Standards → Open Questions](../08-crosscutting/ctn-coding-standards.md)); a React + TypeScript SPA is one candidate, a server-rendered Quarkus + Qute approach another | Frontend framework choice deferred to a dedicated ADR | | **JWT Token Standards** | VAD tokens must use JWS format (RS256) | Standardised token generation and validation | | **RESTful APIs** | All external interfaces must follow REST principles | Consistent API design across registers | | **Container Transport Focus** | System optimized for container logistics | Domain-specific data models and workflows | ### Security Constraints [Section titled “Security Constraints”](#security-constraints) | Constraint | Description | Rationale | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | **Keycloak Authentication** | All users must authenticate via Keycloak (single IdP, with outward federation per [ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md)) | Single sign-on (SSO), centralised identity management | | **VAD-only authorisation** | Every data request carries a valid VAD; data providers validate it locally and apply their own policy. No central PDP, no second token in 2026. | See [Solution Strategy](../04-solution-strategy/ctn-solution-strategy.md) and [Roadmap](../13-roadmap/ctn-roadmap.md) for later-phase additions. | | **Zero Trust Model** | No implicit trust between participants | Security-by-design principle | | **HTTPS Only** | All traffic must use TLS 1.3+ | GDPR and eIDAS compliance requirements | | **Certificate-Based Signing** | Private keys for token generation | Non-repudiation of issued tokens | | **RBAC Authorization** | Role-based access control mandatory | Principle of least privilege, audit requirements | | **Secret Management** | All secrets in Azure Key Vault | No secrets in code, environment variables, or config files | | **Data Encryption** | Data at rest (AES-256) and in transit encrypted | GDPR Article 32 requirements | | **Public Key Discovery** | JWKS endpoints for verification | Federated trust verification | ### Integration Constraints [Section titled “Integration Constraints”](#integration-constraints) | Constraint | Description | Impact | | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | **BDI Compliance** | Must implement all seven BDI core principles | Framework determines token format, signing, claims | | **KvK API Rate Limits** | Max 100 requests/minute to KvK API | Implement caching, batch processing for bulk operations | | **Association Register Dependency** | Requires ASR for member verification | ASR must be operational for other registers | | **Excel Compatibility** | Support standard spreadsheet formats for bulk operations | Plain CSV also allowed | | **X.509 Certificates** | TLS certificates for secure communications | Certificate management required | | **OAuth 2.0** | Client Credentials flow for authentication | Standard authentication pattern | | **CI/CD Pipeline** | Build, test and release via GitHub Actions (source hosted in GitHub); Infrastructure-as-Code via OpenTofu and Kubernetes deployment via Helm + ArgoCD (GitOps) | Consistent, auditable deployment pipeline | ### Performance Constraints [Section titled “Performance Constraints”](#performance-constraints) | Operation | Target | Notes | | -------------------- | ----------------- | -------------------------------- | | **Token Generation** | <100ms | VAD creation | | **Bulk Processing** | 100+ participants | Single bulk-onboarding operation | | **Event Delivery** | <10 seconds | Notification to all subscribers | | **Cache Operations** | <50ms | Party/order lookups | | **API Response** | <500ms | Standard queries | ## Organizational Constraints [Section titled “Organizational Constraints”](#organizational-constraints) ### Team Structure [Section titled “Team Structure”](#team-structure) | Constraint | Description | Impact | | -------------------------- | ------------------------------------ | ---------------------------------------------------------------- | | **Small Team** | Development team of 1-2 developers | Emphasis on automation, simple architecture, minimal maintenance | | **Part-Time Availability** | Developers work on multiple projects | Clear documentation required, minimal context switching | | **No Dedicated QA** | Developers responsible for testing | Automated testing critical, CI/CD quality gates mandatory | ### Business Rules [Section titled “Business Rules”](#business-rules) | Rule | Description | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | **Single Shared Register** | One ASR instance serves the association (no per-orchestrator registers in 2026) | | **Local Authorisation** | Each data provider decides access locally on validated VAD claims | | **Tiered Access Levels** | Access tiers are policy-owned per data provider (no central scheme) | | **Periodic Re-verification** | Participant legitimacy re-checked at least annually against KvK / KBO / VIES / GLEIF; quarterly where risk requires it | ### Timeline Constraints [Section titled “Timeline Constraints”](#timeline-constraints) The original 2025 milestones (MVP Launch, BDI Integration, Production Scale) have been superseded by the phased plan in the [Implementation Roadmap](../13-roadmap/ctn-roadmap.md). Refer to that document for current target dates and status. ### Budget Constraints [Section titled “Budget Constraints”](#budget-constraints) | Item | Monthly Budget | Annual Budget | | ------------------------ | -------------- | ------------- | | **Azure Infrastructure** | €500 | €6,000 | | **Third-Party APIs** | €200 | €2,400 | | **Development Tools** | €100 | €1,200 | | **Total** | €800 | €9,600 | **Impact:** Must use cost-effective Azure services: * Right-sized Kubernetes pods with autoscaling (AKS as the Azure reference) * Azure Static Web Apps (free tier where possible) * PostgreSQL Flexible Server (Burstable tier for dev) ### Operational Constraints [Section titled “Operational Constraints”](#operational-constraints) | Constraint | Target | | -------------------------- | ------------------------------------- | | **Availability** | 99.9% for active transport operations | | **Disaster Recovery** | RTO 1 hour, RPO 15 minutes | | **Backward Compatibility** | Support existing integration patterns | | **Gradual Rollout** | Phased implementation approach | | **24/7 Operations** | Support global logistics operations | ## Compliance and Legal Constraints [Section titled “Compliance and Legal Constraints”](#compliance-and-legal-constraints) ### GDPR Requirements [Section titled “GDPR Requirements”](#gdpr-requirements) | Requirement | Implementation | Verification | | ------------------------- | -------------------------------------------- | --------------------------------------- | | **Data Minimization** | Only collect necessary member data | Privacy impact assessment | | **Right to be Forgotten** | Member deletion workflow with cascade | E2E test for data deletion | | **Data Portability** | Export member data in JSON/CSV | Export functionality in member portal | | **Consent Management** | Explicit consent for data processing | Consent checkboxes in registration flow | | **Audit Logging** | Log all data access and modifications | Application Insights integration | | **Data Sovereignty** | Organizations maintain control of their data | Per-organization data isolation | ### eIDAS Compliance [Section titled “eIDAS Compliance”](#eidas-compliance) | Requirement | Implementation | | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Qualified Electronic Signatures** | RS256 signing for BDI tokens | | **Identity Verification** | KvK registry integration for legal entities | | **Trust Services** | Keycloak for identity management, federating to eIDAS-compatible national schemes (eHerkenning for NL) per [ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md) | | **UBO Verification** | Ultimate Beneficial Owner compliance checks | ### Regulatory Reporting [Section titled “Regulatory Reporting”](#regulatory-reporting) * Support for authority oversight requirements * Complete audit trail of all access and modifications ### NIS2 / Cyberbeveiligingswet [Section titled “NIS2 / Cyberbeveiligingswet”](#nis2--cyberbeveiligingswet) The NIS2 Directive (EU 2022/2555), implemented in the Netherlands as the Cyberbeveiligingswet (in force since mid-2025), constrains how CTN must be designed and operated. Most CTN participants are themselves NIS2-obliged entities under the Transport sector; DIL as orchestrator may itself fall under “Digital Infrastructure” / “ICT service management”. Even where DIL is not directly obliged, participants must include CTN in their **supply-chain security assessments** under Article 21(2)(d). | Aspect | Requirement | Status | | ---------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------- | | **Article 21(2) controls** | 10 categories of risk-management measures | See [NIS2 mapping](../08-crosscutting/ctn-nis2-compliance.md) | | **Incident reporting** | 24h early warning, 72h notification, 1M final report | Runbook to be defined | | **Supply-chain attestation** | Auditable security posture for participants | ISO 27001 / SOC 2 on roadmap | | **Vulnerability disclosure** | Published CVD policy with `security.txt` | To be published | | **Cryptography** | Approved-algorithms policy, key rotation | TLS 1.3, JWS implemented; rotation policy to document | | **Management body training** | Mandatory NIS2 awareness for directors (personal liability) | DIL board training to be scheduled | Penalties for non-compliance reach **€10M or 2%** of worldwide turnover (essential entities) or **€7M or 1.4%** (important entities). See [NIS2 compliance mapping](../08-crosscutting/ctn-nis2-compliance.md) for a full Article 21(2) gap analysis and the readiness summary. ## Political/Industry Constraints [Section titled “Political/Industry Constraints”](#politicalindustry-constraints) ### Stakeholder Agreements [Section titled “Stakeholder Agreements”](#stakeholder-agreements) | Agreement | Description | | --------------------------- | ---------------------------------------------------- | | **Association Governance** | Membership rules and verification standards | | **Data Sharing Agreements** | Bilateral commercial arrangements respected | | **Competitive Sensitivity** | Protect commercial information between competitors | | **Industry Standards** | Align with existing logistics standards (e.g., DCSA) | ### Unresolved Business Decisions [Section titled “Unresolved Business Decisions”](#unresolved-business-decisions) | Decision | Status | | ------------------------------ | -------------------------------------------------------------- | | **Liability Framework** | Responsibility for data sharing between participants (PENDING) | | **Dispute Resolution Process** | Handling access disputes between parties (PENDING) | ## Development Constraints [Section titled “Development Constraints”](#development-constraints) ### Code Quality Standards [Section titled “Code Quality Standards”](#code-quality-standards) ``` graph LR A[Commit Code] --> B{Lint & Format Check} B -->|Pass| C{Build & Test} B -->|Fail| D[Block Commit] C -->|Pass| E{Security Scan} C -->|Fail| D E -->|Pass| F[Allow Deployment] E -->|Critical Issues| D E -->|Minor Issues| G[Warn but Allow] ``` | Standard | Tool | Enforcement | | --------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------- | | **Build & Test** | Maven + JUnit 5 (Quarkus) | Build failure on errors; coverage gate in CI | | **Code Style** | `google-java-format` (see [Coding Standards](../08-crosscutting/ctn-coding-standards.md)) | Enforced in CI; no wildcard/unused imports | | **Security Scanning** | Aikido | Pipeline quality gate (warnings allowed) | ### Testing Requirements [Section titled “Testing Requirements”](#testing-requirements) | Test Type | Coverage Target | When Required | | ------------------ | -------------------------------- | ---------------------------- | | **Unit Tests** | >70% for business logic | Critical functions only | | **E2E Tests** | All critical user journeys | Before production deployment | | **API Tests** | All CRUD operations | Before UI testing | | **Security Tests** | Automated vulnerability scanning | Every commit | ### Documentation Standards [Section titled “Documentation Standards”](#documentation-standards) | Document Type | Location | Update Frequency | | -------------------------- | --------------------------------- | ----------------------------- | | **API Documentation** | Swagger/OpenAPI spec | Every API change | | **Architecture Decisions** | `docs/arc42/09-decisions/` (ADRs) | When a major decision is made | | **Deployment Guide** | `docs/arc42/07-deployment/` | Every infrastructure change | ## Conventions [Section titled “Conventions”](#conventions) ### Naming Conventions [Section titled “Naming Conventions”](#naming-conventions) | Entity | Convention | Example | | ------------------------- | ------------------------------- | ----------------------------- | | **Azure Resources** | `{type}-{project}-{env}` | `aks-ctn-dev` (AKS cluster) | | **Database Tables** | `snake_case`, singular | `participant`, `legal_entity` | | **Java Files** | match the public class / record | `ParticipantResource.java` | | **API Routes** | `/v1/{resource}` (kebab-case) | `/v1/participants` | | **Environment Variables** | `SCREAMING_SNAKE_CASE` | `QUARKUS_DATASOURCE_JDBC_URL` | > The full naming conventions live in [Coding Standards](../08-crosscutting/ctn-coding-standards.md); this table is only a summary. ### Git Workflow [Section titled “Git Workflow”](#git-workflow) ``` gitGraph commit id: "Initial commit" branch feature/new-feature checkout feature/new-feature commit id: "Implement feature" commit id: "Add tests" checkout main merge feature/new-feature tag: "v1.1.0" commit id: "Deploy to dev" commit id: "Deploy to prod" ``` **Branching Strategy:** * `main` - Production-ready code * `feature/*` - Feature development branches * `fix/*` - Hotfix branches **Commit Message Format:** ```plaintext (): Co-Authored-By: Claude ``` Types: `feat`, `fix`, `docs`, `test`, `refactor`, `chore` ## Technology Decisions [Section titled “Technology Decisions”](#technology-decisions) ### Frozen Decisions [Section titled “Frozen Decisions”](#frozen-decisions) These technology choices are **fixed** and cannot be changed without significant re-architecture: 1. **Azure as reference cloud** - Cloud-agnostic at container level (OCI image) 2. **PostgreSQL Database** - Relational data model 3. **Quarkus on Java 25 LTS** - Backend runtime/framework (per [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md)) 4. **Kubernetes (Helm + ArgoCD GitOps)** - Container runtime for the backend (OCI image; AKS as the Azure reference, portable to any Kubernetes) 5. **Keycloak** — Identity and authentication (single IdP with outward federation, per [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md) and [ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md)) > The frontend stack is **not** frozen — it is an open decision (see Flexible Decisions below and [Coding Standards → Open Questions](../08-crosscutting/ctn-coding-standards.md)). ### Flexible Decisions [Section titled “Flexible Decisions”](#flexible-decisions) These can be changed with moderate effort: 1. **Frontend framework** - Open decision (see [Coding Standards → Open Questions](../08-crosscutting/ctn-coding-standards.md)); a React + TypeScript SPA is one candidate, server-rendered Quarkus + Qute another 2. **Frontend state / HTTP libraries** - To be chosen together with the frontend framework 3. **Cloud target** - AKS is the Azure reference; the OCI image + Helm chart run on any Kubernetes (EKS, GKE, self-hosted) *** **Related Sections:** * [1. Introduction and Goals](../01-introduction/ctn-initiative-overview.md) * [3. System Scope and Context](../03-context/ctn-context.md) * [9. Architecture Decisions](../09-decisions/) (ADRs to be added) # Cloud Tower (CDA App) — Migration Assessment onto the CTN Stack > **Scope note.** This document assesses re-platforming the existing **Cloud Tower / CDA app** — the implementation of the 2026 use case *“Visibility Achterland Containerlogistiek”* — from its current Supabase backend onto the CTN [preferred stack](../09-decisions/00002-define-a-preferred-stack.md) (**Quarkus, PostgreSQL, Keycloak**). It is a pre-scoping assessment, not a committed plan; effort figures are order-of-magnitude pending detailed estimation by the implementing team. The app currently lives in a separate (private) repository, `Basic-Data-Infrastructure/cloud-tower-beginnings`. ## Why this matters for CTN [Section titled “Why this matters for CTN”](#why-this-matters-for-ctn) The Cloud Tower app already realises the [2026 use case](../../README.md) — combining Portbase data with inland-terminal data into a dashboard for import containers moving over barge into the hinterland. It was originally generated on **Supabase** (PostgREST data API, Supabase Auth, Deno Edge Functions, Supabase Realtime, \~53 Row-Level-Security policies). Porting it onto the CTN stack converges the use-case dashboard with the target architecture and lets it consume the [Association Register (ASR)](../05-building-blocks/) and endpoint registration as a first-class **data consumer**. ## Position in the CTN context [Section titled “Position in the CTN context”](#position-in-the-ctn-context) | CTN concept (see [System Scope and Context](ctn-context.md)) | Cloud Tower mapping | | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Use case *“Visibility Achterland Containerlogistiek”* | This app — the import-container hinterland dashboard | | **Data consumer** | The app — it consumes Portbase (CargoController) and inland-terminal (ITV / Van Berkel) data | | **Data custodians / providers** | Poort8 CargoController function (Portbase), Van Berkel ITV (via iSHARE) | | **Association Register (ASR)** | Participant onboarding, authentication of authorised representatives, **M2M-credential issuance** — the app’s external calls should obtain credentials this way instead of hardcoded keys | | **Endpoint registration** | Where the app registers as a consumer, obtains OAuth client credentials, and discovers provider endpoints | | **Keycloak (IAM)** | Replaces Supabase Auth for human login | | **Local authorisation (VAD-only, 2026)** | The app enforces authorisation locally — exactly where the current RLS logic must move (into the Quarkus app layer). No central PDP/OPA in 2026 scope. | ## Assessment summary [Section titled “Assessment summary”](#assessment-summary) Porting is **feasible and medium-sized — a backend re-platform, not a rewrite**. The database is already PostgreSQL, and the heavy business logic (demurrage/detention calculation, Excel import/deduplication, release tri-state) lives in the React frontend, so both move almost for free. Effort concentrates in three areas: 1. **Data API (🔴 largest).** Replace the auto-generated Supabase PostgREST API with hand-written Quarkus REST endpoints (`quarkus-rest` + Hibernate ORM/Panache) and rewrite the frontend data hooks to call them. Frontend data access is centralised in a handful of hooks/pages, which bounds the surface. 2. **Security / authorisation (🔴 highest risk).** \~53 Postgres RLS policies are **silently bypassed** the moment a Quarkus service connects with a privileged role. They must be re-expressed in the app layer (JAX-RS `@RolesAllowed` + service-layer visibility/ownership checks), with **parity tests** asserting each former RLS rule before cutover. This matches the CTN 2026 “local authorisation at the data provider” model. 3. **Realtime (🔴).** Supabase `postgres_changes` channels (live per-row sync spinners, live navigation counts) → Quarkus **SSE/WebSocket** backed by Postgres `LISTEN/NOTIFY` (or CDC). Several remaining concerns are **net simplifications** on the CTN stack: * The custom branded-auth-email function is **deleted** — Keycloak renders branded auth emails via **themes** (the repo’s `keycloak-theme/` module can be reused). * Human login moves to **Keycloak OIDC** (`quarkus-oidc`), backed by eHerkenning per the CTN model; OTP / magic-link are native Keycloak features. * Roles (`user_roles` + a `has_role()` DB function) become **Keycloak realm/client roles** carried in the token. * External-API credentials (today hardcoded Portbase key + iSHARE client credentials) become **M2M credentials issued via the ASR**, with endpoints discovered via endpoint registration — removing hardcoded secrets and aligning with CTN. * The sync edge functions (`portbase-sync`, `itv-sync`, `scheduled-sync`, queue/trigger functions) become Quarkus `@Scheduled` jobs + MicroProfile REST clients; logic ports cleanly (Deno-TS → Java is mechanical). * Email decouples from the current third-party (Lovable) email API to a standard provider via `quarkus-mailer`. * Telemetry moves from a hardcoded Azure App Insights connection string to native OpenTelemetry/Micrometer. ## Recommended approach [Section titled “Recommended approach”](#recommended-approach) A **strangler-fig** migration keeps the dashboard shippable throughout: 1. **Phase 0** — Stand up the Quarkus service alongside Supabase; Liquibase owns a copy of the schema ([TDR-0001](../../ai/adrs/TDR-0001-liquibase-owns-schema.md)); read-only `containers` endpoints; validate Keycloak login against a CTN realm. 2. **Phase 1** — Auth cutover to Keycloak/OIDC; migrate roles into Keycloak. (Lowest-risk visible change; unblocks the rest.) 3. **Phase 2** — Move reads (dashboards, counts, tracking) to Quarkus REST + SSE; re-implement visibility/role checks with RLS-parity tests. 4. **Phase 3** — Move writes and the sync jobs; switch external calls to ASR-issued M2M credentials; retire the third-party email path. 5. **Phase 4** — Decommission Supabase; final data cutover. **Order of magnitude:** a few weeks for an experienced Quarkus team, gated by Phases 2–3 (data API + RLS parity + realtime). The repo’s `asr-service` provides a working Quarkus + Keycloak template to clone conventions from. ## Key risks [Section titled “Key risks”](#key-risks) * **🔴 Invisible RLS.** Most access control is currently enforced *in the database*, not in code. Migrating to a privileged Quarkus DB connection removes that protection unless re-implemented and parity-tested first. This is the item most likely to introduce a security gap if rushed. * **🔴 Realtime is a feature, not polish.** Per-row sync feedback and live counts depend on it; the SSE + `LISTEN/NOTIFY` replacement needs deliberate design (reconnection, fan-out, stream auth). * **🟡 `auth.uid()` coupling.** Migrations/policies reference Supabase’s `auth` schema; these must be translated to app-supplied identity during the Liquibase port. * **🟡 Two identity models.** Human representatives (eHerkenning via Keycloak) vs. machine sync (M2M via ASR) — keep them distinct in the token/authorisation design. * **🟡 iSHARE specifics carry over.** The inland-terminal sync already tolerates iSHARE `403 "No Policy"`; preserve that when re-issuing the call via ASR-managed credentials. ## Open questions for the architecture board [Section titled “Open questions for the architecture board”](#open-questions-for-the-architecture-board) * [ ] Confirm the CTN model for this app: **data-consumer** registration + **M2M credentials via the ASR** for the Portbase / inland-terminal (iSHARE) calls. * [ ] Confirm **Keycloak** handles human login (eHerkenning-backed) for the dashboard’s operator/admin roles. * [ ] Confirm **Liquibase** ownership of the ported schema ([TDR-0001](../../ai/adrs/TDR-0001-liquibase-owns-schema.md)). * [ ] Decide timing relative to the 2026 ASR / endpoint-registration deliverables. ## References [Section titled “References”](#references) * CTN [System Scope and Context](ctn-context.md) * ADR-00002 — [Preferred stack: Quarkus / PostgreSQL / Keycloak](../09-decisions/00002-define-a-preferred-stack.md) * [Solution Strategy](../04-solution-strategy/ctn-solution-strategy.md) · [Building Blocks](../05-building-blocks/) · [Deployment](../07-deployment/ctn-deployment.md) * Full technical inventory (per-file change list, backend building blocks) lives in the app repo at `docs/MIGRATION-Quarkus-Keycloak.md` (`Basic-Data-Infrastructure/cloud-tower-beginnings`, private). # System Scope and Context > **Cloud-agnostic note.** This context view describes CTN in terms of capabilities and interface types, not products. Concrete platform choices (Azure as the current reference implementation, with AWS and self-hosted variants explicitly kept open) live in the [Solution Strategy](../04-solution-strategy/ctn-solution-strategy.md) and the [Deployment View](../07-deployment/ctn-deployment.md). Vendor names below appear only as examples (“e.g.”) to make a capability tangible. ## Business Context [Section titled “Business Context”](#business-context) The Connected Trade Network (CTN) operates within the container transport ecosystem, enabling secure data sharing between multiple parties involved in moving containers from origin to destination. The system bridges the gap between data custodians (who hold operational information) and data consumers (who need access to that information). ### System Purpose [Section titled “System Purpose”](#system-purpose) CTN creates a trusted environment where: * **Cargo Owners and Forwarders** orchestrate transport operations * **Transport Parties** (carriers, terminals, truckers) execute physical movements * **Data Custodians** control access to their operational data * **Regulatory Authorities** maintain oversight of transport activities ### Business Scope [Section titled “Business Scope”](#business-scope) **Within CTN Scope (2026):** * Association membership management and verification * Endpoint registration and discovery * VAD token generation and validation * Local authorisation at the data provider (no central PDP) * Compliance verification and audit trails **Outside CTN Scope:** * Physical container movements * Commercial negotiations and contracts * Payment processing and invoicing * Customs clearance procedures * Container tracking hardware * Route optimization algorithms ## Technical Context [Section titled “Technical Context”](#technical-context) ### External Interfaces [Section titled “External Interfaces”](#external-interfaces) | System/Actor | Interface Type | Protocol | Purpose | | --------------------------------- | -------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Transport Management Systems** | REST API | HTTPS/JSON | Submit transport orders, receive status updates | | **Port Community Systems** | Event Stream | AMQP/MQTT | Container movement events | | **Identity Providers** | OAuth/OIDC | HTTPS | User authentication via Keycloak (chosen IdP, per [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md)), with outward federation to Microsoft Entra ID / eHerkenning / local accounts ([ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md)) | | **Certificate Authorities** | ACME | HTTPS | TLS certificate management | | **Document-Intelligence Service** | AI Service | HTTPS | Excel/PDF processing for bulk participant onboarding (e.g. Azure Document Intelligence, AWS Textract) | | **Monitoring Systems** | Metrics API | HTTPS | System health and performance (e.g. Azure Monitor, Prometheus/Grafana) | ### Communication Patterns [Section titled “Communication Patterns”](#communication-patterns) **Synchronous Communications:** * Token validation requests * Participant verification queries * Authorization decisions * API data requests **Asynchronous Communications:** * Event notifications via an event broker (e.g. Azure Event Grid, AWS EventBridge, Apache Pulsar, Kafka) * Bulk participant onboarding * Compliance verification workflows * Transport status updates ### Data Flows [Section titled “Data Flows”](#data-flows) **Inbound Data:** ```plaintext 1. Participant Registration Data Source: Organisations joining the association Format: JSON via REST API Content: Legal entity details, compliance info 2. Endpoint Registration Data Source: Data-providing participants Format: OpenAPI (YAML/JSON) + metadata Content: Endpoint label, base URL, supported data products 3. Authorisation Requests Source: Data consumers Format: JWT token (VAD) Content: Identity / participation claims 4. Event Notifications Source: Various transport systems Format: CloudEvents Content: Status updates, milestones ``` **Outbound Data:** ```plaintext 1. Verification Tokens Destination: Authorised parties Format: JWT (JWS signed, RS256) Type: VAD (participation) 2. Event Notifications Destination: Subscribed parties Format: CloudEvents via webhooks Content: Transport updates, party changes 3. Authorization Decisions Destination: API Gateways Format: JSON response Content: Allow/Deny with context 4. Audit Logs Destination: Compliance systems Format: Structured JSON logs Content: Access records, modifications ``` ## Integration Architecture [Section titled “Integration Architecture”](#integration-architecture) ### API Specifications [Section titled “API Specifications”](#api-specifications) All external APIs follow OpenAPI 3.0 specifications with: * Semantic versioning (v1, v2) * Standardized error responses * Rate limiting headers * Correlation ID tracking ### Event Schema [Section titled “Event Schema”](#event-schema) Events use CloudEvents specification with: * Standard envelope format * Type-based routing * Schema registry for payload validation * Guaranteed delivery semantics ### Security Boundaries [Section titled “Security Boundaries”](#security-boundaries) * **Public Zone**: Participant and admin portals, API documentation * **DMZ**: API gateways, event-broker endpoints, webhook receivers * **Private Zone**: Core registers, relational databases, cache layer * **Restricted Zone**: Secret stores and HSM-backed key material (e.g. Azure Key Vault, AWS KMS, HashiCorp Vault) *** *This context view shows CTN’s position within the container transport ecosystem and its integration points.* # Solution Strategy ## Strategic Overview [Section titled “Strategic Overview”](#strategic-overview) The Connected Trade Network solves the container-transport data-sharing challenge by introducing a single, verifiable identity layer (the **Association Register**) and a standard way to **register endpoints** so that data providers and consumers can discover and authenticate each other without bespoke bilateral integration. Authorisation stays where the data already lives: each provider decides locally — on the basis of the verified identity — whether the request is allowed. The 2026 delivery sticks to this minimum so participants can start exchanging operational data quickly; broader components (orchestrator-bound registers, dossier-level proof of involvement, full PKI federation) are deliberately deferred to later phases. ### CTN as BDI Implementation [Section titled “CTN as BDI Implementation”](#ctn-as-bdi-implementation) CTN is a specific implementation of the **Basic Data Infrastructure (BDI) framework** for the container-transport domain. BDI provides the principles and patterns for sovereign data exchange between organisations; CTN applies them to container-transport coordination. **Key BDI Foundations Used by CTN:** * **Association-Based Model**: container-transport associations onboard participants and manage compliance. * **Digital Trust Framework**: identity management and access control using OAuth 2.0 and signed JWTs. * **Event-Driven Communication**: real-time notifications of operational changes (subscribe-and-receive from the source). * **Zero Trust Security**: continuous verification; no implicit trust. * **Data Sovereignty**: data remains with the provider, accessed through controlled interfaces. * **Federative Architecture**: distributed governance, no central data repository. For complete BDI framework specifications, see [BDI Framework Principles](../01-introduction/bdi-framework-principles.md). ## Core Strategic Decisions [Section titled “Core Strategic Decisions”](#core-strategic-decisions) ### 1. The Association Register (ASR) is the foundation [Section titled “1. The Association Register (ASR) is the foundation”](#1-the-association-register-asr-is-the-foundation) **Decision**: The 2026 delivery is built around a production-ready ASR. Everything else hangs off it. **Rationale**: * Identity and discovery are the prerequisites for any other CTN capability. Until parties can find each other and verify each other, no data product matters. * A single source of truth for participant status (active, suspended, re-verified, off-boarded) avoids the bilateral re-verification overhead that today slows onboarding. **Implementation**: * Self-service onboarding with eHerkenning-based authentication of authorised representatives. * Automated checks against KvK, KBO, VIES and GLEIF; re-verification at least annually, quarterly where risk requires. * Issuance of VAD tokens (JWS-signed JWTs) and M2M OAuth client credentials. * JWKS endpoint for public-key discovery. ### 2. Endpoint registration over bespoke integration [Section titled “2. Endpoint registration over bespoke integration”](#2-endpoint-registration-over-bespoke-integration) **Decision**: Data providers register their endpoints in the ASR (OpenAPI spec + metadata); data consumers register their applications and obtain credentials. The ASR validates and publishes both sides; consumers and providers find each other through the discovery API. **Rationale**: * Removes the N×M integration cost that today blocks adoption of new data products. * Makes data products discoverable across the network at a single well-known location. * Standardises authentication: one OAuth client-credentials flow regardless of provider. ### 3. Local authorisation at the data provider [Section titled “3. Local authorisation at the data provider”](#3-local-authorisation-at-the-data-provider) **Decision**: CTN does not run a central policy decision point. Each data provider validates the VAD and applies its own policy locally. **Rationale**: * Keeps the data owner in control of every release decision. * Avoids a central rule engine that everyone would have to trust to enforce their policies correctly. * Aligns with BDI principles 5 (Data at the Source) and 6 (Local Decision-Making). ### 4. Use case first: Visibility Achterland Containerlogistiek [Section titled “4. Use case first: Visibility Achterland Containerlogistiek”](#4-use-case-first-visibility-achterland-containerlogistiek) **Decision**: The first practical case combines Portbase data with inland-terminal operational data into one dashboard for import containers moving over barge into the hinterland. **Rationale**: * Low operational risk — no intervention in primary terminal processes. * Touches the parties that benefit most from end-to-end visibility (terminals, barge operators, BCOs, forwarders). * Reuses the same plumbing (ASR + endpoint registration + OAuth) that subsequent use cases will need; the next use case is mostly endpoint registration. ### 5. Phased delivery; defer the heavyweight components [Section titled “5. Phased delivery; defer the heavyweight components”](#5-phased-delivery-defer-the-heavyweight-components) **Decision**: Orchestration Register, dossier-level authorisation, dual-token validation, an OPA-based policy engine, and full PKI federation are deferred to later phases. See the [Implementation Roadmap](../13-roadmap/ctn-roadmap.md). Design notes for those components are parked under `docs/_out-of-scope-2026/` (gitignored) until they are needed. **Rationale**: * Smaller blast radius for the 2026 delivery; clear win condition for the delivery team. * Avoids forcing every participant to operate components they do not yet need. * Lets the architecture grow when adoption justifies it, not before. ## Technology Decisions [Section titled “Technology Decisions”](#technology-decisions) Concrete vendor and product choices are kept out of this document where possible; the [Constraints chapter](../02-constraints/02-constraints.md) lists the binding choices, and the [Deployment View](../07-deployment/ctn-deployment.md) lists the deployed services. The cloud-agnostic intent is real: Azure is the reference implementation, but the architecture avoids constructs that cannot be expressed on AWS or a self-hosted equivalent. ### Reference selections [Section titled “Reference selections”](#reference-selections) | Capability | Reference choice | Notes | | --------------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | OAuth / OIDC server | Keycloak (self-hosted) | EU-controlled; supports client credentials, federation, SAML; integrates with eHerkenning. | | Database | PostgreSQL (managed) | JSON support; ACID; portable across clouds. | | API gateway | Cloud-managed API gateway (e.g. Azure API Management) | OAuth introspection, rate limiting. | | Secret store | Cloud-managed key vault | HSM-backed signing keys for VAD JWS. | | Identifier enrichment | KvK API, KBO API, GLEIF, VIES | See [Identifier Enrichment Architecture](../05-building-blocks/ctn-identifier-enrichment-architecture.md). | ### Data architecture [Section titled “Data architecture”](#data-architecture) * **Transactional data**: PostgreSQL for ACID guarantees (participant records, endpoint registrations, OAuth clients). * **Audit logs**: structured JSON to the cloud-provider’s log platform; export available for compliance use. * **No central data lake**: data products stay at the provider; CTN brokers identity and discovery, not data content. ### Security architecture [Section titled “Security architecture”](#security-architecture) **Defence in depth**: 1. Network: private endpoints, WAF, TLS 1.3 only. 2. Identity: OAuth 2.0 client credentials for M2M; eHerkenning for human authorised representatives. 3. Application: JWS-signed JWT validation against JWKS; short-lived tokens. 4. Data: encryption at rest (AES-256) and in transit. 5. Monitoring: centralised audit log and threat detection. **Zero Trust:** every request authenticated and authorised; no implicit trust between services. ## Integration Patterns [Section titled “Integration Patterns”](#integration-patterns) ### API design [Section titled “API design”](#api-design) * **OpenAPI 3.x**: contract-first; required for endpoint registration. * **REST**: standard HTTP verbs; resource-oriented URLs. * **Versioning**: URL path versioning (`/v1`, `/v2`). * **Pagination**: cursor-based for large datasets. * **Idempotency**: safe retries on POST via idempotency keys. ### Async patterns [Section titled “Async patterns”](#async-patterns) * **Subscribe-and-receive at the source**: consumers subscribe to provider events over the provider’s own webhook/event channel; CTN does not relay event payloads. * **Circuit breaker**: resilience for external calls. * **Retry with backoff**: transient failure handling. ## Quality Tactics [Section titled “Quality Tactics”](#quality-tactics) ### Performance [Section titled “Performance”](#performance) * Database optimisation: indexes, partitioning on participant ID. * Async processing for re-verification. * Connection pooling. ### Scalability [Section titled “Scalability”](#scalability) * Horizontal scaling for stateless services. * Auto-scaling based on metrics. * Read replicas for the discovery API. ### Reliability [Section titled “Reliability”](#reliability) * Health checks; proactive monitoring. * Graceful degradation: providers can keep operating on locally-cached VAD validation when the ASR is briefly unavailable (within token expiry window). * Idempotency for safe retries. ## BDI Framework Alignment [Section titled “BDI Framework Alignment”](#bdi-framework-alignment) ### BDI Principle Implementation Matrix [Section titled “BDI Principle Implementation Matrix”](#bdi-principle-implementation-matrix) | BDI Principle | CTN Implementation (2026) | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------- | | **1. Physical-Digital Connection** | Registered endpoints map physical transport events (gate-in/out, ETAs) to discoverable digital data products. | | **2. Event-Driven Architecture** | Provider-published events over standard webhooks; consumers subscribe at the source. | | **3. Zero Trust** | VAD-validated authorisation on every request; no implicit trust. | | **4. Dynamic Data** | Data fetched on demand from the source; no central cached copy goes stale. | | **5. Data at Source** | Data providers retain ownership and control; CTN never relays data content. | | **6. Local Decision-Making** | Each provider runs its own authorisation policy on validated VAD claims. | | **7. Coherent Security** | TLS 1.3, JWS-signed JWTs, OAuth client credentials, JWKS discovery, eHerkenning. | ### Interoperability [Section titled “Interoperability”](#interoperability) * **With other BDI Associations**: standard OAuth 2.0 and JWT enable cross-association identity verification; the ASR exposes a JWKS endpoint and a discovery API that any BDI-compliant peer can query. * **With other data spaces**: standards-based protocols (OAuth 2.0, OpenID Connect, JWT) keep the door open to IDSA, GAIA-X, and similar initiatives. Common semantic models for logistics (e.g. DCSA standards) are reused where applicable. ## Risk Mitigation [Section titled “Risk Mitigation”](#risk-mitigation) | Risk | Mitigation Strategy | | --------------------------- | -------------------------------------------------------------------------------------------- | | Slow participant onboarding | Self-service flow with eHerkenning; automated registry checks; clear administrator workflow. | | Endpoint quality variance | Mandatory OpenAPI spec + automated validation; reachability check at registration. | | Integration challenges | One standard auth flow; discovery API; reference connector. | | Security breaches | Defence in depth; continuous monitoring; periodic external pentest. | | Vendor lock-in | Cloud-agnostic design; portable components (Keycloak, PostgreSQL). | ## Success Metrics [Section titled “Success Metrics”](#success-metrics) **Technical KPIs**: * API response time <500ms (P95) * Token issuance <100ms * ASR availability >99.9% **Business KPIs**: * Participant onboarding time <24 hours (happy path) * Endpoint registration (validate + publish) <10 minutes * Visibility use case live with the agreed set of inland terminals and barge operators *** *This solution strategy provides the architectural foundation for the 2026 delivery of the Connected Trade Network. Later-phase components are tracked in the [Roadmap](../13-roadmap/ctn-roadmap.md).* # ASR Data Model The data model for the Association Register (ASR) — the authoritative store for CTN participants, their legal identities, identifiers, verifications, endpoints, dataspace roles and onboarding state. This document describes the **target (SOLL) model**: the schema the ASR should converge on, aligned with [ADR-00003 — Separate users and participants](../09-decisions/00003-separate-users-and-participants.md). > The diagram in this document is large by design. A zoom-friendly, **ERD-only** companion lives at [ctn-asr-data-model-erd.md](ctn-asr-data-model-erd.md) — use that when you want to pan and zoom the entity relationships without the surrounding prose. ## 1. Scope and Ownership Boundary [Section titled “1. Scope and Ownership Boundary”](#1-scope-and-ownership-boundary) The ASR is **not** the system of record for identities. Per ADR-00003, a strict separation applies: human and machine identities live in **Keycloak**; participant business data lives in the **ASR database**. The two are linked by a single cross-reference — the Keycloak Organization ID on the ASR participant record, and the ASR participant ID carried as the Keycloak Organization `alias`. | Lives in **Keycloak** (identity) | Lives in **ASR** (participants) | | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------- | | User credentials, MFA, sessions | Participant record (legal entity registration) | | Minimal user profile (email, name) | Business identifiers: KVK / KBO / HRB, EUID, LEI, EORI, VAT | | User-to-participant **membership** | Verification records and their status | | Per-participant **user roles** (admin / member / signatory) via Organization Groups | | | **M2M client credentials** (client secret) | Dataspace **participation roles** (data owner / consumer / provider) | | Per-participant IdP federation | Onboarding workflow state, evidence, audit trail | | | A *reference* to each Keycloak M2M client (no secret) | **Consequence for the model:** the ASR stores no passwords and no client secrets. Where the ASR needs to point at an identity (the participant owner, an application reviewer, a provisioned M2M client) it stores the **Keycloak ID as an opaque string reference**, not a foreign key — there is no database-level FK across the system boundary. ## 2. Meta-field Convention (mandatory) [Section titled “2. Meta-field Convention (mandatory)”](#2-meta-field-convention-mandatory) Every ASR table uses **one** audit/meta block. No variants. This replaces the inconsistent mix that existed across earlier prototypes (`dt_created`, `dt_modified`, `created`, `created_on`, `createdAt`, `dt_updated`, `submitted_at`-as-created, mixed `created_by` types, etc.). | Field | Type | Rule | | ------------ | -------------- | ----------------------------------------------------------------------------- | | `created_at` | `timestamptz` | `NOT NULL DEFAULT now()`; never written by the application after insert | | `updated_at` | `timestamptz` | `NOT NULL DEFAULT now()`; maintained by an `BEFORE UPDATE` trigger | | `created_by` | `varchar(100)` | Keycloak user ID, or `system` for automated processes | | `updated_by` | `varchar(100)` | Keycloak user ID, or `system` | | `is_deleted` | `boolean` | `NOT NULL DEFAULT false`; soft delete — all reads filter `is_deleted = false` | **Exceptions, by design:** * `audit_record` is an append-only log: it has `created_at` only (no `updated_at`, no `is_deleted`). * Keycloak entities (`kc_*` in the diagram) are **external**; they are managed by Keycloak and are not subject to this convention. They appear in the diagram only to show the link points. Domain status fields (e.g. `participant.status`, `verification.status`, `endpoint.lifecycle_status`) are **separate** from `is_deleted` and describe business lifecycle, not record deletion. ## 3. Entity Relationship Diagram [Section titled “3. Entity Relationship Diagram”](#3-entity-relationship-diagram) Solid lines are intra-ASR foreign keys (`||` = mandatory, `|o` = nullable). Dashed lines (`..`) are **logical cross-system links** to Keycloak records — resolved by ID at runtime, with no database foreign key. ``` erDiagram participant ||--o{ legal_entity : "has" legal_entity ||--o{ legal_entity_identifier : "has" legal_entity ||--o{ contact : "has" participant ||--o{ endpoint : "registers" participant ||--o{ verification : "verified by" participant ||--o{ participant_role : "plays (dataspace roles)" participant ||--o{ m2m_client_ref : "owns" participant |o--o{ onboarding_request : "results in" kc_organization ||--o{ kc_user : "members" kc_organization ||--o{ kc_organization_group : "has (admin/signatory)" kc_organization ||--o{ kc_m2m_client : "has" participant ||..|| kc_organization : "org-id link (no DB FK)" m2m_client_ref ||..|| kc_m2m_client : "client-id link (no DB FK)" participant }o..o| kc_user : "owner / initial admin (id ref)" onboarding_request }o..o| kc_user : "applicant / reviewed_by (id ref)" participant { uuid participant_id PK varchar keycloak_organization_id UK "link to Keycloak Org (no FK)" varchar owner_user_id "Keycloak user id (id ref)" varchar status "Prospective|Active|Suspended" timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } legal_entity { uuid legal_entity_id PK uuid participant_id FK varchar primary_legal_name varchar address_line1 varchar postal_code varchar city varchar country_code boolean verified timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } legal_entity_identifier { uuid identifier_id PK uuid legal_entity_id FK varchar identifier_type "KVK|KBO|HRB|HRA|EUID|LEI|EORI|VAT|RSIN|SIREN|SIRET|UID|VESTIGINGSNUMMER|PEPPOL" varchar identifier_value varchar registry_name varchar identifier_scope "ENTITY|ESTABLISHMENT" varchar validation_status "Pending|Verified|Rejected" varchar validation_source "KvK|KBO|VIES|GLEIF|Handelsregister|OffeneRegister|recherche-entreprises|Zefix" timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } contact { uuid contact_id PK uuid legal_entity_id FK varchar contact_type "PRIMARY|BILLING|TECHNICAL" varchar full_name varchar email varchar phone timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } endpoint { uuid endpoint_id PK uuid participant_id FK varchar endpoint_name varchar endpoint_url varchar data_category varchar endpoint_type varchar lifecycle_status "Registered|Validated|Suspended|Retired" timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } verification { uuid verification_id PK uuid participant_id FK varchar type "KvkCheck|LeiCheck|VatCheck|DnsVerification|EmailVerification|EHerkenning|OnboardingApproval" varchar category "Admin|Automatic|Organization" varchar status "Pending|Approved|Rejected|Revoked" timestamptz handled_at varchar handled_by text evidence text reason timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } participant_role { uuid role_id PK uuid participant_id FK varchar role "DataOwner|DataConsumer|DataProvider|ServiceProvider" date start_date date end_date varchar loa boolean compliancy_verified timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } m2m_client_ref { uuid m2m_client_ref_id PK uuid participant_id FK varchar keycloak_client_id UK "link to Keycloak client (no FK)" varchar keycloak_organization_id "link" varchar client_name text assigned_scopes "array, mirrored from Keycloak" varchar status "Active|Suspended|Revoked" timestamptz last_rotated_at timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } onboarding_request { uuid request_id PK uuid participant_id FK "nullable, set on approval" varchar applicant_user_id "Keycloak user id (id ref)" varchar applicant_email varchar applicant_name varchar legal_name varchar kvk_number varchar status "Submitted|UnderReview|Approved|Rejected" timestamptz submitted_at timestamptz reviewed_at varchar reviewed_by "Keycloak user id (id ref)" timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } audit_record { uuid audit_record_id PK varchar entity_type varchar entity_key varchar action "Added|Modified|Deleted" varchar performed_by timestamptz created_at jsonb payload } kc_user { varchar user_id PK varchar email varchar given_name varchar family_name } kc_organization { varchar org_id PK varchar alias "carries participant_id" varchar name } kc_organization_group { varchar group_id PK varchar org_id FK varchar path "/admin, /signatory" } kc_m2m_client { varchar client_id PK varchar org_id FK varchar client_name varchar secret "stored in Keycloak only" text scopes } ``` ## 4. Entity Reference [Section titled “4. Entity Reference”](#4-entity-reference) ### ASR (PostgreSQL) [Section titled “ASR (PostgreSQL)”](#asr-postgresql) | Entity | Purpose | Key relationships | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | | `participant` | Root entity: a registered CTN participant. Holds lifecycle status, and the **link to its Keycloak Organization**. | Parent of legal entities, endpoints, verifications, roles, M2M references | | `legal_entity` | The verified legal identity (name, address) behind a participant. Usually 1:1 with `participant`. | FK → `participant` | | `legal_entity_identifier` | Business identifiers with validation status and enrichment source. **Authoritative here, never in Keycloak.** Carries `identifier_scope` to distinguish **entity-level** identifiers (KVK, KBO, HRB/HRA, SIREN, UID, EUID, LEI, EORI, VAT, RSIN) from **establishment-level** ones (vestigingsnummer, SIRET) — see *Two-Level Identity Model* below. | FK → `legal_entity` | | `contact` | Business contact persons (primary / billing / technical). **Not** authentication users — those live in Keycloak. | FK → `legal_entity` | | `endpoint` | Provider API endpoints with lifecycle status (registered → validated → suspended → retired). | FK → `participant` | | `verification` | One row per verification check, with status, evidence and handler. Replaces scattered `validation_status` flags; gives a full auditable trail. | FK → `participant` | | `participant_role` | **Dataspace participation roles** (DataOwner / DataConsumer / DataProvider / ServiceProvider) with validity dates and assurance. Distinct from Keycloak user roles. | FK → `participant` | | `m2m_client_ref` | ASR-side **reference** to a Keycloak M2M client provisioned for the participant: client name, scopes (mirrored for display/audit), status. **No secret.** | FK → `participant`; logical link → `kc_m2m_client` | | `onboarding_request` | Intake/onboarding request; links to the participant once approved. | FK → `participant` (nullable); id-ref → `kc_user` | | `audit_record` | Append-only change log across all entities (entity type, key, action, actor, payload). | None (standalone) | #### Two-Level Identity Model (entity vs establishment) [Section titled “Two-Level Identity Model (entity vs establishment)”](#two-level-identity-model-entity-vs-establishment) *(Added 2026-06 — from batch-enrichment learnings; see [Identifier Enrichment Architecture](./ctn-identifier-enrichment-architecture.md#batch-enrichment-of-existing-records-learnings-2026-06).)* Business identifiers live at two scopes that must not be conflated: * **Entity / rechtspersoon** — KVK, HRB/HRA, KBO, SIREN, UID (CHE). The carriers of **LEI, RSIN, VAT, EUID**. One value per legal entity, regardless of how many locations it has. * **Establishment / vestiging** — a physical location with its own address but no own LEI/VAT, identified by **vestigingsnummer** (NL), **SIRET** (FR), or equivalent. The current model keeps both on `legal_entity_identifier`, tagged with `identifier_scope` (`ENTITY` | `ESTABLISHMENT`). DE/CH branches are usually **not** separately numbered (a Zweigniederlassung refers back to the parent HRB/UID); for those, the establishment is distinguished by address (and, in batch enrichment, a geocoded location code). A future refinement could promote establishments to their own `establishment` table (`legal_entity 1 ──< establishment N`) if per-location endpoints/roles are needed. ### Keycloak (external — shown for the link only) [Section titled “Keycloak (external — shown for the link only)”](#keycloak-external--shown-for-the-link-only) | Entity | Purpose | | ----------------------- | -------------------------------------------------------------------------------------------- | | `kc_user` | Human or service identity. Owns credentials and minimal profile. | | `kc_organization` | One Keycloak Organization per ASR participant. Its `alias` carries the ASR `participant_id`. | | `kc_organization_group` | Per-participant user roles (`/admin`, `/signatory`) as Organization Groups. | | `kc_m2m_client` | The M2M client and **its secret** — the credential the ASR provisions but never stores. | ## 5. M2M Client Credentials and the Keycloak Link [Section titled “5. M2M Client Credentials and the Keycloak Link”](#5-m2m-client-credentials-and-the-keycloak-link) Client credentials are created and held **exclusively in Keycloak**. The ASR fronts the lifecycle (create / rotate / revoke) via the Keycloak Admin REST API — see [Keycloak M2M Authentication](ctn-keycloak-m2m-auth.md) — and keeps only a non-secret reference row (`m2m_client_ref`) for listing, audit and participant scoping. The links between the two systems, all by opaque ID (no database FK): * `participant.keycloak_organization_id` ⇄ `kc_organization.org_id` (and the reverse via the Organization `alias`). * `m2m_client_ref.keycloak_client_id` ⇄ `kc_m2m_client.client_id`. * `participant.owner_user_id`, `onboarding_request.applicant_user_id` / `reviewed_by` ⇄ `kc_user.user_id`. **Never stored in the ASR:** client secrets, passwords, MFA state, session/refresh tokens, or business identifiers mirrored as Keycloak attributes (an anti-pattern called out in ADR-00003). ## 6. Reference DDL [Section titled “6. Reference DDL”](#6-reference-ddl) A PostgreSQL `CREATE TABLE` script implementing this model is maintained alongside this document: **the SQL DDL shown in full below ([jump](#reference-ddl))** (PostgreSQL 15+). It contains the ten ASR tables, the shared `updated_at` trigger, `CHECK` constraints for the controlled vocabularies, foreign keys (`ON DELETE CASCADE`, except `onboarding_request` which is `ON DELETE SET NULL`), unique constraints, and indexes — including the partial unique index on `participant.keycloak_organization_id` and `uq_identifier`. The Keycloak entities are intentionally absent: they are not created or owned by this database. This consolidated script is a **reference** of the target schema. In the implementation repo the schema is applied through **Liquibase YAML changesets** (TDR-0001, see [Database Schema Management — PostgreSQL on Azure](ctn-database-schema-management-azure.md)); this file is the basis for the baseline changelog, after which every change is its own reviewed changeset rather than an edit to a create-all script. ## 7. Related Documents [Section titled “7. Related Documents”](#7-related-documents) * [ADR-00003 — Separate users and participants](../09-decisions/00003-separate-users-and-participants.md) * [ADR-00006 — Token Claim Composition](../09-decisions/00006-token-claim-composition.md) * [ADR-00007 — Identity Mutation Proxy](../09-decisions/00007-identity-mutation-proxy.md) * [Association Register — Architecture](ctn-association-register.md) * [Keycloak M2M Authentication](ctn-keycloak-m2m-auth.md) * [Identifier Enrichment Architecture](ctn-identifier-enrichment-architecture.md) * [Member Onboarding Workflow](ctn-member-onboarding-workflow.md) * [Database Schema Management](ctn-database-schema-management-cloud-agnostic.md) * ERD-only companion: [ctn-asr-data-model-erd.md](ctn-asr-data-model-erd.md) ### Reference DDL [Section titled “Reference DDL”](#reference-ddl) **ctn-asr-schema.sql** — full PostgreSQL DDL ```sql -- ============================================================================= -- CTN Association Register (ASR) — PostgreSQL schema (SOLL / target state) -- Reference DDL for the data model described in ctn-asr-data-model.md -- Target: PostgreSQL 15+ -- -- Scope: ASR-owned tables only. Human/machine identities and M2M client -- secrets live in Keycloak (see ADR-00003) and are referenced here by opaque -- ID string, with NO database foreign key across the system boundary. -- -- Meta-field convention (every table except audit_record): -- created_at, updated_at, created_by, updated_by, is_deleted -- -- NOTE: This is a CONSOLIDATED REFERENCE of the target schema. The ASR applies -- schema through Flyway versioned migrations (src/main/resources/db/migration/, -- V__*.sql) per ctn-database-schema-management-azure.md — not via a single -- create-all script or ORM auto-DDL. Use this file to seed V001__initial_schema.sql. -- ============================================================================= BEGIN; -- gen_random_uuid() is built into PostgreSQL 13+. -- On older servers, enable it explicitly: -- CREATE EXTENSION IF NOT EXISTS pgcrypto; -- ----------------------------------------------------------------------------- -- Shared trigger function: maintain updated_at on every UPDATE -- ----------------------------------------------------------------------------- CREATE OR REPLACE FUNCTION asr_set_updated_at() RETURNS trigger AS $ BEGIN NEW.updated_at := now(); RETURN NEW; END; $ LANGUAGE plpgsql; -- ----------------------------------------------------------------------------- -- participant — root entity; links to its Keycloak Organization -- ----------------------------------------------------------------------------- CREATE TABLE participant ( participant_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), keycloak_organization_id varchar(255), owner_user_id varchar(255), status varchar(20) NOT NULL DEFAULT 'Prospective' CONSTRAINT participant_status_check CHECK (status IN ('Prospective','Active','Suspended','Terminated')), metadata jsonb, created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(), created_by varchar(100), updated_by varchar(100), is_deleted boolean NOT NULL DEFAULT false ); COMMENT ON COLUMN participant.keycloak_organization_id IS 'Link to Keycloak Organization (opaque ID, no DB FK; see ADR-00003)'; COMMENT ON COLUMN participant.owner_user_id IS 'Keycloak user ID of the initial admin (opaque ID, no DB FK)'; CREATE UNIQUE INDEX uq_participant_kc_org ON participant (keycloak_organization_id) WHERE keycloak_organization_id IS NOT NULL; CREATE INDEX idx_participant_status ON participant (status) WHERE is_deleted = false; CREATE TRIGGER trg_participant_updated BEFORE UPDATE ON participant FOR EACH ROW EXECUTE FUNCTION asr_set_updated_at(); -- ----------------------------------------------------------------------------- -- legal_entity — verified legal identity behind a participant -- ----------------------------------------------------------------------------- CREATE TABLE legal_entity ( legal_entity_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), participant_id uuid NOT NULL REFERENCES participant (participant_id) ON DELETE CASCADE, primary_legal_name varchar(255) NOT NULL, address_line1 varchar(255), address_line2 varchar(255), postal_code varchar(20), city varchar(255), province varchar(255), country_code varchar(2), verified boolean NOT NULL DEFAULT false, metadata jsonb, created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(), created_by varchar(100), updated_by varchar(100), is_deleted boolean NOT NULL DEFAULT false ); CREATE INDEX idx_legal_entity_participant ON legal_entity (participant_id); CREATE INDEX idx_legal_entity_name ON legal_entity (primary_legal_name); CREATE INDEX idx_legal_entity_active ON legal_entity (participant_id) WHERE is_deleted = false; CREATE TRIGGER trg_legal_entity_updated BEFORE UPDATE ON legal_entity FOR EACH ROW EXECUTE FUNCTION asr_set_updated_at(); -- ----------------------------------------------------------------------------- -- legal_entity_identifier — business identifiers (authoritative in ASR) -- ----------------------------------------------------------------------------- CREATE TABLE legal_entity_identifier ( identifier_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), legal_entity_id uuid NOT NULL REFERENCES legal_entity (legal_entity_id) ON DELETE CASCADE, identifier_type varchar(20) NOT NULL CONSTRAINT legal_entity_identifier_type_check CHECK (identifier_type IN ('KVK','KBO','HRB','EUID','LEI','EORI','VAT','SIREN','CRN')), identifier_value varchar(100) NOT NULL, registry_name varchar(255), registry_url varchar(500), valid_from timestamptz, valid_to timestamptz, validation_status varchar(20) NOT NULL DEFAULT 'Pending' CONSTRAINT legal_entity_identifier_status_check CHECK (validation_status IN ('Pending','Verified','Rejected')), validation_source varchar(20) CONSTRAINT legal_entity_identifier_source_check CHECK (validation_source IN ('KvK','KBO','VIES','GLEIF','Manual')), validation_date timestamptz, created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(), created_by varchar(100), updated_by varchar(100), is_deleted boolean NOT NULL DEFAULT false, CONSTRAINT uq_identifier UNIQUE (legal_entity_id, identifier_type, identifier_value) ); CREATE INDEX idx_identifier_entity ON legal_entity_identifier (legal_entity_id); CREATE INDEX idx_identifier_value ON legal_entity_identifier (identifier_type, identifier_value); CREATE TRIGGER trg_identifier_updated BEFORE UPDATE ON legal_entity_identifier FOR EACH ROW EXECUTE FUNCTION asr_set_updated_at(); -- ----------------------------------------------------------------------------- -- contact — business contacts (NOT authentication users; those live in Keycloak) -- ----------------------------------------------------------------------------- CREATE TABLE contact ( contact_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), legal_entity_id uuid NOT NULL REFERENCES legal_entity (legal_entity_id) ON DELETE CASCADE, contact_type varchar(20) NOT NULL CONSTRAINT contact_type_check CHECK (contact_type IN ('PRIMARY','BILLING','TECHNICAL','LEGAL')), full_name varchar(255) NOT NULL, first_name varchar(100), last_name varchar(100), email varchar(255) NOT NULL, phone varchar(50), mobile varchar(50), job_title varchar(100), is_primary boolean NOT NULL DEFAULT false, is_active boolean NOT NULL DEFAULT true, created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(), created_by varchar(100), updated_by varchar(100), is_deleted boolean NOT NULL DEFAULT false ); COMMENT ON TABLE contact IS 'Business contact persons. Authentication users are managed in Keycloak (ADR-00003).'; CREATE INDEX idx_contact_entity ON contact (legal_entity_id); CREATE INDEX idx_contact_email ON contact (email); CREATE TRIGGER trg_contact_updated BEFORE UPDATE ON contact FOR EACH ROW EXECUTE FUNCTION asr_set_updated_at(); -- ----------------------------------------------------------------------------- -- endpoint — provider API endpoints with lifecycle status -- ----------------------------------------------------------------------------- CREATE TABLE endpoint ( endpoint_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), participant_id uuid NOT NULL REFERENCES participant (participant_id) ON DELETE CASCADE, endpoint_name varchar(255) NOT NULL, endpoint_url varchar(500), endpoint_description text, data_category varchar(100), endpoint_type varchar(50) NOT NULL DEFAULT 'REST_API', authentication_method varchar(50), lifecycle_status varchar(20) NOT NULL DEFAULT 'Registered' CONSTRAINT endpoint_lifecycle_check CHECK (lifecycle_status IN ('Registered','Validated','Suspended','Retired')), is_active boolean NOT NULL DEFAULT true, created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(), created_by varchar(100), updated_by varchar(100), is_deleted boolean NOT NULL DEFAULT false ); CREATE INDEX idx_endpoint_participant ON endpoint (participant_id); CREATE INDEX idx_endpoint_status ON endpoint (lifecycle_status) WHERE is_deleted = false; CREATE TRIGGER trg_endpoint_updated BEFORE UPDATE ON endpoint FOR EACH ROW EXECUTE FUNCTION asr_set_updated_at(); -- ----------------------------------------------------------------------------- -- verification — one auditable row per verification check (authoritative in ASR) -- ----------------------------------------------------------------------------- CREATE TABLE verification ( verification_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), participant_id uuid NOT NULL REFERENCES participant (participant_id) ON DELETE CASCADE, type varchar(40) NOT NULL CONSTRAINT verification_type_check CHECK (type IN ('KvkCheck','KboCheck','LeiCheck','VatCheck','DnsVerification', 'EmailVerification','EHerkenning','OnboardingApproval', 'BusinessRegisterExtract','ConditionsOfUseAcceptance')), category varchar(20) NOT NULL CONSTRAINT verification_category_check CHECK (category IN ('Admin','Automatic','Organization')), status varchar(20) NOT NULL DEFAULT 'Pending' CONSTRAINT verification_status_check CHECK (status IN ('Pending','Approved','Rejected','Revoked')), handled_at timestamptz, handled_by varchar(100), evidence text, reason text, created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(), created_by varchar(100), updated_by varchar(100), is_deleted boolean NOT NULL DEFAULT false ); CREATE INDEX idx_verification_participant ON verification (participant_id); CREATE INDEX idx_verification_status ON verification (status) WHERE is_deleted = false; CREATE TRIGGER trg_verification_updated BEFORE UPDATE ON verification FOR EACH ROW EXECUTE FUNCTION asr_set_updated_at(); -- ----------------------------------------------------------------------------- -- participant_role — dataspace participation roles ONLY -- (user/auth roles admin/member/signatory live in Keycloak Organization Groups) -- ----------------------------------------------------------------------------- CREATE TABLE participant_role ( role_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), participant_id uuid NOT NULL REFERENCES participant (participant_id) ON DELETE CASCADE, role varchar(20) NOT NULL CONSTRAINT participant_role_check CHECK (role IN ('DataOwner','DataConsumer','DataProvider','ServiceProvider')), start_date date, end_date date, loa varchar(20), compliancy_verified boolean NOT NULL DEFAULT false, created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(), created_by varchar(100), updated_by varchar(100), is_deleted boolean NOT NULL DEFAULT false ); COMMENT ON TABLE participant_role IS 'Dataspace participation roles only. User/authentication roles live in Keycloak (ADR-00003).'; CREATE INDEX idx_participant_role_participant ON participant_role (participant_id); CREATE TRIGGER trg_participant_role_updated BEFORE UPDATE ON participant_role FOR EACH ROW EXECUTE FUNCTION asr_set_updated_at(); -- ----------------------------------------------------------------------------- -- m2m_client_ref — ASR-side reference to a Keycloak M2M client (NO secret) -- ----------------------------------------------------------------------------- CREATE TABLE m2m_client_ref ( m2m_client_ref_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), participant_id uuid NOT NULL REFERENCES participant (participant_id) ON DELETE CASCADE, keycloak_client_id varchar(255) NOT NULL, keycloak_organization_id varchar(255), client_name varchar(255) NOT NULL, assigned_scopes text[] NOT NULL DEFAULT '{}', status varchar(20) NOT NULL DEFAULT 'Active' CONSTRAINT m2m_client_ref_status_check CHECK (status IN ('Active','Suspended','Revoked')), last_rotated_at timestamptz, created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(), created_by varchar(100), updated_by varchar(100), is_deleted boolean NOT NULL DEFAULT false, CONSTRAINT uq_m2m_client_ref_kc_client UNIQUE (keycloak_client_id) ); COMMENT ON TABLE m2m_client_ref IS 'Reference to a Keycloak M2M client. The client secret is held ONLY in Keycloak (ADR-00003); never stored here.'; COMMENT ON COLUMN m2m_client_ref.assigned_scopes IS 'Mirrored from Keycloak for display/audit; Keycloak remains the source of truth.'; CREATE INDEX idx_m2m_client_ref_participant ON m2m_client_ref (participant_id) WHERE is_deleted = false; CREATE TRIGGER trg_m2m_client_ref_updated BEFORE UPDATE ON m2m_client_ref FOR EACH ROW EXECUTE FUNCTION asr_set_updated_at(); -- ----------------------------------------------------------------------------- -- onboarding_request — intake; links to participant once approved -- ----------------------------------------------------------------------------- CREATE TABLE onboarding_request ( request_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), participant_id uuid REFERENCES participant (participant_id) ON DELETE SET NULL, applicant_user_id varchar(255), applicant_email varchar(255) NOT NULL, applicant_name varchar(255) NOT NULL, applicant_job_title varchar(255), applicant_phone varchar(50), legal_name varchar(255) NOT NULL, kvk_number varchar(50), lei varchar(20), company_address text, postal_code varchar(20), city varchar(100), country varchar(100), status varchar(20) NOT NULL DEFAULT 'Submitted' CONSTRAINT onboarding_request_status_check CHECK (status IN ('Submitted','UnderReview','Approved','Rejected')), submitted_at timestamptz NOT NULL DEFAULT now(), reviewed_at timestamptz, reviewed_by varchar(255), created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(), created_by varchar(100), updated_by varchar(100), is_deleted boolean NOT NULL DEFAULT false ); COMMENT ON COLUMN onboarding_request.participant_id IS 'Set when the application is approved and a participant is created.'; COMMENT ON COLUMN onboarding_request.reviewed_by IS 'Keycloak user ID of the reviewer (opaque ID, no DB FK).'; CREATE INDEX idx_onboarding_request_status ON onboarding_request (status); CREATE INDEX idx_onboarding_request_email ON onboarding_request (applicant_email); CREATE TRIGGER trg_onboarding_request_updated BEFORE UPDATE ON onboarding_request FOR EACH ROW EXECUTE FUNCTION asr_set_updated_at(); -- ----------------------------------------------------------------------------- -- audit_record — append-only change log (no updated_at / is_deleted by design) -- ----------------------------------------------------------------------------- CREATE TABLE audit_record ( audit_record_id uuid PRIMARY KEY DEFAULT gen_random_uuid(), entity_type varchar(100) NOT NULL, entity_key varchar(255) NOT NULL, action varchar(20) NOT NULL CONSTRAINT audit_record_action_check CHECK (action IN ('Added','Modified','Deleted')), performed_by varchar(100), created_at timestamptz NOT NULL DEFAULT now(), payload jsonb ); CREATE INDEX idx_audit_record_entity ON audit_record (entity_type, entity_key); CREATE INDEX idx_audit_record_created ON audit_record (created_at); COMMIT; ``` # ASR Data Model — ERD The ASR target (SOLL) entity-relationship diagram, on its own for easy zooming and panning. For entity descriptions, the Keycloak ownership boundary, and the meta-field convention, see [ctn-asr-data-model.md](ctn-asr-data-model.md). Solid lines are intra-ASR foreign keys (`||` mandatory, `|o` nullable). Dashed lines (`..`) are logical cross-system links to Keycloak records (resolved by ID, no DB FK). ``` erDiagram participant ||--o{ legal_entity : "has" legal_entity ||--o{ attribute : "has" legal_entity ||--o{ contact : "has" participant ||--o{ endpoint : "registers" participant ||--o{ verification : "verified by" participant ||--o{ participant_role : "plays (dataspace roles)" participant ||--o{ m2m_client_ref : "owns" participant |o--o{ onboarding_request : "results in" kc_organization ||--o{ kc_user : "members" kc_organization ||--o{ kc_organization_group : "has (admin/signatory)" kc_organization ||--o{ kc_m2m_client : "has" participant ||..|| kc_organization : "org-id link (no DB FK)" m2m_client_ref ||..|| kc_m2m_client : "client-id link (no DB FK)" participant }o..o| kc_user : "owner / initial admin (id ref)" onboarding_request }o..o| kc_user : "applicant / reviewed_by (id ref)" participant { uuid participant_id PK varchar keycloak_organization_id UK "link to Keycloak Org (no FK)" varchar owner_user_id "Keycloak user id (id ref)" varchar status "Prospective|Active|Suspended" timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } legal_entity { uuid legal_entity_id PK uuid participant_id FK varchar primary_legal_name varchar address_line1 varchar postal_code varchar city varchar country_code boolean verified timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } attribute { uuid attribute_id PK uuid legal_entity_id FK varchar identifier_type "KVK|KBO|HRB|EUID|LEI|EORI|VAT" varchar identifier_value varchar registry_name varchar validation_status "Pending|Verified|Rejected" varchar validation_source "KvK|KBO|VIES|GLEIF" timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } contact { uuid contact_id PK uuid legal_entity_id FK varchar contact_type "PRIMARY|BILLING|TECHNICAL" varchar full_name varchar email varchar phone timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } endpoint { uuid endpoint_id PK uuid participant_id FK varchar endpoint_name varchar endpoint_url varchar data_category varchar endpoint_type varchar lifecycle_status "Registered|Validated|Suspended|Retired" timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } verification { uuid verification_id PK uuid participant_id FK varchar type "KvkCheck|LeiCheck|VatCheck|DnsVerification|EmailVerification|EHerkenning|OnboardingApproval" varchar category "Admin|Automatic|Organization" varchar status "Pending|Approved|Rejected|Revoked" timestamptz handled_at varchar handled_by text evidence text reason timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } participant_role { uuid role_id PK uuid participant_id FK varchar role "DataOwner|DataConsumer|DataProvider|ServiceProvider" date start_date date end_date varchar loa boolean compliancy_verified timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } m2m_client_ref { uuid m2m_client_ref_id PK uuid participant_id FK varchar keycloak_client_id UK "link to Keycloak client (no FK)" varchar keycloak_organization_id "link" varchar client_name text assigned_scopes "array, mirrored from Keycloak" varchar status "Active|Suspended|Revoked" timestamptz last_rotated_at timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } onboarding_request { uuid request_id PK uuid participant_id FK "nullable, set on approval" varchar applicant_user_id "Keycloak user id (id ref)" varchar applicant_email varchar applicant_name varchar legal_name varchar kvk_number varchar status "Submitted|UnderReview|Approved|Rejected" timestamptz submitted_at timestamptz reviewed_at varchar reviewed_by "Keycloak user id (id ref)" timestamptz created_at timestamptz updated_at varchar created_by varchar updated_by boolean is_deleted } audit_record { uuid audit_record_id PK varchar entity_type varchar entity_key varchar action "Added|Modified|Deleted" varchar performed_by timestamptz created_at jsonb payload } kc_user { varchar user_id PK varchar email varchar given_name varchar family_name } kc_organization { varchar org_id PK varchar alias "carries participant_id" varchar name } kc_organization_group { varchar group_id PK varchar org_id FK varchar path "/admin, /signatory" } kc_m2m_client { varchar client_id PK varchar org_id FK varchar client_name varchar secret "stored in Keycloak only" text scopes } ``` # Association Register - Architecture ## 1. Introduction and Goals [Section titled “1. Introduction and Goals”](#1-introduction-and-goals) ### Business Context [Section titled “Business Context”](#business-context) The Association Register is the core application within the Association Domain that manages the complete lifecycle of association members - from prospective member onboarding through active member management. It serves as the authoritative source for member information, credentials, and verification status. ### Primary Goals [Section titled “Primary Goals”](#primary-goals) * **Member Lifecycle Management**: Handle the complete journey from prospective to active member * **Document Verification**: Automate verification of legal documents (Chamber of Commerce statements, etc.) * **VAD Token Management**: Issue and manage Verifiable Association Data (VAD) tokens for members * **External Identity Resolution**: Provide fast lookup services for LEI, CoC number, and EORI to Client ID mapping * **Tiered Access Management**: Determine access levels for both members and non-member parties * **Audit Trail**: Maintain comprehensive records of all member-related activities ### Quality Goals [Section titled “Quality Goals”](#quality-goals) | Priority | Quality Attribute | Description | Measurable Goal | | -------- | ------------------- | ----------------------------------------------------- | ---------------------------------------- | | 1 | **Security** | Protect member data and ensure only authorized access | Zero security breaches, SOC 2 compliance | | 2 | **Reliability** | System must be available 24/7 for member operations | 99.9% uptime SLA | | 3 | **Usability** | Intuitive interface for both admin and member users | <5 clicks to complete common tasks | | 4 | **Maintainability** | Easy to update, extend, and debug | <2 hours to deploy updates | | 5 | **Performance** | Fast response times for all operations | <500ms API response time (p95) | ### Stakeholders [Section titled “Stakeholders”](#stakeholders) #### Internal Stakeholders [Section titled “Internal Stakeholders”](#internal-stakeholders) | Role | Responsibilities | Expectations | | ------------------------- | ----------------------------------------------------- | ------------------------------------------------------------- | | **CTN Admin Staff** | Manage members, approve registrations, resolve issues | Efficient workflow tools, clear visibility into member status | | **System Administrators** | Deploy, monitor, maintain infrastructure | Reliable deployment process, comprehensive logging | | **Development Team** | Build features, fix bugs, maintain codebase | Clean architecture, good documentation, automated testing | #### External Stakeholders [Section titled “External Stakeholders”](#external-stakeholders) | Role | Responsibilities | Expectations | | ------------------------ | ----------------------------------------------------------- | ----------------------------------------------------------- | | **CTN Participants** | Register organizations, manage endpoints, maintain profiles | Self-service portal, quick onboarding, clear error messages | | **Integration Partners** | Integrate with CTN API endpoints | Well-documented API, stable contracts, versioning support | | **Auditors** | Verify security and compliance | Audit logs, access control reports, security documentation | | **Data Providers** | Validate VAD tokens before releasing data | Fast JWKS lookups, reliable VAD tokens | | **Data Consumers** | Obtain VAD tokens and discover endpoints | Discovery API, stable OAuth client-credentials flow | ### Key Business Goals [Section titled “Key Business Goals”](#key-business-goals) 1. **Reduce Onboarding Time** - From weeks to days for new participants 2. **Increase Trust** - Verified identities for all participants 3. **Enable Automation** - API-first design for system integrations 4. **Ensure Compliance** - GDPR, eIDAS, and industry standards 5. **Support Growth** - Scale to 500+ participant organizations ### Success Metrics [Section titled “Success Metrics”](#success-metrics) | Metric | Current | Target (6 months) | | -------------------------------- | ------- | ----------------- | | Participant Onboarding Time | 14 days | <3 days | | System Uptime | 98.5% | 99.9% | | API Response Time (p95) | 800ms | <500ms | | User Satisfaction Score | 3.2/5 | >4.5/5 | | Active Participant Organizations | 45 | 150+ | ## 2. Architecture Constraints [Section titled “2. Architecture Constraints”](#2-architecture-constraints) ### Technical Constraints [Section titled “Technical Constraints”](#technical-constraints) * **Cloud-agnostic (Azure reference)**: packaged as an OCI image; Azure is the reference cloud * **OAuth 2.0 Standard**: M2M authentication using client credentials flow * **Event-Driven Architecture**: notifications via provider webhooks (a CTN-wide event broker is deferred — see [Roadmap](../13-roadmap/ctn-roadmap.md)) * **RESTful APIs**: External interfaces must follow REST principles ### Organizational Constraints [Section titled “Organizational Constraints”](#organizational-constraints) * **Data Residency**: Member data must remain within EU boundaries * **Compliance**: GDPR compliance for member personal data * **Budget**: Cost-optimized solution using Azure PaaS services ### Standards and Conventions [Section titled “Standards and Conventions”](#standards-and-conventions) * **LEI Standards**: Global Legal Entity Identifier Foundation compliance * **Chamber of Commerce**: Integration with national CoC registries * **EUID Standards**: The EUID is the European standard for company identification introduced by the European Commission in 2017, combining country code, trade register identifier, and company registration number. It enables identification of companies and their branches across EU Member States through the Business Registers Interconnection System (BRIS). For example: `DEK1101R.HRB116737` where DE = Germany, K1101R = Hamburg trade register, HRB116737 = company number * **VIES Standard**: VIES is the European Commission’s system for validating VAT numbers across the EU, providing a single access point to query VAT numbers from individual member state registries. It allows businesses to validate the existence and validity of VAT numbers for trading partners across the EU, helping ensure correct VAT treatment and reducing fraud . Each EU country has its own VAT number format, but VIES provides centralized validation * EORI Standards\*\*: Economic Operators Registration and Identification compliance\*\* * **JWT Standards**: Use of JSON Web Tokens for verifiable documents ## 3. System Scope and Context [Section titled “3. System Scope and Context”](#3-system-scope-and-context) ### Business Context [Section titled “Business Context”](#business-context-1) The Association Register operates within a container transport ecosystem where multiple parties (BCOs, forwarders, terminals, carriers) need to share data securely. The ASR issues VAD tokens proving verified participation; each data provider then decides locally what a given participant may access (no central tier or involvement model in 2026). ### Technical Context [Section titled “Technical Context”](#technical-context) * **Upstream Systems**: External verification APIs (Kamer van Koophandel, GLEIF Foundation, EORI API) * **Downstream Systems**: Data providers (terminals, carriers, logistics providers) that validate VAD tokens before releasing data * **Integration Points**: OAuth Provider (Keycloak), provider webhooks, email/notification service ## 4. Solution Strategy [Section titled “4. Solution Strategy”](#4-solution-strategy) ### Key Architectural Decisions [Section titled “Key Architectural Decisions”](#key-architectural-decisions) * **Event-Driven Design**: Decoupled components communicating via provider webhooks (CTN-wide event broker deferred — see [Roadmap](../13-roadmap/ctn-roadmap.md)) * **Microservices Pattern**: Separation of concerns with dedicated services for different functions * **Cache-First Strategy**: Redis cache for high-performance member lookups * **Document AI Integration**: Automated document processing using Azure AI services ### Technology Choices [Section titled “Technology Choices”](#technology-choices) **Frontend:** * **Frontend stack not yet decided** — a React + TypeScript SPA is one candidate (see [Coding Standards → Open Questions](../08-crosscutting/ctn-coding-standards.md)); a separate ADR will record the choice * **Azure Static Web Apps**: Hosting with CDN for the portal (reference deployment) * **Keycloak (OIDC)**: User authentication for Admin and Participant portals **Backend:** * **Quarkus (Java 25+ LTS)**: REST API packaged as OCI image — per [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md) * **PostgreSQL 15**: Azure Flexible Server for member data * **Redis**: Caching for high-performance lookups * **Azure Blob Storage**: Document storage with lifecycle management * **Azure Front Door**: WAF protection and global load balancing **Authentication & Authorization:** * **Keycloak (OIDC)**: Human user authentication (Admin / Participant portals) * **Keycloak**: Machine-to-Machine (M2M) authentication for external systems * **RBAC Model** (5 roles): 1. `SystemAdmin` - All permissions 2. `AssociationAdmin` - Manage all entities 3. `MemberAdmin` - Manage own entity 4. `MemberUser` - Self-service access 5. `MemberReadOnly` - Read-only access **DevOps:** * **GitHub Actions**: CI/CD with path-filtered triggers; IaC via OpenTofu, Kubernetes deployment via Helm + ArgoCD (GitOps) * **JUnit 5 + RestAssured**: Backend tests (see [Coding Standards §7](../08-crosscutting/ctn-coding-standards.md)) * **Application Insights**: Monitoring and telemetry ## 5. Building Block View [Section titled “5. Building Block View”](#5-building-block-view) ### Core Components [Section titled “Core Components”](#core-components) * **ASR Web UI**: Administrator interface for member management * **Member Data Storage**: PostgreSQL database with member profiles and status * **Document Storage**: Azure Blob Storage for legal documents and certificates * **Verification Logic**: Quarkus services (on Kubernetes/AKS) handling document and data verification * **Asynchronous Processing**: background jobs in the Quarkus services for long-running verification * **Event Notifications**: provider webhooks (a CTN-wide event broker is deferred — see [Roadmap](../13-roadmap/ctn-roadmap.md)) ### External Integrations [Section titled “External Integrations”](#external-integrations) * **API Management**: Gateway for external API calls with caching and rate limiting * **Key Vault**: Secure storage of API keys and certificates * **Document Processor**: AI-powered document analysis and data extraction * **Keystore Cache**: Redis cache for LEI/CoC to Client ID mappings ### Data Model [Section titled “Data Model”](#data-model) The participant data model (entities, relationships, identifier and verification storage) is documented separately in **[ASR Data Model](ctn-asr-data-model.md)**, with a zoom-friendly [ERD-only view](ctn-asr-data-model-erd.md). It follows the user/participant separation of [ADR-00003](../09-decisions/00003-separate-users-and-participants.md): identities and M2M client secrets live in Keycloak, while participants, business identifiers, verifications and dataspace roles live in the ASR, linked to Keycloak by ID. ## 6. Runtime View [Section titled “6. Runtime View”](#6-runtime-view) ### Key Scenarios [Section titled “Key Scenarios”](#key-scenarios) 1. **Member Onboarding Flow**: Document upload → AI processing → External verification (KvK, GLEIF, EORI) → VAD token creation 2. **VAD Token Delivery**: Event-driven secure link generation and MFA verification for token delivery 3. **Member Lookup**: External party queries LEI/CoC/EORI → Redis cache hit → Client ID and VAD token response 4. **Status Updates**: Member status changes → Event publication → Stakeholder notifications ### Integration Patterns [Section titled “Integration Patterns”](#integration-patterns) * **Event Sourcing**: All state changes published as events * **CQRS**: Separate read/write models for optimal performance * **Circuit Breaker**: Protection against external API failures *Link to detailed sequence diagrams: [OAuth Provider Member Enrollment Flow](mermaid-chart-link)* ## 7. Deployment View [Section titled “7. Deployment View”](#7-deployment-view) ### Azure Services Architecture (reference deployment) [Section titled “Azure Services Architecture (reference deployment)”](#azure-services-architecture-reference-deployment) * **Compute**: Azure Kubernetes Service (AKS) running the Quarkus ASR API + background jobs; Static Web Apps for the portal * **Storage**: PostgreSQL (Structured Data), Blob Storage (Documents), Redis (Cache) * **Integration**: API Management (a CTN-wide event broker is deferred — see [Roadmap](../13-roadmap/ctn-roadmap.md)) * **Security**: Key Vault, Azure Front Door, Managed Identity * **AI/ML**: Document Intelligence ### Scaling Strategy [Section titled “Scaling Strategy”](#scaling-strategy) * **Horizontal Scaling**: Kubernetes Horizontal Pod Autoscaler (HPA) * **Database Scaling**: PostgreSQL read replicas for query optimization * **Cache Strategy**: Redis cluster for high availability * **CDN**: Azure Front Door for global performance ## 8. Cross-cutting Concepts [Section titled “8. Cross-cutting Concepts”](#8-cross-cutting-concepts) ### Security Patterns [Section titled “Security Patterns”](#security-patterns) * **Zero Trust**: No implicit trust, verify everything * **Managed Identity**: Service-to-service authentication without stored credentials * **Key Vault Integration**: Centralized secret management * **API Gateway Security**: Rate limiting, IP filtering, OAuth validation ### Error Handling [Section titled “Error Handling”](#error-handling) * **Retry Policies**: Exponential backoff for external API calls * **Circuit Breaker**: Fail-fast pattern for unreliable dependencies * **Dead Letter Queues**: Failed message handling and recovery * **Correlation IDs**: End-to-end request tracking ### Monitoring and Logging [Section titled “Monitoring and Logging”](#monitoring-and-logging) * **Application Insights**: Comprehensive telemetry and performance monitoring * **Azure Monitor**: Infrastructure and service health monitoring * **Structured Logging**: JSON-formatted logs with correlation IDs * **Alerting**: Proactive notification of issues and SLA breaches ## 10. Quality Requirements [Section titled “10. Quality Requirements”](#10-quality-requirements) ### Performance Requirements [Section titled “Performance Requirements”](#performance-requirements) * **Response Time**: <200ms for cached member lookups * **Throughput**: Support 1000+ concurrent member verification requests * **Document Processing**: Process uploaded documents within 30 seconds * **Event Processing**: Event delivery within 5 seconds ### Security Requirements [Section titled “Security Requirements”](#security-requirements) * **Data Encryption**: All data encrypted at rest and in transit * **Access Control**: Role-based access with least privilege principle * **Audit Logging**: Complete audit trail of all administrative actions * **MFA**: Multi-factor authentication for credential delivery ### Availability Requirements [Section titled “Availability Requirements”](#availability-requirements) * **Uptime**: 99.9% availability (8.7 hours downtime per year) * **Recovery Time**: RTO of 1 hour, RPO of 15 minutes * **Geographic Distribution**: Multi-region deployment for disaster recovery ## 11. Risks and Technical Debts [Section titled “11. Risks and Technical Debts”](#11-risks-and-technical-debts) ### Current Risks [Section titled “Current Risks”](#current-risks) * **External API Dependencies**: Reliance on Kamer van Koophandel and GLEIF APIs * **Data Quality**: Inconsistent or outdated information in external systems * **Scalability Limits**: PostgreSQL scaling limitations with very large member bases * **Compliance Changes**: Evolving GDPR and data residency requirements ### Technical Debts [Section titled “Technical Debts”](#technical-debts) * **Legacy Document Formats**: Some older document types may not be AI-processable * **API Versioning**: Need for backward compatibility as system evolves * **Monitoring Gaps**: Some edge cases in error scenarios not fully covered ### Future Improvements [Section titled “Future Improvements”](#future-improvements) * **Machine Learning**: Implement ML-based fraud detection for document verification * **Global Expansion**: Support for non-EU legal entity verification systems * **Real-time Sync**: Live synchronization with external registries * **Advanced Analytics**: Member behavior and system usage analytics *** *This document follows the arc42 template and should be updated as the architecture evolves.* # CTN Association Register - Format and Data Service Specification *Based on BDI Association Register Format and Data Service Documentation* ## Overview [Section titled “Overview”](#overview) This document specifies the data formats, API contracts, and service interfaces for the BDI Association Register. It defines how data is structured, how services are accessed, and the protocols used for communication. ## Data Formats [Section titled “Data Formats”](#data-formats) ### Participant Profile Format [Section titled “Participant Profile Format”](#participant-profile-format) ```json { "participant_id": "uuid", "client_id": "string", "legal_entity": { "primary_legal_name": "string", "lei": "string (20 chars)", "coc_number": "string", "eori": "string", "country_code": "string (ISO 3166-1 alpha-2)", "address": { "line1": "string", "line2": "string (optional)", "postal_code": "string", "city": "string", "province": "string (optional)", "country": "string (ISO 3166-1 alpha-2)" }, "entity_legal_form": "string", "registered_at": "string", "direct_parent_lei": "string (optional)", "ultimate_parent_lei": "string (optional)" }, "participation": { "status": "enum (pending|active|suspended|terminated)", "identity_assurance_level": "enum (ial1|ial2|ial3)", "joined_date": "datetime (ISO 8601)", "renewal_date": "datetime (ISO 8601)" }, "contacts": [{ "contact_id": "uuid", "type": "enum (primary|technical|billing)", "name": "string", "email": "string", "phone": "string (optional)" }], "created_at": "datetime", "updated_at": "datetime" } ``` ### VAD Token Format (JWT) [Section titled “VAD Token Format (JWT)”](#vad-token-format-jwt) **Header:** ```json { "alg": "RS256", "typ": "JWT", "kid": "key-id" } ``` **Payload:** ```json { "iss": "https://association.ctn.network", "sub": "client_id", "aud": "https://ctn.network", "iat": 1704063600, "exp": 1704150000, "nbf": 1704063600, "jti": "unique-token-id", "participant_claims": { "legal_name": "string", "lei": "string", "coc_number": "string", "eori": "string", "country_code": "string", "participation_status": "active", "identity_assurance_level": "substantial", "verified_at": "datetime" } } ``` ## API Service Interfaces [Section titled “API Service Interfaces”](#api-service-interfaces) ### Participant Management API [Section titled “Participant Management API”](#participant-management-api) #### Create Participant Application [Section titled “Create Participant Application”](#create-participant-application) ```http POST /v1/applications Content-Type: application/json Authorization: Bearer { "legal_name": "string", "coc_number": "string", "country_code": "string", "contact_email": "string" } Response: 201 Created { "application_id": "uuid", "status": "pending_documents", "upload_url": "string" } ``` #### Get Participant Profile [Section titled “Get Participant Profile”](#get-participant-profile) ```http GET /v1/participants/{clientId} Authorization: Bearer Response: 200 OK { // Participant profile as defined above } ``` #### Update Participant Profile [Section titled “Update Participant Profile”](#update-participant-profile) ```http PATCH /v1/participants/{clientId} Content-Type: application/json Authorization: Bearer { "contacts": [/* updated contacts */], "address": {/* updated address */} } Response: 200 OK ``` ### Directory Lookup API [Section titled “Directory Lookup API”](#directory-lookup-api) #### Lookup by LEI [Section titled “Lookup by LEI”](#lookup-by-lei) ```http GET /v1/directory/lei/{lei} Authorization: Bearer Response: 200 OK { "client_id": "string", "legal_name": "string", "participation_status": "string", "verified": true } ``` #### Lookup by CoC Number [Section titled “Lookup by CoC Number”](#lookup-by-coc-number) ```http GET /v1/directory/coc/{country_code}/{coc_number} Authorization: Bearer Response: 200 OK { "client_id": "string", "legal_name": "string", "participation_status": "string", "verified": true } ``` #### Lookup by EORI [Section titled “Lookup by EORI”](#lookup-by-eori) ```http GET /v1/directory/eori/{eori} Authorization: Bearer Response: 200 OK { "client_id": "string", "legal_name": "string", "participation_status": "string", "verified": true } ``` ### VAD Token API [Section titled “VAD Token API”](#vad-token-api) #### Request VAD Token [Section titled “Request VAD Token”](#request-vad-token) ```http POST /v1/tokens/vad Content-Type: application/json Authorization: Bearer { "client_id": "string", "audience": ["string"], "validity_period": "number (seconds)" } Response: 200 OK { "vad_token": "string (JWT)", "expires_at": "datetime", "token_type": "Bearer" } ``` #### Validate VAD Token [Section titled “Validate VAD Token”](#validate-vad-token) ```http POST /v1/tokens/validate Content-Type: application/json { "token": "string (JWT)" } Response: 200 OK { "valid": true, "claims": {/* token claims */}, "expires_at": "datetime" } ``` ## Data Service Specifications [Section titled “Data Service Specifications”](#data-service-specifications) ### Caching Strategy [Section titled “Caching Strategy”](#caching-strategy) **Redis Cache Usage:** * **Key Pattern**: `lei:{lei}` → `{client_id, legal_name, participation_status}` * **Key Pattern**: `coc:{country}:{number}` → `{client_id, legal_name, participation_status}` * **Key Pattern**: `eori:{eori}` → `{client_id, legal_name, participation_status}` * **TTL**: 24 hours * **Invalidation**: On participant data update ### Database Schema [Section titled “Database Schema”](#database-schema) **Key Tables:** * `party_reference`: Core party information * `legal_entity`: Legal entity details * `legal_entity_number`: External identifiers (LEI, CoC, EORI) * `agreement`: Membership agreements * `agreement_item`: Agreement line items * `legal_entity_contact`: Contact information * `legal_entity_endpoint`: API endpoints ### Event Schema [Section titled “Event Schema”](#event-schema) **Participant Events:** ```json { "event_id": "uuid", "event_type": "enum", "timestamp": "datetime", "source": "association-register", "data": { "client_id": "string", "event_details": {} } } ``` **Event Types:** * `participant.application.created` * `participant.application.approved` * `participant.application.rejected` * `participant.activated` * `participant.suspended` * `participant.terminated` * `participant.profile.updated` * `vad.issued` * `vad.revoked` ## Integration Protocols [Section titled “Integration Protocols”](#integration-protocols) ### OAuth 2.0 Client Credentials Flow [Section titled “OAuth 2.0 Client Credentials Flow”](#oauth-20-client-credentials-flow) ``` sequenceDiagram participant Client participant AuthServer as OAuth Server participant API as Association Register Client->>AuthServer: POST /oauth/token Note right of Client: grant_type=client_credentials
client_id, client_secret AuthServer->>Client: Access Token (JWT) Client->>API: GET /v1/participants/{id} Note right of Client: Authorization: Bearer API->>AuthServer: Validate Token AuthServer->>API: Token Valid + Claims API->>Client: Participant Data ``` ### VAD Token Usage Flow [Section titled “VAD Token Usage Flow”](#vad-token-usage-flow) ``` sequenceDiagram participant Consumer as Data Consumer participant ASR as Association Register participant Provider as Data Provider Consumer->>ASR: Request VAD Token (OAuth client credentials) ASR->>ASR: Verify Participation ASR->>Consumer: VAD Token (JWS) Consumer->>Provider: Request data (Bearer VAD) Provider->>ASR: Fetch JWKS (cached) ASR->>Provider: Public keys Provider->>Provider: Validate VAD, apply local policy Provider->>Consumer: Data (or 403 Forbidden) ``` ## Data Validation Rules [Section titled “Data Validation Rules”](#data-validation-rules) ### LEI Validation [Section titled “LEI Validation”](#lei-validation) * Format: 20 alphanumeric characters * Checksum validation (ISO 17442) * GLEIF verification required ### CoC Number Validation [Section titled “CoC Number Validation”](#coc-number-validation) * Format varies by country * Netherlands: 8 digits * Cross-reference with Chamber API ### EORI Validation [Section titled “EORI Validation”](#eori-validation) * Format: Country code + up to 15 characters * EU EORI validation service * Required for customs operations ## Error Codes and Handling [Section titled “Error Codes and Handling”](#error-codes-and-handling) ### HTTP Status Codes [Section titled “HTTP Status Codes”](#http-status-codes) * `200 OK`: Success * `201 Created`: Resource created * `400 Bad Request`: Invalid input * `401 Unauthorized`: Missing/invalid authentication * `403 Forbidden`: Insufficient permissions * `404 Not Found`: Resource not found * `409 Conflict`: Duplicate resource * `422 Unprocessable Entity`: Validation error * `429 Too Many Requests`: Rate limit exceeded * `500 Internal Server Error`: Server error * `503 Service Unavailable`: Service down ### Error Response Format [Section titled “Error Response Format”](#error-response-format) ```json { "error": { "code": "string", "message": "string", "details": [ { "field": "string", "issue": "string" } ], "request_id": "uuid" } } ``` ## Performance Requirements [Section titled “Performance Requirements”](#performance-requirements) ### Response Times [Section titled “Response Times”](#response-times) * Directory lookups: < 200ms (cached) * Participant profile retrieval: < 500ms * VAD token generation: < 1 second * Participant updates: < 2 seconds ### Throughput [Section titled “Throughput”](#throughput) * Support 1000+ concurrent requests * Handle 10,000+ directory lookups/minute * Process 100+ participant applications/day ### Availability [Section titled “Availability”](#availability) * 99.9% uptime SLA * Graceful degradation * Automatic failover ## Security Specifications [Section titled “Security Specifications”](#security-specifications) ### Authentication [Section titled “Authentication”](#authentication) * OAuth 2.0 Client Credentials * JWT-based access tokens * Mutual TLS (optional) ### Authorization [Section titled “Authorization”](#authorization) * Role-based access control (RBAC) * Scope-based permissions * Participation-status enforcement ### Data Protection [Section titled “Data Protection”](#data-protection) * Encryption at rest (AES-256) * Encryption in transit (TLS 1.3) * PII masking in logs * GDPR compliance ## Monitoring and Observability [Section titled “Monitoring and Observability”](#monitoring-and-observability) ### Metrics [Section titled “Metrics”](#metrics) * API request rate * Response times (p50, p95, p99) * Error rates * Cache hit rates * Database query performance ### Logging [Section titled “Logging”](#logging) * Structured JSON logs * Correlation IDs * Request/response logging * Audit trail ### Alerting [Section titled “Alerting”](#alerting) * SLA violations * Error rate spikes * Service degradation * Security events *** ## Related Documentation [Section titled “Related Documentation”](#related-documentation) * [Association Register Architecture](ctn-association-register.md) * [Member Onboarding Workflow](ctn-member-onboarding-workflow.md) ## Source [Section titled “Source”](#source) Based on: `BDI Association Register Format and Data Service.docx` (processed 2025-10-05) # BDI Association Register - Trust Lists Architecture *Based on BDI Association Trust Lists Specification* ## Overview [Section titled “Overview”](#overview) Trust Lists are a critical component of the BDI Association Register that maintain curated lists of trusted entities, certificates, and services within the network. They serve as the foundation for establishing trust relationships and enabling secure communication between network participants. ## Trust List Types [Section titled “Trust List Types”](#trust-list-types) ### 1. Member Trust List [Section titled “1. Member Trust List”](#1-member-trust-list) **Purpose**: Authoritative list of verified association members **Contents**: * Member Client IDs * Legal entity information (name, LEI, CoC, EORI) * Membership tier and status * Valid from/until dates * Public keys for signature verification * Service endpoints **Update Frequency**: Real-time (event-driven) **Format**: ```json { "trust_list_id": "uuid", "type": "member_trust_list", "version": "1.0", "issued_at": "datetime", "valid_until": "datetime", "issuer": "association.ctn.network", "members": [ { "client_id": "string", "legal_name": "string", "lei": "string", "coc_number": "string", "eori": "string", "participation_status": "string", "status": "active", "public_key_jwk": {/* JWK */}, "endpoints": [ { "type": "api", "url": "https://...", "protocols": ["https", "mtls"] } ], "valid_from": "datetime", "valid_until": "datetime" } ], "signature": "string" } ``` ### 2. Certificate Trust List [Section titled “2. Certificate Trust List”](#2-certificate-trust-list) **Purpose**: List of trusted certificate authorities and their certificates **Contents**: * CA certificates * Intermediate certificates * Certificate revocation information * Certificate policies * Validation rules **Update Frequency**: Daily or on-demand **Format**: ```json { "trust_list_id": "uuid", "type": "certificate_trust_list", "version": "1.0", "issued_at": "datetime", "certificates": [ { "certificate_id": "string", "subject": "string (DN)", "issuer": "string (DN)", "serial_number": "string", "not_before": "datetime", "not_after": "datetime", "public_key": "string (PEM)", "signature_algorithm": "string", "purposes": ["signing", "encryption"], "status": "valid", "revocation_info": { "crl_url": "string", "ocsp_url": "string" } } ], "revoked_certificates": [ { "serial_number": "string", "revocation_date": "datetime", "reason": "string" } ], "signature": "string" } ``` ### 3. Service Trust List [Section titled “3. Service Trust List”](#3-service-trust-list) **Purpose**: List of trusted services and their endpoints within the BDI network **Contents (2026)**: * Registered provider endpoints (data products) * External services * Service capabilities * SLA commitments **Update Frequency**: On service registration/update **Format**: ```json { "trust_list_id": "uuid", "type": "service_trust_list", "version": "1.0", "issued_at": "datetime", "services": [ { "service_id": "string", "service_name": "string", "service_type": "enum (data_provider|custodian|external)", "provider_client_id": "string", "endpoints": [ { "url": "https://...", "protocol": "https", "mtls_required": true } ], "capabilities": ["endpoint_registration", "data_access"], "sla": { "availability": "99.9%", "response_time_p95": "500ms" }, "status": "active", "valid_from": "datetime", "valid_until": "datetime" } ], "signature": "string" } ``` ## Trust List Management [Section titled “Trust List Management”](#trust-list-management) ### Publication [Section titled “Publication”](#publication) ``` graph TB subgraph "Association Register" TLM[Trust List Manager] DB[(Database)] EventBus[Event Bus] end subgraph "Publication Channels" API[Public API] CDN[CDN Distribution] EventSub[Event Subscribers] end subgraph "Consumers" OR[Data Providers] Custodian[Custodian
Systems] Member[Participant
Applications] end DB-->|Read|TLM TLM-->|Publish|API TLM-->|Distribute|CDN TLM-->|Notify|EventBus EventBus-->EventSub API-->OR API-->Custodian CDN-->Member EventSub-->OR EventSub-->Custodian ``` ### Update Workflow [Section titled “Update Workflow”](#update-workflow) ``` sequenceDiagram participant Event as Event Source participant TLM as Trust List Manager participant Validator as Validator participant Signer as Signing Service participant Store as Storage participant Publish as Publisher Event->>TLM: Member Status Change TLM->>TLM: Generate New Trust List TLM->>Validator: Validate List Structure Validator->>TLM: Validation OK TLM->>Signer: Sign Trust List Signer->>TLM: Signed Trust List TLM->>Store: Store New Version TLM->>Publish: Publish Update Publish->>Publish: Notify Subscribers Publish->>Publish: Update CDN Publish->>Publish: Update API ``` ### Versioning [Section titled “Versioning”](#versioning) * **Semantic Versioning**: Major.Minor.Patch * **Version History**: Last 30 versions retained * **Backwards Compatibility**: Breaking changes = major version bump * **Deprecation Policy**: 90-day notice for major changes ## Trust Validation [Section titled “Trust Validation”](#trust-validation) ### Trust Chain Verification [Section titled “Trust Chain Verification”](#trust-chain-verification) ``` graph TB Start[Start Validation] Start-->CheckMembership{Is Member in
Trust List?} CheckMembership-->|No|Reject[Reject: Not Trusted] CheckMembership-->|Yes|CheckStatus{Status =
Active?} CheckStatus-->|No|Reject CheckStatus-->|Yes|CheckExpiry{Within Valid
Period?} CheckExpiry-->|No|Reject CheckExpiry-->|Yes|CheckSignature{Trust List
Signature Valid?} CheckSignature-->|No|Reject CheckSignature-->|Yes|CheckCertificate{Certificate in
Certificate TL?} CheckCertificate-->|No|Reject CheckCertificate-->|Yes|CheckRevocation{Certificate
Revoked?} CheckRevocation-->|Yes|Reject CheckRevocation-->|No|Accept[Accept: Trusted] ``` ### Validation Process [Section titled “Validation Process”](#validation-process) 1. **Retrieve Latest Trust List** * Check version compatibility * Verify signature * Validate expiry 2. **Check Member Entry** * Lookup by client\_id * Verify membership status * Check validity period 3. **Verify Certificates** * Check certificate chain * Validate against Certificate Trust List * Verify not revoked 4. **Assess Trust Level** * Determine trust score based on: * Membership tier * Verification level * Historical reliability * Service quality metrics ## Distribution Mechanisms [Section titled “Distribution Mechanisms”](#distribution-mechanisms) ### Pull Model [Section titled “Pull Model”](#pull-model) **API Endpoint**: ```http GET /v1/trust-lists/{type}/{version} Authorization: Bearer Response: 200 OK { // Trust list as defined above } ``` **CDN Distribution**: * Global CDN for fast access * Automatic regional replication * Cache invalidation on updates * Public read access (no auth required) ### Push Model [Section titled “Push Model”](#push-model) **Event Notification**: ```json { "event_type": "trust_list.updated", "trust_list_type": "member_trust_list", "new_version": "1.5.0", "download_url": "https://cdn.../trust-lists/member/1.5.0", "changes": { "added": ["client_id_1", "client_id_2"], "removed": ["client_id_3"], "modified": ["client_id_4"] }, "timestamp": "datetime" } ``` **Webhook Delivery**: * Subscribers register webhook URLs * POST notifications on updates * Retry logic for failed deliveries * Signature verification for authenticity ## Security Considerations [Section titled “Security Considerations”](#security-considerations) ### Trust List Integrity [Section titled “Trust List Integrity”](#trust-list-integrity) **Digital Signatures**: * RSA-SHA256 or ECDSA signatures * Private key stored in Azure Key Vault * Public key published in well-known location * Signature validation mandatory **Tamper Detection**: * Hash verification * Version sequence validation * Timestamp verification * Audit trail of all changes ### Access Control [Section titled “Access Control”](#access-control) **Read Access**: * Member Trust List: Public (no auth) * Certificate Trust List: Public (no auth) * Service Trust List: Members only (VAD required) **Write Access**: * Association administrators only * Multi-factor authentication required * Approval workflow for sensitive changes * Audit logging ### Secure Distribution [Section titled “Secure Distribution”](#secure-distribution) **HTTPS Only**: * TLS 1.3 required * Certificate pinning recommended * HSTS headers * CORS policies **CDN Security**: * Signed URLs for sensitive data * Rate limiting * DDoS protection * Geographic restrictions (optional) ## Performance and Caching [Section titled “Performance and Caching”](#performance-and-caching) ### Caching Strategy [Section titled “Caching Strategy”](#caching-strategy) **Client-Side Caching**: * Cache-Control headers * ETags for conditional requests * Recommended TTL: 1 hour * Revalidation on critical operations **CDN Caching**: * Edge caching with 5-minute TTL * Instant purge on updates * Regional distribution * Failover to origin ### Performance Targets [Section titled “Performance Targets”](#performance-targets) * Trust list download: < 1 second * Signature verification: < 100ms * Member lookup in list: < 10ms * Update propagation: < 5 minutes globally ## Monitoring and Audit [Section titled “Monitoring and Audit”](#monitoring-and-audit) ### Metrics [Section titled “Metrics”](#metrics) * Trust list versions published * Download frequency * Validation success rate * Signature verification times * Update propagation latency ### Audit Events [Section titled “Audit Events”](#audit-events) * Trust list generated * Member added/removed/modified * Certificate added/revoked * Service registered/updated * Trust list downloaded * Validation performed ### Compliance [Section titled “Compliance”](#compliance) * GDPR: Personal data handling * NIS2: Critical infrastructure protection * eIDAS: Trust services * ISO 27001: Information security ## Integration Examples [Section titled “Integration Examples”](#integration-examples) ### Consumer Usage [Section titled “Consumer Usage”](#consumer-usage) ```javascript // Fetch latest participant trust list const trustList = await fetch( 'https://association.ctn.network/v1/trust-lists/participant/latest', { headers: { 'Authorization': `Bearer ${vadToken}` } } ).then(r => r.json()); // Verify signature const isValid = await verifyTrustListSignature(trustList); // Check if member is trusted function isTrustedMember(clientId) { const member = trustList.members.find(m => m.client_id === clientId); if (!member) return false; if (member.status !== 'active') return false; const now = new Date(); if (now < new Date(member.valid_from)) return false; if (now > new Date(member.valid_until)) return false; return true; } ``` ### Custodian System Usage [Section titled “Custodian System Usage”](#custodian-system-usage) ```javascript // Subscribe to trust list updates eventBus.subscribe('trust_list.updated', async (event) => { if (event.trust_list_type === 'member_trust_list') { // Download new version const newTrustList = await downloadTrustList(event.download_url); // Verify and update local cache if (await verifyTrustListSignature(newTrustList)) { await updateLocalTrustListCache(newTrustList); console.log(`Trust list updated to version ${event.new_version}`); } } }); ``` *** ## Related Documentation [Section titled “Related Documentation”](#related-documentation) * [Association Register Architecture](ctn-association-register.md) ## Source [Section titled “Source”](#source) Based on: `BDI Association Trust Lists.docx` (processed 2025-10-05) # Building Block View - CTN Components ## Component Overview [Section titled “Component Overview”](#component-overview) The 2026 delivery is centred on the **Association Register (ASR)**: participant management, identifier enrichment, VAD issuance, endpoint registration, and discovery. Authorisation stays **local at the data provider**. There is no central CTN policy decision point in 2026; each provider validates the VAD and applies its own policy. Components for later phases — Orchestration Register, dossier-bound VOD tokens, OPA-based policy engine, orchestrator portal federation — have their design notes parked under `docs/_out-of-scope-2026/` (gitignored). See the [Roadmap](../13-roadmap/ctn-roadmap.md) for when they come back into scope. ## Core Registers [Section titled “Core Registers”](#core-registers) ### Association Register - Detailed Architecture [Section titled “Association Register - Detailed Architecture”](#association-register---detailed-architecture) The CTN Association Register system consists of three main building blocks providing member management, self-service portals, and API services. #### System Overview (Level 0) [Section titled “System Overview (Level 0)”](#system-overview-level-0) ``` graph TB subgraph "CTN Association Register" A[Admin Portal] B[Participant Portal] C[API Layer] D[Database] end E[CTN Admin Users] -->|Manage Members| A F[CTN Member Users] -->|Self-Service| B A --> C B --> C C --> D G[Keycloak] -.->|Authentication| A G -.->|Authentication| B H[KvK API] -.->|Verification| C ``` ##### Level 0 Components [Section titled “Level 0 Components”](#level-0-components) | Component | Responsibility | Technology | | ---------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Admin Portal** | CTN staff interface for participant management | Frontend stack to be decided (see [Coding Standards → Open Questions](../08-crosscutting/ctn-coding-standards.md)); a React + TypeScript SPA is one candidate | | **Participant Portal** | Self-service interface for participant organisations | Same — frontend stack TBD | | **API Layer** | Business logic, data access, external integrations | Quarkus on Java 25+ LTS, packaged as OCI image | | **Database** | Persistent data storage | PostgreSQL 15+ (managed; Azure Flexible Server in the reference deployment, RDS / Crunchy as alternatives) | #### Level 1: Admin Portal [Section titled “Level 1: Admin Portal”](#level-1-admin-portal) ``` graph LR subgraph "Admin Portal" A[Authentication Module] B[Member Management] C[Legal Entity Management] D[Identifier Management] E[Contact Management] F[Endpoint Management] G[Audit Log Viewer] end A --> B A --> C A --> D A --> E A --> F A --> G B --> H[API Client] C --> H D --> H E --> H F --> H G --> H I[Keycloak] -.-> A ``` ##### Admin Portal Modules [Section titled “Admin Portal Modules”](#admin-portal-modules) ###### Authentication Module [Section titled “Authentication Module”](#authentication-module) **Responsibility:** User login, session management, token refresh **Key Components (framework-agnostic):** * Auth state/context — holds the current session * Keycloak integration module (OIDC) * Route guards for protected views **Dependencies:** * Keycloak OIDC adapter for the chosen frontend stack (e.g. `keycloak-js` for a JavaScript portal) * Keycloak realm/client configuration ###### Member Management Module [Section titled “Member Management Module”](#member-management-module) **Responsibility:** Member CRUD operations, status changes, approval workflows **Key Components:** * Participants grid — data table for participant listing * Participant detail view * Approval workflow — multi-step approval process **API Endpoints:** * `GET /v1/participants` — list participants (paginated) * `GET /v1/participants/{participantId}` — get participant details * `POST /v1/participants` — create participant * `PUT /v1/participants/{participantId}` — update participant * `DELETE /v1/participants/{participantId}` — delete participant ###### Legal Entity Management Module [Section titled “Legal Entity Management Module”](#legal-entity-management-module) **Responsibility:** Legal entity verification, KvK integration, international registries **Key Components:** * Legal-entity form — entity creation/editing * KvK lookup — KvK number search widget * Verification-status indicator **API Endpoints:** * `POST /v1/legal-entities` - Create legal entity * `GET /v1/legal-entities/{legalEntityId}` - Get entity details * `PUT /v1/legal-entities/{legalEntityId}/verify` - Trigger KvK verification #### Level 1: Participant Portal [Section titled “Level 1: Participant Portal”](#level-1-participant-portal) ``` graph LR subgraph "Participant Portal" A[Authentication Module] B[Organization Profile] C[Contact Management] D[Endpoint Management] E[Identifier Management] F[Support Page] end A --> B A --> C A --> D A --> E A --> F B --> G[API Client] C --> G D --> G E --> G H[Keycloak] -.-> A ``` ##### Participant Portal Modules [Section titled “Participant Portal Modules”](#participant-portal-modules) ###### Organization Profile Module [Section titled “Organization Profile Module”](#organization-profile-module) **Responsibility:** View and update organization information **Key Components:** * Profile view — organisation profile display * Profile edit form — editable organisation fields * Identifier list — display registered identifiers **Features:** * Read-only fields: KvK number, legal entity name * Editable fields: Address, phone, email * Auto-save on field blur ###### Endpoint Management Module [Section titled “Endpoint Management Module”](#endpoint-management-module) **Responsibility:** Register and manage API endpoints **Key Components:** * Endpoint grid — list of registered endpoints * Endpoint form — add/edit endpoint details * Health-status indicator **Validation:** * URL format validation (HTTPS required) * Duplicate endpoint detection * Version format validation (semver) #### Level 1: API Layer [Section titled “Level 1: API Layer”](#level-1-api-layer) ``` graph TB subgraph "API Layer (Quarkus on Java 25+)" A[REST Resources] B[Authentication quarkus-oidc] C[Business Logic Services] D[Data Access Layer Panache] E[External Integrations] end F[Admin Portal] --> A G[Participant Portal] --> A A --> B B --> C C --> D C --> E D --> H[PostgreSQL] E --> I[KvK API] E --> J[BDI Framework] K[Keycloak] -.->|Token Validation| B ``` ##### REST Resources [Section titled “REST Resources”](#rest-resources) ``` graph LR subgraph "REST Resources (JAX-RS)" A[ParticipantResource] B[LegalEntityResource] C[IdentifierResource] D[ContactResource] E[EndpointResource] end G[HTTP Request] --> A G --> B G --> C G --> D G --> E ``` **ParticipantResource:** * `GET /v1/participants` - List/search participants * `GET /v1/participants/{participantId}` - Retrieve single participant * `POST /v1/participants` - Register new participant * `PUT /v1/participants/{participantId}` - Modify participant details * `DELETE /v1/participants/{participantId}` - Remove participant (cascade delete) **LegalEntityResource:** * `POST /v1/legal-entities` - Register legal entity * `GET /v1/legal-entities/{legalEntityId}` - Retrieve entity * `PUT /v1/legal-entities/{legalEntityId}/verify` - Trigger KvK verification **IdentifierResource:** * `POST /v1/legal-entities/{legalEntityId}/identifiers` - Register identifier (KvK, EUID, LEI, etc.) * `GET /v1/legal-entities/{legalEntityId}/identifiers` - List identifiers for a legal entity * `PUT /v1/legal-entities/{legalEntityId}/identifiers/{identifierId}` - Modify identifier metadata * `DELETE /v1/legal-entities/{legalEntityId}/identifiers/{identifierId}` - Remove identifier **Token (VAD) endpoints:** * `GenerateVAD` - Generate VAD token * `GET /.well-known/jwks.json` - Retrieve RSA public keys for verification (JWKS) ##### Authentication (quarkus-oidc) [Section titled “Authentication (quarkus-oidc)”](#authentication-quarkus-oidc) **Responsibilities:** * Validate the bearer JWT against Keycloak (signature, expiry) as an OIDC resource server * Extract user claims (roles, email, organization) * Enforce `@Authenticated` / `@RolesAllowed` on resources * Return 401 Unauthorized if validation fails **Implementation:** * `quarkus-oidc` resource-server configuration (see [Coding Standards §4.3](../08-crosscutting/ctn-coding-standards.md)) * JWKS caching (\~10 minutes; rotation handled via Keycloak’s key-rotation events) * Role extraction for RBAC ##### Business Logic Services [Section titled “Business Logic Services”](#business-logic-services) ``` graph TB subgraph "Services Layer" A[ParticipantService] B[LegalEntityService] C[IdentifierService] D[ContactService] E[EndpointService] F[TokenService] end A --> G[Database Repository] B --> G C --> G D --> G E --> G F --> H[Crypto Service] F --> I[JWT Service] B --> J[KvK Integration] ``` **Service Responsibilities:** | Service | Responsibility | External Dependencies | | -------------------- | ------------------------------------------------------ | ------------------------------ | | `ParticipantService` | Participant CRUD, status management, approval workflow | None | | `LegalEntityService` | Entity creation, KvK verification | KvK API | | `IdentifierService` | Identifier registration, validation, deduplication | KvK API (for KvK numbers) | | `ContactService` | Contact management, email validation | None | | `EndpointService` | Endpoint registration, health checks | None | | `TokenService` | VAD token generation, signature verification | None (crypto / HSM-backed key) | ##### Data Access Layer [Section titled “Data Access Layer”](#data-access-layer) ``` graph LR subgraph "Database Repository (Panache)" A[ParticipantRepository] B[LegalEntityRepository] C[IdentifierRepository] D[ContactRepository] E[EndpointRepository] end A --> F[PostgreSQL Connection Pool] B --> F C --> F D --> F E --> F F --> G[PostgreSQL Database] ``` **Repository Pattern:** * Each repository handles CRUD for one entity * Parameterized queries (never string concatenation) * Transaction support for multi-table operations * Connection pooling (max 20 connections) **Example: `ParticipantRepository` (Panache)** ```java @ApplicationScoped public class ParticipantRepository implements PanacheRepositoryBase { public List search(ParticipantFilters filters, Page page) { ... } // findById, persist, update and delete are provided by Panache } ``` ##### External Integrations [Section titled “External Integrations”](#external-integrations) **KvK API Integration:** ``` sequenceDiagram participant API participant KvKService participant KvKAPI API->>KvKService: verifyKvKNumber(kvkNumber) KvKService->>KvKAPI: GET /basisprofiel/{kvkNumber} KvKAPI-->>KvKService: Company Details KvKService->>KvKService: Map to LegalEntity KvKService-->>API: Verified Entity Data ``` **Responsibilities:** * Verify KvK numbers exist * Fetch company details (name, address, status) * Map KvK response to internal data model * Cache KvK responses (24-hour TTL) * Handle rate limiting (max 100 req/min) **BDI Framework Integration:** ``` sequenceDiagram participant Client participant BDIService participant CryptoService participant Database Client->>BDIService: Generate VAD Token BDIService->>Database: Fetch Member Data Database-->>BDIService: Member Details BDIService->>BDIService: Build JWT Claims BDIService->>CryptoService: Sign with RSA Private Key CryptoService-->>BDIService: Signed Token BDIService-->>Client: VAD Token ``` **Responsibilities:** * Generate VAD tokens (JWT with custom claims) * Manage RSA key pairs (private key in Key Vault / HSM) * Implement BDI specification v2.0 compliance (VAD portion) #### Database Schema (Level 1) [Section titled “Database Schema (Level 1)”](#database-schema-level-1) The ASR data model has its own document: see **[ASR Data Model](ctn-asr-data-model.md)** for the full entity reference, the Keycloak ownership boundary (per [ADR-00003](../09-decisions/00003-separate-users-and-participants.md)), and the meta-field convention. A zoom-friendly, ERD-only companion is at [ctn-asr-data-model-erd.md](ctn-asr-data-model-erd.md). The high-level view below shows the core entities and their links to Keycloak. Client credentials live in Keycloak; the ASR holds only a reference (no secret). Dashed lines are logical cross-system links (resolved by ID, no database FK). ``` erDiagram participant ||--o{ legal_entity : "has" legal_entity ||--o{ legal_entity_identifier : "has" legal_entity ||--o{ contact : "has" participant ||--o{ endpoint : "registers" participant ||--o{ verification : "verified by" participant ||--o{ participant_role : "plays" participant ||--o{ m2m_client_ref : "owns" participant |o--o{ onboarding_request : "results in" participant ||..|| kc_organization : "org-id link (no FK)" m2m_client_ref ||..|| kc_m2m_client : "client-id link (no FK)" participant { uuid participant_id PK varchar keycloak_organization_id UK varchar status } legal_entity { uuid legal_entity_id PK uuid participant_id FK varchar primary_legal_name } legal_entity_identifier { uuid identifier_id PK uuid legal_entity_id FK varchar identifier_type varchar validation_status } contact { uuid contact_id PK uuid legal_entity_id FK varchar contact_type } endpoint { uuid endpoint_id PK uuid participant_id FK varchar lifecycle_status } verification { uuid verification_id PK uuid participant_id FK varchar type varchar status } participant_role { uuid role_id PK uuid participant_id FK varchar role } m2m_client_ref { uuid m2m_client_ref_id PK uuid participant_id FK varchar keycloak_client_id UK } onboarding_request { uuid request_id PK uuid participant_id FK varchar status } kc_organization { varchar org_id PK varchar alias } kc_m2m_client { varchar client_id PK varchar secret "in Keycloak only" } ``` Every ASR table carries the same meta fields — `created_at`, `updated_at`, `created_by`, `updated_by`, `is_deleted` — and UUID primary keys. Full per-entity detail, including the `audit_record` log and the Keycloak user / organization-group entities, is in the [ASR Data Model](ctn-asr-data-model.md) document. #### Deployment View (Simplified) [Section titled “Deployment View (Simplified)”](#deployment-view-simplified) ``` graph TB subgraph "Azure West Europe" subgraph "Static Web Apps" A[Admin Portal] B[Participant Portal] end subgraph "Kubernetes (AKS)" C[ASR API Quarkus] end subgraph "Database" D[PostgreSQL Flexible Server] end subgraph "Security" E[Keycloak] F[Azure Key Vault] end end A --> C B --> C C --> D A -.->|Auth| E B -.->|Auth| E C -.->|Secrets| F G[Internet Users] --> A G --> B ``` ### Association Register Components (2026) [Section titled “Association Register Components (2026)”](#association-register-components-2026) * **[SPI & Seam Architecture (as-built)](ctn-spi-seam-architecture.md)** — Developer-facing IST view of the implemented SPIs and seams (onboarding providers, lifecycle, claim verification), with per-element implemented/stubbed/shape-only status. Companion to this (SOLL) document. * **[Association Register](ctn-association-register.md)** — Participant management system. * **[ASR Data Model](ctn-asr-data-model.md)** — Target (SOLL) database model and ERD, with the Keycloak ownership boundary ([ERD-only view](ctn-asr-data-model-erd.md)). * **[Member Onboarding Workflow](ctn-member-onboarding-workflow.md)** — Vetting, verification, and credential issuance. * **[DNS Record Validation](ctn-dns-record-validation.md)** — Alternative authentication method for domain ownership. * **[Register Format and Data Service](ctn-association-register-format-data-service.md)** — API specifications and data formats, including endpoint registration. * **[Trust Lists Architecture](ctn-association-trust-lists.md)** — Trust list management and distribution. * **[Endpoint Lifecycle](ctn-endpoint-lifecycle.md)** — Registration, validation, suspension and retirement of provider endpoints. * **[Identifier Enrichment Architecture](ctn-identifier-enrichment-architecture.md)** — Automated KvK / KBO / VIES / GLEIF enrichment. #### Related Diagrams [Section titled “Related Diagrams”](#related-diagrams) The Association Register data model and ERD are maintained as Mermaid diagrams in [ASR Data Model](ctn-asr-data-model.md) and the [ERD-only view](ctn-asr-data-model-erd.md). Trust-list internals are described in [Trust Lists Architecture](ctn-association-trust-lists.md). ### Local authorisation (2026) [Section titled “Local authorisation (2026)”](#local-authorisation-2026) There is **no central policy decision component** in the 2026 architecture. Each data provider: 1. Validates the consumer’s VAD against the ASR’s JWKS endpoint. 2. Applies its own local policy (participant status, jurisdiction, data product, requester tier). 3. Returns the data shaped by that policy, or denies the request. The [BDI Connector](ctn-connector.md) is a reference implementation that providers can adopt to get VAD validation and policy hooks out of the box. ## Supporting Components [Section titled “Supporting Components”](#supporting-components) ### Authentication Components [Section titled “Authentication Components”](#authentication-components) * **[Centralised Auth](ctn-centralized-auth.md)** — OAuth provider services for user authentication. * **[Keycloak M2M Authentication](ctn-keycloak-m2m-auth.md)** — Self-hosted M2M authentication via OAuth 2.0 client credentials, per [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md) and [ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md). ### Integration Components [Section titled “Integration Components”](#integration-components) * **[BDI Connector](ctn-connector.md)** — CTN integration connector. ### Data Consumer [Section titled “Data Consumer”](#data-consumer) * A lean 2026 specification for the Visibility Achterland Containerlogistiek portal will be added here. The earlier draft (which assumed federated Orchestration Registers and an OPA PDP) is parked under `docs/_out-of-scope-2026/`. ## Component Architecture Layers [Section titled “Component Architecture Layers”](#component-architecture-layers) ``` graph TB subgraph "Presentation Layer" AdminPortal[Admin Portal] MemberPortal[Participant Portal] DataConsumerPortal[Data-Consumer Portal
Visibility] end subgraph "Business Logic Layer" AssocReg[Association Register] VADGen[VAD Generator] DNS[DNS Validator] Enrich[Identifier Enrichment
KvK / KBO / VIES / GLEIF] EndpointReg[Endpoint Registry & Discovery] end subgraph "Data Layer" PG[(PostgreSQL)] end subgraph "Infrastructure Layer" KeyVault[Secret Store / HSM-backed keys] IdP[OAuth / OIDC Server
Keycloak] end AdminPortal --> AssocReg MemberPortal --> AssocReg DataConsumerPortal --> AssocReg AssocReg --> VADGen AssocReg --> DNS AssocReg --> Enrich AssocReg --> EndpointReg AssocReg --> PG VADGen --> KeyVault AssocReg --> IdP ``` ## Component Interactions [Section titled “Component Interactions”](#component-interactions) ### VAD Issuance Flow [Section titled “VAD Issuance Flow”](#vad-issuance-flow) 1. The participant’s authorised representative authenticates via eHerkenning. 2. The ASR runs the enrichment + verification checks (KvK / KBO / VIES / GLEIF). 3. On success the IdP issues an OAuth client credential. 4. When the consumer requests a VAD via the OAuth client-credentials grant, the ASR signs and returns the VAD (JWS). ### Data Access Flow (local authorisation) [Section titled “Data Access Flow (local authorisation)”](#data-access-flow-local-authorisation) ``` sequenceDiagram participant Consumer participant ASR as Association Register participant Provider as Data Provider Consumer->>ASR: OAuth client-credentials grant ASR-->>Consumer: VAD (JWS-signed JWT) Consumer->>Provider: GET /data-product (Bearer VAD) Provider->>ASR: GET /.well-known/jwks (cached) ASR-->>Provider: JWKS Provider->>Provider: Validate signature, claims, expiry Provider->>Provider: Apply local policy alt Authorised Provider-->>Consumer: 200 OK + data else Denied Provider-->>Consumer: 403 Forbidden end ``` ## Component Specifications [Section titled “Component Specifications”](#component-specifications) ### Standard Interfaces [Section titled “Standard Interfaces”](#standard-interfaces) * **REST APIs**: OpenAPI 3.x specifications. * **Tokens**: JWS-signed JWTs (RS256). * **Health**: standard health-check endpoints. * **Discovery**: ASR exposes a discovery API listing registered endpoints and their OpenAPI specs. ### Common Patterns [Section titled “Common Patterns”](#common-patterns) * **Idempotency**: safe retries on POST. * **Circuit breaker**: resilience for external calls (KvK, KBO, VIES, GLEIF). * **Audit logging**: structured JSON for every state change and authorisation decision. ## Component Dependencies [Section titled “Component Dependencies”](#component-dependencies) ``` graph LR subgraph "External Dependencies" IDP[OAuth / OIDC IdP] KVK[Chamber of Commerce] KBO[KBO Belgium] GLEIF[LEI Registry] VIES[VIES VAT] EH[eHerkenning] end subgraph "CTN Components" AssocReg[Association Register] end AssocReg --> KVK AssocReg --> KBO AssocReg --> GLEIF AssocReg --> VIES AssocReg --> IDP AssocReg --> EH ``` ## Configuration Management [Section titled “Configuration Management”](#configuration-management) ### Association Register [Section titled “Association Register”](#association-register) ```yaml association: verification: kvk_enabled: true kbo_enabled: true vies_enabled: true gleif_enabled: true dns_enabled: true token: signing_algorithm: RS256 validity_hours: 24 re_verification: interval_months: 12 # at least annually high_risk_interval_months: 3 # quarterly where risk requires ``` ## Deployment Considerations [Section titled “Deployment Considerations”](#deployment-considerations) ### Scaling Requirements (2026) [Section titled “Scaling Requirements (2026)”](#scaling-requirements-2026) | Component | Min Instances | Max Instances | Scaling Metric | | -------------------- | ------------- | ------------- | --------------------- | | Association Register | 2 | 10 | CPU > 70% | | Discovery API | 2 | 10 | Request rate | | IdP (Keycloak) | 2 | 8 | CPU + active sessions | ### Resource Allocation [Section titled “Resource Allocation”](#resource-allocation) * **Database**: General Purpose tier with read replicas. * **Kubernetes pods (AKS)**: right-sized requests/limits with autoscaling. * **Storage**: hot tier for active documents. ## Security Considerations [Section titled “Security Considerations”](#security-considerations) ### Component Security [Section titled “Component Security”](#component-security) * **Managed Identity**: service-to-service authentication. * **Private Endpoints**: network isolation for databases. * **Key Vault Integration**: HSM-backed secret management for signing keys. * **TLS Everywhere**: encrypted communication, TLS 1.3. ### Token Security [Section titled “Token Security”](#token-security) * **Private Key Storage**: HSM-backed keys in Key Vault * **Certificate Rotation**: Automated 90-day rotation * **Token Expiration**: Time-limited validity * **Signature Validation**: JWKS endpoint verification ## Next Steps [Section titled “Next Steps”](#next-steps) For implementation details, refer to: 1. Individual component documentation (linked above) 2. [Runtime scenarios](../06-runtime) for dynamic behavior 3. [Deployment view](../07-deployment/ctn-deployment.md) for infrastructure 4. [Cross-cutting concepts](../08-crosscutting/ctn-crosscutting.md) for patterns *** [← Back to Solution Strategy](../04-solution-strategy/ctn-solution-strategy.md) | [↑ Back to Index](../ctn-arc42-index.md) | [Next: Runtime View →](../06-runtime/ctn-association-onboarding.md) # Centralised Authentication — Architecture Documentation ## 1. Introduction and Goals [Section titled “1. Introduction and Goals”](#1-introduction-and-goals) ### Business Context [Section titled “Business Context”](#business-context) The Centralised Authentication component provides the OAuth / OIDC server used by the Association Register. It issues OAuth client credentials for M2M flows and signs the **VAD** tokens that data providers validate locally. The 2026 architecture has a **single shared authentication layer**; there is no separate authentication service per CTN domain. ### Primary Goals [Section titled “Primary Goals”](#primary-goals) * **Unified authentication** — one OAuth / OIDC server for all CTN participants and ASR-issued tokens. * **VAD token management** — issuance and validation of VAD (Verifiable Association Data) tokens signed with the association’s private key. * **EU data compliance** — full GDPR alignment; EU-resident infrastructure and key material. * **High availability** — reliable authentication for critical trade operations. * **Standards compliance** — OAuth 2.0 (client credentials), OpenID Connect, JWS RS256. ### Quality Goals [Section titled “Quality Goals”](#quality-goals) * **Security** — robust signing, JWKS-based key distribution, HSM-backed private keys. * **Performance** — <200ms token issuance, <100ms token validation. * **Reliability** — 99.95% uptime. * **Compliance** — EU data residency, GDPR. * **Scalability** — thousands of concurrent authentication requests. ### Stakeholders [Section titled “Stakeholders”](#stakeholders) * **Participants** — organisations consuming or providing data. * **Association administrators** — managing onboarding and OAuth client lifecycle. * **Data providers** — validating VAD tokens before releasing data. * **EU regulatory bodies** — ensuring compliance with data-protection regulations. ## 2. Architecture Constraints [Section titled “2. Architecture Constraints”](#2-architecture-constraints) ### Technical [Section titled “Technical”](#technical) * **OAuth 2.0** — full client-credentials support; OIDC where human authentication is involved. * **EU data residency** — all authentication data inside EU jurisdiction. * **JWS** — JWT signing with RS256 minimum. * **High availability** — multi-region deployment within the EU. ### Regulatory [Section titled “Regulatory”](#regulatory) * **GDPR** — data protection. * **eIDAS** — compatible with eHerkenning for authorised-representative login. * **Audit** — complete audit trail. ### Integration [Section titled “Integration”](#integration) * **Single OAuth server** — serves the entire CTN ASR estate; no per-domain split. * **Webhook support** — events for OAuth-client lifecycle. ## 3. System Scope and Context [Section titled “3. System Scope and Context”](#3-system-scope-and-context) ### Technical Context [Section titled “Technical Context”](#technical-context) * **Upstream**: KvK, KBO, VIES, GLEIF, eHerkenning. * **Downstream**: the Association Register and every data provider that validates VAD tokens. * **Integration points**: JWKS endpoint, OAuth APIs, webhook receivers, audit pipeline. ## 4. Solution Strategy [Section titled “4. Solution Strategy”](#4-solution-strategy) ### Key Decisions [Section titled “Key Decisions”](#key-decisions) * **Self-hosted OAuth server** — Keycloak is the chosen IdP (see [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md)). * **EU-first design** — all services and data storage within EU boundaries. * **Standards-based integration** — OAuth 2.0, OpenID Connect, JWS RS256. ### Technology choice [Section titled “Technology choice”](#technology-choice) **Keycloak (self-hosted)** is the chosen IdP — see [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md). The detailed M2M integration is described in [Keycloak M2M Authentication](ctn-keycloak-m2m-auth.md); per-participant federation (Entra ID, eHerkenning, local) is decided in [ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md). Key material lives in an HSM-backed key vault (e.g. Azure Key Vault Managed HSM, AWS KMS Managed HSM, or HashiCorp Vault). ## 5. Building Block View [Section titled “5. Building Block View”](#5-building-block-view) ### Core Components [Section titled “Core Components”](#core-components) * **OAuth provider** — authentication and authorisation server. * **VAD token service** — generates VAD tokens with participation claims. * **Key management** — cryptographic key storage and rotation. * **Admin interface** — administration of users and OAuth clients. * **JWKS endpoint** — public-key discovery for token validation. ### Integration Components [Section titled “Integration Components”](#integration-components) * **API gateway** — rate limiting, monitoring, API management. * **Webhook service** — events for OAuth-client lifecycle. * **Audit service** — logging for compliance and security monitoring. * **Identity federation** — integration with eHerkenning and eIDAS. ### Security Components [Section titled “Security Components”](#security-components) * **Certificate management** — X.509 lifecycle. * **MFA** — for administrative access. * **Rate limiting** — protection against authentication attacks. * **Anomaly detection** — suspicious authentication patterns. ## 6. Runtime View [Section titled “6. Runtime View”](#6-runtime-view) ### VAD Issuance (M2M, client credentials) [Section titled “VAD Issuance (M2M, client credentials)”](#vad-issuance-m2m-client-credentials) ```plaintext Consumer → OAuth server: POST /oauth/v2/token (client_credentials) OAuth server → Key management: retrieve signing key OAuth server → VAD token service: generate VAD with participation claims VAD token service → OAuth server: signed VAD (JWS RS256) OAuth server → Consumer: access_token (VAD) OAuth server → Audit service: log issuance ``` ### VAD Validation (at the data provider) [Section titled “VAD Validation (at the data provider)”](#vad-validation-at-the-data-provider) ```plaintext Provider → JWKS endpoint: GET /.well-known/jwks (cached) JWKS endpoint → Provider: public keys Provider → Provider: verify signature, claims, expiry Provider → Audit service: log validation result ``` ### Token Format Example [Section titled “Token Format Example”](#token-format-example) **VAD Token Structure** ```json { "iss": "https://association.ctn.network", "sub": "participant-organisation-id", "aud": "https://ctn.network", "iat": 1754764800, "exp": 1754851200, "token_type": "VAD", "participation_status": "active", "compliance_verified": true, "https://schemas.ctn.network/claims/legal_entity/name": "Transport Company BV", "https://schemas.ctn.network/claims/legal_entity/registry/kvk": "12345678" } ``` ### Token validation checklist (data service provider) [Section titled “Token validation checklist (data service provider)”](#token-validation-checklist-data-service-provider) A provider validates every incoming VAD/access token in this order; reject on the first failure: 1. **Signature** — valid against the ASR’s JWKS public keys → 401 on failure. 2. **Expiry** — `exp` is in the future (allow \~30s clock skew) → 401. 3. **Issuer** — `iss` equals the expected ASR issuer → 401. 4. **Audience** — `aud` contains the provider’s own client ID → 403. **Critical:** without this check, a token minted for another API would be accepted here. 5. **Organisation** — read the participant identity from the token for local business rules. Operational rules: cache JWKS and refresh on an unknown `kid`; reject `alg: none`; never log raw tokens (log the `jti` for traceability). ### Integration Patterns [Section titled “Integration Patterns”](#integration-patterns) * **OAuth 2.0 client-credentials flow** — M2M. * **JWT bearer pattern** — stateless authentication with cryptographic verification. * **JWKS discovery** — automated public-key distribution. * **Webhook notifications** — event-driven updates for OAuth-client lifecycle. ## 7. Deployment View [Section titled “7. Deployment View”](#7-deployment-view) ### EU Deployment [Section titled “EU Deployment”](#eu-deployment) * **Primary region**: EU-West (Frankfurt or Amsterdam). * **Secondary region**: EU-Central (Dublin or Paris) for disaster recovery. * **Data residency**: all authentication data inside EU boundaries. ### Scaling Strategy [Section titled “Scaling Strategy”](#scaling-strategy) * **Horizontal scaling** — multiple OAuth server instances behind a load balancer. * **Database scaling** — read replicas for token validation, master for token issuance. * **Caching** — JWKS cache at the consumer and provider edge. ## 8. Cross-cutting Concepts [Section titled “8. Cross-cutting Concepts”](#8-cross-cutting-concepts) ### Security Patterns [Section titled “Security Patterns”](#security-patterns) * Zero Trust — every request authenticated; no implicit trust. * Defence in depth — multiple security layers. * Principle of least privilege. * Continuous monitoring. ### EU Compliance [Section titled “EU Compliance”](#eu-compliance) * GDPR-compliant data processing. * Data portability. * Right to be forgotten. * Consent management. ### Token Lifecycle (simplified example) [Section titled “Token Lifecycle (simplified example)”](#token-lifecycle-simplified-example) ```javascript const generateVad = async (claims, audience) => { const token = await tokenService.create({ type: 'VAD', claims: validateClaims(claims), audience: audience, expirySeconds: 86400, signingKey: await keyManager.getCurrentKey(), }); await auditService.logTokenGeneration(token, claims); return token; }; ``` ### Error Handling [Section titled “Error Handling”](#error-handling) * Graceful degradation if a component fails. * Circuit breaker for downstream calls. * Audit preservation for all errors. * User-facing error messages without exposing system internals. ## 10. Quality Requirements [Section titled “10. Quality Requirements”](#10-quality-requirements) ### Performance [Section titled “Performance”](#performance) * VAD issuance <200ms. * Token validation <100ms. * 10,000+ token operations per minute supported. * JWKS endpoint <50ms. ### Security [Section titled “Security”](#security) * RSA-2048 (or stronger) signing. * Automated key rotation every 90 days. * 100% of authentication events logged. * <4h response time for security incidents. ### Compliance [Section titled “Compliance”](#compliance) * GDPR-compliant. * 100% authentication data in the EU. * Minimum 7 years audit retention. * SOC 2 Type II (or equivalent) target. ### Availability [Section titled “Availability”](#availability) * 99.95% uptime. * RTO 15 minutes, RPO 1 minute. * Multi-region failover within 15 minutes. * 24/7 monitoring with automated alerting. ## 11. Risks and Technical Debt [Section titled “11. Risks and Technical Debt”](#11-risks-and-technical-debt) ### Current Risks [Section titled “Current Risks”](#current-risks) * **Vendor lock-in** — mitigated by Keycloak’s open-source status and the standard OAuth/OIDC protocols (other compliant IdPs could replace it if ever needed). * **Regulatory change** — privacy and identity rules evolve; design assumes regular review. * **Single point of failure** — the OAuth server is central by design; multi-region failover mitigates this. * **Non-EU participants** — handle case by case. ### Technical Debt [Section titled “Technical Debt”](#technical-debt) * **Token versioning** — no formal versioning strategy yet (see [risks register](../11-risks/ctn-risks.md), TD-2). * **Provider migration playbook** — to be documented. * **Compliance automation** — some controls are still manual. ### Future Improvements [Section titled “Future Improvements”](#future-improvements) * Multi-provider support for resilience. * AI-powered anomaly detection. * Full eIDAS-2 wallet integration. * Quantum-safe cryptography readiness. *** *This document is part of the 2026 CTN scope. Components related to dossier-bound authorisation (the Orchestration Register and dual-token validation) are out of scope for 2026 — see the [Roadmap](../13-roadmap/ctn-roadmap.md).* # BDI Reference Architecture – BDI Connector The **BDI Connector** is defined as the standard integration point for secure communication, signing and event handeling: interface between internal IT landscape and BDI network, implemented by every participant. The BDI Connector is a set of functions that can be implemented building upon existing infrastructure such as reverse proxies, proxies and messaging platforms/brokers, or implemented as a container with all functions. ## 1. Core Concept BDI [Section titled “1. Core Concept BDI”](#1-core-concept-bdi) The BDI (Basic Data Infrastructure) Reference Architecture is designed to minimize complexity for participants by standardizing the core components and confining complexity of authentication, signing certficates handling and event handling to the set of core components. The main standardised components in the 2026 scope are: * **ASR (Association Register)** — Registration of onboarding information, authentication enrolment, local Certificate Authority that issues a BDI Signing Certificate to a Connector, issuer of signed Assurance (VAD) tokens, maintenance of synced trust lists, event notification to participants/users. * **BDI Connector** — Standard integration point for secure communication, signing and event handling: interface between the internal IT landscape and the BDI network, implemented by every participant. The implementation of an ASR will typically use a BDI Connector as its standard integration point. Additional components (Orchestration Register, Local Policy Engine as a separate PDP service) are part of later phases — see the [Roadmap](../13-roadmap/ctn-roadmap.md). ## 2. BDI Connector [Section titled “2. BDI Connector”](#2-bdi-connector) {This section needs to be expanded by Jomco and others] Forward proxy, with TLS. Proxy API Gateway Webhook or messaging platform for event notification * subcribing entities to specific topics * sending notifications * receiving notifications from others OAuth client credentials flow handling: enrollment ASR, access token requests JWT validation (JWKS endpoint of signing authorithy, content, expiration etc.) PKI handling * generation of key pair * storage of private key * ACME protocol initial verification with ASR, CSR requests * ACME key rotation with ASR * signing of tokens or data Trust list handling (domain, public keys Data requests: receiving, sending. Sending data. # Database Schema Management — PostgreSQL on Azure (CTN ASR) > This is the Azure-concrete companion to the platform-neutral [Database Schema Management](ctn-database-schema-management-cloud-agnostic.md) principles. The CTN stack is **PostgreSQL + Quarkus** (see [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md)), so schema is managed with **Liquibase** ([TDR-0001](../../ai/adrs/TDR-0001-liquibase-owns-schema.md)), not with SQL Server / SSDT or ORM auto-DDL. ## Recommendation [Section titled “Recommendation”](#recommendation) Use **Liquibase** (built into Quarkus) with **YAML changesets** kept in version control. Do **not** let Hibernate generate or update the schema (`hibernate-orm.schema-management.strategy` stays `none`/`validate` in every profile), and do not manage schema through an ORM’s migration feature. ## Context [Section titled “Context”](#context) The CTN Association Register (ASR) database runs on **Azure Database for PostgreSQL — Flexible Server** and is consumed by multiple parties: * Admin Portal and Participant Portal (via the ASR API — they never talk to the database directly) * ASR API on Kubernetes/AKS (the single writer of the schema) * Potentially external integrations via the API The ASR API owns the schema definition; every other consumer reaches data through the API, not through its own ORM. A second driver shapes the tool choice: the dataspace model is federated, and federated operators are expected — much later — to run their own ASR on the DBMS of their choice. Migrations must therefore not lock the schema history to the PostgreSQL dialect (see [TDR-0001](../../ai/adrs/TDR-0001-liquibase-owns-schema.md)). Only PostgreSQL is supported and CI-tested today. ## Why Liquibase with YAML changesets [Section titled “Why Liquibase with YAML changesets”](#why-liquibase-with-yaml-changesets) | Aspect | ORM auto-DDL (e.g. Hibernate `update`) | Liquibase YAML changesets | | ---------------------- | --------------------------------------- | ----------------------------------------------------------------------------------------------- | | Schema definition | Inferred from Java entities | Explicit, declarative changesets | | Deployment coupling | Tied to the app’s ORM | Versioned, deliberate step | | Reviewability | Requires reading Java + ORM conventions | Declarative YAML, reviewable by anyone | | Destructive changes | Silent / unpredictable | Explicit, reviewed per changeset | | Audit trail | Hidden in entity diffs | `DATABASECHANGELOG` + Git history | | Multi-DBMS portability | Per-dialect DDL generation, unreviewed | Portable constructs rendered per dialect; non-portable DDL isolated in `dbms`-scoped changesets | | Rollback | None | Forward-fix changesets + verified backups (no rollback blocks authored — see TDR-0001) | Flyway plain-SQL was the original proposal and remains a fine tool, but every migration would be PostgreSQL dialect throughout; a second DBMS would mean a parallel migration history per dialect, rewritten under an operator’s deadline. Liquibase writes portable constructs once. See TDR-0001 for the full trade-off. ## Approach [Section titled “Approach”](#approach) * **Changelogs live in the repo** under `src/main/resources/db/changelog/`: `changelog-master.yaml` lists explicit `include` entries in execution order; one YAML file per migration unit, with one changeset per table/index/logical change. They are reviewed in pull requests like any other code. Applied changesets are never edited — corrections are new forward changesets. * **Contexts are always pinned**: Liquibase runs context-tagged changesets when *no* runtime context is given, so every Quarkus profile pins one (`dev`/`test`/`prod`), plus an unprofiled `prod` default for custom profiles. The dev demo seed is a `context: dev`, `runOnChange` changeset and therefore exists only in dev databases. * **Apply at boot only in dev/test**: `quarkus.liquibase.migrate-at-start` is `true` in dev/test (ephemeral databases) and `false` in prod. * **Apply via init container in production**: the application process never migrates. A CI-built migrations image (`FROM liquibase/liquibase` with the changelog tree copied in, tagged in lockstep with the app image) runs `liquibase update --contexts=prod` as a Kubernetes **init container** before the app container starts. Concurrent replicas serialize on `DATABASECHANGELOGLOCK`; already-applied changesets no-op. The `--contexts=prod` flag is mandatory — a context-less run would apply the dev seed. * **Privilege split**: the init container connects with a DDL-capable PostgreSQL role; the application role is DML-only. Read-only consumers get a dedicated role. Only the migration role may alter the schema. * **Destructive changes use expand-contract** (add column → backfill → switch → drop in a later release), never an in-place rename/drop in one step. ## Type mapping — the hybrid rule [Section titled “Type mapping — the hybrid rule”](#type-mapping--the-hybrid-rule) This table is the canonical home of TDR-0001’s hybrid type rule. Changesets use Liquibase **logical types** where the dialect mapping is trustworthy, and **verbatim PostgreSQL types** where it is not: | Write in the changeset | Renders on PostgreSQL | Renders on e.g. MSSQL | Why | | --------------------------------------- | --------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `uuid` (logical) | `uuid` | `uniqueidentifier` | Trustworthy mapping | | `boolean` (logical) | `boolean` | `bit` | Trustworthy mapping | | `timestamp(6) with time zone` (logical) | `timestamptz(6)` | `datetimeoffset` | Trustworthy mapping; precision preserved | | `text` (verbatim) | `text` | *(decide at port time)* | Deliberately **not** `clob`: Liquibase’s `clob` can render as `oid`/LOB on PostgreSQL, breaking Hibernate mapping parity | Two construct rules ride along: * **Check constraints are raw `sql` changes** (ANSI `ALTER TABLE … ADD CONSTRAINT … CHECK`, un-scoped, inside the owning table’s changeset): Liquibase OSS silently drops the `checkConstraint` column attribute on `createTable`, and `addCheckConstraint` is Pro-only. Constraint names are spelled explicitly to match what PostgreSQL derives for Hibernate’s inline checks. * **Non-portable DDL is a raw `sql` changeset scoped with `dbms:`** — e.g. the partial unique index `uq_onboarding_open_kvk` (`dbms: postgresql`). A future DBMS adds a sibling changeset with the **same constraint/index name**, since application code matches names when classifying violations. The porting surface is enumerable: search the changelog for `dbms:`. The rendered PostgreSQL DDL mirrors what Hibernate would generate from the entities, so the planned graduation to `hibernate-orm.schema-management.strategy=validate` passes without drift. ## Azure specifics [Section titled “Azure specifics”](#azure-specifics) * **Service**: Azure Database for PostgreSQL — Flexible Server (PostgreSQL 15+), reached over a private endpoint. * **Backups**: geo-redundant, point-in-time restore; a verified backup is a precondition for any destructive changeset. * **Secrets**: the migration and runtime credentials are two distinct Key Vault secrets (DDL role for the init container, DML role for the app), injected via Quarkus config / pod env — never committed. * **CI/CD**: the pipeline builds the app image and the migrations image from the same commit; the init container applies the changelog before the new application pods become ready. ## Cloud portability [Section titled “Cloud portability”](#cloud-portability) Portable changesets are rendered per dialect by Liquibase, so the same `db/changelog/` tree runs against PostgreSQL on AWS (RDS/Aurora), GCP (Cloud SQL) or a self-hosted cluster — and is the basis for a future second DBMS when a federated operator names one. Database-specific DDL is isolated in `dbms`-scoped changesets. See the [platform-neutral principles](ctn-database-schema-management-cloud-agnostic.md) for the full workflow, dangerous-operation table and data-migration patterns. ## Conclusion [Section titled “Conclusion”](#conclusion) For CTN’s PostgreSQL + Quarkus stack, **Liquibase with reviewed YAML changesets** gives explicit, auditable, dialect-portable schema management with a clear owner (the ASR API), a privilege-split apply path in production, and a bounded porting surface for the federated multi-DBMS future — without coupling the schema to an ORM or to SQL Server tooling. # Database Schema Management ## The Core Problem [Section titled “The Core Problem”](#the-core-problem) Database schema changes are inevitable. Without discipline, they cause data loss, production incidents, and emergency restores. This document establishes platform-agnostic principles and practices to prevent these issues. ## Principles [Section titled “Principles”](#principles) ### 1. Schema is Infrastructure, Not Application Code [Section titled “1. Schema is Infrastructure, Not Application Code”](#1-schema-is-infrastructure-not-application-code) The database schema serves multiple consumers (portals, APIs, background jobs, integrations). No single application should own or deploy the schema through its ORM. **Implications:** * Dedicated repository for schema definitions * Separate CI/CD pipeline for database deployments * Applications connect to the database; they don’t define it ### 2. Declarative Over Imperative [Section titled “2. Declarative Over Imperative”](#2-declarative-over-imperative) Define the **desired state** of your schema, not the steps to get there. Let tooling calculate the delta. | Approach | Example | Risk | | ----------- | ----------------------------------- | ---------------------------------------------------------- | | Imperative | ”Add column X, then rename Y to Z” | Steps can conflict, order matters, hard to review | | Declarative | ”Table should have columns A, B, C” | Tool compares current vs desired, generates safe migration | ### 3. Data Migration is Explicit, Never Automatic [Section titled “3. Data Migration is Explicit, Never Automatic”](#3-data-migration-is-explicit-never-automatic) Schema tools handle structure. Data transformations require explicit scripts written and reviewed by humans. **Golden rule:** If a schema change could affect existing data, a data migration script must accompany it. ## The Dangerous Operations [Section titled “The Dangerous Operations”](#the-dangerous-operations) These operations can cause data loss if handled incorrectly: | Operation | Risk | Safe Approach | | ------------------ | ---------------------------------- | ------------------------------------------------------- | | Column rename | Tool may interpret as drop + add | Use tool’s refactor feature or write explicit migration | | Column type change | Implicit conversion may truncate | Add new column, migrate data, drop old column | | Column removal | Data gone | Verify column is unused, backup, then remove | | Table rename | References break, data may be lost | Use refactor tooling or staged migration | | NOT NULL addition | Fails if NULLs exist | Add default value or backfill first | ## Recommended Workflow [Section titled “Recommended Workflow”](#recommended-workflow) ### Step 1: Schema Changes in Version Control [Section titled “Step 1: Schema Changes in Version Control”](#step-1-schema-changes-in-version-control) All schema definitions live in Git: ```plaintext /database /tables members.sql organizations.sql /views /functions /migrations # Data migration scripts /pre-deploy # Scripts that run before schema changes /post-deploy # Scripts that run after schema changes ``` ### Step 2: Review Process [Section titled “Step 2: Review Process”](#step-2-review-process) Every schema change requires: 1. **Structural review** — Is the SQL correct? 2. **Data impact review** — Does this affect existing data? 3. **Rollback plan** — How do we undo this if needed? ### Step 3: Deployment Pipeline [Section titled “Step 3: Deployment Pipeline”](#step-3-deployment-pipeline) ```plaintext [Schema Repo] → [Build/Validate] → [Generate Diff] → [Review Diff] → [Deploy to Test] → [Deploy to Prod] ``` **Critical:** Always review the generated diff before production deployment. Automated deployment without human review of the actual changes is how data gets lost. ### Step 4: Protect Against Data Loss [Section titled “Step 4: Protect Against Data Loss”](#step-4-protect-against-data-loss) Configure your deployment tooling to: * **Block on data loss** — Abort if the operation would drop columns/tables with data * **Require explicit override** — Force developers to acknowledge destructive changes * **Generate rollback script** — Always have a way back ## Tool Options by Platform [Section titled “Tool Options by Platform”](#tool-options-by-platform) ### PostgreSQL (Azure, AWS, GCP) [Section titled “PostgreSQL (Azure, AWS, GCP)”](#postgresql-azure-aws-gcp) | Tool | Approach | Strengths | | -------------------- | --------------------------- | -------------------------------------- | | **Flyway** | Versioned migration scripts | Simple, widely adopted, CI/CD friendly | | **Liquibase** | Declarative (XML/YAML/SQL) | Rollback support, database refactoring | | **pgAdmin/pg\_dump** | Manual scripts | Full control, no tooling dependency | | **Sqitch** | Change management | Dependency tracking between changes | **Recommendation for PostgreSQL:** Flyway or Liquibase. Both are cloud-agnostic, work with any PostgreSQL instance, and integrate with CI/CD. CTN chose **Liquibase with YAML changesets** for its dialect portability ([TDR-0001](../../ai/adrs/TDR-0001-liquibase-owns-schema.md)). ### SQL Server (Azure SQL, AWS RDS, On-premise) [Section titled “SQL Server (Azure SQL, AWS RDS, On-premise)”](#sql-server-azure-sql-aws-rds-on-premise) | Tool | Approach | Strengths | | ------------- | -------------------- | ------------------------------------------ | | **SSDT** | Declarative (dacpac) | Native Microsoft tooling, refactor support | | **Flyway** | Versioned migrations | Cross-platform consistency | | **Liquibase** | Declarative | Same tooling across database platforms | ### Cloud-Agnostic Strategy [Section titled “Cloud-Agnostic Strategy”](#cloud-agnostic-strategy) If portability between clouds/databases is required: 1. Use **Liquibase** or **Flyway** — they work across PostgreSQL, SQL Server, MySQL, Oracle 2. Keep SQL as ANSI-compliant as possible 3. Isolate database-specific syntax in clearly marked files ## Data Migration Patterns [Section titled “Data Migration Patterns”](#data-migration-patterns) ### Pattern 1: Expand-Contract (Safe Column Rename) [Section titled “Pattern 1: Expand-Contract (Safe Column Rename)”](#pattern-1-expand-contract-safe-column-rename) Instead of renaming directly: ```sql -- Step 1: Add new column ALTER TABLE members ADD COLUMN client_name VARCHAR(255); -- Step 2: Migrate data (in post-deploy script) UPDATE members SET client_name = customer_name WHERE client_name IS NULL; -- Step 3: Application updated to use new column (separate deployment) -- Step 4: Remove old column (next release, after verification) ALTER TABLE members DROP COLUMN customer_name; ``` This is slower but eliminates data loss risk. ### Pattern 2: Feature Flags for Schema [Section titled “Pattern 2: Feature Flags for Schema”](#pattern-2-feature-flags-for-schema) For major schema changes: 1. Deploy new schema structures alongside old 2. Application writes to both (feature flagged) 3. Migrate historical data 4. Switch reads to new structure 5. Remove old structure after verification ### Pattern 3: Blue-Green for Databases [Section titled “Pattern 3: Blue-Green for Databases”](#pattern-3-blue-green-for-databases) For high-risk migrations: 1. Create new database with new schema 2. Migrate data with transformation 3. Switch application to new database 4. Keep old database as rollback option ## Pre-Deployment Checklist [Section titled “Pre-Deployment Checklist”](#pre-deployment-checklist) Before any schema deployment: * [ ] Schema change reviewed by second person * [ ] Data impact assessed — which rows/tables affected? * [ ] Migration script written for data transformations * [ ] Rollback plan documented * [ ] Deployment tested on non-production environment * [ ] Backup verified and accessible * [ ] Deployment window communicated if downtime required ## Anti-Patterns to Avoid [Section titled “Anti-Patterns to Avoid”](#anti-patterns-to-avoid) 1. **ORM-driven migrations in production** — ORMs are for development convenience, not production schema management 2. **Auto-apply migrations on application startup** — Schema changes should be deliberate, reviewed, and separate from app deployment 3. **Trusting the tool blindly** — Always review generated SQL before execution 4. **No rollback plan** — Every change needs a way back 5. **Schema changes without data assessment** — The question isn’t “will this SQL run?” but “what happens to existing data?” ## Conclusion [Section titled “Conclusion”](#conclusion) Database schema management is a discipline, not a tool choice. The principles remain constant regardless of whether you use PostgreSQL on AWS, SQL Server on Azure, or any other combination: 1. Treat schema as infrastructure with its own lifecycle 2. Use declarative definitions where possible 3. Handle data migrations explicitly 4. Review every change for data impact 5. Always have a rollback plan The specific tool matters less than the process around it. Flyway, Liquibase, SSDT — all can work well if the team follows safe practices. All can cause disasters if used carelessly. # DNS Record Validation *Alternative Authentication Method for CTN* ## Overview [Section titled “Overview”](#overview) DNS Record Validation provides a lightweight, low-assurance alternative to traditional business registry verification (LEI, KvK) for organization authentication in the Connected Trade Network. This method leverages existing DNS infrastructure to enable organizations to demonstrate domain control with minimal administrative overhead. ## Motivation [Section titled “Motivation”](#motivation) While legal registry-based verification provides strong assurance, it can be resource-intensive and may not be available for all organizations. DNS validation offers: * **Low Friction**: Requires only DNS record access * **Automation-Friendly**: Machine-verifiable with minimal manual oversight * **Industry Precedent**: Already used in SSL/TLS (ACME), email auth (SPF/DKIM/DMARC) * **Quick Bootstrap**: Enables rapid onboarding for pilot programs ## Method [Section titled “Method”](#method) ### Challenge Mechanism [Section titled “Challenge Mechanism”](#challenge-mechanism) 1. Association Register generates unique **validation token** for organization 2. Organization publishes token in DNS under reserved namespace 3. Association Register verifies DNS record via standard resolution ### Record Format [Section titled “Record Format”](#record-format) ```dns _ctn-register.example.org. IN TXT "ctn-validation-token-12345" ``` Alternative CNAME approach for delegated validation: ```dns _ctn-register.example.org. IN CNAME _validated.ctn.network. ``` ### Verification Process [Section titled “Verification Process”](#verification-process) ``` sequenceDiagram participant Org as Organization participant ASR as Association Register participant DNS as DNS System Org->>ASR: Request DNS validation ASR->>ASR: Generate unique token ASR-->>Org: Return token & instructions Org->>DNS: Create TXT record Note over DNS: _ctn-register.example.org
TXT "ctn-validation-token-12345" Org->>ASR: Request verification ASR->>DNS: Query TXT record DNS-->>ASR: Return token ASR->>ASR: Validate token match alt Token matches ASR-->>Org: Validation successful ASR->>ASR: Mark as DNS-validated else Token mismatch ASR-->>Org: Validation failed end ``` ### Implementation Details [Section titled “Implementation Details”](#implementation-details) #### Token Generation [Section titled “Token Generation”](#token-generation) * Use cryptographically secure random generator * Minimum 128 bits of entropy * Base64url encoding for DNS compatibility * Include timestamp for expiration tracking #### Validation Logic [Section titled “Validation Logic”](#validation-logic) ```python def validate_dns_record(domain: str, expected_token: str) -> bool: """Validate DNS TXT record for domain ownership.""" try: # Query DNS for TXT records records = dns.resolver.query(f"_ctn-register.{domain}", "TXT") # Check each TXT record for token for record in records: if expected_token in str(record): return True return False except dns.resolver.NXDOMAIN: return False ``` #### Refresh Requirements [Section titled “Refresh Requirements”](#refresh-requirements) * Initial validation: Valid for 90 days * Renewal validation: Valid for 365 days * Automated monitoring for record removal * Grace period of 7 days for DNS issues ## Security Considerations [Section titled “Security Considerations”](#security-considerations) ### Assurance Level [Section titled “Assurance Level”](#assurance-level) * **Classification**: LOW ASSURANCE * **Capability**: Proves domain control only * **Limitation**: Does not verify legal entity identity * **Risk**: Temporary domain hijacking could enable impersonation ### Threat Mitigation [Section titled “Threat Mitigation”](#threat-mitigation) | Threat | Mitigation | | ------------------- | ------------------------------- | | DNS Cache Poisoning | DNSSEC validation required | | Subdomain Takeover | Validate apex domain only | | Token Reuse | Time-limited, single-use tokens | | Social Engineering | Clear labeling as low-assurance | ### Policy Integration [Section titled “Policy Integration”](#policy-integration) Organizations validated via DNS receive restricted VAD claims: ```json { "https://schemas.ctn.network/claims/validation/method": "DNS", "https://schemas.ctn.network/claims/validation/assurance_level": "LOW", "https://schemas.ctn.network/claims/validation/domain": "example.org", "https://schemas.ctn.network/claims/validation/validated_at": "2025-01-15T10:00:00Z" } ``` ## Comparison with Other Methods [Section titled “Comparison with Other Methods”](#comparison-with-other-methods) | Method | Assurance | Complexity | Time | Cost | Use Case | | ------------------------- | --------- | ---------- | ------------ | ---- | ----------------------------------- | | **DNS Validation** | Low | Low | Minutes | Free | Quick onboarding, pilots | | **KvK Registry** | High | Medium | Days | €€ | Dutch companies | | **LEI Registration** | High | High | Weeks | €€€ | International corps | | **EORI Verification** | High | Medium | Days | €€ | EU traders | | **eHerkenning (EH3/EH4)** | High | Medium | Minutes–Days | €€ | Dutch entities; optional, see below | | **Manual Review** | Variable | High | Weeks | €€€€ | Special cases | ## eHerkenning Authentication (optional) [Section titled “eHerkenning Authentication (optional)”](#eherkenning-authentication-optional) > **Optional.** eHerkenning is **not** part of the 2026 baseline and is **not required** for any participant. It applies to Dutch entities only. eHerkenning (EH3/EH4) is the Dutch government’s authorised-representation scheme. In CTN it is used as a **step-up verification** of organisational representation (a KvK cross-check), not as a domain-validation method and not as a login. The verification flow, evidence format, and security guardrails are described in [Member Onboarding §3.6](ctn-member-onboarding-workflow.md). It ranks as a HIGH-assurance identity verification alongside KvK and LEI; the resulting `EHerkenning` verification contributes to the participant’s trust level, which a data provider may consult when applying its local authorisation policy. ## Implementation Phases [Section titled “Implementation Phases”](#implementation-phases) ### Phase 1: Pilot (Current) [Section titled “Phase 1: Pilot (Current)”](#phase-1-pilot-current) * Manual token generation * Email-based instructions * Manual verification trigger * Limited to pilot participants ### Phase 2: Automated (Q2 2025) [Section titled “Phase 2: Automated (Q2 2025)”](#phase-2-automated-q2-2025) * Self-service portal * Automated verification * API integration * Monitoring dashboard ### Phase 3: Enhanced (Q3 2025) [Section titled “Phase 3: Enhanced (Q3 2025)”](#phase-3-enhanced-q3-2025) * Multi-domain validation * Subdomain delegation * DNSSEC enforcement * CAA record integration ## Configuration [Section titled “Configuration”](#configuration) ### Association Register Settings [Section titled “Association Register Settings”](#association-register-settings) ```yaml dns_validation: enabled: true namespace: "_ctn-register" token_length: 32 token_charset: "alphanumeric" ttl_initial: 7776000 # 90 days ttl_renewal: 31536000 # 365 days grace_period: 604800 # 7 days dnssec_required: false # Phase 3 allowed_record_types: - TXT - CNAME ``` ### Access Restrictions [Section titled “Access Restrictions”](#access-restrictions) Organizations validated via DNS: * Cannot access high-sensitivity data * Limited to read-only operations initially * Require additional verification for write access * Must transition to higher assurance within 12 months ## References [Section titled “References”](#references) * RFC 8555: Automatic Certificate Management Environment (ACME) * RFC 7208: Sender Policy Framework (SPF) * RFC 6376: DomainKeys Identified Mail (DKIM) * RFC 7489: Domain-based Message Authentication (DMARC) * RFC 8659: DNS Certification Authority Authorization (CAA) *** [← Back to Building Blocks](ctn-building-blocks.md) | [↑ Back to Index](../ctn-arc42-index.md) # CTN Endpoint Lifecycle ## Overview [Section titled “Overview”](#overview) This document describes the complete lifecycle of a data endpoint within the Connected Trade Network (CTN) — from registration by a provider to data retrieval by a consumer. Keycloak is used as the identity and token service for all M2M (machine-to-machine) authentication. *** ## Terminology [Section titled “Terminology”](#terminology) | Term | Definition | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Provider** | Organization that makes data available via an endpoint (data owner) | | **Consumer** | Organization that retrieves data from a provider | | **Endpoint** | An API/data source operated by a provider from which data can be retrieved | | **System** | A technical system (application/service) that operates on behalf of an organization. Can be *single-tenant* (own system) or *multi-tenant* (shared, e.g., a SaaS platform) | | **ASR** | Association Register — the system that manages association participation, performs compliance checks, and issues VAD tokens. In the endpoint lifecycle, the ASR also manages registrations, relations, and access grants | | **M2M Client** | A Keycloak OAuth 2.0 client using the Client Credentials grant, used for system-to-system communication | | **Keycloak** | Open-source identity provider used by CTN for authentication, token issuance, and M2M client management | *** ## Lifecycle Overview [Section titled “Lifecycle Overview”](#lifecycle-overview) ``` flowchart LR subgraph provider["PROVIDER (data owner)"] direction LR R["1 Registration
Enter endpoint URL"] --> V["2 Validation
Prove ownership"] V --> P["3 Publication
Visible in service directory
+ set access model"] end subgraph consumer["CONSUMER (data user)"] direction LR A["4 Request Access
Register system in ASR,
create M2M client,
request endpoint access"] --> AU{"5 Authorization"} AU -->|"open: auto-approved"| D["6 Data Exchange
Fetch token from
Keycloak, retrieve data"] AU -->|"request: provider approves"| D AU -->|"restricted: invite only"| D end P --> A style provider fill:#e8f4fd,stroke:#1a73e8,stroke-width:2px style consumer fill:#e8f8e8,stroke:#34a853,stroke-width:2px style R fill:#fff,stroke:#1a73e8 style V fill:#fff,stroke:#1a73e8 style P fill:#fff,stroke:#1a73e8 style A fill:#fff,stroke:#34a853 style AU fill:#fff3cd,stroke:#e6a800 style D fill:#fff,stroke:#34a853 ``` *** ## 1. Registration [Section titled “1. Registration”](#1-registration) The provider registers an endpoint in the CTN Member Portal. **Fields captured:** | Field | Description | | ----------------- | ------------------------------------------------------------------------------------- | | `endpoint_url` | Base URL of the endpoint (e.g., `https://api.company.nl/ctn`) | | `name` | Human-readable name for the endpoint | | `description` | Description of the available data | | `data_type` | Type of data the endpoint serves (e.g., `container_status`, `eta`, `transport_order`) | | `provider_org_id` | The organization that owns the endpoint | **Status after this step:** `REGISTERED` *** ## 2. Validation [Section titled “2. Validation”](#2-validation) After registration, the provider must prove they actually control the endpoint. This is done through a **Callback Challenge-Response** mechanism (industry standard, used by Stripe, GitHub, Slack). ### Wizard Flow [Section titled “Wizard Flow”](#wizard-flow) | Step | Action | | ----------- | -------------------------------------------------- | | 1. Details | Provider enters endpoint URL | | 2. Verify | CTN sends challenge to endpoint → auto-verified | | 3. Test | Health check (`GET /health`) confirms connectivity | | 4. Activate | Endpoint status set to `VERIFIED` | ### Sequence Diagram [Section titled “Sequence Diagram”](#sequence-diagram) ``` sequenceDiagram autonumber participant M as Provider
(Portal UI) participant CTN as CTN Backend participant E as Provider's
Endpoint rect rgb(240, 248, 255) Note over M,E: Step 1 — Register Endpoint M->>CTN: Register endpoint URL
POST /api/endpoints CTN->>CTN: Validate URL format
Generate challenge token
(UUID + HMAC signature) CTN-->>M: 202 Accepted
Status: "Verifying..." end rect rgb(240, 255, 240) Note over M,E: Step 2 — Challenge-Response Verification CTN->>E: POST {endpoint_url}/.well-known/ctn/verify
Headers: X-CTN-Signature
Body: { "challenge": "abc123...", "timestamp": "..." } alt ✅ Endpoint responds correctly E-->>CTN: 200 OK
{ "challenge": "abc123..." } CTN->>CTN: Validate challenge match
+ response time < 10s CTN-->>M: ✅ Verification Passed else ❌ Wrong challenge / timeout / error E-->>CTN: 4xx / 5xx / timeout CTN-->>M: ❌ Verification Failed
Show error details Note over M: Provider can retry
after fixing endpoint end end rect rgb(255, 248, 240) Note over M,E: Step 3 — Connection Test CTN->>E: GET {endpoint_url}/health
Headers: Authorization (Bearer) E-->>CTN: 200 OK
{ "status": "healthy", "version": "1.0" } CTN-->>M: ✅ Connection Test Passed end rect rgb(248, 240, 255) Note over M,E: Step 4 — Activate M->>CTN: Activate endpoint CTN->>CTN: Status: VERIFIED CTN-->>M: ✅ Endpoint verified end ``` ### Primary: Callback Challenge-Response [Section titled “Primary: Callback Challenge-Response”](#primary-callback-challenge-response) **CTN → Provider Endpoint** ```http POST {endpoint_url}/.well-known/ctn/verify Content-Type: application/json X-CTN-Signature: sha256= X-CTN-Request-Id: { "challenge": "ctn_v1_a3b8f2e1-4c6d-4e8f-9a1b-2c3d4e5f6789", "timestamp": "2026-02-04T09:30:00Z", "endpoint_id": "ep_abc123" } ``` **Expected response (within 10 seconds)** ```http HTTP/1.1 200 OK Content-Type: application/json { "challenge": "ctn_v1_a3b8f2e1-4c6d-4e8f-9a1b-2c3d4e5f6789" } ``` **Validation rules:** * Response must be HTTP 200 * `challenge` value must match exactly * Response must arrive within **10 seconds** * `X-CTN-Signature` header allows the provider to verify the request genuinely came from CTN **Challenge token format:** ```plaintext ctn_v1_{uuid} ``` * Prefix `ctn_v1_` for versioning * UUID v4 for uniqueness * Single-use, expires after 15 minutes * HMAC-SHA256 signature of the body using a shared signing secret ### Provider Implementation Requirement [Section titled “Provider Implementation Requirement”](#provider-implementation-requirement) The provider implements a single endpoint: ```plaintext POST /.well-known/ctn/verify ``` Minimal implementation (illustrative; the actual Quarkus version lives in the ASR codebase): ```javascript app.post('/.well-known/ctn/verify', (req, res) => { const { challenge } = req.body; // Optionally verify X-CTN-Signature header res.json({ challenge }); }); ``` ### Fallback: Well-Known URL Verification [Section titled “Fallback: Well-Known URL Verification”](#fallback-well-known-url-verification) If the callback challenge fails (e.g., firewall blocks inbound POST, or the verify handler hasn’t been deployed yet), CTN falls back to file-based verification. 1. CTN generates a verification token 2. Provider places a JSON file at a well-known path on their endpoint domain 3. CTN fetches the file to confirm the provider controls the server **Location:** ```plaintext GET {endpoint_url}/.well-known/ctn/verification.json ``` **Expected content:** ```json { "ctn_verification": "ctn_v1_a3b8f2e1-4c6d-4e8f-9a1b-2c3d4e5f6789", "endpoint_id": "ep_abc123", "created_at": "2026-02-04T09:30:00Z" } ``` * File must be accessible via HTTPS * CTN polls up to 3 times at 30-second intervals * Token expires after 24 hours **Status after this step:** `VERIFIED` *** ## 3. Publication [Section titled “3. Publication”](#3-publication) After successful validation, the provider publishes the endpoint, making it visible in the CTN service directory for other participants. ### What the Provider Configures [Section titled “What the Provider Configures”](#what-the-provider-configures) | Setting | Description | | ---------------- | ------------------------------------------------------------------------------- | | **Access model** | `open`, `request`, or `restricted` (see below) | | **Scopes** | Which data operations are available (e.g., `read:container_status`, `read:eta`) | | **Rate limits** | Maximum requests per consumer per time period | | **Description** | Documentation about the endpoint for potential consumers | ### Access Models [Section titled “Access Models”](#access-models) The access model determines how consumers gain access to the endpoint. This choice affects the flow through steps 4 and 5. | Model | Behavior | Steps 4d + 5 | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- | | **`open`** | Any registered CTN member can access the endpoint immediately. No approval required. | Consumer requests access → **auto-approved** instantly. Keycloak scopes and policies are created automatically. | | **`request`** | Consumers must request access. The provider reviews and approves or denies each request. | Consumer requests access → status `PENDING` → provider approves/denies → Keycloak updated on approval. | | **`restricted`** | Only organizations explicitly invited by the provider can access the endpoint. The endpoint is visible in the directory but the “Request Access” button is hidden for non-invited organizations. | Provider sends invite → consumer accepts → **auto-approved**. | ### Publication Action [Section titled “Publication Action”](#publication-action) The provider clicks “Publish” in the Member Portal. CTN registers the endpoint in the service directory. ```http POST /api/endpoints/{endpoint_id}/publish Authorization: Bearer {provider_token} { "access_model": "open", "available_scopes": ["read:container_status", "read:eta"], "rate_limit": { "requests_per_minute": 60 }, "description": "Real-time container status updates for terminal X" } ``` **Status after this step:** `PUBLISHED` The endpoint is now discoverable in the service directory. For `open` endpoints, consumers can immediately request and receive access. For `request` and `restricted` endpoints, additional steps are required. *** ## 4. Request Access [Section titled “4. Request Access”](#4-request-access) A consumer that wants to retrieve data from a published endpoint goes through the following steps. ### 4a. Register System in ASR [Section titled “4a. Register System in ASR”](#4a-register-system-in-asr) Before a consumer can request access, the system they use to retrieve data must be registered in the ASR. | Field | Description | | -------------- | --------------------------------------------------------------------- | | `system_name` | Name of the system | | `system_type` | `single_tenant` (own system) or `multi_tenant` (SaaS/shared platform) | | `owner_org_id` | Organization that manages the system | **Single-tenant** systems can only be used by the owning organization. **Multi-tenant** systems (e.g., a SaaS platform from a software vendor) can be linked by multiple organizations. > **Design decision:** Software vendors offering multi-tenant systems are registered as organizations in the ASR with the role `system_provider`. They register the system; participants then create a relation with that system. ### 4b. Create Relation: Organization ↔ System [Section titled “4b. Create Relation: Organization ↔ System”](#4b-create-relation-organization--system) A consumer organization registers in the ASR that a specific system may operate on its behalf. ```http POST /asr/relations Authorization: Bearer {consumer_org_token} { "organization_id": "org_consumer_456", "system_id": "sys_tms_789", "relation_type": "authorized_consumer" } ``` This means: “System X may retrieve data on behalf of Organization Y.” ### 4c. Create M2M Client in Keycloak [Section titled “4c. Create M2M Client in Keycloak”](#4c-create-m2m-client-in-keycloak) When a system is registered in the ASR and linked to an organization, a corresponding M2M client is created in Keycloak. This happens automatically via the CTN backend or can be triggered via the Member Portal (”+ Add M2M Client”). #### Keycloak Configuration: M2M Client [Section titled “Keycloak Configuration: M2M Client”](#keycloak-configuration-m2m-client) **Realm:** `ctn` **Client settings:** | Setting | Value | | ---------------------------- | --------------------------------------------------------------------- | | Client ID | `m2m-{org_id}-{system_id}` (e.g., `m2m-org_consumer_456-sys_tms_789`) | | Client Protocol | `openid-connect` | | Access Type | `confidential` | | Service Accounts Enabled | `true` (required for Client Credentials grant) | | Standard Flow Enabled | `false` | | Direct Access Grants Enabled | `false` | | Authorization Enabled | `true` | **Client Credentials:** After creation, Keycloak generates a `client_id` and `client_secret`. These are presented once to the consumer in the Member Portal and must be stored securely by the consumer. **Client Scopes:** Each M2M client is assigned scopes that match the granted endpoint access. Scopes follow the pattern: ```plaintext endpoint:{endpoint_id}:{operation} ``` Examples: `endpoint:ep_abc123:read`, `endpoint:ep_abc123:write` #### Keycloak Configuration: Custom Token Mapper [Section titled “Keycloak Configuration: Custom Token Mapper”](#keycloak-configuration-custom-token-mapper) To include organization and system context in the JWT, configure a custom protocol mapper on the M2M client: | Mapper Setting | Value | | ------------------- | ------------------------------------ | | Name | `ctn-org-claims` | | Mapper Type | `Hardcoded claim` or `Script Mapper` | | Token Claim Name | `ctn_org_id` | | Claim Value | `{organization_id}` | | Add to access token | `true` | Additional claims to include: | Claim | Value | Purpose | | ----------------- | -------------------------------- | -------------------------------------------------- | | `ctn_org_id` | Organization ID | Identifies which organization the token represents | | `ctn_system_id` | System ID | Identifies which system is making the request | | `ctn_system_type` | `single_tenant` / `multi_tenant` | Allows the provider to distinguish system types | ### 4d. Request Access to Endpoint [Section titled “4d. Request Access to Endpoint”](#4d-request-access-to-endpoint) The consumer requests access to a published endpoint via the Member Portal. ```http POST /api/endpoints/{endpoint_id}/access-requests Authorization: Bearer {consumer_token} { "consumer_org_id": "org_consumer_456", "requested_scopes": ["read:container_status"], "purpose": "Real-time tracking for our supply chain" } ``` **What happens next depends on the access model:** | Access model | Result | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `open` | Access request is **auto-approved**. Keycloak scopes and authorization policies are created immediately. Consumer receives a `201 Created` with status `APPROVED`. | | `request` | Access request status is set to `PENDING`. Provider is notified. See step 5. | | `restricted` | If the consumer was invited by the provider, access is **auto-approved**. Otherwise, the request is rejected with `403 Forbidden`. | *** ## 5. Authorization [Section titled “5. Authorization”](#5-authorization) This step differs based on the access model configured by the provider. ### Access Model: `open` [Section titled “Access Model: open”](#access-model-open) No manual authorization step. When the consumer requests access (step 4d), the CTN backend immediately: 1. Creates the AccessGrant with status `APPROVED` 2. Assigns the endpoint scopes to the consumer’s Keycloak M2M client 3. Creates the Keycloak authorization policy and permission The consumer can proceed directly to step 6 (Data Exchange). ### Access Model: `request` [Section titled “Access Model: request”](#access-model-request) The provider receives a notification that a consumer has requested access and reviews the request in the Member Portal. **Provider Approves or Denies:** ```http PUT /api/access-requests/{request_id} Authorization: Bearer {provider_token} { "decision": "approved", "granted_scopes": ["read:container_status"], "expires_at": "2027-02-04T00:00:00Z" } ``` ### Access Model: `restricted` [Section titled “Access Model: restricted”](#access-model-restricted) The provider proactively invites specific organizations via the Member Portal: ```http POST /api/endpoints/{endpoint_id}/invitations Authorization: Bearer {provider_token} { "invited_org_id": "org_partner_789", "granted_scopes": ["read:container_status"], "expires_at": "2027-02-04T00:00:00Z" } ``` When the invited consumer accepts (or requests access), authorization is auto-approved. ### What Happens After Approval [Section titled “What Happens After Approval”](#what-happens-after-approval) 1. The ASR registers the authorization: consumer organization X may use scopes Z on endpoint E 2. The CTN backend updates the Keycloak M2M client for the consumer, adding the granted scopes 3. The Keycloak authorization policy is updated to allow token issuance for this endpoint #### Keycloak Configuration: Authorization Policy [Section titled “Keycloak Configuration: Authorization Policy”](#keycloak-configuration-authorization-policy) For each approved access grant, an authorization policy is configured in Keycloak: **Resource:** | Setting | Value | | ------- | ------------------------------- | | Name | `endpoint-{endpoint_id}` | | Type | `ctn:endpoint` | | URI | `{endpoint_url}/*` | | Scopes | `read`, `write` (as applicable) | **Policy:** | Setting | Value | | ------- | ---------------------------------------- | | Name | `policy-{consumer_org_id}-{endpoint_id}` | | Type | `Client Policy` | | Clients | `m2m-{org_id}-{system_id}` | | Logic | `Positive` | **Permission:** | Setting | Value | | -------- | ---------------------------------------- | | Name | `perm-{consumer_org_id}-{endpoint_id}` | | Type | `Scope-based` | | Resource | `endpoint-{endpoint_id}` | | Scopes | Granted scopes (e.g., `read`) | | Policies | `policy-{consumer_org_id}-{endpoint_id}` | **Status after this step:** Access request `APPROVED` *** ## 6. Data Exchange [Section titled “6. Data Exchange”](#6-data-exchange) With the full trust path established, the consumer system can now retrieve data from the provider endpoint. ### Token Flow via Keycloak [Section titled “Token Flow via Keycloak”](#token-flow-via-keycloak) ``` sequenceDiagram autonumber participant CS as Consumer
System participant KC as Keycloak
(ASR Token Service) participant ASR as ASR
(CTN Backend) participant PE as Provider
Endpoint rect rgb(240, 248, 255) Note over CS,PE: Token Request (OAuth 2.0 Client Credentials) CS->>KC: POST /realms/ctn/protocol/openid-connect/token
grant_type=client_credentials
client_id={m2m_client_id}
client_secret={m2m_client_secret}
scope=endpoint:{endpoint_id}:read KC->>ASR: Validate via custom SPI/policy:
1. System registered?
2. Organization registered?
3. System↔Organization relation?
4. Access to endpoint approved? alt ✅ All checks pass ASR-->>KC: Approved + claims KC-->>CS: 200 OK
{ "access_token": "eyJ...",
"token_type": "Bearer",
"expires_in": 3600 } else ❌ Check fails ASR-->>KC: Denied KC-->>CS: 403 Forbidden
{ "error": "access_denied" } end end rect rgb(240, 255, 240) Note over CS,PE: Data Retrieval CS->>PE: GET {endpoint_url}/data
Authorization: Bearer {access_token} PE->>PE: Validate JWT:
- Signature (Keycloak public key)
- Claims (org_id, scopes)
- Expiry PE-->>CS: 200 OK
{ data } end ``` ### Token Request [Section titled “Token Request”](#token-request) The consumer system requests a token using the standard OAuth 2.0 Client Credentials flow: ```http POST /realms/ctn/protocol/openid-connect/token Content-Type: application/x-www-form-urlencoded grant_type=client_credentials &client_id=m2m-org_consumer_456-sys_tms_789 &client_secret={client_secret} &scope=endpoint:ep_abc123:read ``` ### What Keycloak/ASR Validates at Token Issuance [Section titled “What Keycloak/ASR Validates at Token Issuance”](#what-keycloakasr-validates-at-token-issuance) 1. **System registered?** — Is the consumer system known in the ASR? 2. **Organization registered?** — Is the consumer organization known in the ASR? 3. **System ↔ Organization relation?** — Is this system authorized to operate on behalf of this organization? 4. **Access to endpoint?** — Has the provider granted this consumer organization access to this endpoint? Only when all four checks pass does Keycloak issue an access token. ### JWT Token Claims [Section titled “JWT Token Claims”](#jwt-token-claims) The issued JWT contains the following CTN-specific claims: ```json { "iss": "https://keycloak.ctn.network/realms/ctn", "sub": "m2m-org_consumer_456-sys_tms_789", "aud": "ctn-api", "exp": 1738666200, "iat": 1738662600, "scope": "endpoint:ep_abc123:read", "ctn_org_id": "org_consumer_456", "ctn_system_id": "sys_tms_789", "ctn_system_type": "single_tenant" } ``` ### What the Provider Validates on a Data Request [Section titled “What the Provider Validates on a Data Request”](#what-the-provider-validates-on-a-data-request) The provider only needs to validate the token — all trust logic resides in the ASR/Keycloak. The provider checks: * **Token signature** — Is the token valid and signed by Keycloak? (Verified using Keycloak’s public key, available at `/realms/ctn/protocol/openid-connect/certs`) * **Token claims** — Does the `ctn_org_id` match an authorized consumer? * **Scopes** — Is the consumer only requesting data they’re authorized for? * **Token expiry** — Is the token still valid? This is the core simplification: complexity is removed from both provider and consumer and centralized in Keycloak/ASR. *** ## ASR Entities [Section titled “ASR Entities”](#asr-entities) ``` erDiagram Organization ||--o{ System : "operates via" Organization ||--o{ Endpoint : "owns (as provider)" Organization ||--o{ AccessGrant : "has (as consumer)" System ||--o{ M2MClient : "has" Endpoint ||--o{ AccessGrant : "grants access to" Organization { string org_id PK string name string kvk_number enum role "provider / consumer / system_provider" } System { string system_id PK string name enum type "single_tenant / multi_tenant" string owner_org_id FK } Endpoint { string endpoint_id PK string url string provider_org_id FK enum status "registered / verified / published / active / suspended / revoked" string[] scopes enum access_model "open / request / restricted" } AccessGrant { string grant_id PK string consumer_org_id FK string endpoint_id FK string[] granted_scopes datetime expires_at enum status "pending / approved / denied / revoked" } M2MClient { string client_id PK string system_id FK string org_id FK string keycloak_client_id string[] scopes } ``` *** ## Endpoint Status Lifecycle [Section titled “Endpoint Status Lifecycle”](#endpoint-status-lifecycle) ``` stateDiagram-v2 [*] --> REGISTERED REGISTERED --> VERIFIED : Challenge-response passed VERIFIED --> PUBLISHED : Provider publishes PUBLISHED --> ACTIVE : First consumer granted access ACTIVE --> SUSPENDED : Temporarily unavailable SUSPENDED --> ACTIVE : Reactivated SUSPENDED --> REVOKED : Permanently removed ACTIVE --> REVOKED : Permanently removed REVOKED --> [*] ``` | Status | Meaning | | ------------ | ---------------------------------------------------------- | | `REGISTERED` | URL submitted, not yet verified | | `VERIFIED` | Ownership proven via challenge-response | | `PUBLISHED` | Visible in service directory, consumers can request access | | `ACTIVE` | At least one consumer has active access | | `SUSPENDED` | Temporarily unavailable (by provider or by CTN) | | `REVOKED` | Permanently removed from the service directory | *** ## Keycloak Configuration Summary [Section titled “Keycloak Configuration Summary”](#keycloak-configuration-summary) ### Realm Setup [Section titled “Realm Setup”](#realm-setup) | Setting | Value | | ----------------------------- | ------------------------------------------- | | Realm name | `ctn` | | Token lifespan (access token) | `3600` seconds (1 hour) | | Refresh token | Disabled for M2M clients | | JWKS endpoint | `/realms/ctn/protocol/openid-connect/certs` | ### Client Scopes (Realm-level) [Section titled “Client Scopes (Realm-level)”](#client-scopes-realm-level) Create the following client scopes in Keycloak at the realm level. These are assigned to M2M clients as access is granted. | Scope Name | Description | | ------------------------------ | --------------------------------------------------- | | `endpoint:{endpoint_id}:read` | Read access to a specific endpoint | | `endpoint:{endpoint_id}:write` | Write access to a specific endpoint (if applicable) | Scopes are created dynamically by the CTN backend when an endpoint is published, and assigned to M2M clients when access is granted. ### Provider Token Validation [Section titled “Provider Token Validation”](#provider-token-validation) Providers validate incoming JWTs by fetching Keycloak’s public keys: ```plaintext GET https://keycloak.ctn.network/realms/ctn/protocol/openid-connect/certs ``` Most HTTP frameworks and API gateways support automatic JWKS-based validation. The provider does not need to call the ASR or Keycloak at request time — token validation is done locally using the cached public key. *** ## Security Considerations [Section titled “Security Considerations”](#security-considerations) * **Request signing**: All CTN outbound requests include `X-CTN-Signature` (HMAC-SHA256) * **Token expiry**: Challenge tokens and access tokens are time-limited * **Single-use tokens**: Each verification attempt generates a fresh token * **Rate limiting**: Maximum 5 verification attempts per endpoint per hour * **HTTPS only**: Endpoints must use TLS — no plain HTTP allowed * **IP allowlisting (optional)**: Providers can restrict the verify endpoint to CTN’s egress IPs * **OAuth 2.0 Client Credentials**: M2M communication uses standard OAuth 2.0 flow via Keycloak * **Scope-based access**: Consumers only get access to specific data operations * **JWT local validation**: Providers validate tokens locally using Keycloak’s public keys — no runtime dependency on ASR # CTN Identifier Enrichment & Verification Architecture *Automated Legal Entity Identifier Discovery and Validation* ## Overview [Section titled “Overview”](#overview) The CTN Association Register implements a comprehensive identifier enrichment system that automatically discovers and validates legal entity identifiers from multiple external registries across European countries. This system reduces manual data entry, improves data quality, and ensures regulatory compliance through automated verification. **Purpose:** Enrich member legal entity records with verified identifiers from authoritative sources (KVK, KBO, Handelsregister, GLEIF, VIES, Peppol). **Scope:** Covers NL (Netherlands), BE (Belgium), DE (Germany), and all EU countries for global identifiers (LEI, EUID, Peppol, VAT). *** ## Table of Contents [Section titled “Table of Contents”](#table-of-contents) 1. [System Overview](#system-overview) 2. [Key Principles](#key-principles) 3. [Building Block View](#building-block-view) 4. [Country-Specific Enrichment Flows](#country-specific-enrichment-flows) 5. [External Registry Services](#external-registry-services) 6. [Data Sources & Rate Limits](#data-sources--rate-limits) 7. [Identifier Type Matrix](#identifier-type-matrix) 8. [Service Architecture](#service-architecture) 9. [API Response Format](#api-response-format) 10. [Status Values & State Machine](#status-values--state-machine) 11. [Quality Requirements](#quality-requirements) 12. [Batch Enrichment of Existing Records (Learnings 2026-06)](#batch-enrichment-of-existing-records-learnings-2026-06) 13. [Risks & Technical Debt](#risks--technical-debt) *** ## System Overview [Section titled “System Overview”](#system-overview) ### Design Goals [Section titled “Design Goals”](#design-goals) 1. **Automated Enrichment** - Minimize manual data entry through automatic identifier discovery 2. **Source Authority** - Validate identifiers against authoritative external registries 3. **Country Extensibility** - Modular architecture supports adding new countries without code duplication 4. **Global Standards** - Support pan-European identifiers (EUID, LEI, Peppol) for all EU member states 5. **Defensive Programming** - Handle API failures gracefully, continue partial enrichment on errors 6. **Rate Limit Compliance** - Respect external API rate limits through throttling and caching ### Key Features [Section titled “Key Features”](#key-features) * **Multi-country support:** NL, BE, DE with extensible architecture for additional countries * **Global identifier discovery:** LEI, EUID, Peppol work across ALL EU countries * **Modular services:** Each enrichment type in separate service file (\~100-200 lines vs 900+ monolithic) * **Smart fallback logic:** Registration number first, company name search as fallback (LEI, Peppol) * **Caching layer:** Store external API responses to reduce repeated calls and costs * **Validation status tracking:** Clear result-based terminology: VALID (confirmed), INVALID (failed), PENDING (not checked), EXPIRED, NOT\_VERIFIABLE *** ## Key Principles [Section titled “Key Principles”](#key-principles) ### 1. Global Identifiers Apply to ALL EU Countries [Section titled “1. Global Identifiers Apply to ALL EU Countries”](#1-global-identifiers-apply-to-all-eu-countries) **EUID, LEI, and Peppol are not country-specific** - they work for all EU member states and should be attempted for every legal entity regardless of country. ```plaintext Example: A French company can have: - LEI from GLEIF (global) - EUID: FR.SIREN.123456789 (generated from SIREN) - Peppol: 0009:12345678901234 (SIRET scheme) - VAT: FR12345678901 (EU-wide) ``` ### 2. Registration Number + Country First, Then Company Name Fallback [Section titled “2. Registration Number + Country First, Then Company Name Fallback”](#2-registration-number--country-first-then-company-name-fallback) For LEI and Peppol searches: 1. **Primary search:** Use national registration number (KVK, KBO, HRB, SIREN) + country code 2. **Fallback search:** If primary fails, search by company name + country code 3. **Name matching logic:** Single result → use it; Multiple results → try exact match → try starts-with → return not found ### 3. Country-Specific Flows for National Identifiers [Section titled “3. Country-Specific Flows for National Identifiers”](#3-country-specific-flows-for-national-identifiers) Each country has unique identifier systems: * **NL:** RSIN → VAT derivation (only for “rechtspersonen”, not “eenmanszaken”) * **DE:** Handelsregister (HRB/HRA) scraping, manual VAT entry required * **BE:** KBO → VAT derivation (simple: BE + KBO number) ### 4. Modular Service Architecture [Section titled “4. Modular Service Architecture”](#4-modular-service-architecture) Each enrichment type is a separate service: * `NlEnrichmentService.java` - RSIN and VAT derivation for Netherlands * `DeEnrichmentService.java` - Handelsregister scraping for Germany * `BeEnrichmentService.java` - KBO lookup and VAT derivation for Belgium * `LeiEnrichmentService.java` - GLEIF LEI lookup (ALL countries) * `PeppolEnrichmentService.java` - Peppol Directory search (ALL countries) * `EuidEnrichmentService.java` - EUID generation (ALL EU countries) * `ViesEnrichmentService.java` - VAT verification against VIES (ALL EU countries) **Benefits:** Testability, readability, separation of concerns, no 900-line God classes. ### 5. Two-Level Identity Model: Legal Entity vs Establishment [Section titled “5. Two-Level Identity Model: Legal Entity vs Establishment”](#5-two-level-identity-model-legal-entity-vs-establishment) *(Added 2026-06 — see [Batch Enrichment of Existing Records](#batch-enrichment-of-existing-records-learnings-2026-06).)* Identifiers live at **two distinct levels** that must not be conflated: * **Legal entity / rechtspersoon** — KVK number, HRB/HRA, KBO, SIREN, UID (CHE). Entity-level identifiers attach here: **LEI, RSIN, VAT, EUID**. One value per company, regardless of how many physical locations it has. * **Establishment / vestiging** — a physical location of that entity with its **own address** but no own LEI/VAT. It carries a **location-level identifier**: vestigingsnummer (NL, 12 digits), **SIRET** (FR, = SIREN + NIC), P-nummer (DK), vestigingseenheidsnummer (BE). Consequences for data modelling and matching: * A single legal entity may correspond to **many rows/locations**; those rows share the same entity identifiers and differ only by establishment ID + address. * **DE and CH branches** are not separately numbered in most cases — a Zweigniederlassung refers back to the parent HRB/UID. There the **geocoded location code (Plus Code / Open Location Code)** is the practical per-location key. * Recommended model: `legal_entity (1) ──< establishment (N)`, with identifiers split by scope. *** ## Building Block View [Section titled “Building Block View”](#building-block-view) ### Level 1: System Context [Section titled “Level 1: System Context”](#level-1-system-context) ``` flowchart TB subgraph ASR["ASR System"] API[ASR API Kubernetes/AKS] DB[(PostgreSQL 15 asr_dev)] end subgraph External["External Registries"] KVK["KVK API Netherlands"] KBO["KBO Public Search Belgium"] HR["Handelsregister Germany"] GLEIF["GLEIF API Global LEI"] VIES["VIES API EU VAT"] PEPPOL["Peppol Directory EU e-Invoicing"] end USER[Admin/Member User] --> API API --> DB API --> KVK API --> KBO API --> HR API --> GLEIF API --> VIES API --> PEPPOL ``` ### Level 2: Enrichment Orchestrator [Section titled “Level 2: Enrichment Orchestrator”](#level-2-enrichment-orchestrator) ``` flowchart TD START([POST /v1/legal-entities/{id}/enrich]) ORCH[Enrichment Orchestrator EnrichmentService.java] subgraph Country["Country-Specific"] NL[NL Enrichment RSIN, VAT derivation] DE[DE Enrichment Handelsregister] BE[BE Enrichment KBO, VAT derivation] end subgraph Global["Global Identifiers"] EUID[EUID Generation All EU countries] LEI[LEI Lookup GLEIF API] PEPPOL[Peppol Lookup Directory API] end DB[(Database Registry Cache)] START --> ORCH ORCH --> NL ORCH --> DE ORCH --> BE ORCH --> EUID ORCH --> LEI ORCH --> PEPPOL NL --> DB DE --> DB BE --> DB EUID --> DB LEI --> DB PEPPOL --> DB ``` ### Level 3: Service Dependencies [Section titled “Level 3: Service Dependencies”](#level-3-service-dependencies) The diagram below shows each country-specific enrichment service with its dedicated API client and registry data table. #### Netherlands (NL) Flow [Section titled “Netherlands (NL) Flow”](#netherlands-nl-flow) ``` flowchart LR NL[nlEnrichmentService] --> KVK[kvkService] --> KVK_DB[(kvk_registry_data)] NL --> VIES[viesService] --> VIES_DB[(vies_registry_data)] NL --> NUM[(legal_entity_number)] ``` #### Belgium (BE) Flow [Section titled “Belgium (BE) Flow”](#belgium-be-flow) ``` flowchart LR BE[beEnrichmentService] --> KBO[kboService] --> BE_DB[(belgium_registry_data)] BE --> VIES[viesService] --> VIES_DB[(vies_registry_data)] BE --> NUM[(legal_entity_number)] ``` #### Germany (DE) Flow [Section titled “Germany (DE) Flow”](#germany-de-flow) ``` flowchart LR DE[deEnrichmentService] --> HR[handelsregisterService] --> BUNDES[bundesApiService] BUNDES --> DE_DB[(german_registry_data)] DE --> LEI[leiService] --> GLEIF_DB[(gleif_registry_data)] DE --> NUM[(legal_entity_number)] ``` #### Cross-Country Services [Section titled “Cross-Country Services”](#cross-country-services) ``` flowchart LR subgraph Global["Global Enrichment Services"] LEI_SVC[leiEnrichmentService] PEPPOL_SVC[peppolEnrichmentService] EUID_SVC[euidEnrichmentService] VIES_SVC[viesEnrichmentService] end subgraph Clients["API Clients"] LEI_CLIENT[leiService] PEPPOL_CLIENT[peppolService] VIES_CLIENT[viesService] end subgraph Storage["Database Tables"] GLEIF_DATA[(gleif_registry_data)] VIES_DATA[(vies_registry_data)] PEPPOL_DATA[(peppol_registry_data)] NUMBERS[(legal_entity_number)] end LEI_SVC --> LEI_CLIENT --> GLEIF_DATA PEPPOL_SVC --> PEPPOL_CLIENT -.->|identifier only| NUMBERS VIES_SVC --> VIES_CLIENT --> VIES_DATA EUID_SVC --> NUMBERS ``` #### Complete Data Flow Summary [Section titled “Complete Data Flow Summary”](#complete-data-flow-summary) | Enrichment Service | API Client(s) | Registry Data Table | | ----------------------- | ---------------------------------------------------- | ---------------------------------------------- | | nlEnrichmentService | kvkService, viesService | kvk\_registry\_data, vies\_registry\_data | | beEnrichmentService | kboService, viesService | belgium\_registry\_data, vies\_registry\_data | | deEnrichmentService | handelsregisterService, bundesApiService, leiService | german\_registry\_data, gleif\_registry\_data | | leiEnrichmentService | leiService | gleif\_registry\_data | | peppolEnrichmentService | peppolService | *(identifier stored in legal\_entity\_number)* | | viesEnrichmentService | viesService | vies\_registry\_data | | euidEnrichmentService | *(local generation)* | *(identifier stored in legal\_entity\_number)* | *** ## Country-Specific Enrichment Flows [Section titled “Country-Specific Enrichment Flows”](#country-specific-enrichment-flows) ### Netherlands (NL) - KVK → RSIN → VAT [Section titled “Netherlands (NL) - KVK → RSIN → VAT”](#netherlands-nl---kvk--rsin--vat) **RSIN Availability:** RSIN (Rechtspersonen en Samenwerkingsverbanden Informatienummer) is only available for legal entities registered as “rechtspersoon” (legal person). **Eenmanszaken (sole proprietorships)** do not have RSIN because they are not separate legal entities—the business and owner are legally the same person. For eenmanszaken, only the KVK number is available, and VAT cannot be auto-derived. ``` flowchart TD START([Start NL Enrichment]) subgraph KVK["KVK → RSIN Flow"] CHECK_KVK{KVK exists?} CHECK_CACHED{Cache exists?} FETCH_KVK[Call KVK API] EXTRACT_RSIN[Extract RSIN from response] STORE_KVK[Store kvk_registry_data] STORE_RSIN[Store RSIN identifier] NO_RSIN["No RSIN eenmanszaak"] end subgraph VAT["RSIN → VAT Flow"] HAS_RSIN{RSIN available?} DERIVE_VAT["Derive: NL + RSIN + B01"] CALL_VIES[Verify via VIES API] VIES_OK{Valid?} TRY_B0X["Try B02, B03, B04..."] B0X_OK{Valid?} STORE_VAT[Store VAT as VALID] VAT_NA[VAT not available] end subgraph EUID["EUID Generation"] GEN_EUID["Generate: NL.KVK.{kvkNumber}"] STORE_EUID[Store as VALID] end START --> CHECK_KVK CHECK_KVK -->|Yes| CHECK_CACHED CHECK_CACHED -->|No| FETCH_KVK CHECK_CACHED -->|Yes| EXTRACT_RSIN FETCH_KVK --> STORE_KVK STORE_KVK --> EXTRACT_RSIN EXTRACT_RSIN -->|Found| STORE_RSIN EXTRACT_RSIN -->|Not found| NO_RSIN STORE_RSIN --> HAS_RSIN NO_RSIN --> HAS_RSIN HAS_RSIN -->|Yes| DERIVE_VAT HAS_RSIN -->|No| VAT_NA DERIVE_VAT --> CALL_VIES CALL_VIES --> VIES_OK VIES_OK -->|Yes| STORE_VAT VIES_OK -->|No| TRY_B0X TRY_B0X --> B0X_OK B0X_OK -->|Yes| STORE_VAT B0X_OK -->|No| VAT_NA CHECK_KVK --> GEN_EUID GEN_EUID --> STORE_EUID ``` **API Endpoint:** `https://api.kvk.nl/api/v1/basisprofielen/{kvkNumber}` **Data Extracted:** * RSIN from `_embedded.eigenaar.rsin` * Company name, address, legal form * Trade names (handelsnamen) **Rate Limits:** Per API key tier (varies by KVK subscription) *** ### Germany (DE) - Handelsregister → HRB/HRA [Section titled “Germany (DE) - Handelsregister → HRB/HRA”](#germany-de---handelsregister--hrbhra) **VAT for German Companies:** VAT numbers MUST be provided manually. The Handelsregister does not contain VAT data, and VIES API can only validate existing VAT numbers—it cannot derive them. This is a fundamental difference from Dutch companies. ``` flowchart TD START([Start DE Enrichment]) subgraph HR["Handelsregister Flow"] CHECK_CACHED{Cache exists?} HAS_HRB{HRB/HRA exists?} SEARCH_HRB[Search by HRB number] SEARCH_NAME[Search by company name] FOUND{Company found?} STORE_DATA[Store german_registry_data] STORE_HRB[Store HRB/HRA identifier] end subgraph EUID["EUID Generation"] HAS_COURT{Court code available?} GEN_EUID["Generate: DE{court}.{type}{number}"] STORE_EUID[Store as VALID] end START --> CHECK_CACHED CHECK_CACHED -->|No| HAS_HRB HAS_HRB -->|Yes| SEARCH_HRB HAS_HRB -->|No| SEARCH_NAME SEARCH_HRB --> FOUND SEARCH_NAME --> FOUND FOUND -->|Yes| STORE_DATA STORE_DATA --> STORE_HRB STORE_HRB --> HAS_COURT CHECK_CACHED -->|Yes| HAS_COURT HAS_COURT -->|Yes| GEN_EUID GEN_EUID --> STORE_EUID ``` **Data Sources:** * **BundesAPI** (handelsregister.de) - Web scraping, 60 requests/hour rate limit * **GLEIF** (fallback) - Only for companies with LEI * **OffeneRegister.de** (bulk open data, added 2026-06) - Full German register (5M+ companies) as JSONL/SQLite, CC-BY, **no rate limit**. Best for batch enrichment of existing lists. Each record’s `company_number` is already in court-code form `K1101R_HRB150148`, so the **DE EUID is derivable directly** (`DEK1101R.HRB150148`) without a separate court-code lookup — solving the “missing court code” gap of the live scraper. Files: `de_companies_ocdata.jsonl.bz2` (248 MB), `handelsregister.db` (3.5 GB SQLite), `openregister.db.gz` (737 MB SQLite). **Court Codes (Examples):** * `D4601R` - Amtsgericht Neuss * `K1101R` - Amtsgericht Hamburg * `M1301R` - Amtsgericht München * `R2210R` - Amtsgericht Koblenz **EUID Format:** `DE{courtCode}.{type}{number}` → `DEK1101R.HRB116737` **Rate Limits:** 60 requests/hour (1 per minute enforced via `minRequestInterval`) *** ### Belgium (BE) - KBO → VAT [Section titled “Belgium (BE) - KBO → VAT”](#belgium-be---kbo--vat) **VAT for Belgian Companies:** Belgian VAT is directly derived from KBO (Kruispuntbank van Ondernemingen) number. Format: `BE` + 10-digit KBO number (without dots). Example: KBO `0439.291.125` → VAT `BE0439291125` **Data Sources:** * **KBO Public Search** (kbopub.economie.fgov.be) - Free web scraping, always available * **KBO API** (api.kbodata.app) - Paid subscription, richer data (contacts, roles, financials) - **Currently DISABLED** ``` flowchart TD START([Start BE Enrichment]) subgraph KBO["KBO Lookup Flow"] HAS_KBO{KBO exists?} CHECK_API{KBO API enabled?} CALL_API[Call KBO API] API_OK{Data found?} SCRAPE[Scrape KBO Public Search] SCRAPE_OK{Company found?} STORE_DATA[Store belgium_registry_data] NO_KBO["Cannot enrich without KBO"] end subgraph VAT["VAT Derivation"] DERIVE["Derive: BE + KBO number"] VERIFY_VIES[Verify via VIES] VIES_OK{Valid?} STORE_VERIFIED[Store as VALID] STORE_DERIVED[Store as VALID] end subgraph EUID["EUID Generation"] GEN_EUID["Generate: BE.KBO.{kboNumber}"] STORE_EUID[Store as VALID] end START --> HAS_KBO HAS_KBO -->|No| NO_KBO HAS_KBO -->|Yes| CHECK_API CHECK_API -->|Yes| CALL_API CHECK_API -->|No| SCRAPE CALL_API --> API_OK API_OK -->|Yes| STORE_DATA API_OK -->|No| SCRAPE SCRAPE --> SCRAPE_OK SCRAPE_OK -->|Yes| STORE_DATA STORE_DATA --> DERIVE DERIVE --> VERIFY_VIES VERIFY_VIES --> VIES_OK VIES_OK -->|Yes| STORE_VERIFIED VIES_OK -->|No| STORE_DERIVED STORE_VERIFIED --> GEN_EUID STORE_DERIVED --> GEN_EUID GEN_EUID --> STORE_EUID ``` **KBO API Status:** DISABLED (requires subscription at ) **To Enable KBO API:** 1. Subscribe to plan (Search, Medium, or Large) 2. Set environment variables: `KBO_API_KEY`, `KBO_API_ENABLED=true` 3. Restart API service **Rate Limits:** * Public scraping: No documented limit (be respectful) * KBO API: Per subscription tier *** ### Global Identifiers (ALL EU Countries) [Section titled “Global Identifiers (ALL EU Countries)”](#global-identifiers-all-eu-countries) #### LEI Lookup (GLEIF API) [Section titled “LEI Lookup (GLEIF API)”](#lei-lookup-gleif-api) **Critical:** GLEIF API uses specific search parameters that MUST be followed. **Primary Search:** Registration Number + Country ```plaintext GET https://api.gleif.org/api/v1/lei-records ?filter[entity.registeredAs]={registrationNumber} &filter[entity.legalAddress.country]={countryCode} ``` **Parameters:** * `filter[entity.registeredAs]` - Just the registration number (e.g., `33031431` for KVK) * `filter[entity.legalAddress.country]` - Two-letter country code (e.g., `NL`, `DE`) **Important Notes:** * **DO NOT** use combined formats like `NL-KVK/12345678` * The API does **NOT** support `filter[registeredAt.id]` (RA code filtering) * Use country code filtering instead (works reliably for all countries, including Germany’s \~100 local court RA codes) * The API **response** contains `registeredAt.id` (RA code like `RA000463` for NL-KVK) - we store this for reference only **Fallback Search:** Company Name + Country ```plaintext GET https://api.gleif.org/api/v1/lei-records ?filter[entity.legalName]={companyName}* &filter[entity.legalAddress.country]={countryCode} &page[size]=20 ``` **Name Matching Logic:** 1. Single result → Use it 2. Multiple results → Try exact match (normalized, alphanumeric only) 3. No exact match → Try starts-with match 4. No match → Return not\_found > **Learning (2026-06):** The `filter[entity.legalName]` parameter performs a near-**exact** match, *not* a prefix/fuzzy search — a trailing `*` does little. For name-only discovery (no registration number, e.g. enriching an existing list) use the **fuzzy-completion endpoint** instead: `GET https://api.gleif.org/api/v1/fuzzycompletions?field=entity.legalName&q={name}`, collect the candidate LEIs, then batch-fetch full records via `filter[lei]=lei1,lei2,...` and rank by name similarity + `legalAddress.country`. Always enforce a **same-country** constraint so a foreign sister/subsidiary LEI is never attached to a branch row. #### Peppol Lookup (Directory API) [Section titled “Peppol Lookup (Directory API)”](#peppol-lookup-directory-api) **Peppol** is a pan-European e-invoicing network. Companies registered in Peppol Directory can send/receive electronic invoices. **Search Strategy:** 1. Search by national registry number (KVK for NL, KBO for BE, etc.) 2. Then search by VAT number 3. Fallback: search by company name (less reliable) ``` flowchart TD START([Start Peppol Enrichment]) CHECK{PEPPOL exists?} HAS_COC{CoC number exists?} SEARCH_COC["Search by CoC: NL: scheme=0106 (KVK) BE: scheme=0208 (KBO) DE: N/A (no scheme)"] COC_FOUND{Found?} HAS_VAT{VAT exists?} SEARCH_VAT["Search by VAT: NL: scheme=9944 BE: scheme=9925 DE: scheme=9930"] VAT_FOUND{Found?} SEARCH_NAME["Search by company name (fallback)"] NAME_FOUND{Found?} STORE[Store PEPPOL identifier] NOT_FOUND["Not in Peppol Directory"] START --> CHECK CHECK -->|No| HAS_COC HAS_COC -->|Yes| SEARCH_COC HAS_COC -->|No| HAS_VAT SEARCH_COC --> COC_FOUND COC_FOUND -->|Yes| STORE COC_FOUND -->|No| HAS_VAT HAS_VAT -->|Yes| SEARCH_VAT HAS_VAT -->|No| SEARCH_NAME SEARCH_VAT --> VAT_FOUND VAT_FOUND -->|Yes| STORE VAT_FOUND -->|No| SEARCH_NAME SEARCH_NAME --> NAME_FOUND NAME_FOUND -->|Yes| STORE NAME_FOUND -->|No| NOT_FOUND ``` **Peppol Identifier Schemes:** | Country | CoC Scheme | VAT Scheme | Example | | ------- | ------------ | ---------- | --------------------- | | **NL** | 0106 (KVK) | 9944 | `0106:12345678` | | **BE** | 0208 (KBO) | 9925 | `0208:0439291125` | | **DE** | - | 9930 | `9930:DE123456789` | | **FR** | 0009 (SIRET) | 9957 | `0009:12345678901234` | | **GB** | 0088 (CRN) | 9932 | `0088:12345678` | | **DK** | 0184 (CVR) | 9902 | `0184:12345678` | **Rate Limit:** 2 requests/second (500ms minimum between requests) > **Batch alternative (2026-06):** for bulk enrichment do **not** loop over the search API — download the **Peppol Directory full export** instead and match offline (no rate limit). See [Peppol via bulk export](#peppol-via-bulk-export-learnings-2026-06) under Batch Learnings. #### EUID Generation [Section titled “EUID Generation”](#euid-generation) **EUID (European Unique Identifier)** is available for ALL EU member states via BRIS system. **Format by Country:** | Country | Format | Example | Source | | ------- | -------------------------- | -------------------- | ---------- | | NL | `NL.KVK.{number}` | `NL.KVK.12345678` | KVK | | BE | `BE.KBO.{number}` | `BE.KBO.0123456789` | KBO/BCE | | DE | `DE{court}.{type}{number}` | `DEK1101R.HRB116737` | HRB/HRA | | FR | `FR.SIREN.{number}` | `FR.SIREN.123456789` | SIREN | | AT | `AT.FB.{number}` | `AT.FB.123456A` | Firmenbuch | | IT | `IT.REA.{number}` | `IT.REA.RM-123456` | REA | | ES | `ES.CIF.{number}` | `ES.CIF.A12345678` | CIF | | DK | `DK.CVR.{number}` | `DK.CVR.12345678` | CVR | **Registry:** #### Ultimate Parent / Corporate Hierarchy (GLEIF Level-2) [Section titled “Ultimate Parent / Corporate Hierarchy (GLEIF Level-2)”](#ultimate-parent--corporate-hierarchy-gleif-level-2) Establishes the **corporate group head** (“global ultimate”) an entity belongs to, so that many legal entities and establishments of one group (e.g. all DSV rows) can be rolled up to a single concern. This is **not** the AML *UBO* (Ultimate Beneficial Owner = the natural person): the NL UBO register is access-restricted, and a listed group such as *DSV A/S* has no single >25% owner. For rolling up locations the corporate parent is the correct key; the UBO is neither available nor needed. **Source:** GLEIF **Level-2 relationship data** (accounting-consolidation parents), free, authoritative. Only available for entities that **have an LEI** and whose parent **reported** the relationship. ```plaintext GET https://api.gleif.org/api/v1/lei-records/{lei}/ultimate-parent # global ultimate GET https://api.gleif.org/api/v1/lei-records/{lei}/direct-parent # immediate parent ``` **Resolution logic (per LEI):** 1. `ultimate-parent` → use it (parent LEI + legal name). 2. else `direct-parent` → use it. 3. else (HTTP 404 on both) → the entity is its **own** group head / non-consolidating → use its own LEI+name. **Notes & limitations:** * A single `ultimate-parent` call does not always reach the absolute top (inconsistent group reporting), e.g. *Containerships – CMA CGM GmbH* instead of *CMA CGM S.A.* Optionally **climb transitively** (resolve the parent’s parent until 404) for a cleaner top. * Coverage equals LEI coverage: in a 1.202-row cross-client test only **55 unique LEIs** existed; 27 of those carried a reported ultimate parent. Rows without an LEI get no GLEIF parent → fall back to a **name/brand canonicalisation** as a full-coverage proxy (see Batch Learnings). * For 100 % hierarchy coverage (incl. non-LEI entities, full chain) a commercial source is required: **D\&B “Global Ultimate” (DUNS), Bureau van Dijk / Orbis,** or **OpenCorporates** (API key). **Example results (cross-client test, 2026-06):** | Brand | GLEIF ultimate parent | Parent LEI | | ---------------- | --------------------------------- | ---------------------- | | DSV | DSV A/S | `529900X41C0BSLK67H70` | | Kühne+Nagel | Kühne + Nagel International AG | — | | DB Schenker | Deutsche Bahn AG | — | | FrieslandCampina | Koninklijke FrieslandCampina N.V. | — | | Maersk | A.P. Møller Holding A/S | — | *** ## External Registry Services [Section titled “External Registry Services”](#external-registry-services) ### Service Overview [Section titled “Service Overview”](#service-overview) ``` flowchart LR subgraph Services["Quarkus Services (Java)"] KVK_SVC[KvkService for NL] KBO_SVC[KboService for BE - scraping] KBO_API_SVC[KboApiService for BE - API] VIES_SVC[ViesService] VIES_ENRICH[ViesEnrichmentService] LEI_SVC[LeiService] HR_SVC[HandelsregisterService for DE] PEPPOL_SVC[PeppolService] EUID_SVC[EuidService] end subgraph APIs["External APIs"] KVK_API["KVK API api.kvk.nl"] KBO_WEB["KBO Public Search kbopub.economie.fgov.be (web scraping - free)"] KBO_API["KBO API api.kbodata.app (paid - DISABLED)"] VIES_API["VIES API ec.europa.eu/taxation_customs/vies"] GLEIF_API["GLEIF API api.gleif.org/api/v1"] PEPPOL_API["Peppol Directory directory.peppol.eu"] HR_WEB["Handelsregister.de (web scraping)"] end subgraph Data["Registry Cache Tables"] KVK_DATA[(kvk_registry_data)] BE_DATA["(belgium_registry_data) PLANNED - not deployed"] VIES_DATA[(vies_registry_data)] GLEIF_DATA[(gleif_registry_data)] DE_DATA[(german_registry_data)] PEPPOL_DATA["(peppol_registry_data) EXISTS - not used"] end KVK_SVC --> KVK_API KBO_SVC --> KBO_WEB KBO_API_SVC -.->|disabled| KBO_API VIES_SVC --> VIES_API VIES_ENRICH --> VIES_API LEI_SVC --> GLEIF_API HR_SVC --> HR_WEB PEPPOL_SVC --> PEPPOL_API KVK_API --> KVK_DATA KBO_WEB -.->|pending| BE_DATA KBO_API -.->|pending| BE_DATA VIES_API --> VIES_DATA GLEIF_API --> GLEIF_DATA HR_WEB --> DE_DATA PEPPOL_API -.->|not stored| PEPPOL_DATA ``` **Notes:** * KBO API (kbodata.app) is disabled pending subscription * `peppol_registry_data` table exists but is not used (only identifier stored in `legal_entity_number`) *** ## Data Sources & Rate Limits [Section titled “Data Sources & Rate Limits”](#data-sources--rate-limits) ### Web Scraping Services [Section titled “Web Scraping Services”](#web-scraping-services) ⚠️ **Important:** These services use web scraping rather than official APIs. They have rate limits to avoid being blocked and may be more fragile than API-based services. #### German Handelsregister (DE) [Section titled “German Handelsregister (DE)”](#german-handelsregister-de) | Service | URL | Method | Rate Limit | | -------------------- | ------------------ | ------------ | ----------------------------------- | | **BundesAPI** | handelsregister.de | Web scraping | **60 requests/hour** (1 per minute) | | **GLEIF** (fallback) | api.gleif.org | REST API | No hard limit (fair use) | **Implementation:** `BundesApiService.java` ```java // Rate limit: 60 queries/hour per Terms of Use private static final Duration MIN_REQUEST_INTERVAL = Duration.ofMinutes(1); ``` #### Belgian KBO Public Search (BE) [Section titled “Belgian KBO Public Search (BE)”](#belgian-kbo-public-search-be) | Service | URL | Method | Rate Limit | | ------------------ | ----------------------- | ------------ | ----------------------------------- | | **KBO Public** | kbopub.economie.fgov.be | Web scraping | No documented limit (be respectful) | | **KBO API** (paid) | api.kbodata.app | REST API | Per subscription tier | **Implementation:** `KboService.java` **Notes:** * Public search requires enterprise number (KBO) - name search not supported * For high-volume production use, paid KBO API is recommended ### Official API Services [Section titled “Official API Services”](#official-api-services) | Service | API | Rate Limit | Auth Required | | -------------------- | ----------------------------------- | ---------------- | ------------- | | **KVK** (NL) | api.kvk.nl | Per API key tier | Yes (API key) | | **GLEIF** | api.gleif.org | Fair use policy | No | | **VIES** | ec.europa.eu/taxation\_customs/vies | Fair use policy | No | | **Peppol Directory** | directory.peppol.eu | 2 req/second | No | **Peppol Rate Limiting:** `PeppolService.java` ```java // Rate limit: 2 queries per second (500ms minimum between requests) private static final long MIN_REQUEST_INTERVAL_MS = 500; ``` *** ## Identifier Type Matrix [Section titled “Identifier Type Matrix”](#identifier-type-matrix) ### Country-Specific Identifiers [Section titled “Country-Specific Identifiers”](#country-specific-identifiers) | Country | Identifier | Source | Derivation Logic | | ------- | ---------- | --------------- | --------------------------------------------------------- | | NL | KVK | Input | Manual entry or application | | NL | RSIN | KVK API | Extracted from `_embedded.eigenaar.rsin` | | NL | VAT | Generated | `NL` + `RSIN` + `B01` (or B02, B03, B04 for fiscal units) | | NL | EUID | Generated | `NL.KVK.{kvkNumber}` | | DE | HRB/HRA | Handelsregister | Scraped or manual entry | | DE | EUID | Generated | `DE{courtCode}.{type}{number}` | | DE | VAT | Manual | Cannot be auto-derived | | BE | KBO/BCE | Input/KBO | Belgian business register (10 digits) | | BE | VAT | Generated | `BE` + KBO number (without dots) | | BE | EUID | Generated | `BE.KBO.{kboNumber}` | ### Global Identifiers (ALL EU Countries) [Section titled “Global Identifiers (ALL EU Countries)”](#global-identifiers-all-eu-countries-1) | Identifier | Source | Lookup Strategy | | ------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------- | | **EUID** | Generated | From national identifier (KVK, KBO, HRB, SIREN, etc.) | | **LEI** | GLEIF | 1. Registration number + country 2. Company name + country fallback | | **Ultimate parent** | GLEIF Level-2 | `lei-records/{lei}/ultimate-parent` → else `direct-parent` → else self (corporate group head, not AML UBO) | | **PEPPOL** | Peppol Directory | 1. CoC/VAT by country scheme 2. Company name + country fallback | ### VAT Derivation Rules [Section titled “VAT Derivation Rules”](#vat-derivation-rules) | Country | Rule | Example | Auto-Derivable? | | --------- | --------------------------------------------- | ---------------- | --------------------------------- | | **NL** | `NL` + RSIN + `B01` (try B02, B03 if invalid) | `NL861785721B01` | ✅ Yes (if RSIN available) | | **BE** | `BE` + KBO number (10 digits, no dots) | `BE0439291125` | ✅ Yes (if KBO available) | | **DE** | Manual entry only | `DE123456789` | ❌ No (Handelsregister has no VAT) | | **Other** | Manual entry | Country-specific | ❌ No (no derivation rule) | ### Supported CoC Types for LEI Lookup [Section titled “Supported CoC Types for LEI Lookup”](#supported-coc-types-for-lei-lookup) | Type | Country | GLEIF Registration Authority | | --------- | ------- | --------------------------------------- | | KVK | NL | RA000463 (Kamer van Koophandel) | | HRB/HRA | DE | RA000197-RA000296 (\~100 local courts) | | KBO/BCE | BE | RA000025 (Kruispuntbank) | | RCS/SIREN | FR | RA000192, RA000189 (Infogreffe, SIRENE) | | CRN | GB | RA000585-RA000587 (Companies House) | | REA | IT | RA000407 (Registro Delle Imprese) | | CIF | ES | RA000533, RA000780 (Registro Mercantil) | | CVR | DK | RA000170 (Central Business Register) | *** ## Service Architecture [Section titled “Service Architecture”](#service-architecture) ### Enrichment Services (Modular Design) [Section titled “Enrichment Services (Modular Design)”](#enrichment-services-modular-design) | Service | File | Purpose | Scope | | --------------------- | -------------------------------------------------------------- | ---------------------------------------- | ---------------- | | **Orchestrator** | `nl.ctn.asr.enrichment.EnrichmentService` | Main coordinator - `enrichLegalEntity()` | ALL | | **EUID Enrichment** | `nl.ctn.asr.enrichment.EuidEnrichmentService` | EUID generation | ALL EU countries | | **NL Enrichment** | `nl.ctn.asr.enrichment.NlEnrichmentService` | RSIN, VAT derivation, KVK registry | NL only | | **DE Enrichment** | `nl.ctn.asr.enrichment.DeEnrichmentService` | HRB/HRA, registry scraping | DE only | | **BE Enrichment** | `nl.ctn.asr.enrichment.BeEnrichmentService` | KBO, VAT derivation | BE only | | **LEI Enrichment** | `nl.ctn.asr.enrichment.LeiEnrichmentService` | GLEIF lookup | ALL countries | | **Peppol Enrichment** | `nl.ctn.asr.enrichment.PeppolEnrichmentService` | Peppol Directory lookup | ALL countries | | **VIES Enrichment** | `nl.ctn.asr.enrichment.ViesEnrichmentService` | Batch VAT verification | ALL EU countries | | **Types** | `nl.ctn.asr.enrichment.EnrichmentContext` / `EnrichmentResult` | Shared records | - | ### External API Services [Section titled “External API Services”](#external-api-services) | Service | File | Purpose | Method | Status | | --------------- | ---------------------------------------------- | --------------------------- | ------------------------ | ----------------------------------- | | KVK | `nl.ctn.asr.enrichment.KvkService` | Dutch CoC lookup | REST API | ✅ Active | | KBO (scraping) | `nl.ctn.asr.enrichment.KboService` | Belgian KBO public search | **Web Scraping** | ✅ Active | | KBO API | `nl.ctn.asr.enrichment.KboApiService` | Belgian KBO official API | REST API | ⏸️ Disabled (requires subscription) | | VIES | `nl.ctn.asr.enrichment.ViesService` | EU VAT validation | SOAP/REST | ✅ Active | | LEI | `nl.ctn.asr.enrichment.LeiService` | GLEIF LEI lookup + storage | REST API | ✅ Active | | Handelsregister | `nl.ctn.asr.enrichment.HandelsregisterService` | German registry coordinator | Mixed | ✅ Active | | BundesAPI | `nl.ctn.asr.enrichment.BundesApiService` | Handelsregister.de scraping | **Web Scraping** (60/hr) | ✅ Active | | Peppol | `nl.ctn.asr.enrichment.PeppolService` | Peppol directory lookup | REST API (2/sec) | ✅ Active | | EUID | `nl.ctn.asr.enrichment.EuidService` | EUID format generation | Local generation | ✅ Active | ### Service Structure [Section titled “Service Structure”](#service-structure) ```plaintext src/main/java/nl/ctn/asr/enrichment/ ├── EnrichmentService.java # Main orchestrator - enrichLegalEntity() ├── EnrichmentContext.java # record (input context) ├── EnrichmentResult.java # record (result) ├── EuidEnrichmentService.java # enrichEuid() - ALL EU ├── NlEnrichmentService.java # enrichRsin(), enrichVat() - NL ├── DeEnrichmentService.java # enrichGermanRegistry() - DE ├── BeEnrichmentService.java # enrichBelgianRegistry() - BE ├── LeiEnrichmentService.java # enrichLei() - ALL via CoC/name ├── PeppolEnrichmentService.java # enrichPeppol() - ALL via CoC/VAT/name ├── ViesEnrichmentService.java # verifyVatAgainstVies() - batch ├── KvkService.java # KVK API client ├── KboService.java # KBO scraping (free) ├── KboApiService.java # KBO API (paid, DISABLED) ├── ViesService.java # VIES API client ├── LeiService.java # GLEIF API + storage ├── HandelsregisterService.java # Handelsregister logic ├── BundesApiService.java # Handelsregister scraping ├── PeppolService.java # Peppol Directory client └── EuidService.java # EUID generation ``` ### Enrichment Context [Section titled “Enrichment Context”](#enrichment-context) ```java public record EnrichmentContext( UUID legalEntityId, // UUID of entity String companyName, // For fallback searches (nullable) String countryCode, // Two-letter code List existingIdentifiers, // Already stored Set existingTypes // Quick lookup ) { } public record EnrichmentResult( String identifier, // e.g., "LEI", "VAT", "EUID" EnrichmentStatus status, // ADDED | EXISTS | ERROR | NOT_AVAILABLE String value, // Identifier value if found (nullable) String message // Human-readable status (nullable) ) { } ``` > The PostgreSQL connection is injected via Quarkus/Panache, not passed in the context. ### Benefits of Modular Architecture [Section titled “Benefits of Modular Architecture”](#benefits-of-modular-architecture) 1. **Separation of concerns** - Each enrichment type in its own service 2. **Testability** - Services can be unit tested independently 3. **Readability** - \~100-200 lines per service vs 900+ lines inline 4. **Global scope** - EUID, LEI, Peppol work for ALL EU countries 5. **Extensibility** - Easy to add new country-specific formats 6. **Maintainability** - Bug fixes isolated to specific services *** ## API Response Format [Section titled “API Response Format”](#api-response-format) ### Enrichment Endpoint [Section titled “Enrichment Endpoint”](#enrichment-endpoint) **POST** `/v1/legal-entities/{id}/enrich` **Response:** ```json { "success": true, "added_count": 3, "company_details_updated": true, "updated_fields": ["primary_legal_name", "city", "address_line1"], "german_registry_fetched": false, "results": [ { "identifier": "RSIN", "status": "added", "value": "123456789" }, { "identifier": "VAT", "status": "added", "value": "NL123456789B01" }, { "identifier": "EUID", "status": "added", "value": "NL.KVK.12345678" }, { "identifier": "LEI", "status": "exists", "value": "724500Y6YT5CU09QUU37" }, { "identifier": "PEPPOL", "status": "not_available", "message": "No Peppol participant found" } ], "summary": { "added": ["RSIN: 123456789", "VAT: NL123456789B01", "EUID: NL.KVK.12345678"], "already_exists": ["LEI"], "not_available": ["PEPPOL (No Peppol participant found)"], "errors": [], "company_fields_updated": ["primary_legal_name", "city", "address_line1"] } } ``` *** ## Status Values & State Machine [Section titled “Status Values & State Machine”](#status-values--state-machine) ### Enrichment Status (API Response) [Section titled “Enrichment Status (API Response)”](#enrichment-status-api-response) When enrichment runs, each identifier returns one of these statuses: | Enrichment Status | Meaning | Maps to Validation Status | | ----------------- | ---------------------------------- | ------------------------- | | `added` | New identifier found and stored | `VALID` | | `exists` | Identifier already in database | (unchanged) | | `not_available` | Identifier cannot be found/derived | (not stored) | | `error` | API or processing error occurred | (not stored) | ### Validation Status (Database) [Section titled “Validation Status (Database)”](#validation-status-database) The `validation_status` field in `legal_entity_number` table: | Status | Meaning | Set When | | ---------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------- | | `VALID` | Confirmed valid by registry API or derivation rules | LEI from GLEIF, VAT from VIES, RSIN from KVK, EUID generated, Belgian VAT derived | | `INVALID` | Registry confirmed identifier does not exist or failed validation | VIES returned invalid, GLEIF not found | | `PENDING` | Not yet checked/verified | Manual entry, awaiting API check | | `EXPIRED` | Was valid but verification has expired | Periodic re-verification needed | | `NOT_VERIFIABLE` | Cannot be verified (no registry API exists) | Identifier types without registry APIs | ### Status Mapping [Section titled “Status Mapping”](#status-mapping) | Source | Enrichment Status | Validation Status | | ------------------------------ | ----------------- | ----------------- | | LEI from GLEIF API | `added` | `VALID` | | VAT from VIES API | `added` | `VALID` | | RSIN from KVK API | `added` | `VALID` | | EUID auto-generated | `added` | `VALID` | | Peppol from Directory | `added` | `VALID` | | Belgian VAT (derived from KBO) | `added` | `VALID` | | KBO from Belgian registry | `added` | `VALID` | | HRB/HRA from German registry | `added` | `VALID` | | Manual entry | N/A | `PENDING` | | VIES validation failed | N/A | `INVALID` | ### State Transition Diagram [Section titled “State Transition Diagram”](#state-transition-diagram) ``` stateDiagram-v2 [*] --> PENDING: Manual entry [*] --> VALID: API confirms / Derivation succeeds PENDING --> VALID: API verification succeeds PENDING --> INVALID: Verification fails VALID --> EXPIRED: Time-based expiry EXPIRED --> VALID: Re-verification succeeds EXPIRED --> INVALID: Re-verification fails INVALID --> VALID: Manual correction + re-verification ``` *** ## Quality Requirements [Section titled “Quality Requirements”](#quality-requirements) ### Performance [Section titled “Performance”](#performance) | Requirement | Target | Measurement | | ------------------------- | -------------------------- | -------------------------------------------- | | **Enrichment Duration** | < 5 seconds per entity | Total API call time | | **Rate Limit Compliance** | 100% adherence | No 429 HTTP errors | | **Cache Hit Rate** | > 80% for repeated queries | Registry cache usage | | **Parallel Processing** | ALL identifiers attempted | No sequential blocking (defensive try-catch) | ### Reliability [Section titled “Reliability”](#reliability) | Requirement | Target | Measurement | | ------------------- | ---------------------------------- | ---------------------------------------- | | **Partial Success** | Continue on individual failures | At least 1 identifier added if available | | **Error Handling** | Graceful degradation | No cascading failures | | **Retry Logic** | 3 retries with exponential backoff | HTTP 5xx errors | | **Cache Staleness** | 90 days | Re-fetch after threshold | ### Data Quality [Section titled “Data Quality”](#data-quality) | Requirement | Target | Measurement | | ------------------------- | ------------------------------ | ------------------------------------------------ | | **Verification Coverage** | > 90% VALID status | % of identifiers confirmed via API or derivation | | **Validation Accuracy** | 100% format compliance | Regex pattern matches | | **Duplicate Prevention** | 0 duplicate identifiers | Unique constraint enforcement | | **Audit Trail** | 100% registry responses stored | Cache table completeness | ### Maintainability [Section titled “Maintainability”](#maintainability) | Requirement | Target | Measurement | | ------------------------- | ----------------------- | ------------------ | | **Service Size** | < 250 lines per service | Lines of code | | **Cyclomatic Complexity** | < 10 per function | Static analysis | | **Test Coverage** | > 80% | Unit tests | | **API Documentation** | Complete OpenAPI spec | Swagger validation | *** ## Batch Enrichment of Existing Records (Learnings 2026-06) [Section titled “Batch Enrichment of Existing Records (Learnings 2026-06)”](#batch-enrichment-of-existing-records-learnings-2026-06) The flows above describe **per-entity, on-demand** enrichment (registration number known, single `POST /enrich`). A second, complementary mode emerged from enriching an **existing list of \~380 customer rows** that had only *name + address + country* and no registration numbers. The learnings below extend the architecture for this **bulk / discovery** mode. ### Discovery is name-first, and registers beat LEI for coverage [Section titled “Discovery is name-first, and registers beat LEI for coverage”](#discovery-is-name-first-and-registers-beat-lei-for-coverage) LEI is **opt-in** and exists only at legal-entity level, so it is a poor primary key: in the test set 127 LEIs mapped to only **35 unique legal entities** — the rest of the rows were establishments of the same companies. **National registers are mandatory** and therefore the right primary source. Strategy: *register-first*, with LEI/EUID/Peppol as supplementary entity identifiers, and a geocoded **Plus Code** as a universal location fallback. ### Bulk / discovery data sources [Section titled “Bulk / discovery data sources”](#bulk--discovery-data-sources) | Country | Identifier obtained | Source | Auth | Notes | | ------- | --------------------------------- | ------------------------------------------ | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | DE | HRB/HRA + Amtsgericht + EUID | **OffeneRegister.de** bulk (offline) | none | 5M+ companies; no rate limit; EUID derivable from `company_number` | | FR | SIREN + **SIRET (establishment)** | **recherche-entreprises.api.gouv.fr** | none | Free; `code_postal` filter resolves the branch SIRET per location | | CH | UID (CHE-…) | **Zefix PublicREST** | **account required** | Now needs free Basic-auth account (request via `zefix@bj.admin.ch`); alternatives: LINDAS SPARQL `register.ld.admin.ch`, opendata.swiss bulk | | NL | **vestigingsnummer** (12) | KVK `/basisprofielen/{kvk}/vestigingen` | API key | Matched to the row address by postcode | | ALL | Plus Code (Open Location Code) | Nominatim/Photon geocode + local OLC | none | Universal per-location identifier when no register ID exists | | ALL | **Peppol participant ID** | **Peppol Directory bulk export** (offline) | none | 4.4M participants; daily refresh; also yields registration numbers (e.g. BE KBO) as by-catch | > **Environment caveats observed:** the KVK **Zoeken** (name-search) endpoint was network-blocked in the batch environment while `basisprofielen`/`vestigingen` worked; the **Google Geocoding** API returned `REQUEST_DENIED` (billing disabled); **Photon** refused connections and **Nominatim** throttles hard above 1 req/s. Plan geocoding as a slow, resumable, single-threaded job. ### Batch name-matching technique (and pitfalls) [Section titled “Batch name-matching technique (and pitfalls)”](#batch-name-matching-technique-and-pitfalls) Matching a source name to a registry name needs to be **direction-aware** and location-aware: 1. **Strip legal forms** (GmbH, AG, SA, B.V., …) and compare on core brand tokens. 2. Require the **registry name to contain all brand tokens** of the source row (the source name is usually the more specific one). A registry entry that is a *subset* of the source (fewer tokens) is the clean parent entity → good; a *superset* with extra qualifiers is likely a different subsidiary → reject or flag. 3. **Two-pass on location-in-name:** first match with the full name (keeps a city that is part of the legal name, e.g. *Rhenus Port Logistics Weil am Rhein GmbH*); only if that fails, strip the row’s city token and retry (for *Name + branch-city* like *Kühne & Nagel Koblenz*). 4. Use **city/postcode only as a tie-breaker**, never to override a weak name match — otherwise a same-city but unrelated company wins. 5. **Enforce same-country**; never attach a foreign entity’s identifier to a branch row. Source-data quality matters: \~12 rows were labelled `DE` but were actually foreign (Rhenus *Singapore/China/Dubai*, *DSV Pfister Basel* = CH, *Schenker Nederland* = NL). The country label should be validated/cleaned before matching. **Transliteration & normalisation.** Match on a normalised form, never mutate source data. Three layers proved necessary in practice: 1. **Diacritic folding** (NFKD → ASCII: é→e, ø→o) — but German has its own convention (ü→ue, ß→ss: *Müller* ≈ *Mueller*); Swiss registers (Zefix/LINDAS) only match the ue-variant, so generate **both** forms. 2. **Exonyms** — *Copenhagen* ≠ *København*, *Gothenburg* ≠ *Göteborg* for an exact matcher; try English and local place names (bit us in UN/LOCODE assignment). 3. **Non-Latin scripts** — Cyrillic/Han/Greek sources need true transliteration: a St. Petersburg address only geocoded with the **Cyrillic** query, and for non-EU entities GLEIF publishes a transliterated ASCII legal name (`transliteratedOtherEntityNames`) that should be used as the match key. For systematic handling use an ICU-based transliterator rather than ad-hoc tables. Also watch concatenation variants: *Friesland Campina* vs *FrieslandCampina* — token-set matching misses these; compare a whitespace-squashed form as well. ### VAT learning: fiscale eenheid [Section titled “VAT learning: fiscale eenheid”](#vat-learning-fiscale-eenheid) For NL, deriving `NL + RSIN + B0x` and validating via VIES can return **INVALID for every B0x suffix** even though the RSIN is correct. This is expected when the entity sits in a **fiscale eenheid (VAT group)** — it has no own VIES-registered VAT. Treat this as a valid outcome (`no own VAT — fiscal unity`), not an error. ### Implementation notes [Section titled “Implementation notes”](#implementation-notes) * Decompressing the 248 MB OffeneRegister `.bz2` (≈5M lines) exceeds a single batch window; splitting with `bzip2recover` into blocks and filtering per batch on target brand tokens is a workable offline approach. The 737 MB / 3.5 GB SQLite variants are simpler to query ad-hoc. * `openpyxl` gotcha: `ws.cell(row, col, value=None)` does **not** clear a cell (the `None` default means “no value passed”); use `ws.cell(row, col).value = None`. ### Result (test set of 378 rows) [Section titled “Result (test set of 378 rows)”](#result-test-set-of-378-rows) | Identifier level | Rows | Source | | -------------------------------------------- | ---: | --------------------------------------------------- | | Legal entity (LEI / HRB / SIREN / KVK / UID) | 220 | GLEIF + national registers | | Establishment (SIRET / vestigingsnummer) | 24 | recherche-entreprises, KVK | | Location only (Plus Code) | 67 | geocoding | | None | 67 | CH (Zefix account pending), non-EU/mislabelled rows | **≈82% of rows received a unique identifier**, up from 33% (LEI-only). Largest remaining gap: CH (needs a Zefix account) and cleaning mislabelled country codes. ### Peppol via bulk export (Learnings 2026-06) [Section titled “Peppol via bulk export (Learnings 2026-06)”](#peppol-via-bulk-export-learnings-2026-06) For batch enrichment the per-query Directory API (2 req/s) is the wrong tool. The Peppol Directory publishes **full exports**, refreshed daily, linked from the footer of `https://directory.peppol.eu/public` (“Export data”): | Export | URL path | Size (2026-06) | Use | | -------------------------------------- | ---------------------------------------- | -------------- | ----------------------------------- | | BusinessCards XML **without doctypes** | `/export/businesscards-xml-no-doc-types` | **1.08 GB** | **preferred for matching** | | BusinessCards CSV (incl. doctypes) | `/export/businesscards-csv` | 4.7 GB | doctype analysis only | | Participant IDs CSV/XML/JSON | `/export/participants-csv` | 178 MB | ID-existence checks only (no names) | **Pipeline:** download → flatten to TSV (`participant ID, country, names, geo, extra elements`) with a streaming regex parser (`ElementTree.iterparse` leaks memory on the root element unless `root.clear()` is called; a chunked regex over the raw bytes is \~3× faster) → **invert the matching**: build small lookup sets from the *customer* rows, then stream the 4.4M Peppol rows once against them. Whole run incl. download ≈ 3 minutes, no rate limits. **Match keys, in order of confidence:** 1. **Hard identifiers** against the participant scheme and the business-card `` elements: KVK→`0106`, KBO→`0208`, CVR→`0184`, SIREN(ET)→`0002/0009/0225`, NO org→`0192`, CHE-UID→`0183/9927` (**bare 9 digits**, no `CHE` prefix, often with suffixes like `ch02`), VAT DE→`9930`, AT→`9914/9915`, NL→`9944`, BE→`9925`, SE→`0007`. Free-text `` schemes (`VAT=CHE-184.676.741 MWST`) need tolerant parsing. 2. **Normalised name + same country** (legal forms stripped) — exact, then prefix match with a length-ratio guard (≥ 0.55) labelled separately as “loose”. **Results (1,202-row cross-client test):** 177 rows matched (93 hard-ID, 66 exact-name, 18 loose). Coverage is bounded by **Peppol adoption, not matching**: the directory is dense for BE (2.07M), FR (0.87M), NO/SE/DK and NL (106k) but thin for **DE (10.5k), AT (477), CH (280)** — zero CH/AT hits despite full CHE-UID coverage on the input side. **By-catch:** a Peppol name-match returns the participant’s registered `` values, so for BE it **backfills the KBO number** (scheme `0208`) that free KBO name-search cannot provide — 20 BE rows gained a KBO this way, from which BE VAT is then derivable (`BE`+KBO → VIES). This is a concrete instance of the iterative-enrichment pattern below. ### Iterative enrichment — identifiers as a graph, not a sequence (proposed) [Section titled “Iterative enrichment — identifiers as a graph, not a sequence (proposed)”](#iterative-enrichment--identifiers-as-a-graph-not-a-sequence-proposed) The current orchestrator runs **one sequential pass** (country service → LEI → Peppol → …). The batch runs showed identifiers are **mutually derivable** — each found identifier unlocks new lookups that may recover identifiers an earlier step missed: ```plaintext LEI ──(GLEIF record: registeredAs + RA code)──▶ national reg.nr ──▶ EUID national reg.nr ──(scheme 0106/0208/…)──▶ Peppol ID Peppol ID ──(business-card elements)──▶ KBO/KVK/VAT (by-catch) KBO ──(BE + number)──▶ VAT ──(VIES)──▶ validated RSIN ──(NL + B0x)──▶ VAT reg.nr ──(GLEIF registeredAs, number instead of name)──▶ LEI (retry with higher precision) HRB+court (OffeneRegister) ──▶ DE EUID ``` **Proposal:** treat enrichment as **fixpoint iteration** over this derivation graph. After each round, re-run every resolver whose *inputs* gained a value; stop when a round adds nothing (typically 2–3 rounds). Concretely valuable paths observed: name→LEI→reg.nr (rows that had no register number), Peppol→KBO→VAT→VIES (BE), reg.nr→LEI retry on `registeredAs` (more precise than the name-based fuzzy search). Guards: every derived value keeps its **provenance chain** (`VAT ← KBO ← Peppol name-match`), confidence is the **minimum** along the chain, and same-country/conflict rules apply at every hop so an early weak name-match cannot launder itself into a “hard” identifier. ### Iteration results & convergence (2026-06) [Section titled “Iteration results & convergence (2026-06)”](#iteration-results--convergence-2026-06) The fixpoint iteration proposed above was executed on the 1,202-row cross-client set: * **Round 1:** Peppol→KBO write-back (12 rows), KBO→VAT→VIES (7/9 unique numbers VALID), reg.nr→LEI retry via `filter[entity.registeredAs]` (+192 LEIs after name-guard), OffeneRegister for remaining DE rows (+30 HRB/HRA+EUID). * **Round 2:** re-ran the LEI retry for the 30 newly found DE register numbers → **0 additions; fixpoint reached.** Every GLEIF hit was an **HRB collision**: German HRB numbers are only unique per Amtsgericht, and the GLEIF API cannot filter on court. `registeredAs` lookups for DE therefore **require a name guard** — without it, *Rhenus* rows would have received the LEI of a meat-products company sharing the same HRB number in another court. * **Name guard tuning:** token-overlap alone rejects legitimate concatenation variants (*Friesland Campina* vs *FrieslandCampina*); add a whitespace-squashed containment check. Guard rejected 55 collisions and passed 211 legitimate matches. Conclusion: 2 rounds suffice on a static source set; further rounds only pay off when a **new source** (or key) enters the graph. Re-clustering the deduplicated organisation list with the newly derived identifiers merged 9 organisation pairs that shared a hard ID (name variants, location aliases), 457 → 448 unique organisations. ### Location codes (UN/LOCODE, SMDG, GLN, OTM) — Learnings 2026-06 [Section titled “Location codes (UN/LOCODE, SMDG, GLN, OTM) — Learnings 2026-06”](#location-codes-unlocode-smdg-gln-otm--learnings-2026-06) Location identity is a separate axis from organisation identity. Implemented stack: | Code | Granularity | Source | Coverage achieved | Notes | | ---------------------- | --------------------------- | ----------------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Plus Code** (OLC) | exact point (\~3×3 m) | computed from geocode | 89% | no registry needed; open equivalent of what3words (which is paid/proprietary) | | **UN/LOCODE** | city/place | UNECE list (\~116k places; machine-readable mirror `github.com/datasets/un-locode`) | 95% | matched on country + normalised place name; beware exonyms (*Copenhagen/København*) and postcode fragments in city fields | | **SMDG terminal code** | terminal within a UN/LOCODE | `github.com/smdg-org/Terminal-Code-List` (CSV, \~1,250 terminals) | 27 rows (terminals only) | used in DCSA standards; match on UN/LOCODE + **quay number** (e.g. *Kaai 869* → `BEANR:K869`) first, then brand tokens + distance; **exclude city/country tokens** or offices match terminals | | **GLN** (GS1) | org/location | Peppol scheme `0088` by-catch only | 4 rows | **no open GS1 registry exists** (Verified-by-GS1 needs an account); treat GLN as input data, not derivable | | **OTM location** | data model | generated export | 1,202 rows | OTM ≥5 `location` entities with `geoReference` + all identifiers in `externalAttributes`; the bridge to OTM-based exchange | Matching pitfall worth recording: when matching rows to SMDG terminals, generic tokens (*sea*, *france*, *china*, city names) caused offices to match terminals; the working rule is quay-number first, then brand-token ∩ (facility ∪ company) minus geographic tokens, with a distance guard (<5 km same-name, <400 m for geo-only matches). ### Roll-up to corporate parent (concern) — GLEIF Level-2 [Section titled “Roll-up to corporate parent (concern) — GLEIF Level-2”](#roll-up-to-corporate-parent-concern--gleif-level-2) When consolidating multiple source lists (a 1.202-row ITG × Van Berkel × Contargo cross-client test), the useful question is *“which concern does this row belong to?”* — e.g. rolling 232 DSV rows under **DSV A/S**. Two levels were combined: 1. **Name/brand canonicalisation** (full coverage proxy): scan the *whole* name for known operator brands so customer/operator constructions resolve to the operator — *XEROX C/O DSV* → DSV, *ACER … Kuehne Nagel* → Kühne+Nagel. The terminal’s customer is the **logistics operator**, not its end-client. This groups every row, including those without any identifier. 2. **GLEIF ultimate parent** (verified, partial): for rows with an LEI, attach the `ultimate-parent`/`direct-parent` (see *Ultimate Parent* under Global Identifiers). This both confirms the concern (DSV → DSV A/S) and **disambiguates name-twins** — *DSV & Partners* / *DSV Finances* returned no DSV-group parent and were correctly excluded from the DSV concern. Dedup keying for the legal-entity layer used **multiple identifiers** (LEI → register-nr → EUID → VAT, since coverage varies), with name+country only as a bridge and a **conflict guard** (never merge two clusters that hold different LEIs or register numbers). Establishment-level keys (vestigingsnummer, SIRET) and Plus Code were deliberately **excluded** from organisation dedup — they are location-, not entity-level. *** ## Risks & Technical Debt [Section titled “Risks & Technical Debt”](#risks--technical-debt) ### Current Risks [Section titled “Current Risks”](#current-risks) | Risk | Impact | Probability | Mitigation | | -------------------------- | ------ | ----------- | --------------------------------------------------------------------------------------------------------- | | **Web Scraping Fragility** | HIGH | MEDIUM | HTML structure changes break scrapers → Implement schema validation, version detection, fallback to GLEIF | | **Rate Limit Violations** | MEDIUM | LOW | BundesAPI 60/hr limit → Enforce `minRequestInterval`, implement queue, cache aggressively | | **Peppol Data Loss** | LOW | LOW | `peppol_registry_data` table not used → Store full API response for audit trail | | **GLEIF API Changes** | MEDIUM | LOW | Search parameter changes → Monitor API changelog, implement version detection | | **KBO API Dependency** | LOW | HIGH | Disabled due to subscription cost → Public scraping works but limited data | ### Technical Debt [Section titled “Technical Debt”](#technical-debt) | Debt Item | Severity | Effort | Description | | ------------------------------- | -------- | ------- | ----------------------------------------------------------------------------------------- | | **KBO API Integration** | MEDIUM | 4 hours | Service implemented but disabled - requires subscription decision and configuration | | **Peppol Response Storage** | LOW | 2 hours | Store full Peppol API response in `peppol_registry_data` for compliance audit trail | | **Retry Logic Standardization** | MEDIUM | 4 hours | Implement consistent retry logic across all external API clients (currently inconsistent) | | **Circuit Breaker Pattern** | LOW | 6 hours | Add circuit breakers for failing external APIs to prevent cascade failures | | **Enrichment Scheduling** | LOW | 8 hours | Implement periodic re-enrichment for entities with EXPIRED verification status | ### Future Enhancements [Section titled “Future Enhancements”](#future-enhancements) 1. **Expand Country Coverage** - Add enrichment services for FR (SIREN/SIRET), IT (REA), ES (CIF), UK (Companies House) 2. **Real-time Webhooks** - Subscribe to KVK/KBO change notifications instead of periodic polling 3. **ML-based Name Matching** - Improve company name matching with fuzzy search and ML models 4. **Blockchain Identity** - Integrate with W3C Decentralized Identifiers (DIDs) for verified credentials 5. **API Gateway** - Centralize external API calls through Azure API Management for rate limiting, caching, monitoring *** ## Related Documentation [Section titled “Related Documentation”](#related-documentation) * [CTN Member Onboarding Workflow](./ctn-member-onboarding-workflow.md) - Business vetting and verification process * ASR Database Schema — see `database/asr_dev.sql` in the implementation repository (out of scope for arc42). *** ## Appendix: Decision Trees [Section titled “Appendix: Decision Trees”](#appendix-decision-trees) ### When is RSIN applicable? [Section titled “When is RSIN applicable?”](#when-is-rsin-applicable) ``` flowchart TD Q1{Country = NL?} Q1 -->|Yes| Q2{KVK number exists?} Q1 -->|No| NOT_APPLICABLE["RSIN not applicable (Dutch-specific identifier)"] Q2 -->|Yes| FETCH[Fetch RSIN from KVK API] Q2 -->|No| NOT_POSSIBLE["Cannot derive RSIN without KVK number"] FETCH --> RESULT[Store RSIN identifier] ``` ### When can VAT be auto-derived? [Section titled “When can VAT be auto-derived?”](#when-can-vat-be-auto-derived) ``` flowchart TD Q1{Country?} Q1 -->|NL| Q2{RSIN available?} Q1 -->|BE| Q3{KBO available?} Q1 -->|DE| MANUAL_DE["VAT must be manual (Handelsregister has no VAT)"] Q1 -->|Other| MANUAL_OTHER["VAT must be manual (no derivation rule)"] Q2 -->|Yes| DERIVE_NL["Derive: NL + RSIN + B01"] Q2 -->|No| NO_RSIN["Cannot derive VAT without RSIN"] DERIVE_NL --> VALIDATE_NL[Validate via VIES] VALIDATE_NL -->|Valid| STORE_NL[Store as VALID] VALIDATE_NL -->|Invalid| TRY_B02["Try B02, B03..."] Q3 -->|Yes| DERIVE_BE["Derive: BE + KBO"] Q3 -->|No| NO_KBO["Cannot derive VAT without KBO"] DERIVE_BE --> VALIDATE_BE[Verify via VIES] VALIDATE_BE -->|Valid| STORE_BE_V[Store as VALID] VALIDATE_BE -->|Invalid| STORE_BE_D[Store as VALID] ``` ### When can EUID be generated? [Section titled “When can EUID be generated?”](#when-can-euid-be-generated) ``` flowchart TD Q1{Country = NL?} Q1 -->|Yes| Q2{KVK exists?} Q2 -->|Yes| GEN_NL["Generate: NL.KVK.kvkNumber"] Q2 -->|No| NO_NL["Cannot generate"] Q1 -->|No| Q3{Country = DE?} Q3 -->|Yes| Q4{HRB/HRA exists?} Q4 -->|Yes| Q5{Court code available?} Q5 -->|Yes| GEN_DE["Generate: DEcourt.typenumber"] Q5 -->|No| NO_CODE["Cannot generate (missing court code)"] Q4 -->|No| NO_HRB["Cannot generate (missing register number)"] Q3 -->|No| OTHER["See country-specific EUID format rules"] ``` *** **Document Version:** 1.2 **Last Updated:** 2026-06-10 **Status:** Published **Maintainer:** Ramon de Noronha **Changelog:** * *1.2 (2026-06-10)* — Peppol bulk-export enrichment (full-directory offline matching, scheme mapping, adoption-bounded coverage, KBO by-catch for BE); proposed iterative/fixpoint enrichment over the identifier derivation graph; transliteration & normalisation guidance (diacritic folding incl. German ue/ss convention, exonyms, non-Latin scripts, GLEIF transliterated names, concatenation variants); iteration executed & converged in 2 rounds (HRB-collision learning: GLEIF `registeredAs` for DE requires a name guard); location-code stack (UN/LOCODE, SMDG terminal codes incl. quay-number matching, GLN limitation, OTM location export). * *1.1 (2026-06-04)* — Added two-level identity model (entity vs establishment); batch/discovery enrichment learnings (OffeneRegister bulk, recherche-entreprises FR SIRET, Zefix account requirement, KVK vestigingen, Plus Codes); GLEIF fuzzy-completion endpoint for name discovery; fiscale-eenheid VAT case; name-matching technique and environment caveats. * *1.0 (2025-12-17)* — Initial published architecture. # Keycloak M2M Authentication — Architecture Documentation ## 1. Introduction and Goals [Section titled “1. Introduction and Goals”](#1-introduction-and-goals) ### Business Context [Section titled “Business Context”](#business-context) The Keycloak M2M (Machine-to-Machine) Authentication component provides secure, self-hosted authentication for external organisations and systems to access CTN Association Register APIs without an interactive user login. It lets terminal operators, carriers, logistics portals, and IoT devices authenticate programmatically using the **OAuth 2.0 client-credentials grant**. > **IdP choice.** Keycloak is the chosen IdP for the entire CTN stack (see [ADR-00002 — Preferred stack](../09-decisions/00002-define-a-preferred-stack.md) and [ADR-00008 — External IdP federation strategy](../09-decisions/00008-external-idp-federation-strategy.md)). The same Keycloak instance handles user authentication (with outward federation to Entra ID, eHerkenning or local accounts) and M2M client-credentials flows. ### Primary Goals [Section titled “Primary Goals”](#primary-goals) * **Self-hosted independence** — full control over identity infrastructure; no dependency on US-based managed identity providers. * **Standard M2M authentication** — OAuth 2.0 client-credentials, no custom flows. * **Multi-organisation support** — one Keycloak realm serves all CTN participants; per-participant clients via Keycloak Organizations. * **Fine-grained API authorisation** — scope- and role-based access control via Keycloak Authorization Services. * **Audit compliance** — complete audit trail of M2M authentication and API access. ### Quality Goals [Section titled “Quality Goals”](#quality-goals) * **Security**: RS256 JWT signature validation, token expiration enforcement, JWKS-based key management, HSM-backed signing keys. * **Performance**: <200ms token issuance, <50ms token validation (with JWKS caching at the consumer). * **Reliability**: 99.95% uptime for M2M authentication. * **Independence**: self-hosted, EU-resident. * **Scalability**: hundreds of concurrent M2M clients. ### Stakeholders [Section titled “Stakeholders”](#stakeholders) * **Terminal operators** — accessing container tracking and ETA APIs. * **Carriers** — updating transport status and arrival information. * **External portals** — integrating CTN data into third-party platforms. * **System administrators** — managing M2M client credentials and access control. ## 2. Architecture Constraints [Section titled “2. Architecture Constraints”](#2-architecture-constraints) ### Technical [Section titled “Technical”](#technical) * **OAuth 2.0 client-credentials** — standard grant flow. * **JWT format** — RS256-signed JSON Web Tokens (JWS). * **JWKS discovery** — public-key retrieval for distributed token validation. * **Container deployment** — Keycloak runs as an OCI image with a PostgreSQL backend. ### Regulatory [Section titled “Regulatory”](#regulatory) * **Data sovereignty** — self-hosted infrastructure for full data control. * **EU residency** — Keycloak instance hosted in an EU region; no cross-border secret material. * **GDPR** — audit logging and data-protection controls. ### Integration [Section titled “Integration”](#integration) * **Quarkus compatibility** — every CTN service uses `quarkus-oidc` as resource server against this Keycloak. * **Coexistence with user authentication** — the same Keycloak realm hosts both human users (federated per ADR-00008) and M2M clients; flows do not interfere. * **Per-participant mapping** — M2M clients are scoped to a Keycloak Organization, tying them to a specific participant in the ASR. ## 3. Solution Strategy [Section titled “3. Solution Strategy”](#3-solution-strategy) ### Keycloak realm structure [Section titled “Keycloak realm structure”](#keycloak-realm-structure) ```plaintext Realm: ctn ├── Organizations (per participant — ADR-00003 / 00008) │ ├── acme-bv │ │ ├── users (federated from Acme's Entra, or local accounts) │ │ └── clients │ │ ├── acme-tms-m2m (M2M service account) │ │ └── acme-erp-m2m (M2M service account) │ ├── contargo │ │ └── ... │ └── ... ├── Client scopes │ ├── api.access (basic API access) │ ├── endpoints.read │ ├── endpoints.register │ └── containers.read └── Roles ├── participant-admin └── participant-member ``` ### Client-credentials flow [Section titled “Client-credentials flow”](#client-credentials-flow) ```plaintext 1. M2M client → Keycloak: POST /realms/ctn/protocol/openid-connect/token grant_type=client_credentials Basic Auth: client_id : client_secret (or client_assertion JWT for higher assurance) 2. Keycloak → M2M client: access_token (JWT, RS256, ~1h TTL) 3. M2M client → CTN API: GET /v1/ Authorization: Bearer 4. CTN API (Quarkus + quarkus-oidc): - Fetch JWKS from Keycloak (cached ~10 min) - Validate signature (RS256) - Validate iss, aud, exp, nbf - Read scopes / roles from token 5. CTN API → ASR DB: resolve participant_id from token claims (the client's Keycloak Organization → ASR participant link, per ADR-00003) 6. CTN API: enforce participant-scoped authorisation, serve the request ``` ## 4. Building Block View [Section titled “4. Building Block View”](#4-building-block-view) ### Core components [Section titled “Core components”](#core-components) * **Keycloak realm `ctn`** — single realm hosting users, clients and organisations. * **Service accounts** — one per M2M client, scoped to a participant via the Organization. * **Authorization Services** — fine-grained permissions per resource / scope. * **JWKS endpoint** — `https:///realms/ctn/protocol/openid-connect/certs`. * **Admin REST API** — for ASR-fronted client lifecycle (create / rotate / revoke). ### Supporting components [Section titled “Supporting components”](#supporting-components) * **Quarkus `quarkus-oidc` extension** — resource-server validation in every CTN service. * **Audit pipeline** — Keycloak admin and login events shipped to the central log platform. * **Secret store** — HSM-backed storage for Keycloak’s signing keys (HMAC + RSA). ## 5. Runtime View [Section titled “5. Runtime View”](#5-runtime-view) ### Token issuance [Section titled “Token issuance”](#token-issuance) ``` sequenceDiagram participant Client as M2M Client participant KC as Keycloak participant API as CTN API (Quarkus) Client->>KC: POST /token (client_credentials, client_id+secret) KC->>KC: Authenticate client KC->>KC: Build access token (scopes, organisation claim) KC-->>Client: access_token (JWT, RS256) ``` ### Resource access [Section titled “Resource access”](#resource-access) ``` sequenceDiagram participant Client as M2M Client participant API as CTN API (Quarkus) participant KC as Keycloak (JWKS) participant DB as ASR DB Client->>API: GET /v1/ (Bearer JWT) API->>KC: GET /protocol/openid-connect/certs (cached) KC-->>API: JWKS API->>API: Validate signature, iss, aud, exp API->>API: Read scopes + organisation claim API->>DB: Resolve participant_id → check policy DB-->>API: Allow / Deny alt Allowed API-->>Client: 200 OK + data else Denied API-->>Client: 403 Forbidden end ``` ### Access-token structure (example) [Section titled “Access-token structure (example)”](#access-token-structure-example) ```json { "iss": "https://keycloak.ctn.network/realms/ctn", "sub": "service-account-acme-tms-m2m", "aud": ["ctn-api"], "exp": 1730906400, "iat": 1730902800, "azp": "acme-tms-m2m", "client_id": "acme-tms-m2m", "scope": "openid api.access containers.read", "organization": { "acme-bv": { "id": "11111111-2222-3333-4444-555555555555" } }, "realm_access": { "roles": ["participant-member"] } } ``` The `organization` claim’s exact JSON shape (alias-as-key vs. predictable schema) is the subject of [ADR-00006 — Token Claim Composition](../09-decisions/00006-token-claim-composition.md). Until that ADR closes, consumers should treat the field as opaque and resolve to a participant ID via the ASR’s `/participant/context` endpoint when needed. ## 6. Deployment View [Section titled “6. Deployment View”](#6-deployment-view) ### Reference deployment (Azure) [Section titled “Reference deployment (Azure)”](#reference-deployment-azure) * **Compute**: Azure Kubernetes Service (AKS) running the `quay.io/keycloak/keycloak:` image. * **Database**: Azure Database for PostgreSQL — Flexible Server. * **Secret store**: Azure Key Vault (Managed HSM tier for signing keys). * **Ingress**: Azure Front Door with WAF. ### Cloud-agnostic alternatives [Section titled “Cloud-agnostic alternatives”](#cloud-agnostic-alternatives) | Component | Azure (reference) | AWS | Self-hosted | | ------------ | -------------------------- | --------------------- | ------------------ | | Compute | AKS (Kubernetes) | EKS | Kubernetes / Nomad | | Database | PostgreSQL Flexible Server | RDS for PostgreSQL | Patroni / Crunchy | | Secret store | Key Vault Managed HSM | KMS + Secrets Manager | HashiCorp Vault | | Ingress | Front Door | CloudFront + ALB | Traefik / NGINX | ## 7. Security Considerations [Section titled “7. Security Considerations”](#7-security-considerations) * **Client secrets**: minimum 256-bit entropy; rotated on a scheduled cadence (90 days target). Prefer `client_assertion` (JWT-bearer) over static secrets where the M2M client can hold a private key. * **Scopes**: every M2M client is granted the **minimum** scope set needed for its data products. Default-deny; explicit grants per participant. * **Token lifetime**: access tokens 1 hour; no refresh tokens for client-credentials clients (per OAuth 2.0). * **Audit logging**: every token issuance and every admin action emits a Keycloak event; events ship to the central log platform with a `participant_id` correlation. * **JWKS hardening**: signing keys live in the HSM-backed key vault; rotation through Keycloak’s key-management UI. ## 8. Operations [Section titled “8. Operations”](#8-operations) ### Provisioning a new M2M client (ASR-fronted) [Section titled “Provisioning a new M2M client (ASR-fronted)”](#provisioning-a-new-m2m-client-asr-fronted) 1. Participant admin calls the ASR endpoint (`POST /v1/participants/{id}/m2m-clients`) — see [ADR-00007](../09-decisions/00007-identity-mutation-proxy.md). 2. ASR validates the request (participant exists, requester is admin) and calls Keycloak Admin REST API with its service-account credentials. 3. Keycloak creates the client under the participant’s Organization with the requested scopes. 4. Keycloak returns the freshly-generated client secret. 5. ASR returns the secret to the caller **once** (it is not retrievable afterwards). 6. The participant admin distributes the secret to the M2M system through their own secure channel. ### Rotating a client secret [Section titled “Rotating a client secret”](#rotating-a-client-secret) ```plaintext POST /v1/participants/{id}/m2m-clients/{clientId}/rotate-secret ``` Returns a new secret, valid immediately; the previous secret is invalidated after a grace period (default 24h) to allow rolling deploys. ### Revoking a client [Section titled “Revoking a client”](#revoking-a-client) ```plaintext DELETE /v1/participants/{id}/m2m-clients/{clientId} ``` The client is disabled in Keycloak; outstanding tokens remain valid until expiry (max 1h). Token revocation can be forced via the Keycloak admin event `LOGOUT_ALL`, but that affects every session. ### Error contract [Section titled “Error contract”](#error-contract) M2M error responses follow RFC 9457 (Problem Details), consistent with the [coding standards](../08-crosscutting/ctn-coding-standards.md). The three authentication/authorisation failures a client must handle: **401 — authentication failed.** Returned with a `WWW-Authenticate: Bearer` header per OAuth 2.0. Causes: expired token, invalid RS256 signature, issuer mismatch (`iss`), or a malformed `Authorization` header. Action: request a new access token; if it persists, verify the configured issuer matches the token. ```json { "type": "https://schemas.ctn.network/problems/unauthorized", "title": "Unauthorized", "status": 401, "detail": "Invalid or expired token" } ``` **403 — insufficient scope.** The token is valid but lacks a required scope or role. Scope names are case-sensitive and must match exactly. Action: grant the missing scope to the client in Keycloak. ```json { "type": "https://schemas.ctn.network/problems/insufficient-scope", "title": "Insufficient scope", "status": 403, "detail": "Missing required scope: containers.read", "required_scopes": ["containers.read"], "missing_scopes": ["containers.read"] } ``` **403 — client not linked to a participant.** Authentication succeeded but the client’s Keycloak Organization is not linked to an active ASR participant, so no participant context can be resolved (see [ADR-00003](../09-decisions/00003-separate-users-and-participants.md)). A `participant_id` correlation is logged. Action: link the Organization to a participant in the ASR. ```json { "type": "https://schemas.ctn.network/problems/participant-not-mapped", "title": "Participant not mapped", "status": 403, "detail": "No active ASR participant is linked to this client's Keycloak Organization" } ``` ### Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) | Symptom | Likely cause | Check | | ------------------------- | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | 401 “issuer mismatch” | Configured OIDC issuer ≠ token `iss` | Confirm `quarkus.oidc.auth-server-url` resolves to `https:///realms/ctn`; no trailing slash. | | 401 “JWKS fetch failed” | API cannot reach Keycloak certs endpoint | Network/egress to `/realms/ctn/protocol/openid-connect/certs`; no firewall block on outbound HTTPS. | | 403 after successful auth | Client’s Organization not linked to a participant | Resolve the Organization claim → ASR participant; verify the participant is active. | | 403 “missing scope” | Scope/role not assigned, or case mismatch | Token `scope` / `realm_access.roles` contains the exact (case-sensitive) value; re-issue token after granting. | ## 9. Quality Requirements [Section titled “9. Quality Requirements”](#9-quality-requirements) | Quality | Target | | ------------------------ | ----------------------------------------------- | | Token issuance latency | <200ms (P95) | | Token validation latency | <50ms (P95) with JWKS cached | | Availability | 99.95% | | Audit completeness | 100% of token issuance and admin actions logged | | Secret rotation cadence | ≤90 days | ## 10. Risks and Technical Debt [Section titled “10. Risks and Technical Debt”](#10-risks-and-technical-debt) | Risk | Mitigation | | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Compromise of the Keycloak admin service account → full realm takeover | Least-privilege service-account role; secrets in HSM-backed store; quarterly access review. | | Stolen client secret used elsewhere | Short token TTL; per-client audit; offer `client_assertion` (JWT-bearer) for high-value M2M flows. | | JWKS cache TTL too long → revoked keys still accepted | Standard JWKS cache TTL of 10 minutes at consumers; emergency JWKS push via cache invalidation event. | | Keycloak Admin API list calls are capped (≈100 organisations, 100 members, 1000 admin events per call) | Add pagination before scaling past these thresholds; log a truncation warning when a cap is hit. | | Provisioning is best-effort compensation, not a durable saga (no idempotency keys, no retry/backoff on Admin API calls, no drift reconciliation) | Acceptable at current volume; track as TD — add idempotency keys, a retry/backoff wrapper, and a periodic Keycloak↔ASR reconciliation job when volume grows. | ## 11. Related Documents [Section titled “11. Related Documents”](#11-related-documents) * [ADR-00002 — Define a preferred stack](../09-decisions/00002-define-a-preferred-stack.md) * [ADR-00003 — Separate users and participants](../09-decisions/00003-separate-users-and-participants.md) * [ADR-00006 — Token Claim Composition](../09-decisions/00006-token-claim-composition.md) * [ADR-00007 — Identity Mutation Proxy](../09-decisions/00007-identity-mutation-proxy.md) * [ADR-00008 — External IdP Federation Strategy](../09-decisions/00008-external-idp-federation-strategy.md) * [Centralised Authentication](ctn-centralized-auth.md) * [ASR Data Model](ctn-asr-data-model.md) — the ASR-side `m2m_client_ref` record links to the Keycloak client by ID; the secret stays in Keycloak. * Keycloak documentation: # CTN Member Onboarding Workflow *Vetting, Verification, and Credential Issuance Process* *Last Updated: 2025-11-18* ## Overview [Section titled “Overview”](#overview) This document describes the complete onboarding process for new participants joining the CTN Association Register. The process ensures that all participants are properly vetted, verified, and issued appropriate credentials before gaining access to the network. Within the container transport ecosystem, proper vetting ensures that all parties who are approved participants of the CTN association can be trusted to: * Meet regulatory compliance requirements * Maintain secure data sharing practices * Participate effectively in CTN data sharing use cases *** ## Onboarding Process Summary [Section titled “Onboarding Process Summary”](#onboarding-process-summary) ``` graph LR A[Application] --> B[Vetting] B --> C[Verification] C --> D[Approval] D --> E[Credential Issuance] E --> F[Activation] F --> G[Ongoing Monitoring] ``` *** ## Phase 1: Business Vetting [Section titled “Phase 1: Business Vetting”](#phase-1-business-vetting) ### 1.1 Legal and Regulatory Verification [Section titled “1.1 Legal and Regulatory Verification”](#11-legal-and-regulatory-verification) * **Company Registration**: Confirm legal entity status and business registration * **Business Licenses**: Verify required permits and licenses for their industry * **Regulatory Compliance**: Check adherence to local, national, and international regulations * **Ownership Structure**: Review ultimate beneficial owners (UBOs) and corporate structure * **Sanctions Screening**: Screen against sanctions and restricted party lists ### 1.2 Compliance and Risk Screening [Section titled “1.2 Compliance and Risk Screening”](#12-compliance-and-risk-screening) * **Anti-Money Laundering (AML)**: Perform comprehensive AML checks * **Politically Exposed Persons (PEP)**: Check for PEP status among owners/management * **Adverse Media**: Conduct reputational screening for negative coverage * **Litigation History**: Review any history of legal disputes or proceedings ### 1.3 Capability and Capacity Evaluation [Section titled “1.3 Capability and Capacity Evaluation”](#13-capability-and-capacity-evaluation) * **Operational Capacity**: Assess facilities, equipment, and operational capability * **Certifications**: Review relevant certifications (ISO standards, industry accreditations) * **Quality Control**: Evaluate quality management processes and safety records * **Technology Integration**: Assess IT capabilities and readiness ### 1.4 Past Performance and References [Section titled “1.4 Past Performance and References”](#14-past-performance-and-references) * **Customer References**: Gather testimonials from existing or past clients * **Contract History**: Review performance on previous contracts or projects * **Dispute Resolution**: Investigate history of contract disputes or performance issues * **Industry Reputation**: Assess standing within the logistics/transport industry ### 1.5 Contractual and Ethical Standards [Section titled “1.5 Contractual and Ethical Standards”](#15-contractual-and-ethical-standards) * **Terms Acceptance**: Confirm acceptance of company terms and conditions * **Ethical Standards**: Review adherence to ethical sourcing and corporate responsibility * **Sustainability Practices**: Evaluate environmental and sustainability commitments * **Data Protection**: Assess data protection and cybersecurity practices ### 1.6 Financial Assessment [Section titled “1.6 Financial Assessment”](#16-financial-assessment) > **Note**: Financial assessment is not the responsibility of the CTN association. Each participant should conduct their own financial due diligence. * Credit reporting and financial statements * Payment history and outstanding debts * Financial stability and liquidity * Insurance coverage for transport/logistics operations *** ## Phase 2: Initial Application [Section titled “Phase 2: Initial Application”](#phase-2-initial-application) ### 2.1 Application Submission [Section titled “2.1 Application Submission”](#21-application-submission) 1. **Prospective Participant Applies** * Submits application through member portal * Provides basic company information 2. **Document Collection** * Chamber of Commerce Statement * Identification documents (eHerkenning for NL) ### 2.2 Document Requirements [Section titled “2.2 Document Requirements”](#22-document-requirements) #### Required Documents [Section titled “Required Documents”](#required-documents) 1. **Legal Entity Documents** * Chamber of Commerce statement (< 3 months old) * UBO statement 2. **Identification Documents** * Valid ID for authorized signatories * Proof of authorization * eHerkenning in NL 3. **Business Documents** * VAT registration * Insurance certificates * Compliance certifications #### Document Validation Criteria [Section titled “Document Validation Criteria”](#document-validation-criteria) * **Authenticity**: Documents must be original or certified copies * **Currency**: Documents must be recent (typically < 3 months) * **Completeness**: All required fields must be present * **Consistency**: Information must match across documents *** ## Phase 3: Verification [Section titled “Phase 3: Verification”](#phase-3-verification) ### 3.1 Automated Document Processing [Section titled “3.1 Automated Document Processing”](#31-automated-document-processing) * Azure AI Document Intelligence extracts data * Validates document authenticity when eHerkenning is used * Cross-references with external registries ### 3.2 External Registry Verification [Section titled “3.2 External Registry Verification”](#32-external-registry-verification) #### Chamber of Commerce / national business registers [Section titled “Chamber of Commerce / national business registers”](#chamber-of-commerce--national-business-registers) * Verification using the national register number as parameter (CoC number). * **Multi-country sources** (see [Identifier Enrichment Architecture](./ctn-identifier-enrichment-architecture.md)): KvK (NL), KBO (BE), Handelsregister (DE — live `handelsregister.de` or the **OffeneRegister.de** open-data bulk), **recherche-entreprises** (FR → SIREN + **SIRET** per establishment), **Zefix / LINDAS** (CH → UID, the LINDAS SPARQL endpoint needs no account). #### LEI (Legal Entity Identifier) [Section titled “LEI (Legal Entity Identifier)”](#lei-legal-entity-identifier) * Enrichment via GLEIF using the national register number as parameter; for name-only discovery use the GLEIF **fuzzy-completion** endpoint (the `legalName` filter requires a near-exact match). #### EUID (European Unique Identifier) [Section titled “EUID (European Unique Identifier)”](#euid-european-unique-identifier) * Standardized identifier format for legal entities across the EU. **Canonical definition: see the [Glossary](../12-glossary/ctn-glossary.md) and the dotted form used throughout the architecture.** * Format: `{country}.{register}.{number}` — e.g. `NL.KVK.12345678`, `BE.KBO.0123456789`, `DEK1101R.HRB116737` (DE includes the Amtsgericht/court code, derivable from OffeneRegister). *(A compact BRIS variant `NLHR12345678` also exists; the architecture uses the dotted form.)* #### EORI (Economic Operators Registration and Identification) [Section titled “EORI (Economic Operators Registration and Identification)”](#eori-economic-operators-registration-and-identification) * Used for customs operations and trade with non-EU countries * Required for importing/exporting goods outside the EU * Format: Country code + unique number (e.g., NL123456789000) * Managed by national customs authorities #### VIES (VAT Information Exchange System) [Section titled “VIES (VAT Information Exchange System)”](#vies-vat-information-exchange-system) * Validates VAT numbers for cross-border EU transactions * VIES VAT Validation API: `https://ec.europa.eu/taxation_customs/vies/` * Input: Country code + VAT number * Output: Validation status, company name, address > **Note**: VAT number can be obtained from KvK API and then validated via VIES API. ### 3.3 CTN-Specific Technical Vetting [Section titled “3.3 CTN-Specific Technical Vetting”](#33-ctn-specific-technical-vetting) #### Technical Readiness [Section titled “Technical Readiness”](#technical-readiness) * **Endpoint Registration**: Verify and register URI(s) of system endpoint(s). * **Association Participation**: Register the association of which the entity is a participant. * **Connector Implementation**: Assess API capabilities. #### Security Assessment [Section titled “Security Assessment”](#security-assessment) * **JWKS Endpoint**: Confirm proper JWKS endpoint implementation * **Token Validation**: Test VAD validation capabilities * **Encryption Standards**: Verify compliance with security standards #### Integration Testing [Section titled “Integration Testing”](#integration-testing) * **Connectivity Testing**: Perform end-to-end connectivity tests * **Data Format Compliance**: Verify compliance with CTN data formats * **Event Handling**: Test event notifications and updates * **Error Handling**: Assess recovery mechanisms ### 3.4 Verification Methods [Section titled “3.4 Verification Methods”](#34-verification-methods) #### Automated Verification [Section titled “Automated Verification”](#automated-verification) 1. **Document AI Processing** * OCR extraction of text * Field recognition and validation * Signature detection * Tamper detection 2. **API Verification** * Chamber of Commerce API * GLEIF LEI verification * VIES validation service #### Manual Verification [Section titled “Manual Verification”](#manual-verification) 1. **Document Review** * Visual inspection of documents * Cross-reference with external sources * Flag suspicious documents * Request additional information if needed 2. **Business Vetting** * Industry sector verification * Reputation check * Compliance history review * Reference checks ### 3.5 Verification model [Section titled “3.5 Verification model”](#35-verification-model) Each onboarding check is recorded as a **Verification** with a type, a category, and a status. Verifications are authoritative in the ASR (never mirrored into Keycloak — see [ADR-00003](../09-decisions/00003-separate-users-and-participants.md)). **Status lifecycle**: `Pending` → `Approved` / `Rejected`; an `Approved` verification can later be `Revoked`. **Categories**: * **Automatic** — resolved by a system/API call (e.g. KvK, KBO, VAT, LEI). * **ByAdmin** — requires association/admin approval (e.g. onboarding approval, business-register extract review). * **ByOrganization** — a step the participant performs themselves (e.g. eHerkenning, conditions-of-use acceptance). **Verification types**: `OnboardingApproval`, `KvkCheck`, `KboCheck`, `VatCheck`, `LeiCheck`, `DnsVerification`, `EmailVerification`, `EHerkenning`, `BusinessRegisterExtract`, `ConditionsOfUseAcceptance`. The set is open for extension; a participant’s trust level is derived from which verifications are `Approved`. ### 3.6 eHerkenning step-up verification [Section titled “3.6 eHerkenning step-up verification”](#36-eherkenning-step-up-verification) eHerkenning is used as a **step-up verification**, not a login. The user is already authenticated via their normal session; the eHerkenning assertion proves they represent the organisation by matching the KvK number. Flow: the portal issues an OIDC challenge with `kc_idp_hint=eh` and `prompt=login`; on callback the `kvk` claim is cross-checked (trimmed, case-insensitive) against the participant’s stored KvK. On match, the `EHerkenning` verification is set to `Approved` with compact evidence `{"kvk": "...", "jti": "..."}` (the `jti` links to the Keycloak admin event for audit). Guardrails (proven in the Poort8 prototype): * eHerkenning is **hidden on the Keycloak login page** and only reachable via `kc_idp_hint=eh`, so it cannot be bypassed with username/password — otherwise a verification could be marked “proven via eH” without eH involvement. * The callback **never mutates the main session** (it is an assertion, not account linking). * Verification context is carried in the **OIDC state**, not the session cookie, so the flow survives third-party-cookie blocking (e.g. private-browsing). * Any broker-created Keycloak user is **deleted after the callback** to prevent orphan accounts. *** ## Phase 4: Approval [Section titled “Phase 4: Approval”](#phase-4-approval) ### 4.1 Association Administrator Review [Section titled “4.1 Association Administrator Review”](#41-association-administrator-review) * Manual review of verification results * Assessment of application completeness * Decision on membership approval ### 4.2 Identity Assurance Level Assignment [Section titled “4.2 Identity Assurance Level Assignment”](#42-identity-assurance-level-assignment) Based on the verification methods used, participants are assigned an Identity Assurance Level (IAL): | IAL | Verification Method | Trust Level | | ---- | ------------------------------------------ | ----------- | | IAL1 | Company name + CoC number only | Basic | | IAL1 | DNS verification (DNS-token in TXT record) | Basic | | IAL2 | Recent CoC statement submitted | Substantial | | IAL3 | eHerkenning registration | High | ### 4.3 Risk Classification [Section titled “4.3 Risk Classification”](#43-risk-classification) Based on vetting results, assign risk classification: * **Low Risk**: Full access with minimal monitoring * **Medium Risk**: Standard access with regular monitoring * **High Risk**: Limited access with enhanced monitoring * **Conditional**: Temporary access pending additional verification ### 4.4 Decision Framework [Section titled “4.4 Decision Framework”](#44-decision-framework) #### Full Approval [Section titled “Full Approval”](#full-approval) * All vetting criteria met satisfactorily * Full participation: VAD issued; may register endpoints and consume/provide data * Standard monitoring and review schedule #### Conditional Approval [Section titled “Conditional Approval”](#conditional-approval) * Most criteria met with minor deficiencies * Limited or monitored access initially * Specific conditions for full approval status #### Rejection [Section titled “Rejection”](#rejection) * Critical vetting criteria not met * No VAD issued; cannot participate in the network * Option for re-evaluation after remediation *** ## Phase 5: Credential Issuance [Section titled “Phase 5: Credential Issuance”](#phase-5-credential-issuance) ### 5.1 VAD Token Generation [Section titled “5.1 VAD Token Generation”](#51-vad-token-generation) JWT-based verifiable credential including: * Legal entity information * LEI, CoC number, VIES * Membership tier * Valid from/until dates * Access scopes * Identity Assurance Level ### 5.2 Token Claims Structure [Section titled “5.2 Token Claims Structure”](#52-token-claims-structure) ```json { "iss": "https://association.ctn.network", "sub": "client_id", "aud": "https://ctn.network", "iat": 1704063600, "exp": 1704150000, "nbf": 1704063600, "jti": "unique-token-id", "https://schemas.ctn.network/claims/legal_entity/name": "Company Legal Name", "https://schemas.ctn.network/claims/legal_entity/lei": "LEI123456789012345678", "https://schemas.ctn.network/claims/legal_entity/kvk": "12345678", "https://schemas.ctn.network/claims/participation/status": "active", "https://schemas.ctn.network/claims/identity_assurance_level": "ial2", "https://schemas.ctn.network/claims/terms/version": "v3.2.0" } ``` ### 5.3 Secure Delivery [Section titled “5.3 Secure Delivery”](#53-secure-delivery) 1. **Security Measures** * Multi-factor authentication required * Secure download link via email * Time-limited access to credentials (24 hours) * Confirmation of receipt required 2. **Storage Recommendations** * Secure key vault usage * Encrypted storage * Access control * Regular rotation *** ## Phase 6: Activation [Section titled “Phase 6: Activation”](#phase-6-activation) ### 6.1 Member Portal Access [Section titled “6.1 Member Portal Access”](#61-member-portal-access) * Credentials activated * Initial login and profile setup * OAuth client registration * API key generation ### 6.2 Integration Support [Section titled “6.2 Integration Support”](#62-integration-support) * Documentation provided * Technical support available * Sandbox environment access * Testing period ### 6.3 Internal System Registration [Section titled “6.3 Internal System Registration”](#63-internal-system-registration) Once vetting is complete and approved, the following information is registered: #### Basic Company Information [Section titled “Basic Company Information”](#basic-company-information) * Legal name and trading names * Registration numbers (KvK, LEI, etc.) * Business address and contact information * Key contact persons and roles #### CTN-Specific Information [Section titled “CTN-Specific Information”](#ctn-specific-information) * **Domain**: Primary domain for identification * **Endpoints**: All registered API endpoints * **Association Details**: Which Association they belong to * **Capabilities**: Specific services they can provide or consume #### Operational Details [Section titled “Operational Details”](#operational-details) * Service capabilities and capacity * Geographic coverage * Supported transport modes * Quality certifications and standards *** ## Phase 7: Ongoing Monitoring [Section titled “Phase 7: Ongoing Monitoring”](#phase-7-ongoing-monitoring) ### 7.1 Regular Review Schedule [Section titled “7.1 Regular Review Schedule”](#71-regular-review-schedule) * **Annual Review**: Comprehensive re-vetting for all participants * **Quarterly Monitoring**: Check against Chamber of Commerce API * **Event-Driven Review**: Triggered by significant changes or incidents ### 7.2 Continuous Monitoring [Section titled “7.2 Continuous Monitoring”](#72-continuous-monitoring) * **Sanctions Lists**: Ongoing screening against updated sanctions lists * **Media Monitoring**: Continuous adverse media screening * **Performance Tracking**: Ongoing assessment of contract performance ### 7.3 CTN-Specific Monitoring [Section titled “7.3 CTN-Specific Monitoring”](#73-ctn-specific-monitoring) * **Endpoint Health**: Regular monitoring of endpoint availability * **Security Updates**: Monitoring for security patches and certificate renewals * **Compliance Drift**: Regular assessment of continued compliance * **Integration Performance**: Monitoring of integration performance metrics *** ## Workflow Automation [Section titled “Workflow Automation”](#workflow-automation) ### Event-Driven Architecture [Section titled “Event-Driven Architecture”](#event-driven-architecture) 1. **Application Submitted** → Event published 2. **Documents Uploaded** → AI processing triggered 3. **Verification Complete** → Approval workflow triggered 4. **Application Approved** → VAD generation triggered 5. **VAD Issued** → Notification sent ### Azure Services Used [Section titled “Azure Services Used”](#azure-services-used) * **Kubernetes (AKS, via Helm + ArgoCD)**: hosts the Quarkus ASR services and background jobs * **Document Intelligence**: AI document processing (e.g. Azure Document Intelligence / AWS Textract) * **Key Vault**: Secret management and HSM-backed signing keys *** ## Timeline and SLAs [Section titled “Timeline and SLAs”](#timeline-and-slas) ### Standard Processing [Section titled “Standard Processing”](#standard-processing) | Phase | Duration | | --------------------- | --------------------- | | Application Review | 1-2 business days | | Document Verification | 2-3 business days | | Approval Decision | 1 business day | | **Total** | **4-6 business days** | ### Expedited Processing [Section titled “Expedited Processing”](#expedited-processing) * Available for premium/enterprise tiers * Total: 1-2 business days * Additional fee may apply *** ## Quality Controls [Section titled “Quality Controls”](#quality-controls) ### Data Quality [Section titled “Data Quality”](#data-quality) * Validation rules for all fields * Cross-reference checks * Duplicate detection * Data enrichment from external sources ### Process Quality [Section titled “Process Quality”](#process-quality) * SLA monitoring (e.g., verification within 2 business days) * Quality assurance reviews * Customer feedback collection * Continuous process improvement ### Documentation Requirements [Section titled “Documentation Requirements”](#documentation-requirements) * Complete vetting file for each approved participant * Regular update of participant information * Audit trail of all vetting decisions * Evidence supporting approval/rejection decisions *** ## Compliance and Audit [Section titled “Compliance and Audit”](#compliance-and-audit) ### GDPR Compliance [Section titled “GDPR Compliance”](#gdpr-compliance) * Data minimization * Consent management * Right to access/correction/deletion * Data retention policies * Privacy by design ### Audit Trail [Section titled “Audit Trail”](#audit-trail) * All actions logged with user, timestamp, and changes * Document access tracking * Approval decisions recorded * Token issuance logged *** ## Error Handling [Section titled “Error Handling”](#error-handling) ### Common Issues [Section titled “Common Issues”](#common-issues) 1. **Incomplete Documents** * Automated notification to applicant * Clear indication of missing items * Resubmission workflow 2. **Verification Failures** * Manual review triggered * Applicant contact for clarification * Alternative verification methods 3. **External API Failures** * Retry logic with exponential backoff * Fallback to manual verification * Monitoring and alerting *** ## Support and Communication [Section titled “Support and Communication”](#support-and-communication) ### Communication Channels [Section titled “Communication Channels”](#communication-channels) * Email notifications at each stage * SMS alerts (optional) * Portal notifications * Support ticket system ### Support Availability [Section titled “Support Availability”](#support-availability) * **Basic**: Business hours * **Premium**: Extended hours * **Enterprise**: 24/7 *** ## Integration with CTN [Section titled “Integration with CTN”](#integration-with-ctn) ### Association Register Role [Section titled “Association Register Role”](#association-register-role) The Association Register serves as the trust anchor for the CTN ecosystem: 1. **Participant Directory** * Authoritative source of participant information * LEI/CoC/EORI to Client ID mapping * Membership status and tier 2. **VAD Token Provider** * Issues verifiable membership credentials * Validates participant identity * Provides trust foundation for local authorisation at data providers 3. **Access Control** * Determines participant access levels * Enforces tier-based restrictions * Manages API quotas ### Data-Provider Integration (2026) [Section titled “Data-Provider Integration (2026)”](#data-provider-integration-2026) * Data providers validate the VAD against the ASR’s JWKS endpoint before releasing data. * VAD claims indicate participation tier and verification status. * The provider’s own local policy decides the resulting data shape. *** ## References [Section titled “References”](#references) * [Association Register Architecture](ctn-association-register.md) *** *This document provides implementation guidance for CTN participants and should be adapted to specific organizational requirements and risk tolerance.* # SPI & Seam Architecture (as-built) This document is the **as-built (IST)** view of the asr-service: what the code actually wires *today*, with the focus the architecture board asked for — the **SPIs** (Service Provider Interfaces) and the **seams** the system is being grown along. It is the developer-facing companion to the aspirational (SOLL) [Building Block View](ctn-building-blocks.md): where that document describes the target system, this one describes the running one and marks, per element, whether it is **implemented**, **stubbed**, or **shape-only** (interface present, no driver yet). > **Scope.** Only the seam-bearing parts of the service. The full data model is in [ASR Data Model](ctn-asr-data-model.md); the target component tree is in [Building Blocks](ctn-building-blocks.md). Runtime flows that are *not* about a seam (VAD issuance, M2M) live in their own building-block docs. ## How to keep this document alive [Section titled “How to keep this document alive”](#how-to-keep-this-document-alive) This is a **living document**. Each diagram below carries a `` comment naming the source files and ADRs it reflects. The contract is: * **A ticket that changes a seam changes its diagram in the same PR.** If you add an SPI, add an implementor, add a lifecycle verb, or move the seam from packages to Quarkus extensions ([ADR-00010](../09-decisions/00010-evolving-to-quarkus-extensions.md) phase 2), update the matching diagram here. The `derived-from` front-matter is the watch-list — a PR touching any listed path should touch this file. * **Status markers are part of the diagram.** When a `stub` becomes real or a `shape-only` interface gets its first implementor, flip the marker and the legend note. The markers are the difference between this doc and the SOLL one. * **The glossary is the authority for names.** Terms used here (Claim Verification Provider, Verified Claim Set, Attestation, Lifecycle verb, Enrichment, Intake Country, Actor) are defined in [`docs/ai/CONTEXT.md`](../../ai/CONTEXT.md) and the [arc42 glossary](../12-glossary/ctn-glossary.md). Don’t redefine them here. Legend used throughout: | Marker | Meaning | | ------------- | ----------------------------------------------------------------------------------------------------------- | | ✅ implemented | code exists and is wired into a runtime path | | 🟡 stubbed | wired, but the external edge is faked (e.g. broker claims come from a test source, not a live signed token) | | ⚪ shape-only | interface defined, **no implementor** yet — a marked seam | | 🔒 core-only | exists in the pure core but has no driver (endpoint / scheduler) wired | *** ## 1. System context (C4 Level 1) [Section titled “1. System context (C4 Level 1)”](#1-system-context-c4-level-1) Who and what the seam-bearing service talks to. The two **external seams** — Keycloak-as-broker for identity, and register sources for claims — are where the SPIs sit. ``` graph TB applicant["👤 Applicant / Requestor
(same person, two phases)"] admin["👤 Admin
(reviewer)"] sysacct["⚙️ Expiry service account
🔒 no driver yet"] consumer["🤖 M2M Consumer"] subgraph asr["asr-service (Quarkus, Java 25)"] core["Core + SPI
org.bdinetwork.asr"] end kc["🔑 Keycloak
IdP + broker"] eh["eHerkenning broker
(Signicat OIDC facade)
🟡 simulated realm"] reg["Register sources
KvK / VIES / GLEIF / KBO
⚪ not yet called"] obj["Object store
MinIO / S3
⚪ future — Postgres today"] pg[("PostgreSQL")] applicant -->|"intake (anonymous), edit, step-up"| core admin -->|"review verbs"| core sysacct -.->|"expire sweep 🔒"| core consumer -->|"VAD via client-credentials"| core core -->|"persists (incl. document bytes today)"| pg core -->|"OIDC resource server + admin client"| kc kc -.->|"federates (first-broker-login)"| eh core -.->|"future backend lookups"| reg core -.->|"future document bytes via ObjectStorageProvider"| obj classDef stub stroke-dasharray: 5 5; class eh,reg,obj,sysacct stub; ``` **Read this as:** the applicant and admin drive the service directly; the service never talks to eHerkenning — **Keycloak brokers it** ([ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md)), and ASR only ever consumes a broker-neutral result. Register sources are a declared future seam (the [`BackendEvidenceVerificationProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/BackendEvidenceVerificationProvider.java), §3) with no caller yet. An external object store is likewise a declared future seam (the [`ObjectStorageProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/ObjectStorageProvider.java), §3, [ADR-00021](../09-decisions/00021-document-store-objectstorageprovider-seam.md)) — document bytes live in PostgreSQL today. *** ## 2. Layered packages & the SPI seam (C4 Level 2) [Section titled “2. Layered packages & the SPI seam (C4 Level 2)”](#2-layered-packages--the-spi-seam-c4-level-2) The single deployable is split into **three package layers** ([ADR-00009](../09-decisions/00009-layered-source-code-structure.md), [ADR-00010](../09-decisions/00010-evolving-to-quarkus-extensions.md) phase 1). The [SPI package](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/) is the **seam**: the core defines the interfaces, and the marketplace and consortium layers plug in implementations discovered via CDI. The core never imports downward. ``` graph TB subgraph ctn["nl.ctn.asr — consortium (CTN-specific)"] ctnAssoc["CtnAssociationProvider ✅"] ctnCountry["CtnCountryOnboardingProvider
(abstract skeleton)
+ NL / BE / DE subclasses ✅"] ctnNlPol["CtnNlOnboardingEnrichmentPolicy ✅"] end subgraph mkt["org.bdinetwork.marketplace.asr — reusable implementations"] ehProv["EHerkenningClaimVerificationProvider 🟡"] exAssoc["ExampleMarketplaceAssociationProvider ✅"] end subgraph coreL["org.bdinetwork.asr — core"] subgraph spi["⟦ SPI seam ⟧ org.bdinetwork.asr.spi"] iOnb["OnboardingProvider"] iCb["OnboardingCallback"] iPol["OnboardingEnrichmentPolicy"] iCvp["EvidenceVerificationProvider
(Interactive / Backend)"] iAssoc["AssociationProvider"] iOsp["ObjectStorageProvider"] end svc["OnboardingRequestService
(implements OnboardingCallback) ✅"] shell["OnboardingTransitions
(transition shell) ✅"] lc["OnboardingLifecycle
(pure core) ✅"] defPol["PermissiveOnboardingEnrichmentPolicy
(default) ✅"] docSvc["DocumentService
(store boundary) 🔒 no HTTP driver"] pgOsp["PostgresObjectStorageProvider ✅"] repo["Repositories + JPA model ✅"] end ctnCountry -.implements.-> iOnb ctnNlPol -.implements.-> iPol ctnAssoc -.implements.-> iAssoc exAssoc -.implements.-> iAssoc ehProv -.implements.-> iCvp svc -.implements.-> iCb defPol -.implements.-> iPol pgOsp -.implements.-> iOsp shell -->|"CDI: @All List"| iPol shell --> ehProv shell --> lc svc --> repo docSvc --> iOsp docSvc --> repo classDef seam fill:#fff3cd,stroke:#d39e00; class iOnb,iCb,iPol,iCvp,iAssoc,iOsp seam; ``` **The dependency rule:** arrows of `implements` point **up** into the core’s SPI package; the core depends only on its own interfaces. The three CTN country wizards (NL/BE/DE) now share an abstract skeleton, [`CtnCountryOnboardingProvider`](../../../asr-service/src/main/java/nl/ctn/asr/onboarding/CtnCountryOnboardingProvider.java) — country dispatch, callback handling and the intake sequence are identical; only the country code, wizard route, intake DTO and JAX-RS/OpenAPI surface differ and stay on the concrete subclasses. Swapping eHerkenning for a real broker, or adding a new consortium, is an additive change in the upper layers — the core does not move. [ADR-00010](../09-decisions/00010-evolving-to-quarkus-extensions.md) phase 2 (turning these layers into Quarkus extensions / separate Maven modules) would harden this seam from a package boundary into a module boundary; **today it is package-only**. *** ## 3. SPI catalogue (C4 Level 3) [Section titled “3. SPI catalogue (C4 Level 3)”](#3-spi-catalogue-c4-level-3) The contracts in `org.bdinetwork.asr.spi`, their sub-interfaces, value types, and who implements each. This is the heart of the “specific attention to SPIs” request. ``` classDiagram direction TB class OnboardingProvider { <> +supports(countryCode) boolean +start(OnboardingCallback) String } class OnboardingCallback { <> +save(CreateOnboardingReq) OnboardingReqResp +onboardingSucceeded(...) Response +onboardingFailed() Response } class OnboardingEnrichmentPolicy { <> +supports(country) boolean +isComplete(OnboardingRequest) boolean } class AssociationProvider { <> +getLegalName() String +getFriendlyName() String } class ObjectStorageProvider { <> +put(UUID, InputStream) void +get(UUID) InputStream } class EmailProvider { <> +send(EmailRequest) void } class EvidenceVerificationProvider { <> +validationSource() String } class InteractiveEvidenceVerificationProvider~C~ { <> +start(AttributeWithEvidence) String +complete(AttributeWithEvidence, C) EvidenceOutcome } class BackendEvidenceVerificationProvider { <> +verify(AttributeWithEvidence) Object } class EvidenceDetailsSPI { <> +supports(Evidence.Type) boolean +isControlAttestation(Evidence) boolean +getDetails(Evidence) Map +getAssuranceLevel(Evidence) String } EvidenceVerificationProvider <|-- InteractiveEvidenceVerificationProvider EvidenceVerificationProvider <|-- BackendEvidenceVerificationProvider class AttributeWithEvidence { <> +getEvidences() List~Evidence~ +addEvidence(Evidence) void +getValidationStatus() Status } class AssuranceLevel { <> LOW SUBSTANTIAL HIGH } class EvidenceOutcome { <> Matched Mismatched Abandoned } InteractiveEvidenceVerificationProvider ..> AttributeWithEvidence InteractiveEvidenceVerificationProvider ..> EvidenceOutcome BackendEvidenceVerificationProvider ..> AttributeWithEvidence EvidenceDetailsSPI ..> AssuranceLevel ``` ### SPI contracts (source) [Section titled “SPI contracts (source)”](#spi-contracts-source) The interfaces and value types, all in [`org.bdinetwork.asr.spi`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/): [`OnboardingProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/OnboardingProvider.java) · [`OnboardingCallback`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/OnboardingCallback.java) · [`OnboardingEnrichmentPolicy`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/OnboardingEnrichmentPolicy.java) · [`AssociationProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/AssociationProvider.java) · [`ObjectStorageProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/ObjectStorageProvider.java) · [`EvidenceVerificationProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/EvidenceVerificationProvider.java) ([`Interactive`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/InteractiveEvidenceVerificationProvider.java) / [`Backend`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/BackendEvidenceVerificationProvider.java)) · [`EvidenceDetailsSPI`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/EvidenceDetailsSPI.java) · [`AssuranceLevel`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/AssuranceLevel.java) · [`EvidenceOutcome`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/EvidenceOutcome.java). The interactive/backend subject is the model interface [`AttributeWithEvidence`](../../../asr-service/src/main/java/org/bdinetwork/asr/model/AttributeWithEvidence.java) (implemented by `LegalEntityIdentifier`), and the produced record is an [`Evidence`](../../../asr-service/src/main/java/org/bdinetwork/asr/evidence/entity/Evidence.java) — the former `VerifiedClaimSet`/`VerifiedClaim` value record was retired when claim verification became **evidence** verification (the evidence module carve, ADR-00022 / TDR-0006). ### Implementor & status table [Section titled “Implementor & status table”](#implementor--status-table) | SPI | Sub-interface | Implementor(s) | Layer | Status | | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | -------------------------------------------------- | | [`OnboardingProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/OnboardingProvider.java) | — | [`CtnCountryOnboardingProvider`](../../../asr-service/src/main/java/nl/ctn/asr/onboarding/CtnCountryOnboardingProvider.java) (abstract skeleton) ← [`CtnNlOnboardingProvider`](../../../asr-service/src/main/java/nl/ctn/asr/onboarding/nl/CtnNlOnboardingProvider.java), [`CtnBeOnboardingProvider`](../../../asr-service/src/main/java/nl/ctn/asr/onboarding/be/CtnBeOnboardingProvider.java), [`CtnDeOnboardingProvider`](../../../asr-service/src/main/java/nl/ctn/asr/onboarding/de/CtnDeOnboardingProvider.java) | nl.ctn | ✅ | | [`OnboardingCallback`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/OnboardingCallback.java) | — | [`OnboardingRequestService`](../../../asr-service/src/main/java/org/bdinetwork/asr/onboarding/OnboardingRequestService.java) | core | ✅ | | [`OnboardingEnrichmentPolicy`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/OnboardingEnrichmentPolicy.java) | — | [`PermissiveOnboardingEnrichmentPolicy`](../../../asr-service/src/main/java/org/bdinetwork/asr/onboarding/internal/PermissiveOnboardingEnrichmentPolicy.java) (default), [`CtnNlOnboardingEnrichmentPolicy`](../../../asr-service/src/main/java/nl/ctn/asr/onboarding/nl/CtnNlOnboardingEnrichmentPolicy.java) | core / nl.ctn | ✅ | | [`AssociationProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/AssociationProvider.java) | — | [`CtnAssociationProvider`](../../../asr-service/src/main/java/nl/ctn/asr/CtnAssociationProvider.java), [`ExampleMarketplaceAssociationProvider`](../../../asr-service/src/main/java/org/bdinetwork/marketplace/asr/ExampleMarketplaceAssociationProvider.java) | nl.ctn / marketplace | ✅ | | [`ObjectStorageProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/ObjectStorageProvider.java) | — | [`PostgresObjectStorageProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/document/PostgresObjectStorageProvider.java) | core | ✅ seam wired; document feature 🔒 (no HTTP driver) | | [`EvidenceVerificationProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/EvidenceVerificationProvider.java) | [`InteractiveEvidenceVerificationProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/InteractiveEvidenceVerificationProvider.java) | [`EHerkenningClaimVerificationProvider`](../../../asr-service/src/main/java/org/bdinetwork/marketplace/asr/eherkenning/EHerkenningClaimVerificationProvider.java) | marketplace | 🟡 broker stubbed | | [`EvidenceVerificationProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/EvidenceVerificationProvider.java) | [`BackendEvidenceVerificationProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/BackendEvidenceVerificationProvider.java) | — | — | ⚪ shape-only | | [`EvidenceDetailsSPI`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/EvidenceDetailsSPI.java) | — | [`EHerkenningEvidenceDetailsProvider`](../../../asr-service/src/main/java/org/bdinetwork/marketplace/asr/eherkenning/EHerkenningEvidenceDetailsProvider.java), [`DocumentEvidenceDetailsProvider`](../../../asr-service/src/main/java/org/bdinetwork/marketplace/asr/document/DocumentEvidenceDetailsProvider.java) | marketplace | ✅ | | [`EmailProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/EmailProvider.java) | — | [`SmtpEmailProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/email/SmtpEmailProvider.java) (`@DefaultBean`, Quarkus Mailer + Qute), [`AzureCommunicationEmailProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/email/AzureCommunicationEmailProvider.java) (`@IfBuildProperty` — throws) | core (`email`) | ✅ SMTP default; ⚪ Azure stub | **Why two sub-interfaces ([ADR-00014](../09-decisions/00014-claim-verification-provider-seam-and-eherkenning.md)):** `InteractiveEvidenceVerificationProvider` is `start → redirect`, `complete → outcome` (user present, multi-round) and yields a broker-neutral [`EvidenceOutcome`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/EvidenceOutcome.java) (`Matched` / `Mismatched` / `Abandoned`); `BackendEvidenceVerificationProvider` is `verify → …` (user absent, one synchronous call). Forcing one signature onto both would leak one mode’s accident into the other. The recorded result of a verification is an [`Evidence`](../../../asr-service/src/main/java/org/bdinetwork/asr/evidence/entity/Evidence.java) row (the evidence module, §6); marketplace/country satellites attach their specialised payload via a `@OneToOne` and expose it through the [`EvidenceDetailsSPI`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/EvidenceDetailsSPI.java) (`EHerkenningEvidence`, `DocumentEvidence`). The planned `nl.ctn.asr.enrichment.*` design ([Identifier Enrichment Architecture](ctn-identifier-enrichment-architecture.md), KvK/VIES/GLEIF) is the backend provider *in embryo* — folding it behind this seam is deferred **convergence debt**. **Why `ObjectStorageProvider` ([ADR-00021](../09-decisions/00021-document-store-objectstorageprovider-seam.md), Phase 1):** document **bytes** are owned by the seam (`put`/`get` on `InputStream`, keyed by the Document’s UUID), not by the [`Document`](../../../asr-service/src/main/java/org/bdinetwork/asr/model/Document.java) metadata — so the Postgres-first [`PostgresObjectStorageProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/document/PostgresObjectStorageProvider.java) (a `document_blob` table) can later be swapped for MinIO/S3 without touching the model or any referrer. The contract is `InputStream` in/out (never `byte[]`) and offers **no** presigned/direct-URL operation — document access stays API-mediated. The store boundary [`DocumentService`](../../../asr-service/src/main/java/org/bdinetwork/asr/document/DocumentService.java) (ingest guards, per-type expiry, write-once, authorization via an explicit `CallerContext`) is **HTTP-driver-less today** (🔒) — callers are tests until the Phase 2 endpoint lands. *** ## 4. Seam: onboarding-request lifecycle (ADR-00012) [Section titled “4. Seam: onboarding-request lifecycle (ADR-00012)”](#4-seam-onboarding-request-lifecycle-adr-00012) The `OnboardingRequest` status changes **only** through guarded lifecycle verbs ([ADR-00012](../09-decisions/00012-onboarding-request-lifecycle-transition-seam.md)). The seam is split three ways so that an illegal transition is unrepresentable *even from inside the module*. Since the onboarding module was zoned to the seam pattern (ADR-00022, §6), the pure core lives in `onboarding.internal`, the entity in the published `onboarding.entity` zone, and the CDI shell interface at the module root. ``` stateDiagram-v2 [*] --> Submitted: Intake (actor-less, not a verb) Submitted --> UnderReview: startReview ✅
(admin) UnderReview --> ChangesRequested: sendBack ✅
(admin, note req.) ChangesRequested --> UnderReview: resubmit ✅
(requestor, enrichment-guarded) UnderReview --> Approved: approve ✅
(admin) UnderReview --> Rejected: reject ✅
(admin, note req.) Submitted --> Rejected: expire 🔒
(system, no driver) ChangesRequested --> Rejected: expire 🔒
(system, no driver) Approved --> [*] Rejected --> [*] note right of ChangesRequested Open states = {Submitted, UnderReview, ChangesRequested} Terminal = {Approved, Rejected} Both derived from the edge topology, never hardcoded. end note ``` ### The three-part seam — who is allowed to write `status` [Section titled “The three-part seam — who is allowed to write status”](#the-three-part-seam--who-is-allowed-to-write-status) ``` graph LR res["OnboardingRequestResource
(coarse @RolesAllowed) ✅"] shell["OnboardingTransitions
(CDI shell — only writer) ✅"] entity["OnboardingRequest.transitionTo()
(locked setter) ✅"] core["OnboardingLifecycle
(pure topology table) ✅"] pol["OnboardingEnrichmentPolicy
(resubmit guard, by Intake Country)"] res -->|"verb + Actor"| shell shell -->|"ownership / clock / completeness guards"| shell shell -->|"resolves by intakeCountry"| pol shell -->|"transitionTo(target)"| entity entity -->|"is this edge legal?"| core core -.->|"absent edge → IllegalTransitionException"| entity ``` | Layer | Owns | Lives in | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | **Pure core** [`OnboardingLifecycle`](../../../asr-service/src/main/java/org/bdinetwork/asr/onboarding/internal/OnboardingLifecycle.java) | the legal `(from, verb) → to` edge table; `isOpen` / `isTerminal` derived from it | `onboarding.internal` — no JPA/CDI/IO, unit-test target | | **Entity** [`OnboardingRequest`](../../../asr-service/src/main/java/org/bdinetwork/asr/onboarding/entity/OnboardingRequest.java) | a single non-public `status` + locked `transitionTo()` delegating legality to the core | `onboarding.entity` (published zone) — JPA | | **CDI shell** [`OnboardingTransitions`](../../../asr-service/src/main/java/org/bdinetwork/asr/onboarding/OnboardingTransitions.java) (impl in [`internal`](../../../asr-service/src/main/java/org/bdinetwork/asr/onboarding/internal/OnboardingTransitionsImpl.java)) | the **only** writer of `status`; contextual guards (ownership, clock, per-country completeness), actor/time stamping, persistence | root seam interface + `@ApplicationScoped @Transactional` impl | **Two-layer authorization:** coarse role checks (admin vs requestor) at the HTTP resource; the **data-dependent ownership guard** (`actor.sub == request.applicantUserId`) in the shell, where the aggregate is loaded — so a new endpoint cannot bypass it. The guard runs **before** the state/existence check (`findOwned` / `enrich` / `resubmit`), so a non-owner gets a uniform `403` — never a `404`/`409` that would leak whether the request exists or what state it is in (`SecurityException` not `IllegalTransitionException`). ### Lifecycle notifications — the status-change event fan-out (COMP-23) [Section titled “Lifecycle notifications — the status-change event fan-out (COMP-23)”](#lifecycle-notifications--the-status-change-event-fan-out-comp-23) The audit hook is no longer a no-op: the single transition writer now **fires a CDI event** on every status change, and hidden observers react. The shell [`OnboardingTransitionsImpl`](../../../asr-service/src/main/java/org/bdinetwork/asr/onboarding/internal/OnboardingTransitionsImpl.java) (each verb) and the intake path in [`OnboardingRequestServiceImpl`](../../../asr-service/src/main/java/org/bdinetwork/asr/onboarding/internal/OnboardingRequestServiceImpl.java) publish an [`OnboardingStatusChangedEvent`](../../../asr-service/src/main/java/org/bdinetwork/asr/onboarding/internal/OnboardingStatusChangedEvent.java) `(request, from, to, actor, note)` — `from == null` marks intake creation. Three `@ApplicationScoped` observers, all onboarding-internal, consume it; because the event is fired and observed only inside the module, it stays in `internal` and never widens the seam. ``` graph LR intake["OnboardingRequestServiceImpl
(intake: from = null → Submitted) ✅"] shell["OnboardingTransitionsImpl
(lifecycle verbs) ✅"] evt(["OnboardingStatusChangedEvent
(CDI event, internal)"]) aud["AuditingOnboardingRequestListener
→ AuditRecorder seam ✅
(REQUIRES_NEW, AFTER_SUCCESS)"] eml["EmailOnboardingRequestListener
→ EmailProvider SPI ✅
(async, AFTER_SUCCESS)"] log["LoggingOnboardingRequestListener ✅"] intake -->|"fire"| evt shell -->|"fire"| evt evt --> aud evt --> eml evt --> log ``` The audit and email observers fire **after** the transaction commits (`AFTER_SUCCESS`) so nothing is recorded or mailed for a rolled-back transition; auditing runs in its own `REQUIRES_NEW` transaction, and email is dispatched asynchronously. `AuditingOnboardingRequestListener` is the concrete audit writer the §4 hook always anticipated — it calls the **audit module seam** ([`AuditRecorder`](../../../asr-service/src/main/java/org/bdinetwork/asr/audit/AuditRecorder.java), §6), the only cross-module edge these listeners add (and one onboarding already had). `EmailOnboardingRequestListener` renders a per-status template through the [`EmailProvider`](../../../asr-service/src/main/java/org/bdinetwork/asr/spi/EmailProvider.java) SPI (§3, COMP-156). *** ## 5. Seam: claim verification (ADR-00014) — eHerkenning step-up [Section titled “5. Seam: claim verification (ADR-00014) — eHerkenning step-up”](#5-seam-claim-verification-adr-00014--eherkenning-step-up) The interactive seam in motion ([ADR-00014](../09-decisions/00014-claim-verification-provider-seam-and-eherkenning.md); implementation in [`org.bdinetwork.marketplace.asr.eherkenning`](../../../asr-service/src/main/java/org/bdinetwork/marketplace/asr/eherkenning/)): an authenticated requestor proves control of their KvK via eHerkenning, brokered by Keycloak, and the service writes a **write-once attestation**. The **broker-neutral boundary** is the key seam — all broker-specific mapping is confined to Keycloak’s IdP-mapper layer. ``` sequenceDiagram autonumber actor R as Requestor (browser) participant Res as EHerkenningResource
/v1/onboarding-requests/{id}/eherkenning participant Sh as OnboardingTransitions (shell) participant KC as Keycloak (broker) participant EH as eHerkenning broker
🟡 simulated realm participant P as EHerkenningClaimVerificationProvider participant V as evidence table
(append-only control evidence) R->>Res: POST .../eherkenning/start Res->>Sh: startStepUp(requestId, actor) Sh->>P: start(subject) P-->>R: frontend redirect route (data, not origin) R->>KC: first-broker-login KC-->>EH: SAML/OIDC federate EH-->>KC: brokered claims (kvk, loa, acting_subject, represents) Note over KC: IdP-mapper → user-session-note → protocol-mapper
(claims stay transient — never a user attribute) KC-->>R: token carrying eHerkenning claims R->>Res: POST .../eherkenning/token {token} Res->>Sh: completeStepUpWithToken(requestId, actor, token) Note over Sh: token verifier is a configured CDI bean
(Path A self-signed) — production trusted-key source deferred (KEYC-1192) Sh->>P: complete(subject, claims) P-->>Sh: EvidenceOutcome (Matched / Mismatched / Abandoned) alt Matched Sh->>V: record positive control evidence (write-once) else Mismatched Sh->>V: record failure evidence + flag for admin else Abandoned Note over Sh: nothing persisted — back to edit surface end ``` **The broker-neutral seam (why it is reusable):** ``` graph LR subgraph broker["Broker-specific — confined to Keycloak config"] m["IdP mappers
(literal broker claim names)"] end subgraph neutral["Broker-neutral — what ASR code reads"] t["token: kvk / loa /
acting_subject / represents"] vcs["EvidenceOutcome + Evidence row
(broker-neutral, eIDAS assurance)"] end m -->|"session-note → protocol-mapper"| t t --> vcs note["A real-broker swap (Signicat → other)
= a config change in the mapper layer only.
EHerkenningTokenVerifier still reads kvk/loa/acting_subject/represents."] m -.-> note ``` The service consumes a **broker-neutral** `EvidenceOutcome` (and records a broker-neutral `Evidence` row) regardless of broker; the only broker-specific knowledge is the literal claim names in the Keycloak IdP-mapper layer (the simulated realm: [`eherkenning-sim-realm.json`](../../../asr-service/src/dev-services/keycloak/eherkenning-sim-realm.json)). Replacing the simulated realm with a real Herkenningsmakelaar is therefore a **configuration change**, not a code change ([ADR-00014](../09-decisions/00014-claim-verification-provider-seam-and-eherkenning.md), Path A vs Path B deferred to KEYC-1192). *** ## 6. In-process module seams (ADR-00022) [Section titled “6. In-process module seams (ADR-00022)”](#6-in-process-module-seams-adr-00022) Alongside the *SPI* seams (§2–§5, which let outside layers plug implementations in), the service has *in-process module* seams: boundaries between the business packages under `org.bdinetwork.asr` themselves, so one module cannot reach another’s internals. Each module exposes a small **seam** — its root package (service interface + seam value types) plus the whitelisted sub-packages `dto`, `exceptions` and the published `entity` zone — and hides everything else behind `internal..`; its HTTP `rest..` boundary is visible but **not** injectable by siblings. The rule is enforced by ArchUnit ([`ModuleBoundaryTest`](../../../asr-service/src/test/java/org/bdinetwork/asr/ModuleBoundaryTest.java)), not the compiler. COMP-146 migrated **audit**, **onboarding** and **evidence**; see [ADR-00022](../09-decisions/00022-module-seam-pattern.md) and [TDR-0006](../../ai/adrs/TDR-0006-evidence-module-carve-and-caller-resolved-ownership.md). ``` graph TB subgraph onb["org.bdinetwork.asr.onboarding — module ✅"] onbSeam["root seam: OnboardingRequestService,
OnboardingTransitions, ApplicantAccountProvisioner
+ dto/ + exceptions/ ✅"] onbEnt["entity/ OnboardingRequest
(published zone) ✅"] onbRest["rest/ OnboardingRequestResource,
OnboardingStartResource ✅
(HTTP boundary — not seam)"] onbInt["internal/ impls, lifecycle,
persistence, keycloak ✅ (hidden)"] end subgraph ev["org.bdinetwork.asr.evidence — module ✅"] evSeam["root seam: EvidenceService, NewEvidence,
EvidenceLifecycleAction + dto/ + exceptions/ ✅"] evEnt["entity/ Evidence
(published zone) ✅"] evInt["internal/ EvidenceServiceImpl,
EvidenceLifecycle, persistence ✅ (hidden)"] end common["org.bdinetwork.asr.common
Actor ✅"] model["model/ parking pkg
LegalEntityIdentifier
🔒 dissolve → GitHub #159"] onbSeam -->|"behavior: drives evidence
with resolved context ✅ enforced acyclic"| evSeam evEnt -.->|"data: @ManyToOne FK
legal_entity_identifier → onboarding_request"| onbEnt onbSeam --> common evSeam --> common evInt -.->|"data: navigates LEI.onboardingRequest"| onbEnt onbSeam -.->|"data: reads model.getEvidences() 🔒 deferred cycle"| evEnt ``` **Caller-resolves-ownership (TDR-0006).** The `EvidenceService` seam takes **already-resolved context** — `record(LegalEntityIdentifier, …)`, `listEvidence(LegalEntityIdentifier)`, `evidenceDetailsFor(requestId)` — and does **no** ownership/authorization. Onboarding’s resource/shell resolve the owning request and target identifier first (the two-layer authorization of §4), then call evidence. So evidence imports **zero** onboarding behavior and the onboarding→evidence *behavior* edge is one-directional (evidence is a behavior-graph leaf). `Actor` was promoted to `org.bdinetwork.asr.common` (#161) so neither module depends on the other just to name the acting identity. **Two graphs, staged enforcement (ADR-00022 Amendment 1).** Cross-module dependencies split by target zone into a **behavior graph** (edges into a module root / `dto` / `exceptions`) and a **data graph** (edges into `entity/`), each cycle-checked independently. That is what legalises the deliberate two-way onboarding↔evidence relationship: the behavior edge and the data edge point in opposite directions but live in different graphs. | Rule (`ModuleBoundaryTest`) | Counts | Status | | ------------------------------------ | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `onlySeamReachableAcrossModules` | only root + `dto` + `exceptions` + `entity` reachable from outside a module | ✅ enforced (`@ArchTest`) | | `behaviorGraphIsCycleFree` | cross-module edges into root / `dto` / `exceptions` | ✅ enforced (`@ArchTest`) | | `dataGraphCycleForFutureEnforcement` | cross-module edges into `entity/` | 🔒 **implemented but deferred** — real `onboarding → evidence → onboarding` data-zone cycle routed through the `model` parking package (`model.LegalEntityIdentifier.getEvidences()` + `EvidenceRepository` navigating `legalEntityIdentifier.onboardingRequest`); held as a non-`@ArchTest` method + a tripwire test that fails when the cycle is gone. Promote once `model` dissolves — GitHub #159 | *** ## 7. Seam summary [Section titled “7. Seam summary”](#7-seam-summary) | Seam | SPI / mechanism | Status | ADR | | ----------------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | Country onboarding flow | `OnboardingProvider` + `OnboardingCallback` (CDI, `supports(country)`) | ✅ NL/BE/DE | [00010](../09-decisions/00010-evolving-to-quarkus-extensions.md) | | Per-country enrichment requirements | `OnboardingEnrichmentPolicy` (CDI, `@All List`, by Intake Country) | ✅ default + NL | [00012](../09-decisions/00012-onboarding-request-lifecycle-transition-seam.md) | | Request lifecycle | pure core + locked entity + CDI shell | ✅ verbs; 🔒 expire driver | [00012](../09-decisions/00012-onboarding-request-lifecycle-transition-seam.md) | | Interactive evidence verification | `InteractiveEvidenceVerificationProvider` → `EvidenceOutcome` + `Evidence` | 🟡 eHerkenning, broker stubbed | [00014](../09-decisions/00014-claim-verification-provider-seam-and-eherkenning.md) | | Backend evidence verification | `BackendEvidenceVerificationProvider` | ⚪ shape-only | [00014](../09-decisions/00014-claim-verification-provider-seam-and-eherkenning.md) | | Specialised evidence details | `EvidenceDetailsSPI` (`@All List`, by `Evidence.Type`) | ✅ eHerkenning + document | [00022](../09-decisions/00022-module-seam-pattern.md) | | Document byte storage | `ObjectStorageProvider` (Postgres-first; UUID-keyed `InputStream`) | ✅ seam + Postgres impl; 🔒 no HTTP driver | [00021](../09-decisions/00021-document-store-objectstorageprovider-seam.md) | | Broker independence | Keycloak IdP-mapper layer → broker-neutral token | 🟡 simulated realm | [00008](../09-decisions/00008-external-idp-federation-strategy.md), [00014](../09-decisions/00014-claim-verification-provider-seam-and-eherkenning.md) | | Identity ownership | Keycloak owns identity; ASR owns verification records | ✅ | [00003](../09-decisions/00003-separate-users-and-participants.md) | | In-process module boundary | package-placement seam + ArchUnit allow-list (§6) | ✅ seam + behavior-graph; 🔒 data-graph deferred (#159) | [00022](../09-decisions/00022-module-seam-pattern.md) | | Audit | `AuditRecorder` seam module (COMP-146 pilot) | ✅ seam; ✅ onboarding transition + intake now audited via the status-change event listener (COMP-23) | [00012](../09-decisions/00012-onboarding-request-lifecycle-transition-seam.md), [00022](../09-decisions/00022-module-seam-pattern.md) | | Onboarding lifecycle notifications | CDI `OnboardingStatusChangedEvent` → audit / email / logging observers (§4) | ✅ audit + logging; ✅ email (SMTP) | [00012](../09-decisions/00012-onboarding-request-lifecycle-transition-seam.md) | | Outbound email | `EmailProvider` SPI (Qute templates, `AFTER_SUCCESS`, async) | ✅ SMTP default; ⚪ Azure stub | [00022](../09-decisions/00022-module-seam-pattern.md) | *** ## Related documents [Section titled “Related documents”](#related-documents) * [Building Block View (SOLL)](ctn-building-blocks.md) — target component tree * [ASR Data Model](ctn-asr-data-model.md) — the `verification` / `audit_record` tables behind the attestation * [ADR-00009](../09-decisions/00009-layered-source-code-structure.md), [ADR-00010](../09-decisions/00010-evolving-to-quarkus-extensions.md) — the layering the SPI seam sits on * [ADR-00012](../09-decisions/00012-onboarding-request-lifecycle-transition-seam.md) — lifecycle transition seam * [ADR-00014](../09-decisions/00014-claim-verification-provider-seam-and-eherkenning.md) — claim/evidence verification provider seam * [ADR-00021](../09-decisions/00021-document-store-objectstorageprovider-seam.md) — document store behind the `ObjectStorageProvider` seam * [ADR-00022](../09-decisions/00022-module-seam-pattern.md) — in-process module seam pattern (§6) * [TDR-0006](../../ai/adrs/TDR-0006-evidence-module-carve-and-caller-resolved-ownership.md) — evidence module carve, caller-resolved ownership, two-graph boundary * [`docs/ai/CONTEXT.md`](../../ai/CONTEXT.md) — glossary for every term used here *** [↑ Back to Index](../ctn-arc42-index.md) | [Building Blocks →](ctn-building-blocks.md) # CTN Association Onboarding Process *CTN Implementation of BDI Onboarding* The CTN onboarding process establishes a trusted, verified environment for all participants, ensuring secure and interoperable data exchange. This process leverages the BDI framework principles while applying them specifically to the container transport domain. The onboarding process is started when a participant organisation wants to become Member (voting rights) or User (non-voting) of an Association. The onboarding process in the Association is defined by the Association itself. The description that follows provides guidance on a basic onboarding process to the Association. *** ## 1. Registration & Verification [Section titled “1. Registration & Verification”](#1-registration--verification) New members or users register with the Association of their choice. The registered data is submitted to the **Association Registry (ASR)**: * **CTN Connector endpoints** — the (URI) network addresses of their connectors (option: automated protocol between Connector and ASR). * **Service type attributes** — if a Connector provides **Orchestration Data Services**, this is recorded as a special attribute of that Connector. * **CTN Connector endpoint details** — including URI, method of authentication, and type of TLS certificate. * **Legal entity details** — official name, domain, and legal registration numbers. * **Ownership information** — identification of owners (such as **Ultimate Beneficial Owner (UBO)**), verified according to compliance requirements. * **Compliance Information** - such as - Criminal convictions (for corruption, fraud, terrorism, money laundering, child labor, human trafficking, etc.) * Payment of taxes & social security contributions (must be up to date) * Insolvency or bankruptcy (not in liquidation or subject to winding-up procedures) * Professional misconduct (serious misrepresentation, grave professional errors, collusion, etc.) The Association Administrator validates the information, performs UBO compliance checks, and records the verified (meta-)data in the ASR. *** ## 2. Authentication Enrollment [Section titled “2. Authentication Enrollment”](#2-authentication-enrollment) The CTN Connectors need to be able to set up an encrypted peer-to-peer communication channel, and authenticate themselves to each other. The CTN connector acting as a server **must have a TLS certificate**, preferably issued by **public Certificate Authorities (CAs)** such as Let’s Encrypt. The server authenticates to the client using the TLS-protocol. The CTN Connector acting as a client can authenticate with the OAuth Client Credentials flow, using the ASR of choice as Authentication Server. The connector needs to be enrolled first, receiving a ClientID+Secret from the ASR. The alternative to the client authenticating with OAuth is the client also authenticating with TLS: both the server and the client use the mTLS-protocol. *** ## 3. X.509 CTN Signing Certificate [Section titled “3. X.509 CTN Signing Certificate”](#3-x509-ctn-signing-certificate) A CTN connector may need to use a signing certificate to sign JWT tokens or other data. The ASR of its Association can act as a CTN “localized Certificate Authority”, similar to the SPIFFE “Workload Endpoint”. The ASR can verify the ownership of the public key of an asymmetric key-pair that the CTN connector generates internally, using the ACME protocol. The CTN connector generates a Certificate Signing Request to the ASR and receives a CTN X.509 formatted signing certificate. This certificate and the tokens/content signed with the private key are only recognized and useful within the CTN ecosystem. The security requirements for the storage of the private key may be lower than standard signing certificates as issued by CA’s. The rotation speed of the private key can be relatively high because the ACME protocol with the ASR allows automated rotation, compensating partly for the lower security of the private key storage. *** ## 4. Production Activation [Section titled “4. Production Activation”](#4-production-activation) Once validated, the member’s/users CTN Connector endpoints and certificates are marked **active** in the ASR. From this point, the Connector can participate in live CTN data exchanges according to registered capabilities and policies. *** ## Related BDI Framework [Section titled “Related BDI Framework”](#related-bdi-framework) This onboarding process implements the BDI (Basic Data Infrastructure) framework principles: * **Trust Establishment**: Through association membership and verification * **Decentralized Authentication**: Using connector-based authentication * **Certificate Management**: Localized CA functionality * **Compliance Integration**: Built-in regulatory checks The CTN applies these BDI principles specifically to container transport operations, ensuring compatibility with the broader BDI ecosystem while addressing industry-specific requirements. *** [↑ Back to Index](../ctn-arc42-index.md) | [Next: Deployment View →](../07-deployment/ctn-deployment.md) # CTN ASR Deployment Procedures This document describes how the Association Register is built, deployed, configured and operated. It is **cloud-agnostic by intent** — Azure is the current reference implementation, but every step has a documented equivalent on AWS, GCP, or self-hosted Kubernetes. Cloud-specific examples are labelled accordingly. > **Stack reminder.** [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md) selects **Quarkus + PostgreSQL + Keycloak**, packaged as an OCI image and deployed onto a managed container runtime. The procedures below assume that stack. ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * **Java 25 LTS** (Temurin / Eclipse Adoptium recommended) * **Maven 3.9+** or the Quarkus CLI * **Docker / Podman** for building the OCI image and running local Dev Services * **`kubectl`** (Kubernetes), **`az`** (Azure), **`aws`** (AWS) — only the CLI for your target environment * Access to the target container registry (e.g. `crctnasrdev.azurecr.io`, ECR, GitHub Container Registry) ## Production Deployment [Section titled “Production Deployment”](#production-deployment) ### Build the OCI image [Section titled “Build the OCI image”](#build-the-oci-image) The standard Quarkus container build produces a portable OCI image; the same artefact runs on every supported container runtime. ```bash # Build the JVM image (fast cold-start is typically good enough for the ASR's traffic profile) ./mvnw clean package -Dquarkus.container-image.build=true \ -Dquarkus.container-image.group=ctn-asr \ -Dquarkus.container-image.tag=$(git rev-parse --short HEAD) # Or build a native binary for smallest memory and fastest startup ./mvnw clean package -Pnative -Dquarkus.native.container-build=true \ -Dquarkus.container-image.build=true ``` Push to the registry: ```bash docker push /ctn-asr: ``` ### Deploy the API (reference: Kubernetes / AKS via Helm + ArgoCD) [Section titled “Deploy the API (reference: Kubernetes / AKS via Helm + ArgoCD)”](#deploy-the-api-reference-kubernetes--aks-via-helm--argocd) **Automated (GitOps)** via the CI pipeline that watches `api/`. The pipeline: 1. Builds the OCI image (as above). 2. Pushes to the container registry. 3. Bumps the image tag in the Helm values; ArgoCD reconciles the release onto the cluster. 4. Smoke-tests the resulting endpoint. **Manual — Helm (any conformant Kubernetes cluster, incl. AKS):** ```bash helm upgrade --install ctn-asr ./infrastructure/deploy/helm/ctn-asr \ --namespace ctn-asr-dev --create-namespace \ --set image.repository=/ctn-asr \ --set image.tag= kubectl rollout status deployment/ctn-asr-api -n ctn-asr-dev ``` **Manual — force an ArgoCD sync:** ```bash argocd app sync ctn-asr-dev argocd app wait ctn-asr-dev --health ``` ### Deploy the portals (frontend) [Section titled “Deploy the portals (frontend)”](#deploy-the-portals-frontend) The frontend stack is not yet decided (see [Coding Standards §Open Questions](../08-crosscutting/ctn-coding-standards.md)). When chosen, document the build command and the static-asset deployment target here. The reference implementation uses a CDN-fronted static host (Azure Static Web Apps in the prototype; AWS CloudFront + S3 or Cloudflare Pages are equally valid). ### Verify Deployment [Section titled “Verify Deployment”](#verify-deployment) After deployment, the standard Quarkus health endpoints are available: * **Liveness**: `GET /q/health/live` * **Readiness**: `GET /q/health/ready` * **Info**: `GET /q/info` (build metadata) * **OpenAPI**: `GET /q/openapi` Smoke-test: ```bash curl https:///q/health/ready ``` ## Database Migrations [Section titled “Database Migrations”](#database-migrations) The application uses **Liquibase** (built into Quarkus, [TDR-0001](../../ai/adrs/TDR-0001-liquibase-owns-schema.md)). Changesets are YAML files under `src/main/resources/db/changelog/`, included in order by `changelog-master.yaml`. **dev/test** apply the changelog at application startup (`quarkus.liquibase.migrate-at-start=true`). **Production never migrates from the app process** — a CI-built migrations image runs `liquibase update` as a Kubernetes init container before the app container starts. ```bash # Dev mode applies the changelog (dev context, incl. demo seed) at startup: ./mvnw quarkus:dev # Out-of-band via the Liquibase CLI against any environment. # --contexts is MANDATORY: a context-less run also applies context-tagged # changesets, i.e. the dev demo seed. liquibase update \ --url=jdbc:postgresql://:5432/ \ --username= --password= \ --changelog-file=db/changelog/changelog-master.yaml \ --contexts=prod ``` To verify applied changesets: ```sql SELECT orderexecuted, id, author, dateexecuted, exectype FROM databasechangelog ORDER BY orderexecuted DESC LIMIT 10; ``` ## Configuration [Section titled “Configuration”](#configuration) ### Environment variables [Section titled “Environment variables”](#environment-variables) Quarkus resolves configuration in this order: profile-specific properties → `application.properties` → environment variables → command-line. Secrets are injected at runtime from the configured secret store; properties refer to them indirectly (`${...}`). Typical environment variables (names are stack-stable; the secret store binds them): | Variable | Purpose | | --------------------------------- | --------------------------------------------- | | `QUARKUS_DATASOURCE_JDBC_URL` | PostgreSQL JDBC URL | | `QUARKUS_DATASOURCE_USERNAME` | DB username | | `QUARKUS_DATASOURCE_PASSWORD` | DB password (from secret store) | | `QUARKUS_OIDC_AUTH_SERVER_URL` | Keycloak issuer URL | | `QUARKUS_OIDC_CLIENT_ID` | OIDC client ID | | `QUARKUS_OIDC_CREDENTIALS_SECRET` | OIDC client secret (from secret store) | | `KVK_API_KEY` | KvK API key (from secret store) | | `JWT_SIGNING_KEY_REF` | Reference to the VAD signing key in HSM / KMS | ### Secret stores (cloud-specific examples) [Section titled “Secret stores (cloud-specific examples)”](#secret-stores-cloud-specific-examples) **Azure Key Vault:** ```bash az keyvault secret set \ --vault-name kv-ctn-demo-asr-dev \ --name POSTGRES-PASSWORD \ --value "" # Restart the workload to pick up the new secret value kubectl rollout restart deployment/ctn-asr-api -n ctn-asr-dev ``` **AWS Secrets Manager:** ```bash aws secretsmanager update-secret \ --secret-id ctn-asr/postgres-password \ --secret-string "" ``` **HashiCorp Vault:** ```bash vault kv put secret/ctn-asr postgres-password= ``` In every case the application picks up the new value on the next restart (or via dynamic reloading if configured). ## Monitoring [Section titled “Monitoring”](#monitoring) ### Logs [Section titled “Logs”](#logs) Quarkus emits structured JSON logs. Ship them to the central log platform of your environment: **Kubernetes (reference, AKS)**: pods stream stdout/stderr to the cluster’s log pipeline (Container Insights → Log Analytics on AKS): ```bash kubectl logs deploy/ctn-asr-api -n ctn-asr-dev -f ``` ### Metrics [Section titled “Metrics”](#metrics) Quarkus exposes Prometheus-format metrics on `/q/metrics` via `quarkus-micrometer-registry-prometheus`. Scrape with the cloud’s metric platform: * **Azure**: Azure Monitor managed Prometheus, scraping the `/q/metrics` endpoint. * **AWS**: CloudWatch Container Insights or AMP (Managed Prometheus). * **Self-hosted**: Prometheus + Grafana stack. ### Standard alerts [Section titled “Standard alerts”](#standard-alerts) | Alert | Threshold | | ----------------- | ----------------------------------- | | API 5xx errors | > 10 in 5 min — critical | | Request rate | > 1000 / 5 min — warning | | Memory usage | > 80% of limit for 15 min — warning | | P95 response time | > 5s — warning | ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### API returns 5xx [Section titled “API returns 5xx”](#api-returns-5xx) 1. Check liveness: `curl /q/health/live`. 2. Check readiness: `curl /q/health/ready` — failed readiness usually means a downstream (database, Keycloak, KvK API) is unreachable. 3. Pull the recent logs for stack traces. 4. Verify the running pods and rollout (`kubectl get pods -n ctn-asr-dev`; `kubectl rollout history deployment/ctn-asr-api -n ctn-asr-dev`). ### Workload not scaling [Section titled “Workload not scaling”](#workload-not-scaling) Check the configured autoscaling rule (CPU, request rate, or queue depth) against current metrics. Quarkus apps with native binaries scale up faster (sub-second cold-start) than JVM images — if cold-start is the issue, consider the native build. ### Database connection issues [Section titled “Database connection issues”](#database-connection-issues) ```bash # Connect from inside the workload (Kubernetes): kubectl exec -n ctn-asr-dev deploy/ctn-asr-api -- \ /bin/sh -c 'pg_isready -h $QUARKUS_DATASOURCE_JDBC_URL_HOST -p 5432' ``` Common causes: secret rotation not propagated to the workload, firewall rule missing for the workload subnet, SSL mode mismatch. ### New route not working [Section titled “New route not working”](#new-route-not-working) 1. Verify the resource class exists and is annotated (`@Path`, HTTP-verb annotation). 2. Verify the deployed image contains the change (compare the image digest against the local build). 3. Verify the new route appears in `/q/openapi` — if not, the build did not include the change. ### Production still serving an old version [Section titled “Production still serving an old version”](#production-still-serving-an-old-version) A deploy that “succeeds” in CI but does not change runtime behaviour almost always means the old pods are still serving (image tag not rolled), not that the build was wrong. Diagnose in order: 1. Compare the deployed build/version (e.g. the `git` SHA exposed at `/q/info` or in the image tag) against the commit the pipeline built. 2. Confirm the new pods are actually running the new image and Ready (`kubectl rollout status deployment/ctn-asr-api -n ctn-asr-dev`; `kubectl get pods -n ctn-asr-dev -o wide`); check that ArgoCD shows the app `Synced`/`Healthy`. 3. If old pods still serve, force a fresh rollout (`kubectl rollout restart deployment/ctn-asr-api -n ctn-asr-dev`) or re-sync ArgoCD. 4. Rule out CDN/Front Door caching of API responses — purge if necessary. ## Rollback [Section titled “Rollback”](#rollback) ### Application [Section titled “Application”](#application) **Helm / ArgoCD (GitOps):** ```bash # Roll back to the previous Helm release helm rollback ctn-asr -n ctn-asr-dev # Or revert the image tag in Git and let ArgoCD reconcile argocd app sync ctn-asr-dev ``` **Kubernetes (direct):** ```bash kubectl rollout undo deployment/ctn-asr-api -n ctn-asr-dev ``` ### Database migration [Section titled “Database migration”](#database-migration) The policy is **roll-forward only** (TDR-0001): changesets carry no `rollback` blocks, and `liquibase rollback` is not used. Roll back by **applying the inverse change as a new forward changeset**. Never delete or edit an already-applied changeset. ```sql -- Verify the schema state after the corrective changeset SELECT id, author, dateexecuted, exectype FROM databasechangelog ORDER BY orderexecuted DESC LIMIT 5; ``` ## Infrastructure as Code [Section titled “Infrastructure as Code”](#infrastructure-as-code) Cloud resources are provisioned with **OpenTofu**; the Kubernetes workload is delivered via **Helm + ArgoCD** (GitOps). Both live next to the application code: * `infrastructure/tofu/main.tf` — root module (AKS, PostgreSQL, networking, monitoring) * `infrastructure/tofu/modules/aks/` — cluster + node pools * `infrastructure/deploy/helm/ctn-asr/` — Helm chart for the ASR API * `infrastructure/deploy/argocd/` — ArgoCD Application manifests per environment Provision the cloud resources: ```bash cd infrastructure/tofu tofu init tofu plan -var-file=env/dev.tfvars tofu apply -var-file=env/dev.tfvars ``` ArgoCD then reconciles the Helm release onto the cluster (GitOps); see “Deploy the API” above. AKS is the Azure reference; the same OpenTofu modules and Helm chart target any Kubernetes (EKS, GKE, self-hosted). ## Additional Resources [Section titled “Additional Resources”](#additional-resources) * **Reference environment dashboards / URLs** are environment-specific and live in the operations team’s runbook, not in this repo. * **CI/CD pipelines** are defined under `.github/workflows/` (GitHub Actions); they build the OCI image, run `tofu apply`, and let ArgoCD sync the Helm release. ## Quick Reference [Section titled “Quick Reference”](#quick-reference) ```bash # Health curl https:///q/health/ready curl https:///q/health/live # OpenAPI curl https:///q/openapi # Prometheus metrics curl https:///q/metrics ``` # CI/CD Authentication — GitHub Actions to Azure (OIDC) How the GitHub Actions pipelines authenticate to Azure to run `tofu apply` against the CNCL environment, using **OIDC (workload identity federation)** so no client secrets are stored in GitHub. Complements [ASR Deployment Procedures](./ctn-asr-deployment-procedures.md) (Infrastructure as Code) and the [Deployment View](./ctn-deployment.md). ## Target environment (CNCL-DEV) [Section titled “Target environment (CNCL-DEV)”](#target-environment-cncl-dev) | | Value | | -------------- | --------------------------------------------------------------- | | Tenant | TopsectorLogistiek.nl — `5024789e-bb64-49c4-bb41-5983c2b0ceef` | | Subscription | ”Azure subscription 1” — `66ff09d0-4495-4a1d-8f5a-5b577c96b76c` | | Resource group | `CNCL-DEV` | | Location | West Europe (`westeurope`) | These values are also kept in `.env` (gitignored) for local `tofu` runs and map to the `env/dev.tfvars` var-file used by the root module in `infrastructure/tofu`. ## Authentication model [Section titled “Authentication model”](#authentication-model) The `azurerm` provider authenticates via OIDC: GitHub Actions requests a short-lived ID token, Entra ID trusts it through a *federated credential* on a dedicated app registration, and grants the app’s RBAC roles. No secret to store or rotate. ## Responsibilities [Section titled “Responsibilities”](#responsibilities) The Azure environment is managed by **the platform party** (tenant / Entra ID governance) and **the delivery party** delivers the application. The only hard dependency on the platform party is an Entra **Application Developer** role grant: guest accounts cannot create directory objects (app registrations) by default. With that role, the DIL side creates the app registration and handles the RBAC and state setup itself, because the solution architect already holds **Contributor + User Access Administrator** on `CNCL-DEV`. ### Account model [Section titled “Account model”](#account-model) Azure and deployment work runs under **DIL accounts** (guest accounts in the TopsectorLogistiek tenant, home domain `datainlogistics.nl`). GitHub work uses the delivery party’s **own GitHub accounts**. The OIDC trust binds the *repository* to the service principal, so it is independent of any personal identity. ### Platform party (Entra ID) [Section titled “Platform party (Entra ID)”](#platform-party-entra-id) Guest accounts cannot create app registrations by default, so the platform party: 1. Grants the Entra **Application Developer** role to the DIL accounts so the app registration + federated credentials can be created in-house. *(Requested.)* (The specific DIL accounts are listed in the request to the platform party, not here.) 2. Assigns **Contributor on CNCL-DEV** to the delivery party’s developers’ DIL accounts. *(Not yet assigned.)* ### DIL (architect — has Contributor + UAA on CNCL-DEV) [Section titled “DIL (architect — has Contributor + UAA on CNCL-DEV)”](#dil-architect--has-contributor--uaa-on-cncl-dev) Once Application Developer is granted: 1. Creates the **app registration / service principal** (e.g. `gh-cncl-bdi-deploy`) → provides the **Client ID** used as `ARM_CLIENT_ID`. 2. Adds the **federated credentials**. Issuer is always `https://token.actions.githubusercontent.com`. One credential per scenario: | Scenario | Subject | | ------------------------- | ------------------------------------------------------------- | | Push to main | `repo:Basic-Data-Infrastructure/CNCL-BDI:ref:refs/heads/main` | | Environment `dev` | `repo:Basic-Data-Infrastructure/CNCL-BDI:environment:dev` | | Pull requests (plan only) | `repo:Basic-Data-Infrastructure/CNCL-BDI:pull_request` | > Repo: `https://github.com/Basic-Data-Infrastructure/CNCL-BDI` (public) 3. Assigns the SP **Contributor** on `CNCL-DEV` (UAA permits the role assignment). 4. Creates the OpenTofu remote-state storage account + container and grants the SP **Storage Blob Data Contributor**. ### Delivery party (application) [Section titled “Delivery party (application)”](#delivery-party-application) * Adds `infrastructure/tofu` and `.github/workflows` to the repo. * Sets the GitHub repository variables (below). * Provisions the app’s Azure services in `CNCL-DEV` via OpenTofu and runs the pipeline (`tofu apply`). > Role assignments: creating resources needs Contributor; **assigning** roles (e.g. managed identity → Key Vault / storage) needs UAA. If the OpenTofu assigns roles itself, coordinate with the DIL architect (who has UAA) rather than widening the pipeline identity’s permissions. ## GitHub configuration (no secrets with OIDC) [Section titled “GitHub configuration (no secrets with OIDC)”](#github-configuration-no-secrets-with-oidc) Repository variables (Settings → Secrets and variables → Actions → Variables): | Variable | Value | | --------------------- | -------------------------------------- | | `ARM_CLIENT_ID` | Client ID of the app registration | | `ARM_TENANT_ID` | `5024789e-bb64-49c4-bb41-5983c2b0ceef` | | `ARM_SUBSCRIPTION_ID` | `66ff09d0-4495-4a1d-8f5a-5b577c96b76c` | Workflow requirements: ```yaml permissions: id-token: write # request the OIDC token contents: read env: ARM_USE_OIDC: "true" ARM_CLIENT_ID: ${{ vars.ARM_CLIENT_ID }} ARM_TENANT_ID: ${{ vars.ARM_TENANT_ID }} ARM_SUBSCRIPTION_ID: ${{ vars.ARM_SUBSCRIPTION_ID }} ``` The same `ARM_*` variables are read by the `azurerm` provider for local `tofu` runs (via `.env`); `tofu` variable values (location, rg name) come from `env/dev.tfvars`. ## Remote state [Section titled “Remote state”](#remote-state) Recommended: an `azurerm` backend in a storage account inside `CNCL-DEV`. The deploy service principal needs **Storage Blob Data Contributor** on that account. The DIL architect creates the storage account + container and the role assignment (has Contributor + UAA), or it is handled in a small bootstrap step. ## Environment validated (2026-07-08) [Section titled “Environment validated (2026-07-08)”](#environment-validated-2026-07-08) Live check against `CNCL-DEV` confirmed the environment is further along than this checklist assumed: * RGs `CNCL-NET`/`CNCL-DEV`/`CNCL-TST`, state account `cnclbdidashtfstate` (container `tfstate`) and ACR `cnclbdidashacr` **already exist**; the four required resource providers are Registered. * The DIL architect **already holds Contributor + User Access Administrator on `CNCL-DEV` and `CNCL-NET`** — so the app registration, role assignments and state-backend access can all be done in-house without waiting on the platform party’s Contributor grant. * Entra admin group `CNCL-BDI-DEV-AKS-Admins` created and wired into the `dev` tfvars. * The architect account was granted **Storage Blob Data Contributor** on the state account (for the tofu backend). The *deploy SP* still needs the same role once the app registration exists. ## Handoff checklist [Section titled “Handoff checklist”](#handoff-checklist) **Platform party** * [ ] Entra **Application Developer** role granted to DIL accounts *(only needed if the app registration is not created by a global admin)* * [ ] Contributor on CNCL-DEV for the delivery party’s developers (DIL accounts) **DIL (architect)** * [ ] App registration created → Client ID * [ ] Federated credential(s) per subject (repo `Basic-Data-Infrastructure/CNCL-BDI`) * [ ] Deploy SP assigned Contributor on CNCL-DEV * [x] State storage account + container exist; Blob Data Contributor granted (architect done; deploy SP pending) **Delivery party** * [ ] `infrastructure/tofu` + `.github/workflows` added to the repo * [ ] GitHub repository variables set (`ARM_CLIENT_ID/TENANT_ID/SUBSCRIPTION_ID`) * [ ] App’s Azure services provisioned in CNCL-DEV via OpenTofu * [ ] Pipeline runs `tofu apply` successfully against CNCL-DEV # Deployment View *Infrastructure and deployment architecture for Connected Trade Network* ## Deployment Overview [Section titled “Deployment Overview”](#deployment-overview) The Connected Trade Network is deployed on Microsoft Azure (reference cloud) using a cloud-native architecture designed for high availability, scalability, and security. The 2026 delivery centres on the **Association Register (ASR)**; it is packaged as a portable OCI image so the same artefact can run on AWS or self-hosted Kubernetes. Later-phase components are tracked in the [Roadmap](../13-roadmap/ctn-roadmap.md). ## Environment Strategy [Section titled “Environment Strategy”](#environment-strategy) | Environment | Purpose | Configuration | Data | | --------------- | ------------------------------ | -------------------------------- | --------------------- | | **Development** | Feature development, debugging | Single region, minimal resources | Synthetic test data | | **Testing** | Integration and system testing | Single region, production-like | Test data, anonymized | | **Acceptance** | UAT and performance testing | Multi-region, full stack | Production-like data | | **Production** | Live operations | Multi-region, HA, auto-scaling | Real operational data | ## Azure Resource Architecture [Section titled “Azure Resource Architecture”](#azure-resource-architecture) ### Resource Group Organization [Section titled “Resource Group Organization”](#resource-group-organization) ```plaintext CTN-Production-Subscription ├── RG-CTN-Common-Prod │ ├── Azure Front Door (Global) │ ├── Azure DNS Zones │ └── Shared Key Vault │ ├── RG-CTN-Association-Prod │ ├── Association PostgreSQL │ ├── Association API (AKS workload) │ ├── Participant Portal (Static Web) │ └── Association API Management │ ├── RG-CTN-Common-Prod (API Management for the discovery API) │ └── API Management (Premium) │ └── RG-CTN-Monitor-Prod ├── Application Insights ├── Log Analytics Workspace ├── Azure Monitor └── Azure Sentinel ``` ## Component Deployment Details [Section titled “Component Deployment Details”](#component-deployment-details) ### Association Register Deployment [Section titled “Association Register Deployment”](#association-register-deployment) **As of November 20, 2025:** Migrated from Azure Functions to a containerised Quarkus workload on Kubernetes (AKS), deployed via Helm + ArgoCD, for improved reliability and portability. **Database Tier**: * Service: Azure Database for PostgreSQL 15 - Flexible Server * Configuration: Burstable B1ms (1 vCore, 2GB RAM) for dev * Storage: 32GB Premium SSD with auto-grow * Backup: Geo-redundant, 7-day retention * Server: `psql-ctn-demo-asr-dev.postgres.database.azure.com` **Application Tier (Kubernetes / AKS)**: * Service: Azure Kubernetes Service (AKS); workload deployed via Helm + ArgoCD (GitOps) * Runtime: Java 25 + Quarkus (OCI image) — per [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md) * Image: `crctnasrdev.azurecr.io/ctn-asr-api:latest` * Replicas: 1-10 (Horizontal Pod Autoscaler on CPU / concurrent requests) * Resources: 0.5 vCPU, 1GB RAM requests per pod * Region: West Europe * Namespace: `ctn-asr-dev` * Ingress: `https://asr-api-dev.ctn.network` (via Azure Front Door / ingress controller) **Web Tier**: * Service: Azure Static Web Apps (2 instances) * Admin Portal: `stapp-ctn-demo-asr-dev` * Member Portal: `ctn-member-portal` * Framework: frontend stack not yet decided (see [Coding Standards → Open Questions](../08-crosscutting/ctn-coding-standards.md)); a React + TypeScript SPA is one candidate * CDN: Azure Front Door with WAF protection * Authentication: Keycloak (OAuth 2.0 / OIDC) with RBAC — per [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md), [ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md) **Monitoring & Logging**: * Log Analytics Workspace: `log-ctn-demo` (consolidated, Nov 20, 2025) * 30-day retention, PerGB2018 pricing * Shared by Application Insights and the AKS cluster * Application Insights: `appi-ctn-demo-asr-dev` * Active Alerts (4): * ASR-API-5xx-Errors (Severity 1) * ASR-High-Request-Rate (Severity 2) * ASR-High-Memory-Usage (Severity 2) * ASR-Slow-Response-Time (Severity 2) ### Discovery API Gateway [Section titled “Discovery API Gateway”](#discovery-api-gateway) **API Gateway**: ```yaml API Management: Tier: Premium Units: 2 (multi-region) Features: - OAuth 2.0 validation (VAD) - Rate limiting - Response caching Regions: - West Europe (Primary) - North Europe (Secondary) ``` > Components for later phases (Orchestration Register, OPA policy engine, dual-token validation) are tracked in the [Roadmap](../13-roadmap/ctn-roadmap.md); their deployment-view notes are parked under `docs/_out-of-scope-2026/`. ## Network Architecture [Section titled “Network Architecture”](#network-architecture) ### Virtual Network Design [Section titled “Virtual Network Design”](#virtual-network-design) ```plaintext VNET-CTN-Prod (10.0.0.0/16) ├── Subnet-DMZ (10.0.1.0/24) │ └── API Management, Application Gateway ├── Subnet-Apps (10.0.2.0/24) │ └── AKS node pool (ASR API pods) ├── Subnet-Data (10.0.3.0/24) │ └── PostgreSQL, Redis (Private Endpoints) └── Subnet-Management (10.0.4.0/24) └── Bastion, Management VMs ``` ### Network Security [Section titled “Network Security”](#network-security) **Network Security Groups**: | Subnet | Inbound Rules | Outbound Rules | | ---------- | ------------------------- | ------------------------- | | DMZ | HTTPS (443) from Internet | Any to Apps subnet | | Apps | From DMZ and Management | HTTPS to Data, Internet | | Data | From Apps subnet only | Deny all except responses | | Management | RDP/SSH from Bastion | Any (monitoring) | **Private Endpoints**: * All databases use private endpoints * Storage accounts restricted to VNet * Key Vault with private endpoint * Redis Cache private access only ## High Availability Architecture [Section titled “High Availability Architecture”](#high-availability-architecture) ### Multi-Region Setup [Section titled “Multi-Region Setup”](#multi-region-setup) ``` graph TB subgraph "Global" FD[Azure Front Door] DNS[Azure DNS] end subgraph "West Europe (Primary)" APIM1[API Management] APP1[Application Services] DB1[(PostgreSQL Primary)] REDIS1[(Redis Primary)] end subgraph "North Europe (Secondary)" APIM2[API Management] APP2[Application Services] DB2[(PostgreSQL Replica)] REDIS2[(Redis Replica)] end FD --> APIM1 FD --> APIM2 APIM1 --> APP1 APIM2 --> APP2 DB1 -.-> DB2 REDIS1 -.-> REDIS2 ``` ### Disaster Recovery [Section titled “Disaster Recovery”](#disaster-recovery) **Recovery Targets**: * **RTO (Recovery Time Objective)**: 1 hour * **RPO (Recovery Point Objective)**: 15 minutes **Backup Strategy**: | Component | Backup Frequency | Retention | Location | | ------------- | ----------------- | --------- | --------------------- | | Databases | Continuous | 30 days | Geo-redundant storage | | Blob Storage | Incremental daily | 7 days | RA-GRS | | Configuration | On change | Unlimited | Git repository | | Certificates | Daily | 90 days | Key Vault backup | ## Security Deployment [Section titled “Security Deployment”](#security-deployment) ### Certificate Management [Section titled “Certificate Management”](#certificate-management) ```plaintext Certificate Hierarchy: ├── Root CA (External) │ ├── TLS Certificates (Let's Encrypt) │ └── Wildcard: *.ctn.network ├── BDI Signing (Internal) │ ├── Association Signing Cert (VAD) │ └── Auto-rotation: 90 days └── mTLS Certificates └── Client certificates for connectors ``` ### Key Management [Section titled “Key Management”](#key-management) * **Azure Key Vault Premium**: HSM-backed keys * **Managed Identities**: For service authentication * **Secret Rotation**: Automated (scheduled job) * **Access Policies**: Least privilege principle ## Monitoring and Observability [Section titled “Monitoring and Observability”](#monitoring-and-observability) ### Monitoring Stack [Section titled “Monitoring Stack”](#monitoring-stack) ```yaml Application Monitoring: Service: Application Insights Features: - Distributed tracing - Live metrics - Custom events - Availability tests Infrastructure Monitoring: Service: Azure Monitor Metrics: - CPU, Memory, Network - Database performance - Queue depths - Cache hit rates Security Monitoring: Service: Azure Sentinel Capabilities: - Threat detection - Incident response - Compliance dashboards - SIEM integration ``` ### Dashboards and Alerts [Section titled “Dashboards and Alerts”](#dashboards-and-alerts) **Operations Dashboard**: * System health status * API response times * Token (VAD) generation rates * Active participant / endpoint count **Security Dashboard**: * Failed authentication attempts * Suspicious access patterns * Certificate expiration status * Compliance violations **Business Dashboard**: * Participant registrations * Endpoint registrations * VAD tokens issued * Data access requests ## Deployment Pipeline [Section titled “Deployment Pipeline”](#deployment-pipeline) ### CI/CD Architecture [Section titled “CI/CD Architecture”](#cicd-architecture) ```plaintext GitHub Repository ↓ GitHub Actions (CI) ├── Maven build (Quarkus) ├── Unit & integration tests ├── Security scanning └── OCI image build ↓ Artifact Registry ↓ Release Pipeline ├── Dev Deployment ├── Integration Tests ├── Test Deployment ├── Load Testing ├── Acceptance Deployment ├── Approval Gates └── Production Deployment ``` ### Infrastructure as Code [Section titled “Infrastructure as Code”](#infrastructure-as-code) Infrastructure is provisioned with **OpenTofu**; the Kubernetes workload is deployed via **Helm + ArgoCD** (GitOps). Everything runs from the GitHub Actions workflows — never by hand: the pipeline runs `tofu plan` on pull requests and `tofu apply` on merge, after which ArgoCD reconciles the Helm release onto the cluster. **Repository layout**: ```plaintext /infrastructure ├── /tofu # OpenTofu (cloud resources: AKS, PostgreSQL, networking, monitoring) │ ├── main.tf │ ├── modules/ │ │ ├── networking/ │ │ ├── databases/ │ │ ├── aks/ │ │ └── monitoring/ │ └── env/ │ ├── dev.tfvars │ ├── test.tfvars │ └── prod.tfvars └── /deploy ├── helm/ctn-asr/ # Helm chart for the ASR API └── argocd/ # ArgoCD Application manifests (per environment) ``` ## Scaling Configuration [Section titled “Scaling Configuration”](#scaling-configuration) ### Auto-Scaling Rules [Section titled “Auto-Scaling Rules”](#auto-scaling-rules) | Component | Metric | Scale Out | Scale In | Min/Max | | ----------------- | --------------------------------- | --------- | --------------- | ------- | | API Management | CPU > 70% | +1 unit | CPU < 30% | 2/10 | | ASR API (AKS HPA) | CPU / concurrent HTTP > threshold | +1 pod | below threshold | 1/10 | | PostgreSQL | CPU > 80% | +4 vCores | < 40% | 4/32 | ### Performance Optimization [Section titled “Performance Optimization”](#performance-optimization) * **CDN**: Static content caching * **Redis**: Application data caching * **Database**: Query optimization, indexing * **API**: Response caching, compression *** *This deployment view describes the production infrastructure and operational aspects of the Connected Trade Network.* # Azure Front Door with WAF — Setup Guide > **Scope note.** This document describes the **reference Azure deployment** of CTN ASR — the prototype that ran on Azure Static Web Apps and Azure Functions. The ASR architecture is cloud-agnostic by intent (see [Solution Strategy](../04-solution-strategy/ctn-solution-strategy.md) and [ASR Deployment Procedures](ctn-asr-deployment-procedures.md)). The naming below (“admin portal”, “member portal”) reflects the prototype; the chosen Quarkus implementation will use the same portal layout under the new naming (admin portal + participant portal). ## Overview [Section titled “Overview”](#overview) This guide covers the Azure Front Door and Web Application Firewall (WAF) implementation for the CTN ASR admin and member portals. ## Architecture [Section titled “Architecture”](#architecture) ### Components [Section titled “Components”](#components) 1. **Azure Front Door** - Global load balancer, CDN, and routing service 2. **Web Application Firewall (WAF)** - Security protection layer 3. **Static Web Apps** - Backend origins for admin and member portals ### Data Flow [Section titled “Data Flow”](#data-flow) ```plaintext User Request ↓ [Azure Front Door] ↓ (Global routing + Caching) [WAF Policy Evaluation] ↓ (Security checks) If Allowed → [Admin/Member Portal Static Web App] If Blocked → 403 Forbidden ``` ### Resources Created [Section titled “Resources Created”](#resources-created) * **WAF Policy:** `waf-ctn-asr-{environment}` * **Front Door Profile:** `fd-ctn-asr-{environment}` * **Admin Endpoint:** `admin-ctn-asr-{environment}.z01.azurefd.net` * **Member Endpoint:** `portal-ctn-asr-{environment}.z01.azurefd.net` ## Features [Section titled “Features”](#features) ### Front Door Capabilities [Section titled “Front Door Capabilities”](#front-door-capabilities) 1. **Global Load Balancing** * Routes traffic to nearest/healthiest backend * Session affinity (sticky sessions) enabled * Health probes every 100 seconds 2. **Performance Optimization** * Content caching at edge locations worldwide * SSL/TLS termination * Traffic acceleration via Microsoft global network 3. **High Availability** * Automatic failover * DDoS protection (L3/L4) * 99.99% SLA (Standard) / 99.995% SLA (Premium) ### WAF Security Features [Section titled “WAF Security Features”](#waf-security-features) 1. **OWASP Protection** * Microsoft Default Rule Set 2.1 (based on OWASP CRS) * Blocks SQL injection, XSS, RCE, LFI, RFI, etc. * Automatic threat intelligence updates 2. **Rate Limiting** * 100 requests per minute per IP address * Prevents brute force attacks * Mitigates DDoS attacks 3. **Bot Management** * Microsoft Bot Manager Rule Set 1.0 * Blocks malicious bots * Allows legitimate search engine crawlers 4. **Custom Rules** * **Suspicious User Agent Blocking**: Blocks sqlmap, nikto, masscan, nmap, wpscan * **Geo-Restriction** (Production only): Only allows NL, BE, DE, FR, GB, US 5. **Mode Selection** * **Detection Mode** (dev): Logs threats but doesn’t block * **Prevention Mode** (prod): Actively blocks threats ## Deployment [Section titled “Deployment”](#deployment) ### Prerequisites [Section titled “Prerequisites”](#prerequisites) * Azure subscription with permissions to create resources * Static Web Apps already deployed (admin and member portals) * OpenTofu (applied via GitHub Actions; GitOps with ArgoCD for cluster workloads) ### Deployment via OpenTofu [Section titled “Deployment via OpenTofu”](#deployment-via-opentofu) The Front Door and WAF are provisioned with **OpenTofu** (the `azurerm` provider) as part of the main infrastructure. The resource snippets later in this document are shown in Bicep syntax for readability; in the repo they are authored as OpenTofu modules. ```bash # Provision the whole infrastructure (includes Front Door + WAF) # export DB_PASSWORD=YourSecurePasswordHere cd infrastructure/tofu tofu apply -var-file=env/dev.tfvars -var="database_admin_password=$DB_PASSWORD" ``` ### Deploy Only Front Door and WAF [Section titled “Deploy Only Front Door and WAF”](#deploy-only-front-door-and-waf) ```bash # 1. Deploy WAF Policy az deployment group create \ --resource-group rg-ctn-asr-dev \ --template-file infrastructure/bicep/modules/waf-policy.bicep \ --parameters environment=dev \ --parameters location=westeurope \ --parameters resourcePrefix=ctn-asr # 2. Deploy Front Door (requires WAF Policy ID and Static Web App hostnames) WAF_POLICY_ID=$(az deployment group show \ --resource-group rg-ctn-asr-dev \ --name waf-policy-deployment \ --query properties.outputs.wafPolicyId.value -o tsv) ADMIN_HOSTNAME=$(az staticwebapp show \ --name stapp-ctn-demo-asr-dev \ --query defaultHostname -o tsv) MEMBER_HOSTNAME=$(az staticwebapp show \ --name ctn-member-portal \ --query defaultHostname -o tsv) az deployment group create \ --resource-group rg-ctn-asr-dev \ --template-file infrastructure/bicep/modules/front-door.bicep \ --parameters environment=dev \ --parameters resourcePrefix=ctn-asr \ --parameters wafPolicyId=$WAF_POLICY_ID \ --parameters adminPortalHostname=$ADMIN_HOSTNAME \ --parameters memberPortalHostname=$MEMBER_HOSTNAME ``` ## Configuration [Section titled “Configuration”](#configuration) ### WAF Policy Settings [Section titled “WAF Policy Settings”](#waf-policy-settings) **File:** `infrastructure/bicep/modules/waf-policy.bicep` #### Policy Mode [Section titled “Policy Mode”](#policy-mode) ```bicep mode: isProd ? 'Prevention' : 'Detection' ``` * **Detection Mode (dev)**: Logs threats, allows traffic * **Prevention Mode (prod)**: Blocks threats #### Rate Limiting [Section titled “Rate Limiting”](#rate-limiting) ```bicep { name: 'RateLimitRule' rateLimitThreshold: 100 // requests rateLimitDurationInMinutes: 1 // per minute action: 'Block' } ``` #### Geo-Restriction [Section titled “Geo-Restriction”](#geo-restriction) ```bicep { name: 'GeoRestriction' matchValue: ['NL', 'BE', 'DE', 'FR', 'GB', 'US'] negateCondition: true // Block all EXCEPT these countries action: isProd ? 'Block' : 'Log' enabledState: isProd ? 'Enabled' : 'Disabled' } ``` **Note:** Only enforced in production environment. #### Managed Rule Sets [Section titled “Managed Rule Sets”](#managed-rule-sets) 1. **Microsoft\_DefaultRuleSet 2.1** * OWASP CRS-based rules * SQL injection, XSS, RCE, etc. * Action: Block 2. **Microsoft\_BotManagerRuleSet 1.0** * Bad bot detection * Good bot allowlist * Action: Block ### Front Door Settings [Section titled “Front Door Settings”](#front-door-settings) **File:** `infrastructure/bicep/modules/front-door.bicep` #### Origin Configuration [Section titled “Origin Configuration”](#origin-configuration) ```bicep properties: { hostName: adminPortalHostname httpPort: 80 httpsPort: 443 originHostHeader: adminPortalHostname priority: 1 weight: 1000 enabledState: 'Enabled' enforceCertificateNameCheck: true } ``` #### Health Probes [Section titled “Health Probes”](#health-probes) ```bicep healthProbeSettings: { probePath: '/' probeRequestType: 'GET' probeProtocol: 'Https' probeIntervalInSeconds: 100 } ``` #### Load Balancing [Section titled “Load Balancing”](#load-balancing) ```bicep loadBalancingSettings: { sampleSize: 4 successfulSamplesRequired: 3 additionalLatencyInMilliseconds: 50 } ``` #### HTTPS Enforcement [Section titled “HTTPS Enforcement”](#https-enforcement) ```bicep properties: { supportedProtocols: ['Https'] // HTTPS only forwardingProtocol: 'HttpsOnly' // Always forward as HTTPS httpsRedirect: 'Enabled' // Redirect HTTP → HTTPS } ``` ## Custom Domains (Production) [Section titled “Custom Domains (Production)”](#custom-domains-production) ### Prerequisites [Section titled “Prerequisites”](#prerequisites-1) 1. Domain ownership verification 2. DNS management access 3. SSL certificate (managed by Azure) ### Setup Steps [Section titled “Setup Steps”](#setup-steps) 1. **Add Custom Domain to Front Door** ```bash # Admin Portal az afd custom-domain create \ --profile-name fd-ctn-asr-prod \ --custom-domain-name admin-ctn-nl \ --resource-group rg-ctn-asr-prod \ --host-name admin.ctn.nl \ --minimum-tls-version TLS12 # Member Portal az afd custom-domain create \ --profile-name fd-ctn-asr-prod \ --custom-domain-name portal-ctn-nl \ --resource-group rg-ctn-asr-prod \ --host-name portal.ctn.nl \ --minimum-tls-version TLS12 ``` 2. **Create DNS CNAME Records** ```plaintext admin.ctn.nl CNAME admin-ctn-asr-prod.z01.azurefd.net portal.ctn.nl CNAME portal-ctn-asr-prod.z01.azurefd.net ``` 3. **Enable Managed Certificate** ```bash az afd custom-domain update \ --profile-name fd-ctn-asr-prod \ --custom-domain-name admin-ctn-nl \ --resource-group rg-ctn-asr-prod \ --certificate-type ManagedCertificate ``` 4. **Associate Domain with Route** ```bash az afd route update \ --profile-name fd-ctn-asr-prod \ --endpoint-name admin-ctn-asr-prod \ --route-name admin-portal-route \ --resource-group rg-ctn-asr-prod \ --custom-domains admin-ctn-nl ``` ## Monitoring [Section titled “Monitoring”](#monitoring) ### Key Metrics to Monitor [Section titled “Key Metrics to Monitor”](#key-metrics-to-monitor) 1. **Request Count** - Total requests per endpoint 2. **Error Rate** - 4xx/5xx response codes 3. **Latency** - Response time at edge 4. **WAF Blocks** - Threats blocked by WAF 5. **Origin Health** - Backend availability ### Azure Monitor Queries [Section titled “Azure Monitor Queries”](#azure-monitor-queries) #### WAF Blocks by Rule [Section titled “WAF Blocks by Rule”](#waf-blocks-by-rule) ```kql AzureDiagnostics | where ResourceType == "FRONTDOORWEBAPPLICATIONFIREWALLPOLICIES" | where action_s == "Block" | summarize count() by ruleName_s | order by count_ desc ``` #### Top Blocked IPs [Section titled “Top Blocked IPs”](#top-blocked-ips) ```kql AzureDiagnostics | where ResourceType == "FRONTDOORWEBAPPLICATIONFIREWALLPOLICIES" | where action_s == "Block" | summarize count() by clientIP_s | order by count_ desc | take 10 ``` #### Request Latency P95 [Section titled “Request Latency P95”](#request-latency-p95) ```kql AzureDiagnostics | where ResourceType == "FRONTDOOR" | summarize percentile(timeTaken_d, 95) by bin(TimeGenerated, 5m) | render timechart ``` ### Alerts [Section titled “Alerts”](#alerts) Recommended alerts: 1. **High Error Rate**: > 5% 5xx errors in 5 minutes 2. **WAF Attack Surge**: > 100 blocks in 1 minute 3. **Origin Unhealthy**: Health probe failures 4. **High Latency**: P95 latency > 2 seconds ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Issue: 403 Forbidden [Section titled “Issue: 403 Forbidden”](#issue-403-forbidden) **Possible Causes:** 1. WAF blocking legitimate traffic 2. Geo-restriction blocking user’s country 3. Rate limit exceeded **Solution:** ```bash # Check WAF logs az monitor diagnostic-settings list \ --resource waf-ctn-asr-dev \ --resource-type Microsoft.Network/FrontDoorWebApplicationFirewallPolicies # Temporarily switch to Detection mode for testing az network front-door waf-policy update \ --name waf-ctn-asr-dev \ --resource-group rg-ctn-asr-dev \ --mode Detection ``` ### Issue: Origin Unreachable [Section titled “Issue: Origin Unreachable”](#issue-origin-unreachable) **Possible Causes:** 1. Static Web App down 2. Origin hostname incorrect 3. Certificate validation failure **Solution:** ```bash # Check origin health az afd origin show \ --profile-name fd-ctn-asr-dev \ --origin-group-name admin-portal-origin-group \ --origin-name admin-portal-origin \ --resource-group rg-ctn-asr-dev # Test origin directly curl -I https://calm-tree-03352ba03.1.azurestaticapps.net ``` ### Issue: Slow Performance [Section titled “Issue: Slow Performance”](#issue-slow-performance) **Possible Causes:** 1. Cache not configured properly 2. Origin response time slow 3. TLS handshake overhead **Solution:** ```bash # Check cache statistics az monitor metrics list \ --resource fd-ctn-asr-dev \ --resource-type Microsoft.Cdn/profiles \ --metric CacheHitRatio # Enable query string caching az afd route update \ --profile-name fd-ctn-asr-dev \ --endpoint-name admin-ctn-asr-dev \ --route-name admin-portal-route \ --query-string-caching-behavior IgnoreQueryString ``` ## Security Best Practices [Section titled “Security Best Practices”](#security-best-practices) 1. **Always Use Prevention Mode in Production** * Detection mode logs but doesn’t block * Only use Detection for testing/debugging 2. **Monitor WAF Logs Regularly** * Review blocked requests * Identify false positives * Tune custom rules 3. **Keep Managed Rule Sets Updated** * Azure automatically updates rule sets * Review change logs for new rules 4. **Use Geo-Restriction Wisely** * Only block countries if necessary * Consider legitimate international users * Federated IdP auth (e.g. Entra ID) may require US endpoints if the participant’s tenant home region is US 5. **Test Rate Limits** * Ensure legitimate traffic isn’t blocked * Consider API usage patterns * Adjust thresholds based on metrics 6. **Enable HTTPS Only** * Enforce HTTPS at Front Door level * Redirect HTTP → HTTPS automatically * Use TLS 1.2 minimum 7. **Review Custom Domain Setup** * Verify domain ownership * Use managed certificates * Monitor certificate expiration ## Cost Optimization [Section titled “Cost Optimization”](#cost-optimization) ### SKU Selection [Section titled “SKU Selection”](#sku-selection) * **Standard** (dev/staging): * Basic WAF features * Standard managed rules * Cost: \~$35/month base + usage * **Premium** (production): * Advanced WAF features * Private Link support * Bot protection * Cost: \~$330/month base + usage ### Usage Charges [Section titled “Usage Charges”](#usage-charges) * **Data Transfer**: $0.05/GB (first 10TB) * **Requests**: $0.0075 per 10,000 requests * **WAF Requests**: $0.00025 per request ### Cost Saving Tips [Section titled “Cost Saving Tips”](#cost-saving-tips) 1. Enable caching to reduce origin requests 2. Use Standard SKU for non-production 3. Set appropriate cache TTLs 4. Monitor and optimize request patterns ## References [Section titled “References”](#references) * [Azure Front Door Documentation](https://learn.microsoft.com/en-us/azure/frontdoor/) * [Azure WAF on Front Door](https://learn.microsoft.com/en-us/azure/web-application-firewall/afds/afds-overview) * [OWASP CRS Documentation](https://coreruleset.org/) * [Azure DDoS Protection](https://learn.microsoft.com/en-us/azure/ddos-protection/) ## Support [Section titled “Support”](#support) For issues or questions: 1. Check Azure Monitor logs 2. Review this documentation 3. Contact Azure Support 4. Consult CLAUDE.md for project-specific guidance # CTN ASR Coding Standards This document defines the coding standards for the CTN Association Register project. All code should follow these conventions for consistency and maintainability. > **Stack reminder.** The preferred stack is decided in [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md): **Quarkus + PostgreSQL + Keycloak**, Java-based, open-source, cloud-agnostic. The conventions below assume that stack. The frontend stack for the ASR portals is not yet decided (see Open Questions at the bottom); the front-end section gives only the language-agnostic rules until a framework is chosen. *** ## 1. Naming Conventions [Section titled “1. Naming Conventions”](#1-naming-conventions) ### 1.1 Database Schema (PostgreSQL) [Section titled “1.1 Database Schema (PostgreSQL)”](#11-database-schema-postgresql) **Column Names — `snake_case`** ```sql ✓ GOOD: legal_entity_id dt_created dt_modified kvk_verification_status document_uploaded_at ✗ BAD: legalEntityId dtCreated documentUploadedAt -- missing prefix when part of kvk_* group ``` **Table Names — `snake_case`, singular** ```sql ✓ GOOD: legal_entity legal_entity_contact endpoint_authorization ✗ BAD: LegalEntity legalEntityContact legal_entities -- plural ``` **Consistency Rule:** related columns share a prefix. ```sql ✓ GOOD: kvk_document_url kvk_verification_status kvk_verified_at kvk_verified_by kvk_extracted_company_name ✗ BAD: kvk_document_url verification_status -- missing kvk_ prefix document_uploaded_at -- missing kvk_ prefix ``` ### 1.2 API Routes [Section titled “1.2 API Routes”](#12-api-routes) **Path Segments — `kebab-case`** ```plaintext ✓ GOOD: /v1/legal-entities/{legalEntityId}/identifiers /v1/kvk-verification/flagged-entities /v1/participant-contacts ✗ BAD: /v1/legalentities/... -- no separation /v1/kvk_verification/... -- snake_case in URL /v1/participant/contacts -- inconsistent (missing hyphen) ``` **Path Parameters — `camelCase`** ```plaintext ✓ GOOD: /v1/legal-entities/{legalEntityId} /v1/participants/{participantId}/members/{userId} ✗ BAD: /v1/legal-entities/{legal_entity_id} -- snake_case in URL params /v1/legal-entities/{LegalEntityId} -- PascalCase ``` **Query Parameters — `snake_case`** ```plaintext ✓ GOOD: ?task_type=kvk_verification ?assigned_to=admin@ctn.nl ?include_overdue=true ✗ BAD: ?taskType=... ?assignedTo=... ``` ### 1.3 Java / Quarkus Code [Section titled “1.3 Java / Quarkus Code”](#13-java--quarkus-code) **Packages — lowercase, dotted** ```java ✓ GOOD: package nl.ctn.asr.participant; package nl.ctn.asr.identifier; package nl.ctn.asr.token; ✗ BAD: package nl.ctn.asr.Participant; package nl.CTN.asr.participant; ``` **Classes / Records / Sealed interfaces — `PascalCase`** ```java ✓ GOOD: public record LegalEntity(UUID id, String name, ...) { } public sealed interface VerificationOutcome permits Approved, Pending, Rejected { } public class ParticipantResource { } ✗ BAD: public class participantResource { } public record legal_entity(...) { } ``` **Transport DTOs — `Req` / `Resp` suffixes allowed.** Request/response records in `dto` packages may abbreviate the suffix words *Request* → `Req` and *Response* → `Resp` (`CreateOnboardingReq`, `OnboardingReqResp`). This is the **only** sanctioned abbreviation, and only for the transport-role suffix — glossary terms inside the name stay full words ([ADR-00004](../09-decisions/00004-naming-follows-glossary.md)): the entity stays `OnboardingRequest`. Rationale: avoids stutter like `OnboardingRequestResponse` where the glossary term *Request* collides with the transport suffix. ```java ✓ GOOD: public record CreateOnboardingReq(...) { } (dto package, transport suffix) public class OnboardingRequest { } (domain entity, full glossary term) ✗ BAD: public class OnboardingReq { } (domain term abbreviated) public record CreateOnboardingRqst(...) { } (unsanctioned abbreviation) ``` **Methods / Variables — `camelCase`** ```java ✓ GOOD: public Uni findByEuid(String euid) { } private final ParticipantRepository participantRepository; var legalEntityId = UUID.randomUUID(); ✗ BAD: public Uni Find_By_EUID(String EUID) { } private final ParticipantRepository ParticipantRepository; ``` **Constants — `UPPER_SNAKE_CASE`** ```java ✓ GOOD: public static final String VAD_TOKEN_AUDIENCE = "https://ctn.network"; public static final Duration KVK_CACHE_TTL = Duration.ofHours(24); ✗ BAD: public static final String vadTokenAudience = "..."; public static final Duration KvkCacheTtl = Duration.ofHours(24); ``` **File Names — match the public class / record / interface inside.** ```plaintext ✓ GOOD: ParticipantResource.java (public class ParticipantResource) LegalEntity.java (public record LegalEntity) VerificationOutcome.java (public sealed interface VerificationOutcome) ✗ BAD: participantResource.java legal-entity.java ``` ### 1.4 Records, Sealed Interfaces, Pattern Matching [Section titled “1.4 Records, Sealed Interfaces, Pattern Matching”](#14-records-sealed-interfaces-pattern-matching) Java 25+ features that the codebase actively uses: * **`record`** for DTOs, value objects, and immutable data carriers (legal entities, identifiers, claim payloads). * **`sealed interface` + `permits`** for closed enumerations of states (e.g. `VerificationOutcome`, `OnboardingState`). * **Pattern matching for `switch`** on those sealed hierarchies — exhaustiveness checked by the compiler. ```java ✓ GOOD: public sealed interface VerificationOutcome permits Approved, Pending, Rejected { } public record Approved(Instant at, String verifiedBy) implements VerificationOutcome { } public record Pending(String waitingFor) implements VerificationOutcome { } public record Rejected(String reason) implements VerificationOutcome { } return switch (outcome) { case Approved a -> Response.ok(a).build(); case Pending p -> Response.status(202).entity(p).build(); case Rejected r -> Response.status(409).entity(r).build(); }; ✗ BAD: // Don't expose mutable POJOs in REST layers public class VerificationResultDTO { public String status; public Date when; } ``` *** ## 2. Date and Timezone Standards [Section titled “2. Date and Timezone Standards”](#2-date-and-timezone-standards) ### 2.1 Database Timestamps [Section titled “2.1 Database Timestamps”](#21-database-timestamps) **ALWAYS use `TIMESTAMP WITH TIME ZONE`** ```sql ✓ GOOD: dt_created TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP kvk_verified_at TIMESTAMP WITH TIME ZONE expires_at TIMESTAMP WITH TIME ZONE ✗ BAD: created_at TIMESTAMP WITHOUT TIME ZONE -- no timezone info updated_at TIMESTAMP -- ambiguous ``` **Rationale:** `TIMESTAMP WITH TIME ZONE` stores timestamps in UTC internally and converts to/from any timezone automatically. This prevents timezone bugs. **Unix Timestamp Conversion:** ```sql ✓ GOOD: to_timestamp($1) AT TIME ZONE 'UTC' ✗ BAD: to_timestamp($1) -- assumes server timezone ``` ### 2.2 Java Date Handling [Section titled “2.2 Java Date Handling”](#22-java-date-handling) **Use `java.time.*` only. Never `java.util.Date`, `java.util.Calendar` or `Joda-Time`.** ```java ✓ GOOD: Instant createdAt = Instant.now(); OffsetDateTime expiresAt = OffsetDateTime.now(ZoneOffset.UTC).plusHours(24); LocalDate dueDate = LocalDate.of(2026, 6, 1); ✗ BAD: Date created = new Date(); // legacy, mutable, timezone-naive Calendar.getInstance(); // legacy ``` **Persistence**: store `Instant` (mapped to `TIMESTAMP WITH TIME ZONE`) for events, `LocalDate` for calendar dates without time. ### 2.3 API Date Responses [Section titled “2.3 API Date Responses”](#23-api-date-responses) **Always return dates in ISO 8601 with UTC offset (`Z`).** Jackson is configured to serialise `Instant` and `OffsetDateTime` exactly this way. ```json ✓ GOOD: { "created_at": "2026-05-29T14:30:00.000Z", "expires_at": "2026-05-30T14:30:00.000Z" } ✗ BAD: { "created_at": "2026-05-29T14:30:00", // no timezone "expires_at": "05/30/2026" // ambiguous format } ``` ### 2.4 Frontend Date Formatting [Section titled “2.4 Frontend Date Formatting”](#24-frontend-date-formatting) **The frontend MUST format dates using the user’s locale, not a hard-coded one.** Use the framework’s built-in localisation utilities (when the frontend stack is chosen; see Open Questions). ```plaintext ✓ GOOD: Format with the user's locale (resolved from i18n or browser). ✗ BAD: Hard-coded "nl-NL" or "en-GB" in display code. ``` **For relative times** (e.g. “2 hours ago”), use the platform-standard relative-time formatter. ### 2.5 Timezone Display [Section titled “2.5 Timezone Display”](#25-timezone-display) **Show timezone information for absolute timestamps:** ```plaintext ✓ GOOD: "Created: 2026-05-29 14:30 UTC" "Expires: 2026-05-30 14:30 (in 23 hours)" ✗ BAD: "Created: 2026-05-29 14:30" -- no timezone info ``` **For relative times, timezone is not needed:** ```plaintext ✓ GOOD: "Created 2 hours ago" "Expires in 23 hours" ``` *** ## 3. Error Handling Standards [Section titled “3. Error Handling Standards”](#3-error-handling-standards) ### 3.1 HTTP Status Codes [Section titled “3.1 HTTP Status Codes”](#31-http-status-codes) ```plaintext 200 OK - Successful GET / PUT 201 Created - Successful POST 204 No Content - Successful DELETE 400 Bad Request - Validation error 401 Unauthorized - Missing / invalid auth token 403 Forbidden - Valid auth but no permission 404 Not Found - Resource doesn't exist 409 Conflict - Duplicate resource 422 Unprocessable - Business-logic error 429 Too Many Req. - Rate limit exceeded 500 Server Error - Unexpected server error ``` ### 3.2 Error Response Format (RFC 9457 — Problem Details for HTTP APIs) [Section titled “3.2 Error Response Format (RFC 9457 — Problem Details for HTTP APIs)”](#32-error-response-format-rfc-9457--problem-details-for-http-apis) ```json ✓ GOOD: { "type": "https://docs.ctn.network/errors/validation", "title": "Validation failed", "status": 400, "detail": "kvk_number must be 8 digits", "instance": "/v1/legal-entities/abc-123/identifiers", "errors": [ { "field": "kvk_number", "message": "must be 8 digits" } ] } ✗ BAD: { "message": "Error" } -- not helpful ``` Quarkus exposes `RestResponse` builders; use those rather than rolling a custom body shape. *** ## 4. Security Standards [Section titled “4. Security Standards”](#4-security-standards) ### 4.1 Database Access [Section titled “4.1 Database Access”](#41-database-access) **Use Panache or parameter-bound `PreparedStatement`. Never concatenate user input into SQL.** ```java ✓ GOOD: participantRepository.find("email = ?1", email).firstResult(); em.createQuery("SELECT p FROM Participant p WHERE p.email = :email", Participant.class) .setParameter("email", email) .getSingleResult(); ✗ BAD: em.createQuery("SELECT p FROM Participant p WHERE p.email = '" + email + "'", ...); ``` ### 4.2 Secrets [Section titled “4.2 Secrets”](#42-secrets) **Never commit secrets to git.** Use the configured secret store (e.g. Azure Key Vault, AWS Secrets Manager, HashiCorp Vault) and inject through Quarkus configuration: ```java ✓ GOOD: @ConfigProperty(name = "kvk.api-key") String kvkApiKey; ✗ BAD: private static final String KVK_API_KEY = "abc123xyz"; ``` `application.properties` references the secret indirectly (e.g. `${KVK_API_KEY}`); the secret-store driver resolves it at runtime. ### 4.3 Authentication [Section titled “4.3 Authentication”](#43-authentication) Use `quarkus-oidc` as the resource-server, configured against Keycloak (see [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md), [ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md)). Endpoints carry standard `@Authenticated`, `@RolesAllowed`, and `@PermissionsAllowed` annotations. ```java ✓ GOOD: @Path("/v1/participants/{participantId}") @Authenticated public class ParticipantResource { @GET @RolesAllowed("participant-member") public Participant get(@PathParam("participantId") UUID id) { ... } } ``` ### 4.4 Object-level authorization (ownership) [Section titled “4.4 Object-level authorization (ownership)”](#44-object-level-authorization-ownership) `@RolesAllowed` is coarse role gating only. It does **not** answer *“may **this** caller touch **this** record?”* When an endpoint exposes user-owned data, the role check must be backed by a **data-dependent ownership check** that compares the caller’s identity (Keycloak `sub`) to the record’s owner. Two callers with the same role must not be able to read or mutate each other’s data. **Enforce ownership below the resource layer** (in the service/transition shell), so no endpoint — current or future — can forget the check. **Check ownership *before* state/existence**, so a non-owner cannot probe a record’s lifecycle or existence through differing error codes (a non-owner always gets `403`, never a `404`/`409` that leaks whether the record exists or what state it is in). ```java ✓ GOOD: // OnboardingTransitions — ownership before state public OnboardingRequest findOwned(UUID requestId, Actor actor) { var request = load(requestId); if (!actor.sub().equals(request.applicantUserId)) { // owner == Keycloak sub throw new SecurityException("Actor may only read an onboarding request they own"); } return request; // 403 on mismatch, before any state read } ✗ BAD: @GET @Path("{requestId}") @RolesAllowed(Roles.REQUESTOR) public OnboardingRequest get(@PathParam("requestId") UUID id) { return repository.findById(id); // any requestor reads any requestor's record } ``` **Worked example — `OnboardingRequest`.** After (anonymous) creation, only the **creator** (`applicantUserId == sub`) and the **association admin** (`ASSOCIATION_ADMIN`) may read or modify it. Admin endpoints carry `@RolesAllowed(ASSOCIATION_ADMIN)`; requestor endpoints carry `@RolesAllowed(REQUESTOR)` *and* run the `applicantUserId == actor.sub()` ownership check in the transition shell (`findOwned` / `enrich` / `resubmit`). *** ## 5. Code Organisation [Section titled “5. Code Organisation”](#5-code-organisation) ### 5.1 Project Layout (Quarkus, single module) [Section titled “5.1 Project Layout (Quarkus, single module)”](#51-project-layout-quarkus-single-module) ```plaintext asr/ ├── src/ │ ├── main/ │ │ ├── java/nl/ctn/asr/ │ │ │ ├── participant/ ← Domain package (resource + service + repo) │ │ │ │ ├── ParticipantResource.java │ │ │ │ ├── ParticipantService.java │ │ │ │ ├── Participant.java (record) │ │ │ │ └── ParticipantRepository.java │ │ │ ├── identifier/ │ │ │ ├── token/ │ │ │ └── endpoint/ │ │ └── resources/ │ │ ├── application.properties │ │ └── db/changelog/ ← Liquibase changelogs (YAML) │ └── test/ │ └── java/nl/ctn/asr/ │ ├── participant/ParticipantResourceTest.java │ └── ... └── pom.xml ``` The package layout is **domain-first** (`participant`, `identifier`, `token`, `endpoint`), not layer-first (no `controllers/`, `services/`, `repositories/` top-level packages). ### 5.2 Module seam pattern (ADR-00022) [Section titled “5.2 Module seam pattern (ADR-00022)”](#52-module-seam-pattern-adr-00022) Each top-level business package under `org.bdinetwork.asr` (`audit`, `document`, `verification`, `onboarding`) is a **module** that exposes a small, explicit **seam** and hides everything else. Because the code is a Quarkus modulith (no Spring Modulith) and Java access control cannot express “module-internal across sub-packages” — CDI proxying and JAX-RS resources force types to be `public` — the boundary is defined by **package placement** and enforced by an **ArchUnit allow-list** (`ModuleBoundaryTest`), not the compiler. See [ADR-00022](../09-decisions/00022-module-seam-pattern.md) for the full decision; this section is the working digest. **Audit is the pilot**; `document`/`verification`/`onboarding` migrate opportunistically as they are touched. A module has three package zones: ```plaintext audit/ ← SEAM: root package AuditRecorder (service interface — callers inject this, never the impl) Action (seam enum — root, NOT dto/) dto/ ← SEAM (lazy): the interface's own record types exceptions/ ← SEAM (lazy): seam exception types rest/ ← HTTP boundary — sibling of internal, NOT seam, NOT injectable AuditRecordResource dto/AuditRecordResp (+ from(entity) mapper — HTTP-only wire types) internal/ ← hidden implementation (subtree; add packages freely) AuditRecorderImpl (public, but ArchUnit-hidden) AuditRecordQueryService persistence/ ← internal.persistence: JPA entities + repositories AuditRecord AuditRecordRepository ``` Rules: * **Seam = module root + `dto` + `exceptions`.** Nothing else is reachable from other modules. Cross-module communication goes through the root **service interface** (+ any published events); callers inject the interface, never the impl. Create `dto`/`exceptions` **lazily**, only when a seam type needs them. * **`dto/` holds the interface’s own types**, each a `record` (per `ArchitectureTest.dtos_should_be_records`). Enums and other non-record seam value types live at the **root**, not `dto/`. * **`rest/` is a top-level sibling of `internal`, not part of the seam** (a deliberate deviation from the source spec’s `internal.rest`): a module’s HTTP surface is externally significant, so it is visible at a glance — but the allow-list still forbids sibling modules from injecting it. HTTP-only wire types live under `rest/` (e.g. `rest/dto`), never the seam `dto/`. A resource may freely consume its own module’s `internal` (same-module access is unrestricted). * **`internal/` is a subtree.** Add hidden packages under `internal.*` freely (fail-safe default). Persistence — JPA entities + repositories — lives in `internal.persistence`. Never create hidden packages as seam siblings other than `rest/`. * **`model`/`repository` parking packages dissolve** into each module’s `internal.persistence` as modules migrate. * **Tests are exempt** (`ImportOption.DoNotIncludeTests`): same-module tests may touch `internal`; cross-module/e2e tests go through HTTP (RestAssured), not Java imports. **Watch the Javadoc `{@link}` trap.** ArchUnit reads bytecode, so a `{@link some.other.module.internal.persistence.Entity}` — and the `import` it needs — passes the boundary test while still coupling your source to another module’s internals. Reference the **seam** type (or plain prose) in cross-module Javadoc, and keep no `import` of another module’s `internal..` in production code. ### 5.3 Import Order [Section titled “5.3 Import Order”](#53-import-order) Java import grouping (IntelliJ / `google-java-format` default): ```java // 1. Standard library import java.time.Instant; import java.util.UUID; // 2. Third-party / Quarkus / Jakarta import jakarta.ws.rs.GET; import jakarta.ws.rs.Path; import io.quarkus.security.Authenticated; import org.eclipse.microprofile.config.inject.ConfigProperty; // 3. Project import nl.ctn.asr.participant.Participant; ``` No wildcard imports. No unused imports. Static imports only for test assertions and obvious helpers (e.g. `Assertions.assertThat`). *** ## 6. Documentation Standards [Section titled “6. Documentation Standards”](#6-documentation-standards) ### 6.1 Public API documentation [Section titled “6.1 Public API documentation”](#61-public-api-documentation) Every public REST method has a Javadoc comment that explains *what* and *why* (not just the signature): ```java /** * Creates a new legal-entity identifier (KVK, LEI, EORI, …) for the * given participant. Idempotent on (participant, type, value). * * @param participantId participant the identifier belongs to * @param request identifier details (type, value, country) * @return 201 Created with the persisted identifier * @throws ValidationException if the value does not match the type's format * @throws ConflictException if the identifier already exists for this participant */ @POST @Path("/{participantId}/identifiers") public RestResponse create( @PathParam("participantId") UUID participantId, @Valid CreateIdentifierRequest request) { ... } ``` OpenAPI is generated from the resource classes (`quarkus-smallrye-openapi`) and is the contract for consumers. Because the generated spec *is* the consumer-facing documentation, every endpoint carries OpenAPI annotations: * `@Operation(summary = …, description = …)` on every endpoint method. The description explains the endpoint’s behaviour and intent in current, accurate terms — not a restated method name. Update it whenever behaviour changes; a stale description is a review finding, same as a wrong comment. * `@APIResponse` for each status code the endpoint can produce (success plus the error statuses from §3), with typed `@Schema` content. * `@Parameter` / `@RequestBody` descriptions on inputs whose meaning or format is not obvious from the name and type. * **Examples** — `@ExampleObject` on request/response content, or `example` on `@Schema` fields — unless an example is unreasonable in the given context (trivial primitives, binary payloads, pure pass-through wrappers). * `@Tag` on each resource class so endpoints group sensibly in the spec. ```java @POST @Path("/{participantId}/identifiers") @Operation(summary = "Create a legal-entity identifier", description = "Registers a KVK, LEI or EORI identifier for the participant. " + "Idempotent on (participant, type, value).") @APIResponse(responseCode = "201", description = "Identifier created", content = @Content(schema = @Schema(implementation = Identifier.class), examples = @ExampleObject(name = "kvk", value = """ {"type": "KVK", "value": "12345678", "country": "NL"}"""))) @APIResponse(responseCode = "409", description = "Identifier already exists for this participant") public RestResponse create(...) { ... } ``` ### 6.2 Complex Logic Comments [Section titled “6.2 Complex Logic Comments”](#62-complex-logic-comments) ```java // Calculate expiry date: 24 hours from issuance for VAD tokens Instant expiresAt = Instant.now().plus(Duration.ofHours(24)); // KvK numbers must be exactly 8 digits (Dutch Chamber of Commerce standard) private static final Pattern KVK_REGEX = Pattern.compile("^\\d{8}$"); ``` *** ## 7. Testing Standards [Section titled “7. Testing Standards”](#7-testing-standards) ### 7.1 Unit + Integration Tests (JUnit 5 + RestAssured) [Section titled “7.1 Unit + Integration Tests (JUnit 5 + RestAssured)”](#71-unit--integration-tests-junit-5--restassured) * **Unit tests**: pure Java with JUnit 5; mock external collaborators via Mockito where unavoidable. * **Integration tests**: `@QuarkusTest` with the in-memory database (Dev Services for PostgreSQL via Testcontainers). * Aim for **>70% line coverage** on business logic packages; coverage gates enforced in CI. ```java @QuarkusTest class ParticipantResourceTest { @Test void create_returns_201() { given() .contentType(JSON) .body(new CreateParticipantRequest("Acme BV", "12345678")) .when() .post("/v1/participants") .then() .statusCode(201) .body("name", equalTo("Acme BV")); } } ``` ### 7.2 Authorization tests (mandatory for owner-scoped endpoints) [Section titled “7.2 Authorization tests (mandatory for owner-scoped endpoints)”](#72-authorization-tests-mandatory-for-owner-scoped-endpoints) **Every endpoint that exposes user-owned data MUST have a test proving an unauthorized user cannot reach it.** A happy-path test (owner gets `200`) is not enough — it never exercises the deny path, so a missing or broken ownership check passes CI silently. The ownership rule of §4.4 is only real if a test defends it. For each owner-scoped read/modify endpoint, assert at minimum: * **Owner allowed** — the owning caller succeeds (`200`/`204`). * **Same-role non-owner denied** — a *different* caller with the *same* role is rejected with `403` (this is the test that catches missing object-level checks). * **Deny before leak** — where ordering matters (§4.4), the non-owner gets `403` even when the record is in a state that would otherwise yield `404`/`409`, so existence/state is not leaked. ```java ✓ GOOD: @Test @TestSecurity(user = "stranger-read", roles = Roles.REQUESTOR) void get_asNonOwningRequestor_returns403() { var requestId = seedOwned("33545678", UUID.randomUUID()); // owned by someone else given().when().get(PATH + "/" + requestId).then().statusCode(403); } @Test void resubmit_byNonOwnerOfNonChangesRequested_throwsSecurityExceptionNotIllegalTransition() { // ownership check fires before the state check → 403, not 409 (no state leak) } ✗ BAD: @Test void get_returns200() { // only the owner path — deny path never tested given().when().get(PATH + "/" + ownRequestId).then().statusCode(200); } ``` These tests live next to the resource/shell they protect (e.g. `OnboardingRequestResourceTest`, `OnboardingTransitionsTest`) and carry `@Tag("requires-database")` when they touch the DB. ### 7.3 Contract Tests [Section titled “7.3 Contract Tests”](#73-contract-tests) OpenAPI is the contract. Generate clients from the spec; the API project’s CI compares the generated OpenAPI against the committed `openapi.yaml` and fails on incompatible changes. ### 7.4 End-to-End Tests (UI) [Section titled “7.4 End-to-End Tests (UI)”](#74-end-to-end-tests-ui) The chosen browser E2E framework is **Playwright for Java, driving Chromium**, decided in [ADR-00015](../09-decisions/00015-e2e-playwright-chromium-failsafe.md). It is a **test-scoped, pinned** Maven dependency bound to **Maven Failsafe**: E2E `*IT` classes carry `@Tag("requires-database")` and run under `./mvnw verify` on a Docker host alongside the DB integration tests; a Docker-less run (`-DskipITs=true`) activates the `no-docker` profile and reports them **SKIPPED**, never green (mirroring the DB/IT gating). The E2E suite runs **after** API contract tests pass. This names the **E2E test tooling only**; it does **not** decide the ASR portal frontend stack (still open — see Open Questions). The current E2E coverage drives the throwaway `asr-demo-frontend` static demo, not a portal SPA. *** ## 8. Commit Message Standards [Section titled “8. Commit Message Standards”](#8-commit-message-standards) Follow Conventional Commits: ```plaintext feat: add KvK verification review workflow fix: correct timezone handling in date display docs: update API documentation for identifiers refactor: standardise naming across participant package test: add E2E tests for participant creation chore: bump Quarkus to 3.35.0 ``` **Format:** ```plaintext (): ``` Scope is optional but useful (`feat(participant): …`, `fix(token): …`). *** ## 9. Summary Checklist [Section titled “9. Summary Checklist”](#9-summary-checklist) When writing code, verify: * [ ] Database columns use `snake_case` with consistent prefixes. * [ ] Database timestamps use `TIMESTAMP WITH TIME ZONE`. * [ ] API routes use `kebab-case` for paths; `camelCase` for path params; `snake_case` for query params. * [ ] Java classes/records/interfaces use `PascalCase`; methods/variables `camelCase`; constants `UPPER_SNAKE_CASE`. * [ ] DTOs are immutable (`record`); state machines are `sealed interface`s. * [ ] Dates use `java.time.*`; API serialises as ISO 8601 with `Z`. * [ ] SQL is parameter-bound (Panache or `@Parameter`); no string concatenation of user input. * [ ] No secrets committed; secrets resolved via Quarkus configuration from the secret store. * [ ] Endpoints carry `@Authenticated` / `@RolesAllowed` per ADR-00008. * [ ] Owner-scoped endpoints back the role check with a data-dependent ownership check (§4.4), enforced below the resource layer and evaluated before state/existence. * [ ] Every owner-scoped endpoint has a test proving a same-role non-owner is rejected with `403` (§7.2), not just an owner happy-path test. * [ ] Cross-module access touches only the seam (module root + `dto` + `exceptions`); no production `import` of another module’s `internal..`, not even in a `{@link}` (§5.2, ADR-00022). * [ ] Error responses follow RFC 9457 (Problem Details). * [ ] Public REST methods have Javadoc explaining behaviour and contracts. * [ ] OpenAPI passes the CI compatibility check. * [ ] Test coverage on the touched package is at or above the 70% threshold. *** ## Open Questions [Section titled “Open Questions”](#open-questions) * **Frontend stack**: the ASR portals (admin, participant, data-consumer) need a chosen frontend stack. Candidates discussed include the existing React + TypeScript prototype, a server-rendered Quarkus + Qute approach, or a fully framework-agnostic web-component approach. A separate ADR is expected. * **OpenAPI versioning**: how do we evolve `/v1` → `/v2`? Deprecation policy and concurrent-version support remain TBD. *** **Questions or clarifications?** See [AGENTS.md](../../../AGENTS.md) or ask in project discussions. # Cross-Cutting Concepts *Technical concepts and patterns applied across the Connected Trade Network* ## Security Concepts [Section titled “Security Concepts”](#security-concepts) ### Authentication and Authorization [Section titled “Authentication and Authorization”](#authentication-and-authorization) **Authentication Methods** (per [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md) and [ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md)): ```plaintext Machine-to-Machine (M2M): ├── Keycloak OAuth 2.0 Client Credentials │ ├── Self-hosted IdP (the chosen one) │ ├── RS256 JWT signing │ ├── JWKS-based validation │ └── Participant-scoped authorisation (Keycloak Organizations) └── mTLS (optional high security) User Authentication (single IdP, outward federation per participant): ├── Keycloak realm `ctn` │ ├── Federation → Entra ID (per participant, OIDC) │ ├── Federation → eHerkenning (SAML→OIDC, for NL admin-representation) │ └── Local Keycloak account (email + password + MFA) for participants without an own IdP └── MFA enforced on admin surfaces ``` **Authorisation Pattern (2026) — VAD-only**: ```plaintext Request Authorisation Flow: 1. Validate VAD token (participation proof) against ASR JWKS 2. Apply provider's local policy 3. Make authorisation decision 4. Audit the decision HTTP Headers: Authorization: Bearer X-Correlation-Id: ``` **Keycloak password policy (local accounts).** Local Keycloak accounts enforce a password policy aligned with NIST SP 800-63B: minimum length 12, at least one each of upper/lower/digit/special, password history 10, max length 128 (DoS guard), PBKDF2-SHA256 ≥210 000 iterations, and **no** periodic forced expiry (change only on suspected breach). Federated users (Entra ID, eHerkenning) are governed by their own IdP’s policy. ### Token Design Patterns [Section titled “Token Design Patterns”](#token-design-patterns) **VAD Token Structure**: ```json { "iss": "https://association.ctn.network", "sub": "org-client-id", "aud": "https://ctn.network", "iat": 1704063600, "exp": 1704150000, "jti": "unique-token-id", "https://schemas.ctn.network/claims/legal_entity/name": "Transport Corp", "https://schemas.ctn.network/claims/legal_entity/kvk": "12345678", "https://schemas.ctn.network/claims/compliance/verified": true, "https://schemas.ctn.network/claims/participation/status": "active", "https://schemas.ctn.network/claims/identity_assurance_level": "substantial", "https://schemas.ctn.network/claims/participation/valid_until": "2026-12-31" } ``` **M2M Token Structure (Keycloak)**: ```json { "iss": "https://keycloak.ctn.network/realms/ctn", "sub": "service-account-acme-tms-m2m", "aud": ["ctn-api"], "exp": 1730906400, "iat": 1730902800, "azp": "acme-tms-m2m", "client_id": "acme-tms-m2m", "scope": "openid api.access containers.read", "organization": { "acme-bv": { "id": "11111111-2222-3333-4444-555555555555" } }, "realm_access": { "roles": ["participant-member"] } } ``` **M2M Authentication Flow** (full detail in [Keycloak M2M Authentication](../05-building-blocks/ctn-keycloak-m2m-auth.md)): ```plaintext 1. M2M Client → Keycloak: POST /realms/ctn/protocol/openid-connect/token - grant_type=client_credentials - client_id + client_secret (Basic Auth) — or client_assertion JWT 2. Keycloak → M2M Client: JWT access token (RS256, ~1h TTL) 3. M2M Client → API: HTTP request, Authorization: Bearer 4. API (quarkus-oidc) → Keycloak: Retrieve JWKS (cached ~10 minutes) 5. API: Validate JWT signature, iss, aud, exp, nbf, scopes 6. API → ASR DB: Resolve participant_id from token's organization claim 7. API: Apply participant-scoped policy, serve the request ``` ### Encryption Standards [Section titled “Encryption Standards”](#encryption-standards) **Data Protection**: | Context | Method | Key Management | | ---------------- | ------------------------ | -------------------- | | Data at Rest | AES-256-GCM | Azure Key Vault | | Data in Transit | TLS 1.3 | Managed certificates | | Token Signing | RS256 (RSA + SHA-256) | HSM-backed keys | | Sensitive Fields | Field-level encryption | Application keys | | Backups | Azure Storage encryption | Microsoft-managed | ## Development Concepts [Section titled “Development Concepts”](#development-concepts) ### API Design Standards [Section titled “API Design Standards”](#api-design-standards) **RESTful Principles**: ```yaml URL Structure: Pattern: /{version}/{resource}/{id}/{sub-resource} Example: /v1/participants/12345/endpoints HTTP Methods: GET: Retrieve resource(s) POST: Create new resource PUT: Full update (replace) PATCH: Partial update DELETE: Remove resource Status Codes: 200: Success 201: Created 204: No Content 400: Bad Request 401: Unauthorized 403: Forbidden 404: Not Found 409: Conflict 429: Too Many Requests 500: Internal Server Error ``` **REST Maturity - Level 3 (HATEOAS)**: CTN implements REST Level 3, using hypermedia controls to guide API consumers through available operations. This means API responses include `_links` that tell clients what actions are possible based on the current resource state. > 📖 A separate REST Level 3 / HATEOAS explainer is planned; until then the *Quick Example* below is the canonical reference. **Quick Example**: ```json { "memberId": "ITG001", "status": "active", "_links": { "self": { "href": "/v1/participants/ITG001" }, "requestVadToken": { "href": "/v1/participants/ITG001/tokens/vad", "method": "POST" }, "updateProfile": { "href": "/v1/participants/ITG001", "method": "PATCH" } } } ``` **Benefits**: * Self-documenting APIs * Workflow guidance through state-dependent links * Loose coupling (clients follow links, don’t construct URLs) * Graceful evolution without breaking clients **OpenAPI Specification**: ```yaml openapi: 3.0.3 info: title: CTN Association Register API version: 1.0.0 paths: /v1/participants/{participantId}/endpoints: post: summary: Register a new data endpoint for a participant parameters: - name: participantId in: path required: true schema: { type: string, format: uuid } requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/RegisterEndpointRequest' responses: 201: description: Endpoint registered headers: Location: schema: { type: string } description: URI of the created endpoint ``` ### Rate Limiting [Section titled “Rate Limiting”](#rate-limiting) Rate limiting is applied in two layers. The **edge layer** (Azure Front Door / WAF, see [Front Door with WAF](../07-deployment/ctn-front-door-waf-infrastructure.md)) absorbs volumetric abuse before it reaches the API. The **application layer** enforces finer, purpose-specific limits per caller. Application-layer keys: authenticated requests are limited per caller identity (`participant`/`user`), anonymous requests per source IP. Limits are tiered by endpoint sensitivity: | Tier | Limit (reference) | Applies to | | ----------- | -------------------------- | ------------------------------------ | | API | 100 req / min | General read/write endpoints | | Auth | 10 req / min | Interactive authentication endpoints | | Token | 5 req / hour | Token issuance | | Failed auth | 5 attempts / hour (per IP) | Repeated authentication failures | | Upload | 20 req / hour | File-upload endpoints | Failed authentication attempts carry an extra penalty so brute-force attempts exhaust their budget faster than the nominal limit suggests. Responses advertise the budget via standard headers ([IETF RateLimit headers](https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-ratelimit-headers)): `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`. On exceedance the API returns **429** with `Retry-After` and an RFC 9457 problem body: ```json { "type": "https://schemas.ctn.network/problems/rate-limit-exceeded", "title": "Too Many Requests", "status": 429, "detail": "Rate limit exceeded; retry after 60 seconds", "retry_after": 60 } ``` Consumers must honour `Retry-After` and back off exponentially. Limits are configurable per environment (relaxed in non-production). In-process counters do not share state across instances, so a multi-instance deployment uses a shared backing store (e.g. Redis) or delegates distributed limiting to the API gateway. ### Error Handling [Section titled “Error Handling”](#error-handling) **Standard Error Response**: ```json { "error": { "code": "CTN_AUTH_002", "message": "VAD token validation failed", "details": { "reason": "Token expired", "expired_at": "2026-05-29T10:00:00Z" }, "timestamp": "2026-05-29T11:00:00Z", "correlation_id": "req-abc-123", "documentation": "https://docs.ctn.network/errors/CTN_AUTH_002" } } ``` **Error Categories**: | Code Range | Category | Examples | | -------------- | -------------- | ------------------------------------ | | CTN\_VAL\_\* | Validation | Invalid input, missing fields | | CTN\_AUTH\_\* | Authentication | Invalid token, expired | | CTN\_AUTHZ\_\* | Authorization | Insufficient permissions | | CTN\_BUS\_\* | Business Logic | Invalid state, rule violation | | CTN\_INT\_\* | Integration | External service failure | | CTN\_SYS\_\* | System | Internal error, resource unavailable | ### Logging and Monitoring [Section titled “Logging and Monitoring”](#logging-and-monitoring) **Structured Logging**: ```json { "timestamp": "2026-05-29T10:30:45.123Z", "level": "INFO", "correlation_id": "req-abc-123", "service": "association-register", "operation": "registerEndpoint", "duration_ms": 145, "user_id": "user-456", "participant_id": "PARTICIPANT-789", "endpoint_id": "ENDPOINT-012", "result": "SUCCESS", "metadata": { "participation_status": "active", "verification_level": "FULL" } } ``` **Logging Levels**: * **ERROR**: System errors, failures requiring attention * **WARN**: Degraded performance, potential issues * **INFO**: Normal operations, key business events * **DEBUG**: Detailed diagnostic information **Sensitive data — never log.** Logs are an exfiltration surface and fall under GDPR. The following must never appear in log output, in any field: * Passwords and password hashes * Full API tokens or JWTs (log at most a short prefix, e.g. first 8 chars + `…`) * Client secrets and signing keys * Full personal data (BSN/SSN, payment details, health data) Safe to log: participant IDs and user UUIDs (not emails in security contexts), token prefixes, hashed values, IP addresses (for security monitoring) and user-agent strings. PII that must be correlated is masked or tokenised before logging. **Security-event logging.** Authentication and authorisation outcomes are logged as distinct, queryable security events — not folded into generic request logs — so they can drive alerting: * **Authentication failure** — reason (e.g. invalid signature, expired token), source IP, correlation ID. No token contents. * **Authorisation failure** — subject, required vs. granted scopes/roles, the resource, correlation ID. * **Suspicious activity** — repeated failures from one source, token reuse from multiple IPs, anomalous request rates. ```json { "timestamp": "2026-05-29T10:31:02.500Z", "level": "WARN", "event": "security.authn.failure", "correlation_id": "req-abc-123", "reason": "invalid_token_signature", "ip": "203.0.113.7" } ``` These events feed the audit pipeline and the alerting rules in the [Deployment View](../07-deployment/ctn-deployment.md). **Metrics Collection**: ```plaintext Business Metrics: ├── participants.onboarded (counter) ├── endpoints.registered (counter) ├── vad.tokens.issued (counter) ├── verifications.completed (counter) └── re_verifications.scheduled (counter) Technical Metrics: ├── api.response.time (histogram) ├── database.query.time (histogram) ├── cache.hit.ratio (gauge) ├── external_api.latency (histogram, per provider: KvK / KBO / VIES / GLEIF) └── error.rate (counter) ``` ## Architectural Patterns [Section titled “Architectural Patterns”](#architectural-patterns) ### Caching Strategy [Section titled “Caching Strategy”](#caching-strategy) **Multi-Level Cache**: ```plaintext Level 1: CDN Cache ├── Static content (Portal assets) ├── Public API responses └── TTL: 24 hours Level 2: Consumer-side JWKS cache ├── Public keys of the ASR signing certificate ├── Refreshed on key-rotation events └── TTL: 10 minutes Level 3: Application Cache (in-process) ├── KvK / KBO / VIES / GLEIF lookup results ├── Reference data └── TTL: 5 minutes (lookups), 24 hours (reference) ``` **Cache Invalidation**: * Event-driven invalidation * TTL-based expiration * Manual purge capabilities * Cache-aside pattern for consistency ### Event-Driven Patterns [Section titled “Event-Driven Patterns”](#event-driven-patterns) **Event Schema** (CloudEvents): ```json { "specversion": "1.0", "type": "org.ctn.participant.verified", "source": "/association-register", "id": "evt-123", "time": "2026-05-29T10:30:00Z", "datacontenttype": "application/json", "data": { "participant_id": "PARTICIPANT-789", "verification_source": "KvK", "verified_at": "2026-05-29T10:29:55Z" } } ``` **Event Topics**: ```plaintext Participation Events: ├── participant.registered ├── participant.verified ├── participant.re_verification_due ├── participant.suspended └── participant.offboarded Endpoint Events: ├── endpoint.registered ├── endpoint.suspended └── endpoint.retired Authorisation Events (at data provider, optional): ├── access.granted └── access.denied ``` ### Resilience Patterns [Section titled “Resilience Patterns”](#resilience-patterns) **Circuit Breaker** (Quarkus / SmallRye Fault Tolerance): ```java @CircuitBreaker( requestVolumeThreshold = 10, failureRatio = 0.5, delay = 30, delayUnit = ChronoUnit.SECONDS, successThreshold = 5 ) public Uni verifyKvk(String kvkNumber) { return kvkClient.basisprofiel(kvkNumber); } ``` **Retry Policy**: ```yaml Retry Configuration: MaxAttempts: 3 BackoffStrategy: Exponential InitialDelay: 1s MaxDelay: 30s RetryableErrors: - NetworkError - TimeoutError - ServiceUnavailable ``` ## Data Management [Section titled “Data Management”](#data-management) ### Data Governance [Section titled “Data Governance”](#data-governance) **Data Classification**: | Level | Description | Examples | Protection | | ------------ | ------------------- | -------------------------------------- | ------------------ | | Public | Open information | API docs, schemas | None required | | Internal | Business operations | Participant records, endpoint metadata | Access control | | Confidential | Sensitive business | Pricing, contracts | Encryption + audit | | Restricted | Regulated data | Personal data, UBO | Full protection | **Data Retention**: ```plaintext Retention Policies: ├── Operational Data: 90 days active + 1 year archive ├── Audit Logs: 7 years (regulatory requirement) ├── Tokens: Duration of validity + 30 days ├── Personal Data: As per GDPR requirements └── Backups: 30 days rolling window ``` ### Data Quality [Section titled “Data Quality”](#data-quality) **Validation Rules**: * Input validation at API gateway * Schema validation for events * Business rule validation in services * Referential integrity in database **Data Consistency**: * ACID transactions for critical operations * Eventual consistency for distributed data * Idempotency keys for duplicate prevention * Compensating transactions for rollbacks ## Performance Optimization [Section titled “Performance Optimization”](#performance-optimization) ### Database Optimization [Section titled “Database Optimization”](#database-optimization) ```sql -- Indexes for common queries CREATE INDEX idx_participant_status ON participant(kvk, status) WHERE status = 'ACTIVE'; CREATE INDEX idx_endpoint_participant ON endpoint(participant_id); CREATE INDEX idx_identifier_lookup ON legal_entity_identifier(identifier_type, identifier_value); -- Partitioning for large tables (yearly) CREATE TABLE audit_log_2026 PARTITION OF audit_log FOR VALUES FROM ('2026-01-01') TO ('2027-01-01'); ``` ### API Performance [Section titled “API Performance”](#api-performance) * Response compression (gzip) * Pagination for large datasets * Field filtering (?fields=id,name) * ETag support for caching * Connection pooling ## Compliance and Audit [Section titled “Compliance and Audit”](#compliance-and-audit) ### Audit Requirements [Section titled “Audit Requirements”](#audit-requirements) ```json { "audit_record": { "id": "audit-123", "timestamp": "2026-05-29T10:30:00Z", "actor": { "id": "user-456", "type": "SERVICE_ACCOUNT", "ip_address": "10.0.2.15" }, "action": "DATA_ACCESS", "resource": { "type": "CONTAINER_ETA", "id": "CNT-ABC-123" }, "authorisation": { "vad_token_id": "token-789", "decision": "ALLOW" }, "metadata": { "correlation_id": "req-abc-123", "session_id": "sess-def-456" } } } ``` ### GDPR Compliance [Section titled “GDPR Compliance”](#gdpr-compliance) * Right to access (data export) * Right to rectification (data updates) * Right to erasure (data deletion) * Data portability (standard formats) * Privacy by design principles *** *These cross-cutting concepts ensure consistency and quality across all CTN components.* # NIS2 Compliance — Architecture Mapping *How the Connected Trade Network architecture maps to NIS2 Directive obligations, and what gaps remain.* ## Why this document exists [Section titled “Why this document exists”](#why-this-document-exists) The NIS2 Directive (EU 2022/2555) replaces NIS1 and significantly tightens cybersecurity requirements for “essential” and “important” entities across the EU. In the Netherlands it is implemented through the **Cyberbeveiligingswet (Cbw)**, in force since mid-2025. CTN is touched by NIS2 on multiple layers simultaneously: * **Most CTN participants** (logistics service providers, freight forwarders, shipping lines, terminal operators) are themselves NIS2-obliged entities under the Transport sector. * **DIL as the orchestrator of CTN** may itself fall under “Digital Infrastructure” / “ICT service management” as a B2B platform operator running shared sectoral infrastructure. * **Several connected systems** (KvK, eHerkenning, sectoral data spaces) are operated by entities that are themselves NIS2-obliged. Even where DIL is not directly NIS2-obliged, participants must perform **supply-chain security assessments** on CTN under Article 21(2)(d). CTN must therefore be designed and operated such that participants can confidently include CTN in their own NIS2 attestations. ## Scope and key obligations [Section titled “Scope and key obligations”](#scope-and-key-obligations) NIS2 imposes ten categories of risk-management measures (Article 21(2)) that registered entities must implement, plus governance, reporting, and supervision obligations. The ten measures are: | # | Article 21(2) measure | CTN relevance | | - | ---------------------------------------------------- | ---------------------------------------------------------- | | a | Risk analysis & information system security policies | CTN architecture risk register, security policies | | b | Incident handling | CTN-wide incident response, CSIRT-DSP coordination | | c | Business continuity & crisis management | RTO/RPO targets, DR plan, backup strategy | | d | **Supply-chain security** | Strongest CTN driver — participants assess CTN as supplier | | e | Security in acquisition, development, maintenance | Secure SDLC, code review, dependency management | | f | Policies for assessing effectiveness | Security testing, audit cadence, KPIs | | g | Cyber hygiene & training | Operations team training, secure ops practices | | h | Cryptography policies | TLS 1.3, key management, rotation, JWS algorithms | | i | Human resources, access control, asset management | RBAC, least privilege, joiner/mover/leaver flow | | j | MFA, secured comms, secured emergency comms | M2M auth, MFA on admin surfaces, secure runbooks | In addition: **24-hour early warning** to CSIRT, **72-hour incident notification**, **one-month final report**; mandatory **registration** with the competent authority (RDI in NL); **management body accountability** with personal liability and training obligations. Penalties: up to **€10M or 2% of worldwide turnover** for essential entities, **€7M or 1.4%** for important entities. ## Architecture mapping — Article 21(2) measures [Section titled “Architecture mapping — Article 21(2) measures”](#architecture-mapping--article-212-measures) ### (a) Risk analysis and information security policies [Section titled “(a) Risk analysis and information security policies”](#a-risk-analysis-and-information-security-policies) | Aspect | Status | Reference | | ---------------------------------------- | ------------ | -------------------------------------------------------- | | Architecture-level risk register | ✅ Maintained | [11-risks](../11-risks/ctn-risks.md) | | Security architecture documentation | ✅ Published | [08-crosscutting](ctn-crosscutting.md#security-concepts) | | Information Security Policy (DIL-level) | ⚠️ Gap | DIL to publish; CTN inherits | | Acceptable Use / participant obligations | ⚠️ Gap | Participation agreement to extend | **Gap actions**: DIL-level Information Security Policy needs to be published and referenced from this document. Participant obligations regarding security must be made part of the CTN participation agreement. ### (b) Incident handling [Section titled “(b) Incident handling”](#b-incident-handling) | Aspect | Status | Reference | | ------------------------------------------------ | ------------ | -------------------------------- | | Centralised logging (auth, authorisation, admin) | 🚧 In design | Azure Monitor / Sentinel target | | Detection rules for anomalous token use | 🚧 In design | Sentinel analytics rules | | Incident response runbook (24h/72h/1M) | ⚠️ Gap | Owner: DIL operations | | CSIRT-DSP communication template | ⚠️ Gap | Aligned with NCSC-NL guidance | | Participant notification mechanism | ⚠️ Gap | Status page + email distribution | **Gap actions**: Document the 24h/72h/1M reporting flow (who decides, who signs, who notifies); set up the CSIRT-DSP contact path; define participant notification channels. ### (c) Business continuity and crisis management [Section titled “(c) Business continuity and crisis management”](#c-business-continuity-and-crisis-management) | Aspect | Status | Reference | | --------------------------------------------- | ---------- | ---------------------------------------------------- | | RTO/RPO targets per component | ⚠️ Gap | To define: ASR, IdP | | Backup strategy (PostgreSQL, config, secrets) | 🚧 Partial | Azure Flexible Server backups; Key Vault soft-delete | | Multi-region / failover plan | ⚠️ Gap | Active/passive design needed | | DR test cadence | ⚠️ Gap | Annual full DR test recommended | | Crisis communication plan | ⚠️ Gap | Together with (b) | **Gap actions**: Define RTO/RPO per critical component, document Azure region-failover design, schedule the first DR test. ### (d) Supply-chain security — strongest driver [Section titled “(d) Supply-chain security — strongest driver”](#d-supply-chain-security--strongest-driver) CTN’s role here is dual: CTN is supplier to its participants, and CTN itself depends on suppliers (Azure, KvK, KBO, VIES, GLEIF, eHerkenning, identity providers, etc.). **As supplier to participants:** | Aspect | Status | Reference | | ---------------------------------------------- | ------------ | ------------------------------------ | | ISO 27001 / SOC 2 attestation | ⚠️ Gap | Roadmap item for DIL | | Penetration test reports | ⚠️ Gap | Annual pentest recommended | | SBOM for ASR containers | ⚠️ Gap | Generate via Syft in CI/CD | | Vulnerability disclosure policy (security.txt) | ⚠️ Gap | Publish on every CTN-operated portal | | Audit log accessibility for participants | 🚧 In design | Per-participant audit trail export | **As consumer of suppliers:** | Aspect | Status | Reference | | -------------------------------------------------- | ---------- | --------------------------- | | Supplier inventory with criticality assessment | ⚠️ Gap | Owner: DIL operations | | Contractual security requirements (DPA, SLA) | 🚧 Partial | Azure: enterprise agreement | | Continuous monitoring of supplier security posture | ⚠️ Gap | Periodic review process | **Gap actions**: Pursue ISO 27001 certification, publish CVD policy with `security.txt`, establish SBOM generation in CI/CD, create supplier inventory. ### (e) Security in acquisition, development, and maintenance [Section titled “(e) Security in acquisition, development, and maintenance”](#e-security-in-acquisition-development-and-maintenance) | Aspect | Status | Reference | | --------------------------------- | ------------ | ------------------------------------------------------------------------------------------ | | Secure SDLC defined | 🚧 Partial | Code review, branch protection in GitHub | | Dependency scanning in CI/CD | 🚧 Partial | Maven dependency audit; expand to full SCA (e.g. OWASP Dependency-Check, Dependabot, Snyk) | | Container image scanning | ⚠️ Gap | Trivy or Defender for Containers | | Patching SLA for runtime / images | ⚠️ Gap | Define max-age per severity | | Secure coding standards | ✅ Documented | [08-crosscutting/ctn-coding-standards](ctn-coding-standards.md) | **Gap actions**: Add container image scanning to CI/CD, define and document patching SLA, expand dependency scanning to SCA-grade tooling. ### (f) Policies for assessing effectiveness [Section titled “(f) Policies for assessing effectiveness”](#f-policies-for-assessing-effectiveness) | Aspect | Status | Reference | | -------------------------------- | ------ | ------------------------------------ | | Security KPIs / metrics | ⚠️ Gap | Define: MTTR, patching latency, etc. | | Internal audit cadence | ⚠️ Gap | Annual minimum | | External audit / pentest cadence | ⚠️ Gap | Annual external pentest | | Tabletop exercises | ⚠️ Gap | Annual incident response drill | **Gap actions**: Define security KPIs and reporting cadence, schedule first external pentest and tabletop exercise. ### (g) Cyber hygiene and training [Section titled “(g) Cyber hygiene and training”](#g-cyber-hygiene-and-training) | Aspect | Status | Reference | | --------------------------------- | ------ | ----------------------------------------- | | Operations team security training | ⚠️ Gap | Annual baseline + role-specific | | Phishing simulation | ⚠️ Gap | Quarterly | | Awareness for management body | ⚠️ Gap | NIS2-specific board training is mandatory | **Gap actions**: Establish annual training programme for operations and management body (Cbw makes the latter a personal-liability matter for directors). ### (h) Cryptography policies [Section titled “(h) Cryptography policies”](#h-cryptography-policies) | Aspect | Status | Reference | | ----------------------------------- | ------------- | -------------------------------------------------------------------------- | | TLS 1.3+ enforced on all interfaces | ✅ Constraint | [02-constraints](../02-constraints/02-constraints.md#security-constraints) | | Encryption at rest (AES-256) | ✅ Constraint | Azure default; Key Vault for keys | | JWS signing for VAD tokens | ✅ Implemented | [08-crosscutting](ctn-crosscutting.md#token-design-patterns) | | Key rotation policy | ⚠️ Gap | Document per key type (token signing, TLS, secrets) | | Cryptographic algorithm policy | ⚠️ Gap | Approved-algorithms list; deprecation procedure | **Gap actions**: Document key-rotation policy and cryptographic algorithm policy as architecture decisions (ADRs). ### (i) Human resources, access control, asset management [Section titled “(i) Human resources, access control, asset management”](#i-human-resources-access-control-asset-management) | Aspect | Status | Reference | | -------------------------------------------------------- | ------------ | -------------------------------------------------------------------------- | | RBAC enforced on all admin surfaces | ✅ Constraint | [02-constraints](../02-constraints/02-constraints.md#security-constraints) | | Least privilege on Azure subscriptions | 🚧 In design | RBAC review needed | | Joiner/Mover/Leaver process for CTN ops staff | ⚠️ Gap | DIL HR + Entra ID lifecycle | | Asset inventory (Azure resources, secrets, certificates) | 🚧 Partial | Azure Resource Graph; needs curation | | Privileged Access Management (PAM / PIM) | ⚠️ Gap | Entra PIM for Global Admin / Owner roles | **Gap actions**: Implement Entra PIM for elevated roles, define and document JML process, curate asset inventory. ### (j) MFA, secured communications, secured emergency communications [Section titled “(j) MFA, secured communications, secured emergency communications”](#j-mfa-secured-communications-secured-emergency-communications) | Aspect | Status | Reference | | --------------------------------------------------- | ------------- | ----------------------------------------------------------------------- | | MFA on all admin / management surfaces | ✅ Constraint | Keycloak MFA / federated IdP MFA (per ADR-00008) | | M2M authentication via OAuth 2.0 client credentials | ✅ Implemented | [08-crosscutting](ctn-crosscutting.md#authentication-and-authorization) | | Secured communications (TLS, JWS) | ✅ Implemented | Already covered in (h) | | Out-of-band emergency communication channel | ⚠️ Gap | E.g. Signal-based DIL ops bridge | **Gap actions**: Establish out-of-band crisis communication channel for DIL operations team. ## Authorisation specifics (2026) [Section titled “Authorisation specifics (2026)”](#authorisation-specifics-2026) CTN’s 2026 authorisation model — VAD validated locally at every data provider — keeps the trust surface small. From a NIS2 perspective two artefact categories are critical: 1. **JWKS endpoints** for the VAD signing key are critical infrastructure. Compromise = ability to forge tokens. Hardening: HSM-backed signing keys (Azure Key Vault Managed HSM or equivalent), documented rotation procedure, monitoring of key-usage anomalies. 2. **Trust list management.** Adding or removing parties from the association trust list is a privileged operation — requires its own change-control process equivalent to CA root operations. Both points warrant their own ADRs. When later phases introduce additional artefacts (an OPA-based policy engine, dual-token validation) those will bring their own NIS2 controls; design notes are parked under `docs/_out-of-scope-2026/`. ## NIS2 readiness summary [Section titled “NIS2 readiness summary”](#nis2-readiness-summary) | Category | Items | ✅ Met | 🚧 Partial / In design | ⚠️ Gap | | ---------------------------- | ------ | ------ | ---------------------- | ------ | | (a) Risk analysis & policies | 4 | 2 | 0 | 2 | | (b) Incident handling | 5 | 0 | 2 | 3 | | (c) Business continuity | 5 | 0 | 1 | 4 | | (d) Supply-chain security | 8 | 0 | 2 | 6 | | (e) SDLC security | 5 | 1 | 2 | 2 | | (f) Effectiveness assessment | 4 | 0 | 0 | 4 | | (g) Cyber hygiene & training | 3 | 0 | 0 | 3 | | (h) Cryptography | 5 | 3 | 0 | 2 | | (i) HR / access / assets | 5 | 1 | 2 | 2 | | (j) MFA / secure comms | 4 | 3 | 0 | 1 | | **Totals** | **48** | **10** | **9** | **29** | Roughly 21% of NIS2 control points are met today, 19% partially in place, and 60% remain as gaps. This is consistent with a project still in active build-out and is not unusual at this stage; the value of this document is to make the gaps explicit and actionable. ## Recommended next steps [Section titled “Recommended next steps”](#recommended-next-steps) Short term (Q3 2026): * Publish DIL-level Information Security Policy and reference from CTN docs. * Document 24h / 72h / 1M incident reporting flow with named owners. * Define RTO / RPO per critical component. * Publish vulnerability disclosure policy (`security.txt`) on all CTN portals. * Establish SBOM generation and container image scanning in CI/CD. Medium term (Q4 2026 / Q1 2027): * Pursue ISO 27001 certification for DIL operations. * First external pentest and tabletop exercise. * Implement Entra PIM for elevated roles. * Document key-rotation and cryptographic algorithm policies as ADRs. * Establish supplier inventory and supplier-security review process. Strategic positioning: * A complete NIS2-readiness pack (this mapping + ISO 27001 + pentest reports + SBOM + CVD policy) makes CTN a stronger commercial proposition to participants who feel NIS2 pressure themselves. ## References [Section titled “References”](#references) * EU Directive 2022/2555 (NIS2) * Cyberbeveiligingswet (NL implementation, in force mid-2025) * ENISA NIS2 implementation guidance * NCSC-NL CSIRT-DSP reporting guidance * [02-constraints](../02-constraints/02-constraints.md) — Security and integration constraints * [08-crosscutting](ctn-crosscutting.md) — Cross-cutting security concepts * [10-quality](../10-quality/ctn-quality.md) — Quality requirements * [11-risks](../11-risks/ctn-risks.md) — Risk register # ADR-00001: Record architecture decisions ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | ------------ | ----------- | | CTN Project Team | Approved | 2026-05-29 | | CTN Technical Advisory Board | Approved | 2026-07-08 | | CTN Steering Committee | Not Required | 2026-07-08 | | BDI Conformance Team | Approved | 2026-07-08 | | BDI Framework Team | Not Required | 2026-07-08 | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) Copied and adapted from original Michael Nygard ADR blog post: . Architecture for agile projects has to be described and defined differently. Not all decisions will be made at once, nor will all of them be done when the project begins. Agile methods are not opposed to documentation, only to valueless documentation. Documents that assist the team itself can have value, but only if they are kept up to date. Large documents are never kept up to date. Small, modular documents have at least a chance at being updated. Nobody ever reads large documents, either. Most developers have been on at least one project where the specification document was larger (in bytes) than the total source code size. Those documents are too large to open, read, or update. Bite sized pieces are easier for for all stakeholders to consume. One of the hardest things to track during the life of a project is the motivation behind certain decisions. A new person coming on to a project may be perplexed, baffled, delighted, or infuriated by some past decision. Without understanding the rationale or consequences, this person has only two choices: 1. *Blindly accept the decision.* This response may be OK, if the decision is still valid. It may not be good, however, if the context has changed and the decision should really be revisited. If the project accumulates too many decisions accepted without understanding, then the development team becomes afraid to change anything and the project collapses under its own weight. 2. *Blindly change it.* Again, this may be OK if the decision needs to be reversed. On the other hand, changing the decision without understanding its motivation or consequences could mean damaging the project’s overall value without realizing it. (E.g., the decision supported a non-functional requirement that hasn’t been tested yet.) It’s better to avoid either blind acceptance or blind reversal. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) We will keep a collection of records for “architecturally significant” decisions: those that affect the structure, non-functional characteristics, dependencies, interfaces, or construction techniques. An architecture decision record is a short text file in a format similar to an Alexandrian pattern. (Though the decisions themselves are not necessarily patterns, they share the characteristic balancing of forces.) Each record describes a set of forces and a single decision in response to those forces. Note that the decision is the central piece here, so specific forces may appear in multiple ADRs. We will keep ADRs in the project repository under docs/arc42/09-decisions/NNNNN-active-sentence.md We should use a lightweight text formatting language like Markdown. ADRs will be numbered sequentially and monotonically. Numbers will not be reused. If a decision is reversed, we will keep the old one around, but mark it as superseded. (It’s still relevant to know that it was the decision, but is no longer the decision.) We will use a format as defined in ADR-TEMPLATE.md. The whole document should be one or two pages long. We will write each ADR as if it is a conversation with a future developer. This requires good writing style, with full sentences organized into paragraphs. Bullets are acceptable only for visual style, not as an excuse for writing sentence fragments. (Bullets kill people, even PowerPoint bullets.) ### Rationale [Section titled “Rationale”](#rationale) The adoption of Architecture Decision Records (ADRs) is driven by the need for a sustainable and transparent architectural evolution in an agile environment. * **Business alignment**: ADRs ensure that architectural knowledge is preserved across team changes, protecting the business investment in the software. By documenting the “why” behind decisions, we ensure that future changes are made with a full understanding of the original intent, reducing the risk of costly regressions or architectural drift. * **Technical considerations**: Keeping ADRs in the project repository using Markdown ensures they are easily accessible to developers, version-controlled alongside the code, and simple to update. The bite-sized, modular nature of ADRs makes them more likely to be read and maintained compared to traditional, large architecture documents. * **Risk assessment**: Without ADRs, the project faces the risk of “blind acceptance” (stale decisions hindering progress) or “blind reversal” (changing critical components without understanding consequences). ADRs mitigate these risks by providing the necessary context for informed decision-making. * **Cost/benefit analysis**: While writing ADRs requires an upfront time investment from senior technical staff, the long-term benefits in terms of onboarding speed, architectural consistency, and maintenance efficiency far outweigh these costs. It prevents the “collapse under its own weight” that often occurs when technical debt accumulates due to misunderstood architectural constraints. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: One ADR describes one significant decision for a specific project. It should be something that has an effect on how the rest of the project will run. The consequences of one ADR are very likely to become the context for subsequent ADRs. This is also similar to Alexander’s idea of a pattern language: the large-scale responses create spaces for the smaller scale to fit into. Developers and project stakeholders can see the ADRs, even as the team composition changes over time. The motivation behind previous decisions is visible for everyone, present and future. Nobody is left scratching their heads to understand, “What were they thinking?” and the time to change old decisions will be clear from changes in the project’s context. * **Negative**: As the ADRs become more complex, the consequences and interactions between them become more complex. We’ll cross that bridge when we get there, by merging and re-bootstrapping the resulting adr’s into a coherent architecture document. Also, writing an ADR requires a time investment. We strongly believe this investment offers long-term benefits in terms of project clarity and maintainability. * **Neutral**: ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) This ADR will be the first. We will bootstrap the project with a few more decisions along the way. Writing and approving ADRs need to be embedded in the way of working. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * \[ n/a ] Security review completed * \[ n/a ] Performance impact assessed * \[ n/a ] Integration impact evaluated * \[ X ] Documentation updated * \[ X ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [Michael Nygard ADR blog post](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions) * [arc42 adr](https://github.com/architecture-decision-record/architecture-decision-record/tree/main/locales/en/templates/decision-record-template-by-arc42) ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ---------------------------- | | 2026-05-28 | Proposed | Initial proposal | | 2026-05-29 | Approved | Approved by CTN Project Team | # ADR-00017: Test Automation Using Playwright with TypeScript ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-06-22 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | RdN Question: this seems to overlap with 00015 > **Relationship to other test ADRs.** This ADR defines the **strategic UI-automation framework** for the production portal (Playwright + TypeScript, TAaaS-owned, layered Page Object / Service / Utility architecture). It is complementary to — not a duplicate of — [ADR-00015](00015-e2e-playwright-chromium-failsafe.md), which covers the **tactical, in-repo Java e2e tier** bound to the backend’s Maven build for the throwaway demo frontend. The two use different toolchains on purpose because they test different surfaces; the [overall test strategy in ADR-00023](00023-test-strategy.md) sets out the full picture, layer ownership, and the fact that convergence of the two toolchains is a deferred, future decision. ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) The project requires a reliable, maintainable, scalable, and reusable testing approach to support quality assurance activities across the application lifecycle. Testing will initially be performed manually to validate functionality, understand business workflows, and identify stable, repeatable test scenarios. As the application matures, selected test cases will be prioritized for automation using Playwright with TypeScript. Priority will be given to smoke tests, critical business workflows, regression scenarios, and tests that provide the highest return on investment through repeated execution. This phased approach allows the team to establish confidence in the application while building a sustainable and maintainable automation suite. The selected solution must: * Support modern web application testing. * Enable cross-browser testing. * Integrate with CI/CD pipelines. * Provide reliable and maintainable automated tests. * Support collaboration between developers and testers. * Generate meaningful test execution reports and diagnostics. * Align with the organisation’s quality engineering strategy. In addition, the solution must support the implementation of automation engineering standards, including: * SOLID design principles. * DRY (Don’t Repeat Yourself). * Separation of concerns. * Reusable automation components. * Consistent coding standards. * Scalable framework architecture. * Automated quality gates and continuous integration practices. The solution must enable the creation of a sustainable automation framework that can evolve alongside the application while minimising technical debt and long-term maintenance effort. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Playwright with TypeScript [Section titled “Option 1: Playwright with TypeScript”](#option-1-playwright-with-typescript) * **Description**: A modern automation framework supporting Chromium, Firefox, and WebKit browsers. Combined with TypeScript, Playwright enables the development of maintainable, scalable, and reusable automation solutions through strong typing, modern language features, structured framework design, and robust test execution capabilities. The Test Automation as a Service (TAaaS) team has extensive knowledge and practical experience with Playwright and TypeScript, reducing implementation risk and enabling the rapid establishment of a maintainable automation framework. * **Pros**: * Supports Chromium, Firefox, and WebKit. * Fast execution through parallelisation. * Built-in waiting mechanisms and reliability features. * Native TypeScript support. * Strong support for maintainable framework design. * Enables implementation of SOLID and DRY principles. * Supports reusable Page Object and Component Object patterns. * Facilitates separation of concerns through layered architecture. * Integrated screenshots, video recording, and trace viewer. * Strong CI/CD integration. * API testing capabilities. * Open-source and actively maintained. * Improved maintainability through type safety and compile-time validation. * **Cons**: * Initial framework implementation effort is still required. * Team discipline is needed to maintain framework standards and architecture consistency. * Ongoing maintenance is required as the application and tests evolve. ### Option 2: Cypress with TypeScript [Section titled “Option 2: Cypress with TypeScript”](#option-2-cypress-with-typescript) * **Description**: A popular end-to-end testing framework focused on modern web applications. Cypress provides a strong developer experience and supports TypeScript-based automation development. While effective for many web testing scenarios, its limitations around browser support and enterprise-scale automation architecture make it less suitable for the long-term objectives of this project. * **Pros**: * Easy setup and onboarding. * Strong developer experience and interactive test runner. * Native TypeScript support. * Automatic waiting and retry mechanisms. * Good debugging capabilities. * Large community and ecosystem. * Strong CI/CD integration. * Extensive documentation and learning resources. * **Cons**: * Does not support WebKit, limiting full cross-browser validation. * Limited support for multi-tab and multi-window testing scenarios. * Less comprehensive API testing capabilities than Playwright. * Some advanced execution, reporting, and parallelisation features require commercial services. * Less flexibility for implementing complex enterprise automation architectures. * Reduced suitability for long-term framework scalability and reuse. * Lower alignment with the organisation’s strategic requirement for broad browser coverage and maintainable automation frameworks. ### Option 3: Selenium WebDriver with Java [Section titled “Option 3: Selenium WebDriver with Java”](#option-3-selenium-webdriver-with-java) * **Description**: A mature and widely adopted automation framework supporting multiple browsers and programming languages. While Selenium remains a recognised industry standard, it requires significantly more framework engineering effort to achieve the same capabilities that are available natively in Playwright. * **Pros**: * Mature and widely adopted technology. * Extensive browser support. * Large community and ecosystem. * Broad integration capabilities. * Strong enterprise adoption. * **Cons**: * Higher framework implementation complexity. * Increased maintenance overhead. * More code required to implement reliable synchronisation and waiting strategies. * Slower development lifecycle compared to modern automation frameworks. * Additional tooling required for reporting, tracing, screenshots, and diagnostics. * Higher long-term cost of ownership due to framework maintenance. * Greater risk of inconsistent implementation across teams. * Longer onboarding time for automation engineers. * Increased effort to enforce modern automation engineering standards and reusable framework patterns. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 1 - Playwright with TypeScript. Playwright with TypeScript is selected as the strategic automation solution due to its strong support for modern web application testing, maintainable framework design, cross-browser compatibility, CI/CD integration capabilities, and alignment with the organisation’s quality engineering and software development standards. ### Rationale [Section titled “Rationale”](#rationale) * **Business alignment**: * Supports faster release cycles through automated validation. * Reduces regression effort and associated testing costs. * Improves overall software quality and release confidence. * Enables consistent testing standards across teams. * Provides a scalable foundation for long-term automation growth. * **Technical considerations**: * Native support for Chromium, Firefox, and WebKit browsers. * Strong TypeScript ecosystem and developer tooling. * Built-in reporting, screenshots, video recording, and trace capabilities. * Supports parallel execution for rapid feedback. * Integrates seamlessly with Azure DevOps and GitHub Actions. * Supports both UI and API testing requirements. * Supports the implementation of a layered automation architecture. * Enables separation of concerns through dedicated test, page, component, service, and utility layers. * Supports reusable framework components that reduce duplication and improve maintainability. * Facilitates adherence to SOLID principles through modular design. * Supports DRY principles through shared abstractions, fixtures, services, and utilities. * TypeScript compile-time validation reduces implementation defects and improves code quality. * Aligns with enterprise automation engineering practices and long-term framework scalability. * **Risk assessment**: * Low technology risk due to active community support and industry adoption. * Reduced production risk through increased automated regression coverage. * Framework risks are mitigated through established coding standards, peer reviews, architectural governance, and adherence to SOLID and DRY principles. * Technical debt risks are reduced through reusable framework components and consistent automation design patterns. * **Cost/benefit analysis**: * Costs: * Initial framework implementation effort. * Ongoing maintenance of automation assets. * Continuous enhancement of automation coverage and framework capabilities. * Benefits: * Reduced regression testing effort. * Faster feedback during development. * Improved defect detection. * Better release confidence. * Lower long-term testing costs. * Improved framework maintainability and scalability. * Reduced technical debt through reusable automation components. * Consistent automation implementation across teams. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: * Faster feedback on application quality. * Improved regression coverage. * Increased confidence in cross-browser compatibility. * Reduced manual testing effort. * Strong debugging support through traces, screenshots, and videos. * Improved maintainability through TypeScript. * Improved framework maintainability and scalability. * Reduced technical debt through reusable automation components. * Consistent automation implementation across teams. * Improved code quality through engineering standards and automated quality controls. * Greater long-term sustainability of automation assets. * **Negative**: * Initial investment is required to establish the framework and supporting standards. * Automation assets require continuous maintenance as the application evolves. * Teams must consistently follow agreed architectural and coding standards to preserve maintainability. * **Neutral**: * Existing exploratory testing practices remain unchanged. * Test management processes remain independent of the automation framework. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1: Framework Establishment** * Create the Playwright TypeScript framework repository. * Establish the automation framework architecture. * Implement a layered framework structure consisting of: * Test layer * Page object layer * Component layer * Service layer * Utility layer * Define coding standards and engineering principles. * Establish SOLID and DRY implementation guidelines. * Configure ESLint and Prettier standards. * Integrate reporting capabilities. * Configure CI/CD pipeline execution. * Establish framework documentation. 2. **Phase 2: Automation Rollout** * Execute manual functional and regression testing. * Identify repeatable and high-value test cases suitable for automation. * Prioritise critical business workflows for automation. * Implement smoke test automation. * Introduce regression test coverage incrementally. * Implement API test automation where applicable. 3. **Phase 3: Optimisation and Scaling** * Expand regression coverage. * Optimise execution performance. * Implement test data management strategy. * Introduce quality gates and release criteria. * Monitor framework effectiveness and maintenance metrics. * Continuously improve framework architecture and reusable components. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [ ] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * Test Strategy Document ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ----------------------------------------------------------- | | 2026-06-19 | Proposed | Initial proposal for adoption of Playwright with TypeScript | *** ADR format based on Michael Nygard’s template with CTN-specific enhancements # ADR-00002: Define a preferred stack ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | ------------ | ----------- | | CTN Project Team | Approved | 2026-06-24 | | CTN Technical Advisory Board | Approved | 2026-07-08 | | CTN Steering Committee | Not Required | 2026-07-08 | | BDI Conformance Team | Proposed | 2026-07-08 | | BDI Framework Team | Not Required | 2026-07-08 | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) To align all development efforts and minimize rework, we need to define a preferred technology stack. A common stack allows for easier shifting of functionality between components and team members, ensures better integration, and leverages existing knowledge from previous living labs. We need a stack that is open source, in common use, and integrates well with our chosen Identity and Access Management (IAM) solution. Furthermore, the stack is guided by the following principles: * **Open Source**: Preference for open-source technologies to ensure transparency, avoid vendor lock-in, and leverage community support. * **Broad Adoption**: Choosing technologies with wide industry adoption to ensure a large pool of talent, long-term stability, and rich documentation. * **Uniformity**: Promoting a consistent technology landscape across components to reduce cognitive load for developers and simplify maintenance. * **Alternative Implementations**: While a preferred stack is defined, the architecture should remain open to alternative implementations when there is a strong justification (e.g., specialized performance requirements or specific integration needs), provided they adhere to the overall system architecture and standards. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: The Preferred Stack (Quarkus, PostgreSQL, Keycloak) [Section titled “Option 1: The Preferred Stack (Quarkus, PostgreSQL, Keycloak)”](#option-1-the-preferred-stack-quarkus-postgresql-keycloak) * **Description**: Use Quarkus () as the primary runtime/framework, PostgreSQL () as the relational database, and Keycloak () for IAM. * **Pros**: * Keycloak is already a given based on learnings from living labs. * Quarkus and PostgreSQL align naturally with Keycloak (Red Hat ecosystem, Java-based). * All components are open source and have significant community and industry support. * High performance and developer productivity. * **Cons**: Requires Java/Quarkus expertise. ### What Quarkus is (for thos who aren’t familiar) [Section titled “What Quarkus is (for thos who aren’t familiar)”](#what-quarkus-is-for-thos-who-arent-familiar) Quarkus is Red Hat’s Kubernetes-native Java stack (current LTS is 3.33; latest is 3.35). Its selling point is “supersonic subatomic Java”: it pushes work to build time and supports GraalVM native compilation, so you get fast cold starts and a small memory footprint compared to traditional JVM apps. It offers a unified imperative + reactive model (Vert.x/Mutiny under the hood), Hibernate ORM with Panache, first-class OIDC/OAuth2, and standards-based MicroProfile/Jakarta EE APIs. Quarkus is fully cloud-agnostic at container level. You build a standard OCI image, and it runs identically on AWS (ECS, EKS, App Runner, or Lambda via container images), GCP (Cloud Run, GKE), Azure (AKS), or your own Kubernetes cluster. There’s no vendor-specific runtime lock-in: the same artifact, just a different destination. It lines up well with the envisioned ASR architecture: * **Kubernetes (AKS)** — Quarkus is built for containers; native binaries start in tens of milliseconds, which suits Kubernetes horizontal pod autoscaling and keeps resource use (and cost) down. Deployed via Helm + ArgoCD (GitOps). * **OAuth 2.0 (VAD)** — `quarkus-oidc` is mature as a resource server; Token Exchange and token propagation are available if later phases need them. * **PostgreSQL** — first-class via Panache and the reactive PG client. * **External PDP (later phases)** — when a separate Policy Decision Point is added in a future phase, Quarkus calls it as an external service over REST; the 2026 architecture does not yet include such a component. * **Security posture** — relevant given NIS2/cyber-insurance work: standards-based, regular CVE-patched LTS streams (e.g. the recent CVE-2026-39852 fix). ### Option 2: .NET Stack [Section titled “Option 2: .NET Stack”](#option-2-net-stack) * **Description**: Use the .NET ecosystem for implementation. * **Pros**: Strong enterprise support, good performance. * **Cons**: Does not align as well with Keycloak as the Java-based ecosystem does. Less emphasis on open source in some traditional contexts (though .NET Core is open source). ### Option 3: Clojure Stack [Section titled “Option 3: Clojure Stack”](#option-3-clojure-stack) * **Description**: Use Clojure for implementation. Used in living labs. * **Pros**: Concise, powerful functional programming. * **Cons**: Not in common use compared to the other options, which might make staffing and long-term maintenance more challenging. Does not fit the “common use” criteria as well as Quarkus. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 1: Quarkus, PostgreSQL, and Keycloak. | Component | Technology | Summary | | ----------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Runtime/Framework | **Quarkus** | Java-based framework optimized for cloud-native and serverless environments. | | Database | **PostgreSQL** | Robust, open-source relational database for structured data. | | Schema migration | **Liquibase** | Open-source, dialect-portable schema migrations (YAML changesets) owning the DB schema across dev/test/prod. Detailed rationale in [TDR-0001](../../ai/adrs/TDR-0001-liquibase-owns-schema.md). | | IAM | **Keycloak** | Open-source identity and access management for authentication and authorization. | | Code formatting | **Spotless (Eclipse JDT profile)** | Build-enforced Java formatting via `spotless-maven-plugin` driving a shared Eclipse JDT profile consumed by all IDEs; CI gate fails on unformatted code. First adopted in `asr-service`. Detailed rationale in [ADR-00018](00018-build-enforced-java-formatting-via-spotless.md). | ### Rationale [Section titled “Rationale”](#rationale) Keycloak is selected based on successful results from previous living labs. To complement this, we chose Quarkus and PostgreSQL because they align well with Keycloak’s architecture and deployment patterns. This choice reflects our core principles: * It is entirely **open source**. * It enjoys **broad adoption** in the enterprise Java ecosystem. * It promotes **uniformity** by aligning with the tools used in our IAM solution. By defining this as the preferred stack, we establish a standard path while still allowing for **alternative implementations** if specific requirements necessitate them, as long as they maintain compatibility with the broader system. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: * Alignment across different components and teams. * Reduced rework when moving functionality. * Leverages existing experience from living labs. * Strong integration between IAM, database, and runtime. * **Negative**: * Teams must be proficient in the Java/Quarkus ecosystem. * **Neutral**: * This ADR will be extended over time as the stack evolves or more specific components are added. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1**: Update project documentation and initial scaffolding to reflect the preferred stack. 2. **Phase 2**: Implement core services using Quarkus and PostgreSQL, integrating with Keycloak. 3. **Phase 3**: Extend this ADR with more specific guidelines as implementation progresses. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [ ] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [TDR-0001](../../ai/adrs/TDR-0001-liquibase-owns-schema.md) — Liquibase owns the database schema (the schema-migration choice in the stack table) * [ADR-00018](00018-build-enforced-java-formatting-via-spotless.md) — Build-enforced Java formatting via Spotless (the code-formatting choice in the stack table) ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | --------------------------------------------------------------------------------------------------------------- | | 2026-05-28 | Proposed | Initial proposal | | 2026-05-29 | Approved | Approved by CTN Project Team | | 2026-06-13 | Approved | Extended (per Phase 3 / “extended over time”): added Liquibase as the schema-migration component — see TDR-0001 | | 2026-06-24 | Approved | Extended (per Phase 3 / “extended over time”): added Spotless as the code-formatting component — see ADR-00018 | # ADR-00003: Separate login identities (users) from participant organisations ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | ------------ | ----------- | | CTN Project Team | Approved | 2026-05-29 | | CTN Technical Advisory Board | Approved | 2026-07-08 | | CTN Steering Committee | Not Required | 2026-07-08 | | BDI Conformance Team | Proposed | 2026-07-08 | | BDI Framework Team | Pending | — | ## Terminology (read this first) [Section titled “Terminology (read this first)”](#terminology-read-this-first) Two words in this ADR mean something specific. To avoid confusion: * **User** = a *login identity* (an account that can authenticate), **not necessarily a person**. A user is either: * a **person** logging in on behalf of an organisation (a “human-to-machine” or H2M login — for example an employee using the participant portal), or * a **system/application** logging in without a human (a “machine-to-machine” or M2M login — for example another company’s IT system calling the API). * **Participant** = an **organisation** (a legal entity, e.g. a company) that has been registered in the CTN Association Register. A participant is *not* a person; people and systems act **on behalf of** a participant. So the relationship is: one **participant** (organisation) has one or more **users** (people and/or systems) that act for it. > Terminology note: CTN no longer uses the word “member”; the correct term for an onboarded organisation is **participant** (see [ADR-00004: Naming follows glossary](00004-naming-follows-glossary.md) and the [glossary](../12-glossary/ctn-glossary.md)). “Member” survives only as a Keycloak role name (`admin / member / signatory`) within a participant. ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) The Association Register (ASR) has to keep track of two different kinds of things: **users** (login identities — people or systems, see Terminology above) and **participants** (the organisations they act for). These are fundamentally different: a login identity is about *authentication* (“who is signing in?”), while a participant is about *business registration* (“which verified organisation is this?”). Keycloak has been selected as the identity and access management (IAM) solution (see [ADR-00002](00002-define-a-preferred-stack.md)). The question this ADR answers is: **where do we store user data, and where do we store participant data?** We want a clean separation of concerns, minimal data duplication, and no fragile synchronisation logic between Keycloak and the ASR service — without constraining the ASR’s own participant logic and onboarding workflows. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Clean Separation (Users in Keycloak, participants in ASR) [Section titled “Option 1: Clean Separation (Users in Keycloak, participants in ASR)”](#option-1-clean-separation-users-in-keycloak-participants-in-asr) * **Description**: Store all user-related identification and authentication data in Keycloak. Store all participant-related data in the ASR service’s database. Link them using unique identifiers (e.g., Keycloak User IDs). * **Pros**: * Clean separation of concerns. * No data synchronization required between systems. * Leverages Keycloak’s strengths in identity management. * ASR service remains focused on participant logic. * **Cons**: Requires cross-referencing between two systems for certain operations. ### Option 2: Unified Storage in ASR [Section titled “Option 2: Unified Storage in ASR”](#option-2-unified-storage-in-asr) * **Description**: Store both users and participants in the ASR service database, using Keycloak only for authentication. * **Pros**: All entity data is in one place. * **Cons**: Duplicates user data that Keycloak already manages. Increases complexity in the ASR service. ### Option 3: Unified Storage in Keycloak [Section titled “Option 3: Unified Storage in Keycloak”](#option-3-unified-storage-in-keycloak) * **Description**: Store participant data as attributes or groups within Keycloak. * **Pros**: Single source of truth for all entities. * **Cons**: Keycloak is not designed as a general-purpose participant registry. Management of complex participant hierarchies and metadata would be cumbersome and limited. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 1: Clean Separation. We store **login identities (users)** in Keycloak and **participant organisations** in the ASR, linked by ID. ### Rationale [Section titled “Rationale”](#rationale) This approach follows the principle of using the right tool for the job. Keycloak is excellent for identity and authentication, while the ASR service is specifically designed to manage participant structures and business logic. By separating them, we achieve several key benefits: * **Avoiding Pollution**: We prevent Keycloak from being cluttered with ASR-specific metadata, complex participant hierarchies, and domain-specific logic that doesn’t belong in an IAM solution. * **Scalability**: A separate ASR service can be scaled independently of Keycloak, allowing us to handle high volumes of participant data and complex queries without impacting the performance of the authentication system. * **Custom Onboarding**: A dedicated service allows us to implement custom onboarding workflows for participants (e.g., verification steps, business rule checks) that go beyond the standard user registration and IAM onboarding flows. * **Reduced Complexity**: By linking them via IDs and avoiding synchronization, we reduce system complexity and the risk of data inconsistency. This aligns with our goal of keeping the architecture clean and maintainable. The clean separation pattern was validated in the an earlier CTN-ASR prototype (final delivery review, 2026-03-27): the Keycloak integration itself was assessed as solid (single-writer pattern, service-account provisioning, BFF cookie auth, rollback compensation). The issues that surfaced in that prototype (fabricated EORI numbers in tokens, missing trust levels, unpredictable claim structure, profile mutation dead-ends) were not caused by the separation itself but by violations of it — pushing business identifiers and verification status into Keycloak as attributes instead of treating Keycloak as identity-only. The scope rules below codify those lessons. ### ID Ownership and Linking [Section titled “ID Ownership and Linking”](#id-ownership-and-linking) To maintain a clear boundary, we define strict ownership of identifiers: * **User IDs** are owned and generated by **Keycloak**. These are the unique identifiers used across the system to refer to a specific identity. * **Participant IDs** are owned and generated by the **ASR service**. ### Scope of Data per System [Section titled “Scope of Data per System”](#scope-of-data-per-system) The separation is only meaningful if both sides respect what each system is for. The following rules apply: | Lives in **Keycloak** (identity) | Lives in **ASR** (participants) | | --------------------------------------------------- | ---------------------------------------------------------------------------------- | | User credentials, MFA, password reset state | Participant entity (legal entity record) | | Minimal user profile (email, name) | Business identifiers: KVK / KBO / HRB, EUID, LEI, EORI, VAT | | Authentication sessions, refresh tokens | Verification records and their status (Pending / Approved / Rejected / Revoked) | | Role / group membership for authentication purposes | Trust level and any derived assurance score | | | Participant↔user membership and per-participant roles (admin / member / signatory) | | | Onboarding workflow state, uploaded documents, audit trail | **Anti-pattern (from an earlier prototype, to avoid):** writing business identifiers or verification status into Keycloak as user/organization attributes so they “come along for free” in the access token. This duplicates the source of truth in ASR, invites drift, and — as the prototype demonstrated — leads to fabricated values being presented as authoritative in tokens. Token enrichment with participant attributes is a separate concern: claims should be composed from ASR at token-issuance or token-exchange time, not by mirroring ASR data into Keycloak storage. ### Use of Keycloak Organizations [Section titled “Use of Keycloak Organizations”](#use-of-keycloak-organizations) Keycloak 26.6 ships a first-class **Organizations** feature with hierarchical **Organization Groups** (per-organization isolated, no realm-level collisions), per-organization IdP linking, a built-in `organization` token scope, and a dedicated Admin REST API. This is complementary to — not a replacement for — the separation defined above. We adopt it for the IAM-side concerns that would otherwise require us to reinvent membership and federation in ASR: | Concern | Implementation | | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | User-to-participant membership | A Keycloak Organization exists per ASR participant. The user’s membership of that Keycloak Organization *is* the membership relation. ASR does not maintain a parallel join table. | | Per-participant role (admin / member / signatory) | Modeled as Keycloak Organization Groups (e.g. `/admin`, `/signatory`). Surfaced in tokens via the Organization Group Membership mapper. | | Per-participant identity federation (e.g. the participant’s own Entra ID, eHerkenning, eIDAS) | A federated IdP linked to the Keycloak Organization, with Hardcoded Group or Advanced Claim to Group mappers as needed. The federation strategy per participant type is defined in [ADR-00008](00008-external-idp-federation-strategy.md). | | Linking back to ASR | The Keycloak Organization’s `alias` (or a dedicated attribute) carries the ASR participant ID; the ASR participant record stores the Keycloak Organization ID. This is the only cross-system reference; no other duplication. | **Reinforced anti-patterns specific to Keycloak Organizations:** 1. **No business identifiers as Keycloak Organization attributes.** KVK / KBO / HRB / EUID / LEI / EORI / VAT remain authoritative in ASR. Putting them on the Keycloak Organization recreates an earlier prototype’s class of bug (fabricated values, drift, no verification context). 2. **The built-in `organization` claim shape is alias-as-key** (`{"organization": {"acme-corp": {...}}}`). This is the same anti-pattern as M17 from the prototype review — consumers cannot rely on a predictable JSON path. A separate ADR will decide whether we accept this shape and document the convention, or emit a custom claim structure via a protocol mapper. **Known limitation:** Keycloak Organization Groups cannot be used directly in Keycloak’s own authorisation policies. Fine-grained authorisation is owned by the data provider in 2026 (see [Solution Strategy](../04-solution-strategy/ctn-solution-strategy.md) §3); when a separate PDP service is introduced in a later phase it would consume the same Keycloak claims. ### Mutation Paths [Section titled “Mutation Paths”](#mutation-paths) Because participant-side data does not live in Keycloak, the ASR service must expose explicit endpoints for any data the user is allowed to change that crosses the boundary: * User-owned identity fields (name, email, password) are mutated via ASR endpoints that proxy to the Keycloak Admin API. The portal does not call Keycloak directly. * Participant data is mutated via ASR endpoints, with authorization scoped to the participant the user belongs to. * Membership and per-participant role assignments are mutated via the Keycloak Organizations Admin REST API, fronted by ASR endpoints so that authorization, audit, and business rules (e.g. “cannot remove the last admin”) remain in one place. This avoids the “profile is read-only” dead-end observed in the prototype. #### Addressing the Chicken-and-Egg Problem [Section titled “Addressing the Chicken-and-Egg Problem”](#addressing-the-chicken-and-egg-problem) A potential “chicken-and-egg” problem arises during the onboarding of a new participant by a new user: does the user exist before the participant, or vice versa? We resolve this by defining the following flow: 1. **User Identity First**: The natural person must first be registered in Keycloak to establish a verifiable identity. 2. **Participant Creation**: Once authenticated, the user (now possessing a Keycloak User ID) can initiate the participant onboarding process in the ASR service. 3. **Linking**: The ASR service creates the participant record and stores the Keycloak User ID as the “owner” or “initial administrator” of that participant. This ensures that we always have a valid identity to attribute participant actions to, avoiding orphan participants or unauthenticated registration attempts. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: * Reduced architectural complexity by avoiding sync logic. * Improved data integrity by having a single source of truth for each entity type (Keycloak for identity, ASR for participants and verifications). * Scalable approach for both user and participant management. * Verification status and trust level remain authoritative in ASR — avoids the “fabricated identifier in token” class of bug seen in the prototype. * **Negative**: * Application logic must handle the resolution of IDs between Keycloak and ASR when both user and participant details are needed. * ASR must expose explicit endpoints for any identity-side mutation the user is allowed to perform (proxying to Keycloak Admin API), rather than relying on Keycloak’s account console. * Token enrichment with participant claims requires a deliberate composition step (subject of a future ADR) instead of falling out of Keycloak storage automatically. * **Neutral**: * Internal APIs will need to support fetching details by ID across service boundaries. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1**: Design the ASR database schema to include participant entities with references to Keycloak User IDs where necessary. 2. **Phase 2**: Implement the ASR service to manage participant CRUD operations. 3. **Phase 3**: Configure Keycloak to provide User IDs in tokens or via APIs for the ASR service to consume. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [ ] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [00002-define-a-preferred-stack.md](00002-define-a-preferred-stack.md) * [00006-token-claim-composition.md](00006-token-claim-composition.md) (decides the claim shape for participant context — addresses the alias-as-key issue raised here) * [00007-identity-mutation-proxy.md](00007-identity-mutation-proxy.md) (defines the ASR endpoints that proxy to Keycloak for identity-side mutations) * [00008-external-idp-federation-strategy.md](00008-external-idp-federation-strategy.md) (per-participant-type federation: Entra ID / eHerkenning / local) * an earlier CTN-ASR Final Delivery Review, 2026-03-27 (findings C9, M16, M17, M21 informed the scope and mutation-path sections of this ADR). * Keycloak blog, “Organization Groups: Structure Your Organizations with Hierarchical Group Management” (2026-04-29): ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 2026-05-28 | Proposed | Initial proposal | | 2026-05-29 | Proposed | Reinforced with lessons from an earlier prototype review: explicit scope-of-data rules and mutation-path guidance added | | 2026-05-29 | Proposed | Added “Use of Keycloak Organizations” section adopting Keycloak 26.6 Organizations + Organization Groups for membership, per-participant roles, and per-participant IdP federation | # ADR-00004: Single Consolidated Glossary for BDI and CTN ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | ------------ | ----------- | | CTN Project Team | Approved | 2026-06-11 | | CTN Technical Advisory Board | Not Required | 2026-07-08 | | CTN Steering Committee | Not Required | 2026-07-08 | | BDI Conformance Team | Proposed | 2026-07-08 | | BDI Framework Team | Pending | 2026-07-08 | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) In the development of the Association Registry (ASR) and the broader Connected Trade Network (CTN), we deal with concepts from multiple domains: the Basic Data Infrastructure (BDI) reference architecture, CTN-specific business logic, and general logistics. Inconsistency in terminology leads to confusion in designs, documentation, and source code (e.g., mixing “Organization” vs “Member”, or “User” vs “Participant”). Some terms are a given (“Users” and “Organizations” are terms in use in Keycloak and may not mean the same. We need a stable, single source of truth for terminology that is accessible to both development teams and business stakeholders to ensure clear communication and maintainable code. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Fragmented Glossaries [Section titled “Option 1: Fragmented Glossaries”](#option-1-fragmented-glossaries) * **Description**: Maintain separate glossaries for BDI (external), CTN (local), and ASR (technical docs). * **Pros**: Local control over specific sections. * **Cons**: High risk of drift; conflicting definitions for the same terms; harder for stakeholders to find the authoritative definition. ### Option 2: Single Consolidated Glossary (Chosen) [Section titled “Option 2: Single Consolidated Glossary (Chosen)”](#option-2-single-consolidated-glossary-chosen) * **Description**: Maintain one single, comprehensive glossary at `docs/arc42/12-glossary/ctn-glossary.md` that consolidates BDI and CTN terms. This glossary also includes a dedicated chapter for the ASR bounded context to define its specific entities. * **Pros**: * Single source of truth for all stakeholders (Business and Tech). * Ensures alignment between BDI reference architecture and CTN implementation. * Explicitly defines the ASR bounded context entities, aiding developers. * AI-Ready: A single file is easier to provide as context to LLMs for term-consistent code generation. * Co-location: Stored in the repository, it evolves with the code. * **Cons**: Requires discipline to maintain and keep updated during development. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 2. We will maintain a single consolidated glossary. All naming in code, designs, and documentation MUST follow this glossary. ### Rationale [Section titled “Rationale”](#rationale) Clear and consistent naming is fundamental to a shared understanding of the architecture. By consolidating BDI and CTN terms into one place, we eliminate ambiguity. Adding a dedicated ASR chapter ensures that the technical entities used in the codebase are also business-aligned and documented. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: * Reduced cognitive load and improved clarity for everyone involved. * Seamless alignment between business requirements and technical implementation. * Better AI assistance due to high-quality, consolidated context. * **Negative**: * Developers and architects must ensure the glossary is updated when new significant concepts are introduced. * **Neutral**: * Existing documentation and code might require refactoring to align with the consolidated terms. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1**: Consolidate existing BDI and CTN terms into `docs/arc42/12-glossary/ctn-glossary.md` (Completed). 2. **Phase 2**: Add the ASR Bounded Context chapter to the glossary (Completed). 3. **Phase 3**: Review ASR codebase and data model for alignment with the new glossary sections. 4. **Phase 4**: Enforce glossary usage in PR reviews and design sessions. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [x] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [CTN Glossary](../12-glossary/ctn-glossary.md) * [BDI Reference Architecture Glossary](https://github.com/Basic-Data-Infrastructure/BDI-Reference-Architecture/blob/main/reference-architecture/glossary/bdi-terms.md) * [ASR Data Model](../05-building-blocks/ctn-asr-data-model.md) ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ------------------------------------------------------------ | | 2026-05-28 | Proposed | Initial proposal (Naming follows glossary) | | 2026-06-11 | Proposed | Rewritten to emphasize consolidation and ASR bounded context | *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`). A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* *** *ADR format based on Michael Nygard’s template with CTN-specific enhancements* # ADR-00005: Use AGENTS.md for shared AI guidelines ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | ------------ | ----------- | | CTN Project Team | Approved | 2026-05-29 | | CTN Technical Advisory Board | Not Required | 2026-07-08 | | CTN Steering Committee | Not Required | 2026-07-08 | | BDI Conformance Team | Not Required | 2026-07-08 | | BDI Framework Team | Not Required | 2026-07-08 | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) As the project increasingly uses multiple AI-assisted development tools (e.g., an AI agent, Claude Code, Cursor), there is a risk of fragmented instructions and inconsistent coding patterns. We previously chose `CLAUDE.md` as a shared file, but `AGENTS.md` has emerged as a more tool-agnostic, industry-supported standard stewarded by the Agentic AI Foundation. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Continue using CLAUDE.md as primary [Section titled “Option 1: Continue using CLAUDE.md as primary”](#option-1-continue-using-claudemd-as-primary) * **Description**: Keep `CLAUDE.md` as the source of truth. * **Pros**: Already implemented. * **Cons**: Naming is biased towards one tool (Claude); may not be automatically recognized by other agents. ### Option 2: Adopt AGENTS.md as the primary shared file [Section titled “Option 2: Adopt AGENTS.md as the primary shared file”](#option-2-adopt-agentsmd-as-the-primary-shared-file) * **Description**: Use `AGENTS.md` (a growing open standard) as the central repository of AI instructions and project context. Other tool-specific files (like `CLAUDE.md`) reference this file. * **Pros**: Tool-agnostic; follows emerging industry standards; supported by multiple AI vendors (OpenAI, Anthropic, Google, Cursor); readable by humans and all AI agents. * **Cons**: Requires minor migration from existing setup. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 2: Adopt AGENTS.md as the primary shared file. ### Rationale [Section titled “Rationale”](#rationale) * **Standardization**: `AGENTS.md` is an open format stewarded by the Agentic AI Foundation, making it the most future-proof choice for cross-tool compatibility. * **Clarity**: It clearly separates human-centric documentation (`README.md`) from agent-centric instructions (`AGENTS.md`). * **Scalability**: Supports hierarchical nesting for large repositories and monorepos. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: Single, tool-agnostic source of truth for AI context; easier onboarding for any new AI tools; consistent application of project standards. * **Negative**: Tool-specific config files (e.g. agent-specific guideline files or `CLAUDE.md`) still exist but act as pointers to `AGENTS.md` to ensure compatibility with tools that have fixed defaults. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1**: Create `AGENTS.md` at the project root with core project context, tech stack, and AI best practices. 2. **Phase 2**: Update `CLAUDE.md` to point to `AGENTS.md`. 3. **Phase 3**: Update an AI agent guidelines to reference `AGENTS.md`. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * \[ X ] Documentation updated * \[ X ] Stakeholder approval obtained (via chat) ## Related Documents [Section titled “Related Documents”](#related-documents) * `AGENTS.md` * `CLAUDE.md` * `docs/arc42/12-glossary/ctn-glossary.md` * `docs/arc42/09-decisions/00001-record-architecture-decisions.md` ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | -------------------------------------- | | 2026-05-28 | Approved | Initial adoption of AGENTS.md standard | # ADR-00006: Token Claim Composition for Participant Context ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-05-29 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Proposed | 2026-07-08 | | BDI Framework Team | Pending | — | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) [ADR-00003](00003-separate-users-and-participants.md) split the data between two systems: * **Keycloak** (the login system) stores everything about a **user’s login** (their identity as an account that can sign in). * The **ASR** (the Association Register, CTN’s own back-end service) is the authoritative record for **participant data** — the registered organisations, their business identifiers (such as KVK or LEI numbers) and their verification status (whether those identifiers have been checked). ADR-00003 also decided that Keycloak keeps track of **which users belong to which participant** and **what role** each user has there — using its built-in *Organizations* feature (version 26.6), one Keycloak Organization per participant. Service providers and connectors that receive an access token need to know: 1. **Which participant** the user is acting for (and which alternative participants the user could switch to). 2. **What role** the user holds within that participant (admin / member / signatory). 3. **Which business identifiers** identify the participant authoritatively (EUID, KVK, LEI, EORI, VAT — only those that are verified). Of these, (1) and (2) are pure IAM concerns and can come from Keycloak directly. (3) is authoritative in ASR and must not be mirrored into Keycloak attributes (per ADR-00003). Two prior issues from the an earlier CTN-ASR prototype review (2026-03-27) directly shape this decision: * **M17 — EUID used as attribute key**: Keycloak’s built-in `organization` claim emits `{"organization": {"acme-corp": {...}}}` with the alias as key. Consumers cannot rely on a predictable JSON path. * **M16 — Trust level missing from tokens**: verification status existed in the prototype’s database but was never reflected in tokens. Service providers could not distinguish a fully verified organization from a minimally registered one. This ADR decides **how participant context is composed into the token**. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Keycloak built-in `organization` scope only [Section titled “Option 1: Keycloak built-in organization scope only”](#option-1-keycloak-built-in-organization-scope-only) * **Description**: Use the built-in `organization` scope and the Organization Group Membership mapper. No ASR-side enrichment. Business identifiers are not in the token; consumers fetch them from an ASR endpoint if needed. * **Pros**: Zero custom code in Keycloak. Token stays small. ASR remains single source of truth — token never goes stale relative to ASR. * **Cons**: Token carries no business identifiers — every consumer needs a second call. Alias-as-key shape (M17) remains. Requires a published ASR introspection-style endpoint. ### Option 2: Custom Keycloak protocol mapper that calls ASR at token issuance [Section titled “Option 2: Custom Keycloak protocol mapper that calls ASR at token issuance”](#option-2-custom-keycloak-protocol-mapper-that-calls-asr-at-token-issuance) * **Description**: A custom protocol mapper (Java SPI or JavaScript mapper) in Keycloak that, during token issuance, calls an internal ASR endpoint to fetch participant claims and merges them into the token under a predictable schema (e.g. `participant.euid`, `participant.identifiers.kvk`). * **Pros**: Predictable JSON paths for consumers. Single token round-trip. Claims always reflect current ASR state at issuance. * **Cons**: Couples Keycloak to ASR availability at token-issuance time. Custom mapper is operational surface (deploy, upgrade, test). Latency added to every token issuance. ### Option 3: Token exchange — minimal Keycloak token, ASR-issued participant token [Section titled “Option 3: Token exchange — minimal Keycloak token, ASR-issued participant token”](#option-3-token-exchange--minimal-keycloak-token-asr-issued-participant-token) * **Description**: Keycloak issues a baseline identity token (sub, email, org membership, role). Client exchanges it at an ASR token endpoint for a second token that carries participant claims, signed by ASR. Consumers validate both, or only the ASR token for participant-scoped APIs. * **Pros**: Clean separation — ASR is the issuer for participant claims, matching the storage boundary. ASR can apply business rules (e.g. only emit claims for participants where verifications are current). No Keycloak customization. * **Cons**: Two tokens to manage. Consumers need to understand which token they are validating. More moving parts; documentation burden. ### Option 4: Userinfo-style enrichment — keep token minimal, fetch on demand [Section titled “Option 4: Userinfo-style enrichment — keep token minimal, fetch on demand”](#option-4-userinfo-style-enrichment--keep-token-minimal-fetch-on-demand) * **Description**: Token from Keycloak carries only identity and org membership. ASR exposes a `/participant/context` endpoint that returns the participant payload for the authenticated user. Consumers fetch when they need it; can cache for the token lifetime. * **Pros**: No Keycloak customization. Token always small. ASR claims always fresh. Easy to evolve the payload without breaking token consumers. * **Cons**: Every consumer must implement the second call. Cache invalidation on the consumer side. Performance impact for high-volume APIs. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: TBD. ### Rationale [Section titled “Rationale”](#rationale) TBD. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: TBD. * **Negative**: TBD. * **Neutral**: TBD. ## Open Questions [Section titled “Open Questions”](#open-questions) * Should business identifiers in the token be limited to those with `Approved` verifications, or include all known identifiers with a status field? * What is the desired token TTL relative to verification-status change frequency? * Do connectors / external dataspaces have a preferred claim schema we should align with (BDI, iSHARE, eIDAS-related profiles)? * For Option 2 or 3: how do we handle the case where a user is a member of multiple participants — one token per participant, or one token with an array? ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1**: Inventory token consumers (connectors, service providers, internal services) and document their claim needs. 2. **Phase 2**: Evaluate the four options against those needs; pick one. 3. **Phase 3**: Specify the concrete claim schema and update this ADR with the decision. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [ ] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [00002-define-a-preferred-stack.md](00002-define-a-preferred-stack.md) * [00003-separate-users-and-participants.md](00003-separate-users-and-participants.md) * an earlier CTN-ASR Final Delivery Review, 2026-03-27 (findings M16, M17, C9, m24) * Keycloak blog, “Organization Groups” (2026-04-29): * RFC 9068 — JWT Profile for OAuth 2.0 Access Tokens ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | --------------------------------------------- | | 2026-05-29 | Proposed | Skeleton — options outlined, decision pending | # ADR-00007: Identity-Side Mutations via ASR Proxy ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-05-29 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Proposed | 2026-07-08 | | BDI Framework Team | Pending | — | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) [ADR-00003](00003-separate-users-and-participants.md) split the data between two systems: * **Keycloak** (the login system) stores everything about a **user’s login**: their name, email, password, and multi-factor authentication (MFA). * The **ASR** (the Association Register, CTN’s own back-end service) is the authoritative record for **participants** (the registered organisations) and their **verifications** (the checks that confirm an organisation is who it claims to be). ADR-00003 also decided that Keycloak keeps track of **which users belong to which participant**, and **what role** each user has inside that participant (for example `admin`, `member`, or `signatory`). Keycloak does this with its built-in *Organizations* feature — one Keycloak Organization per participant. This creates a practical question: when a user updates their name or email, or when an admin removes a member, or when a participant admin promotes a member to admin — **who calls Keycloak**? Three constraints apply: 1. **The portal must not call Keycloak directly.** Direct portal-to-Keycloak calls bypass ASR’s audit trail and business rules, and expose Keycloak Admin API surface to the browser. 2. **Business rules belong in ASR**, not split across two systems. Examples: “a participant must always have at least one admin”; “an email change requires re-verification”; “removing the last member dissolves the participant” (or doesn’t — that itself is an ASR-side rule). 3. **The earlier prototype dead-end (M21) must not recur**: user profile read-only in the portal because all mutations require Keycloak admin intervention. This ADR decides **how user-identity-side mutations are performed from the portal**, given that the data lives in Keycloak. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: ASR proxy endpoints calling Keycloak Admin API (service-account) [Section titled “Option 1: ASR proxy endpoints calling Keycloak Admin API (service-account)”](#option-1-asr-proxy-endpoints-calling-keycloak-admin-api-service-account) * **Description**: ASR exposes endpoints such as `PATCH /users/me`, `POST /participants/{id}/members`, `DELETE /participants/{id}/members/{userId}`, `PUT /participants/{id}/members/{userId}/role`. ASR authorizes the request against participant context, applies business rules, then calls Keycloak Admin API (and the Organizations API for membership/groups) using a service account. All actions land in ASR’s audit log. * **Pros**: Single enforcement point for authorization, business rules, and audit. Portal sees one API. Matches the single-writer pattern that the an earlier partner review assessed as solid (O4). Mutations are atomic from the portal’s perspective. * **Cons**: ASR holds powerful Keycloak credentials — high-blast-radius if compromised. Service-account scope must be tightly minimised. ASR must implement compensating rollback if a multi-step operation partially fails. ### Option 2: Direct portal → Keycloak Account Console / Account API [Section titled “Option 2: Direct portal → Keycloak Account Console / Account API”](#option-2-direct-portal--keycloak-account-console--account-api) * **Description**: For user-self-service mutations only (own profile, own password, own MFA), the portal redirects to or embeds Keycloak’s Account Console. Participant-side mutations still go through ASR. * **Pros**: No code in ASR for password/MFA flows; uses Keycloak’s hardened built-in UX. Reduces ASR’s blast radius. * **Cons**: UX split across two surfaces. ASR loses audit visibility on identity-side changes. Business rules that span identity and participant (e.g. email change triggers verification) become harder to enforce. Violates the “portal does not call Keycloak directly” guidance from ADR-00003. ### Option 3: Hybrid — Account Console for secrets, ASR proxy for everything else [Section titled “Option 3: Hybrid — Account Console for secrets, ASR proxy for everything else”](#option-3-hybrid--account-console-for-secrets-asr-proxy-for-everything-else) * **Description**: Password reset and MFA setup go to Keycloak’s native flows (where the security-critical handling is). Profile fields (name, email), membership, and role changes go through ASR proxy endpoints. ASR subscribes to Keycloak admin events to ingest password/MFA changes into its audit log. * **Pros**: Best-of-both — leverages Keycloak’s hardened auth flows for secrets, keeps everything else in ASR. Audit completeness via event subscription. * **Cons**: Two integration surfaces to maintain. Event-driven audit ingestion adds a moving part. Requires careful design of which paths go where. ## Scope of Mutations [Section titled “Scope of Mutations”](#scope-of-mutations) Regardless of option, the following mutations need an explicit path. This list scopes the design space. The mutation scope depends on the user’s identity source, defined per participant type in [ADR-00008](00008-external-idp-federation-strategy.md). Mutations that target user-level identity (name, email, password, MFA, lifecycle) are only ASR’s concern for users on a local Keycloak account. For Entra-federated users (e.g. Contargo) those mutations live in the participant’s Entra tenant; ASR must not attempt to override them. | Mutation | Touches | Local-account user | Entra-federated user (e.g. Contargo) | | ------------------------------------ | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | Update own profile (name) | Keycloak user | ASR proxy → Keycloak Admin API | Out of scope — managed in participant’s Entra | | Update own email | Keycloak user (+ re-verification in ASR) | ASR proxy → Keycloak Admin API | Out of scope — Entra is source of truth; ASR re-reads on next login | | Change password | Keycloak credential | ASR-fronted or Keycloak Account Console (per chosen option) | Not applicable — participant’s Entra owns credentials | | Enroll / reset MFA | Keycloak credential | ASR-fronted or Keycloak Account Console (per chosen option) | Not applicable — participant’s Entra owns MFA | | Invite a new member to a participant | Keycloak (create user / send invite) + Keycloak Organization (add member) | ASR proxy creates Keycloak user + adds to Organization | ASR proxy adds existing Entra user to the Organization (JIT or pre-invite per ADR-00008) | | Remove a member from a participant | Keycloak Organization (remove member) — possibly Keycloak user | ASR proxy removes from Organization; deletes Keycloak user if no other memberships | ASR proxy removes from Organization only; Keycloak federated identity record left as it tracks an Entra `sub` | | Promote a member to admin | Keycloak Organization Group (`/admin`) | ASR proxy → Organizations Admin API | ASR proxy → Organizations Admin API | | Transfer the admin role | Keycloak Organization Group | ASR proxy → Organizations Admin API | ASR proxy → Organizations Admin API | | Deactivate a participant | ASR (status) + Keycloak Organization (disable) + member access propagation | ASR proxy disables Organization; all members lose participant context | Same — disabling the Organization breaks federated-user membership without touching participant’s Entra | ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: TBD. ### Rationale [Section titled “Rationale”](#rationale) TBD. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: TBD. * **Negative**: TBD. * **Neutral**: TBD. ## Open Questions [Section titled “Open Questions”](#open-questions) * For email changes (local accounts): is the new email verified before or after it is written to Keycloak? What is the user’s effective email during the transition window? * RdN Proposal: before (that also discourages script-kiddies trying to register many users by means of a bot) * For email changes (Entra-federated): does ASR re-sync on every login, or only on explicit user action? Stale display-name/email is a UX concern only — authoritative identity stays at Entra. * RdN Proposal: Every login. In reality the number of times that (entra-federated) users will actually login to the ASR will be very limited. * “Last admin cannot be removed” — is this a hard constraint or does the ASR allow a temporary admin-less state with an SLA to resolve? The same question applies to a participant whose only remaining admin is offboarded at Entra. * RdN Proposal: An edge case, which can be resolved by the ASR admin manually for now. Btw, this is a question I would expect to see on the Jira board. * Does removing a user from their last participant delete the Keycloak user (local account), or leave a dormant identity? For Entra-federated, the Keycloak federated-identity record likely stays — its uniqueness key is the Entra `sub`, useful if the same user is re-invited later. * RdN Question: I don’t grasp this question. * Which Keycloak admin events do we subscribe to, and where do we store them? Of particular interest: federated login failures that indicate source-IdP deprovisioning, since those are the lifecycle signal for Entra-federated users. * RdN Question: Didn’t we agree on a basic (audit) loggin mechanism. In this case it is a system-triggered event which should end up in the log and result in a notification to the ASR admin. * How do we scope the Keycloak service account so that compromise of ASR cannot escalate to full realm control? * RdN Question: Please advise. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1**: Confirm the mutation scope above with stakeholders; close gaps. 2. **Phase 2**: Pick an option and define the concrete API surface (one endpoint per mutation, with authorization and audit requirements). 3. **Phase 3**: Define the Keycloak service-account least-privilege role mapping. Document the rollback strategy for multi-step mutations. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [ ] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [00002-define-a-preferred-stack.md](00002-define-a-preferred-stack.md) * [00003-separate-users-and-participants.md](00003-separate-users-and-participants.md) * [00006-token-claim-composition.md](00006-token-claim-composition.md) * [00008-external-idp-federation-strategy.md](00008-external-idp-federation-strategy.md) (mutation scope per participant type derives from the federation strategy defined there) * an earlier CTN-ASR Final Delivery Review, 2026-03-27 (findings M9, M18, M19, M20, M21 — organisation management gaps in the prototype) * Keycloak Admin REST API documentation * Keycloak blog, “Organization Groups” (2026-04-29): ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ------------------------------------------------------------------------------------------------- | | 2026-05-29 | Proposed | Skeleton — mutation scope listed, options outlined, decision pending | | 2026-05-29 | Proposed | Mutation scope split per participant type (local-account vs Entra-federated), driven by ADR-00008 | # ADR-00008: External IdP Federation Strategy ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Approved | 2026-05-29 | | CTN Technical Advisory Board | Approved | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Proposed | — | | BDI Framework Team | Pending | — | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) [ADR-00003](00003-separate-users-and-participants.md) split the data between two systems: **Keycloak** (the login system) holds everything about a **user’s login**, and the **ASR** (the Association Register, CTN’s own back-end service) is the authoritative record for **participant data** (the registered organisations). It also made Keycloak the place that tracks **which users belong to which participant**, using Keycloak’s built-in *Organizations* feature (version 26.6) — one Keycloak Organization per participant, which is also where each participant’s external login provider gets linked. This ADR decides **which external identity providers (IdPs) the ASR trusts for login, per type of participant**, and how a user’s lifecycle (joining, leaving, being deactivated) flows through that setup. An *identity provider* is the system where a user actually authenticates — for example a company’s own Microsoft Entra ID, the Dutch eHerkenning scheme, or a plain email-and-password account in Keycloak itself. ### Which kind of federation this ADR covers [Section titled “Which kind of federation this ADR covers”](#which-kind-of-federation-this-adr-covers) The word “federation” is used for two different things in CTN. This ADR is only about the first: 1. **ASR ↔ a participant’s own login system** (this ADR): the ASR trusts each participant’s identity provider so the participant’s own people can sign in to the ASR portal. 2. **ASR ↔ another ASR** (out of scope here): two Association Registers trusting each other so their participants can interoperate — for example the planned link between Portbase and the upcoming Hinterland ASR. That is on the roadmap for the coming months and will get its own decision. ### The driving concern: deprovisioning at the source [Section titled “The driving concern: deprovisioning at the source”](#the-driving-concern-deprovisioning-at-the-source) The core problem is **deprovisioning** — making sure that when someone leaves a participant organisation, they lose ASR access without CTN having to find out and act manually. A worked example. A large participant runs its own company login system (some run Microsoft Entra ID; others run their own Keycloak). Their IT-admins already manage all internal accounts there, and a small number of those people also need access to the ASR portal to register endpoints. * If the ASR kept its *own* copies of those accounts (the path the DIL prototype took, using Azure AD directly), then CTN would have to learn about and act on every personnel change at the participant. * If instead the ASR **federates login outward** to the participant’s own login system, deprovisioning happens at the participant: the moment they disable someone’s account, that person can no longer authenticate at the source, so their ASR access falls away on its own. ### Relationship to the earlier decision [Section titled “Relationship to the earlier decision”](#relationship-to-the-earlier-decision) This decision was first taken in the prior handover series as **ADR-004 “Single IdP (Keycloak) — federatie naar Entra ID / eHerkenning / lokale account”** (2026-05-06). This ADR promotes that decision into the official BDI ADR series and refines it against the Keycloak *Organizations* feature introduced in version 26.6. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Single IdP with outward federation (chosen) [Section titled “Option 1: Single IdP with outward federation (chosen)”](#option-1-single-idp-with-outward-federation-chosen) * **Description**: Keycloak is the only IdP that ASR talks to. Per participant, Keycloak federates outward to the identity source the participant brings: * **Entra ID** via OIDC federation, linked to the Keycloak Organization, for participants that have it (Contargo, deepsea carriers, terminal operators). * **eHerkenning** via SAML→OIDC for the NL admin-representation step. * **Local Keycloak account** (email + password + MFA) for participants without an own IdP. * **Service accounts** in Keycloak for machine-to-machine clients (TMS / ERP integrations). * **Pros**: * **Lifecycle at the source**: Contargo deactivates an account in Entra → next ASR login fails, existing session expires at the configured TTL, no manual sync needed by ASR. * **Heterogeneous participant base supported**: large parties keep their own IAM; small parties (single-truck operators with no Microsoft stack) get a usable local account without being forced into a third-party tenant. * **Complexity scales at Keycloak**: federation lives in one place; downstream services see one OIDC issuer. * **Audit clarity**: Keycloak’s `idp` claim records the source (`entra-id`, `eherkenning`, `local`), giving us per-event provenance. * **Cons**: * **Per-participant federation setup**: each large participant requires a one-off OIDC trust configuration in Keycloak. Self-service is feasible only for the local-account path. * **No native push-deprovisioning**: revocation is *pull* (next login fails) plus session TTL, not *push* (immediate kill). Aggressive session lifetimes mitigate but do not eliminate the gap. * **JIT-provisioning policy is a real design question**: who gets a Keycloak account on first Entra login — anyone with a valid Entra token, or only after a Contargo admin has invited them? See Open Questions. ### Option 2: ASR holds its own user directory; sync from participant IdPs (SCIM) [Section titled “Option 2: ASR holds its own user directory; sync from participant IdPs (SCIM)”](#option-2-asr-holds-its-own-user-directory-sync-from-participant-idps-scim) * **Description**: ASR (via Keycloak) holds the canonical user records. Participant IdPs push lifecycle events via SCIM 2.0 (create / update / deactivate). * **Pros**: Immediate deprovisioning (push). Standard protocol. * **Cons**: Requires every participant to operate a SCIM connector — entirely unrealistic for the smaller participants. Asymmetric: works for some, not others. Doubles the directory state (drift risk). Rejected. ### Option 3: Direct Entra ID as the primary IdP for all admin users [Section titled “Option 3: Direct Entra ID as the primary IdP for all admin users”](#option-3-direct-entra-id-as-the-primary-idp-for-all-admin-users) * **Description**: The DIL-prototype path. CTN’s own Entra tenant is the IdP for everyone with portal access; participants are invited as guest users. * **Pros**: Simple from CTN’s side — one tenant to operate. * **Cons**: Forces every participant into CTN’s tenant — including the small operators who don’t use Microsoft at all. Lifecycle ownership flips: *CTN* learns about Contargo personnel changes via invitation/guest management, not via Contargo’s own directory. Rejected for the same reasons noted in ADR-004. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 1 — single Keycloak IdP with outward federation per participant type. ### Rationale [Section titled “Rationale”](#rationale) The three reasons that decided ADR-004 still hold and align cleanly with what Keycloak Organizations 26.6 now offers: 1. **Lifecycle is owned where it can actually be observed.** Contargo knows when a Contargo employee leaves; CTN doesn’t. Federating outward delegates that knowledge to the system that has it. 2. **The participant base is heterogeneous.** A solution that requires every participant to operate an IdP excludes the small operators; a solution that puts everyone in CTN’s tenant excludes the operators who don’t use Microsoft. Per-participant IdP choice is the only fit. 3. **Federation matrix complexity belongs in Keycloak, not in middleware.** Quarkus services in ASR see one OIDC issuer regardless of how many participant IdPs are linked. This is exactly what Keycloak Organizations’ per-org IdP linking is designed for. ### Federation Matrix [Section titled “Federation Matrix”](#federation-matrix) | Participant type | Login source | Trust mechanism | Lifecycle ownership | | ------------------------------------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------- | ----------------------- | | Large party with Entra ID (e.g. Contargo, deepsea carriers, terminal operators) | Participant’s Entra ID tenant, linked to the Keycloak Organization | OIDC federation | Participant’s Entra | | NL admin-representation step | eHerkenning (Digidentity, ConnectIS, etc.) | SAML federation via OIDC bridge | Broker (eH provider) | | Small party without an own IdP | Local Keycloak account (email + password + MFA) | Direct in realm | ASR / participant admin | | Machine (TMS / ERP) | Service account in Keycloak, scoped to the participant | OAuth2 client-credentials | ASR (rotation policy) | ### Mapping to Keycloak Organizations [Section titled “Mapping to Keycloak Organizations”](#mapping-to-keycloak-organizations) * Each ASR participant has a corresponding Keycloak Organization (per ADR-00003). * For Entra-federated participants, the participant’s Entra ID tenant is linked as an IdP to that Keycloak Organization. * First-login mapping uses Organization Group mappers: * **Hardcoded Group** mapper for the simple case (“everyone arriving via this IdP joins `/member`”). * **Advanced Claim to Group** mapper when the participant’s Entra exposes a role/group claim that distinguishes who gets `/admin` vs `/member`. * The Keycloak `idp` claim is preserved through to the access token and logged for audit. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: * Deprovisioning at the Entra source: a deactivated Contargo employee cannot obtain a new ASR session and existing sessions die at TTL. * Downstream services see a single OIDC issuer; no per-IdP code paths in Quarkus services. * Audit trail records the source IdP per event. * Participant-specific identity policy (MFA rules, conditional access) is enforced at the source, not duplicated in Keycloak. * **Negative**: * Onboarding a large participant includes a one-time OIDC trust configuration between Keycloak and the participant’s Entra tenant — not self-service. * Revocation latency is bounded by access-token TTL plus refresh-token policy. Requires us to choose conservative lifetimes for the federated paths (see Open Questions). * JIT provisioning rules must be explicit per participant — open SSO from a participant’s entire Entra tenant is not the same as ASR access. * **Neutral**: * For local-account participants, all the identity-side mutations from ADR-00007 apply unchanged. For Entra-federated participants, most of those mutations are *not ASR’s concern* — refining this is a follow-up to ADR-00007. ## Open Questions [Section titled “Open Questions”](#open-questions) 1. **JIT-provisioning policy.** On first federated login from a participant’s Entra: do we create the Keycloak user automatically (open trust), or do we require a prior invitation from a participant admin (closed trust)? Closed is safer; open is operationally cheaper. Likely closed by default with an opt-in per participant. 2. **Session and token lifetimes for federated users.** Short access-token TTL bounds revocation latency but increases load. Proposal: access-token TTL on the order of minutes for federated users; refresh-token TTL bounded by the federated session. 3. **Back-channel logout / session revocation.** Does Keycloak’s OIDC back-channel logout suffice, or do we additionally poll the source IdP? Depends on what the participant’s Entra emits. 4. **Group/claim mapping per participant.** Which Entra claim, group, or app role do we map to ASR’s `/admin` vs `/member`? Each large participant will negotiate this individually. 5. **Multi-Entra-tenant edge cases.** A person who works at two participants that both federate Entra: do they end up as one Keycloak user with two organization memberships, or two separate Keycloak users? Likely one user keyed on a stable cross-tenant identifier (email is not safe; OIDC `sub` is per-IdP). 6. **Audit retention for deactivated source identities.** When the source IdP no longer recognises a user, can we still resolve historical audit entries to a person? Need to capture enough identity context at session time to keep audit interpretable. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1**: Configure a reference Entra ID federation in a non-production Keycloak realm using a willing pilot participant (Contargo if available). Validate the deprovisioning behaviour end-to-end: deactivate a test account, confirm session timeout, confirm refresh-token revocation. 2. **Phase 2**: Resolve the open questions on JIT-provisioning policy and token lifetimes; document as an extension to this ADR. 3. **Phase 3**: Define the operational runbook for onboarding a new federated participant (information needed from the participant, Keycloak configuration steps, claim mapping negotiation). 4. **Phase 4**: Extend ADR-00007 to differentiate identity-side mutation paths per participant type (Entra-federated vs local). ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed (focus: session revocation, JIT-provisioning trust boundary, multi-tenant user identification) * [ ] Performance impact assessed (token TTL, federation latency) * [ ] Integration impact evaluated (downstream services see only Keycloak as issuer) * [ ] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [00002-define-a-preferred-stack.md](00002-define-a-preferred-stack.md) * [00003-separate-users-and-participants.md](00003-separate-users-and-participants.md) (Keycloak Organizations, per-participant IdP linking) * [00006-token-claim-composition.md](00006-token-claim-composition.md) (the `idp` claim is part of what the token carries) * [00007-identity-mutation-proxy.md](00007-identity-mutation-proxy.md) (mutation paths differ per participant type — to be extended) * Prior handover series, ADR-004 “Single IdP (Keycloak) — federatie naar Entra ID / eHerkenning / lokale account” (2026-05-06) — original source of this decision * BDI Association Federation Using Keycloak (2026-01-29) — earlier ASR↔ASR federation work that listed Entra ID as a compatibility target * ASR kickoff notes (2026-05-11) * Keycloak blog, “Organization Groups” (2026-04-29): ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | ---------------------------------- | ---------------------------------------------------------------------------------- | | 2026-05-06 | Accepted (prior handover, ADR-004) | Original decision in handover series | | 2026-05-29 | Proposed | Promoted into official BDI ADR series; refined against Keycloak Organizations 26.6 | # ADR-00009: Layered Source Code Structure for BDI and CTN ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | ------------ | ----------- | | CTN Project Team | Approved | 2026-06-08 | | CTN Technical Advisory Board | Approved | — | | CTN Steering Committee | Not Required | — | | BDI Conformance Team | Proposed | — | | BDI Framework Team | Proposed | — | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) The BDI (Basic Data Infrastructure) network requires a clear separation between core infrastructure, reusable extensions (marketplace), and specific implementations (like CTN). This separation is necessary to ensure that core components remain generic, while allowing for a vibrant ecosystem of plugins and themes without cluttering the core repository or forcing specific requirements onto all participants. Currently, development is focused on CTN-specific functionality, but there is a clear intent to generalize reusable parts later. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Monolithic package structure [Section titled “Option 1: Monolithic package structure”](#option-1-monolithic-package-structure) All code resides under a single package (e.g., `org.bdinetwork.*`). Reusable and specific code are mixed. ### Option 2: Three-layered package and repository structure [Section titled “Option 2: Three-layered package and repository structure”](#option-2-three-layered-package-and-repository-structure) Divide code into three distinct layers based on their purpose and reusability: 1. **BDI Core**: Fundamental protocol and API definitions. 2. **Marketplace**: Generic plugins and extensions. 3. **Association Specific (CTN)**: Specific themes and plugins for a particular network instance. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 2: Three-layered package and repository structure. ### Rationale [Section titled “Rationale”](#rationale) This approach provides a clear path for code evolution: * **BDI Core (`org.bdinetwork.*`)**: Contains core components, protocol/API definitions, and Service Provider Interfaces (SPIs). This is the foundation that everyone uses. * **Marketplace Layer (`org.bdinetwork.marketplace.*`)**: Contains general-purpose plugins that extend the core. These are optional but reusable across different network instances. * **CTN Specific (`nl.ctn.*`)**: Contains themes and plugins specific to the CTN network. Starting with a separate namespace (`nl.ctn`) prevents accidental coupling with core BDI components. The strategy is to “start out by building mostly CTN stuff” and later move code to Core (by defining SPIs) or Marketplace as its general utility becomes clear. **Key Driver: CTN as the Leading Implementation** Currently, CTN is the only concrete implementation of the BDI concepts. As such, it provides the only source of testable requirements. By focusing on CTN first within its own layer, we can discover and validate the necessary abstractions for the BDI Core and Marketplace based on real-world needs rather than theoretical design. * **Business alignment**: Protects the BDI core as a generic standard while allowing CTN to move fast. * **Technical considerations**: Encourages the use of SPIs for extensibility, which is a standard Java pattern for plugin architectures. * **Risk assessment**: Reduces the risk of CTN-specific requirements “polluting” the BDI standard. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: * Clear boundaries for developers. * Explicit path for generalizing code (CTN -> Marketplace -> Core). * Prevents vendor/tenant lock-in at the core level. * **Negative**: * Requires discipline to maintain package boundaries. * Potential overhead in managing multiple repositories or modules. * **Neutral**: * Developers need to be aware of the `nl.ctn` package name even if a final URL is not yet established. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1**: Establish the `nl.ctn` package for current CTN-specific development in the `asr-service` and related repositories. 2. **Phase 2**: Identify reusable patterns in CTN code and define SPIs in `org.bdinetwork` to allow for generalization. 3. **Phase 3**: Migrate generic plugins from `nl.ctn` to `org.bdinetwork.marketplace` as they mature. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [x] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [ADR-00002: Define a preferred stack](./00002-define-a-preferred-stack.md) * [ADR-00004: Naming follows glossary](./00004-naming-follows-glossary.md) * [ADR-00010: Evolving from Package-based Plugins to Quarkus Extensions](./00010-evolving-to-quarkus-extensions.md) ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ---------------- | | 2026-06-08 | Proposed | Initial proposal | # ADR-00010: Evolving from Package-based Plugins to Quarkus Extensions ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | ------------ | ----------- | | CTN Project Team | Proposed | 2026-06-08 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Not Required | — | | BDI Conformance Team | Proposed | — | | BDI Framework Team | Pending | — | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) ADR-00009 established a three-layered package structure (`org.bdinetwork`, `org.bdinetwork.marketplace`, and `nl.ctn`) within a single repository to facilitate rapid development while maintaining logical separation. As the BDI ecosystem grows, we anticipate the need to deliver these layers as independent JAR files that can be “plugged in” to a core distribution. We need a roadmap to move from compile-time package separation to build-time or deployment-time artifact composition without re-architecting the application logic. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Package-based SPI (Status Quo) [Section titled “Option 1: Package-based SPI (Status Quo)”](#option-1-package-based-spi-status-quo) * **Description**: All layers reside in the same Maven module but use distinct package roots. * **Mechanism**: Standard Jakarta CDI injection with `@Alternative` and `@Priority`. * **Pros**: Zero infrastructure overhead, fast refactoring, easy debugging. * **Cons**: Harder to distribute individual plugins without the core source. ### Option 2: Multi-module Artifacts (Intermediate) [Section titled “Option 2: Multi-module Artifacts (Intermediate)”](#option-2-multi-module-artifacts-intermediate) * **Description**: Move `org.bdinetwork.bdi.spi` to a standalone `bdi-spi` Maven module. * **Mechanism**: Marketplace and CTN implementations move to their own modules/repositories. * **Pros**: Enables independent versioning and distribution. * **Cons**: Increased overhead in CI/CD and dependency management. ### Option 3: Quarkus Extensions (Target) [Section titled “Option 3: Quarkus Extensions (Target)”](#option-3-quarkus-extensions-target) * **Description**: Implementation modules are wrapped as Quarkus Extensions. * **Mechanism**: Use Quarkus build-time scanning to optimize plugin discovery and native compilation. * **Pros**: Optimized for GraalVM, best integration with Quarkus ecosystem, supports “drop-in” JARs via classpath scanning. * **Cons**: Requires specialized Quarkus extension development knowledge. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: A phased approach starting with Option 1, transitioning to Option 2 for distribution, and targeting Option 3 for final optimization. ### Rationale [Section titled “Rationale”](#rationale) * **Continuity**: The CDI-based SPI pattern implemented in Phase 1 is the foundation for Quarkus Extensions. Moving to Phase 2 or 3 will not require changes to the `AssociationProvider` interface or its consumers. * **Simplicity**: Starting with a single repo avoids the “dependency hell” of managing multiple versions of SPIs while the API is still volatile. * **Native Support**: Quarkus Extensions are the only way to ensure that “drop-in” logic is properly optimized and compiled when targeting GraalVM Native Images. ## Consequences [Section titled “Consequences”](#consequences) * **Positive**: * Clear migration path from “monolith” to “plugin-host” architecture. * Decouples implementation speed of CTN from the BDI Core stabilization. * **Negative**: * Future transition to extensions requires understanding of the Quarkus Extension API (deployment vs. runtime modules). * **Neutral**: * Developers must ensure every implementation JAR contains a `META-INF/beans.xml` to be discoverable by the CDI container. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1 (Current)**: Maintain the package-based SPI using CDI `@Alternative` and `@Priority` within the `asr-service` module. ### Example Code (Phase 1) [Section titled “Example Code (Phase 1)”](#example-code-phase-1) The following example shows how the `AssociationProvider` SPI is implemented across different layers using CDI. **1. SPI Interface (`org.bdinetwork.asr.spi`)** ```java public interface AssociationProvider { String getLegalName(); String getFriendlyName(); } ``` **2. Default Implementation (`org.bdinetwork.marketplace.asr`)** ```java @ApplicationScoped public class ExampleMarketplaceAssociationProvider implements AssociationProvider { @Override public String getLegalName() { return "Example Legal Name"; } @Override public String getFriendlyName() { return "example plugin name"; } } ``` **3. CTN-Specific Override (`nl.ctn.asr`)** ```java @ApplicationScoped @Alternative @Priority(1) public class CtnAssociationProvider implements AssociationProvider { @Override public String getLegalName() { return "CTN i.o."; } @Override public String getFriendlyName() { return "Connected Trade Network"; } } ``` **4. Consuming the SPI** ```java @Path("/hello") public class GreetingResource { @Inject AssociationProvider associationProvider; @GET @Produces(MediaType.TEXT_PLAIN) public String hello() { return "Hello from ASR Service. Association: " + associationProvider.getFriendlyName(); } } ``` 2. **Phase 2**: Extract the SPI interfaces and implementations into separate Maven modules when independent distribution becomes a requirement. 3. **Phase 3**: Convert the modules into Quarkus Extensions to leverage build-time optimizations and official plugin support. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [x] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [ADR-00009: Layered Source Code Structure](./00009-layered-source-code-structure.md) * [ADR-00002: Define a preferred stack](./00002-define-a-preferred-stack.md) # ADR-00011: URI-Based Legal Entity Identifier Scheme ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | ------------ | ----------- | | CTN Project Team | Accepted | 2026-06-05 | | CTN Technical Advisory Board | Accepted | 2026-07-08 | | CTN Steering Committee | Not Required | 2026-07-08 | | BDI Conformance Team | Proposed | 2026-07-08 | | BDI Framework Team | Proposed | 2026-07-08 | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) The ASR data model records official business register identifiers of legal entities (`LegalEntityIdentifier`). The current design uses a closed Java enum `IdentifierType` (KVK, KBO, HRB, EUID, LEI, EORI, VAT) plus a separate free-form `identifierValue` string. The model review flagged two problems: 1. **Not extensible.** Adding an identifier scheme (e.g. GLN, DUNS, a non-European business register) requires a code change, a release, and — with `EnumType.STRING` persistence — touches every consumer that switches over the enum. A registry service for an open dataspace must accept new schemes as data, not as code. 2. **Type and value are split.** An identifier is only meaningful as the pair (scheme, value). Passing two loosely coupled fields through APIs, tokens, and logs invites mismatches. A single self-describing string removes that class of error. We need a representation that: * conveys the identifier *scheme* (e.g. “this is a Kamer van Koophandel number”), * optionally carries the *value* itself (`68750110`) — scheme-only references are needed for onboarding forms (“please provide your KVK number”), query filters, and validation-source configuration, * is extensible without code changes, * interoperates with the ecosystems CTN already touches: Peppol, GLEIF, BRIS/EUID, VIES (see the [identifier enrichment architecture](../05-building-blocks/ctn-identifier-enrichment-architecture.md)). ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Keep enums (status quo) [Section titled “Option 1: Keep enums (status quo)”](#option-1-keep-enums-status-quo) * **Description**: Retain `IdentifierType` enum + `identifierValue` string as currently modelled. * **Pros**: Type-safe in Java; database-level validation via enum string; no design work. * **Cons**: Every new scheme is a code change; type and value travel separately; no interop story with external scheme registries; explicitly rejected by the model review. ### Option 2: BDI URN namespace with readable tokens [Section titled “Option 2: BDI URN namespace with readable tokens”](#option-2-bdi-urn-namespace-with-readable-tokens) * **Description**: A single custom namespace with human-readable scheme tokens: `urn:bdi:legal-id:kvk:68750110`. A documented mapping table (kvk↔0106, kbo↔0208, …) provides ISO 6523 / Peppol interop. * **Pros**: Human-readable; uniform grammar (one namespace); full control over vocabulary; fallback and primary schemes look identical. * **Cons**: BDI must govern the token vocabulary; every external exchange needs the mapping table applied; readable tokens invite ad-hoc inventions (“kvk” vs “nl-kvk” vs “kamer-van-koophandel”). ### Option 3: ISO 6523-anchored URN with BDI fallback namespace [Section titled “Option 3: ISO 6523-anchored URN with BDI fallback namespace”](#option-3-iso-6523-anchored-urn-with-bdi-fallback-namespace) * **Description**: Use ISO 6523 International Code Designator (ICD) codes as the scheme authority: `urn:iso6523:0106:68750110`. Schemes without an ICD code use a BDI-governed fallback namespace: `urn:bdi:legal-id:eori:NL123456789`. * **Pros**: Anchored on a global (not European) standard — DUNS (0060), GS1 GLN (0088), Australian ABN (0151), Japanese Corporate Number (0188), Singapore UEN (0195) are already registered; zero-translation interop with Peppol participant identifiers; scheme vocabulary governed externally, BDI only governs the small fallback set; new schemes appear by ICD registration, not BDI code or vocabulary changes. * **Cons**: Numeric ICD codes are opaque to humans; two namespaces once the fallback is needed; the `iso6523` URN namespace identifier is not IANA-registered (see Consequences). ### Option 4: HTTPS dereferenceable URIs [Section titled “Option 4: HTTPS dereferenceable URIs”](#option-4-https-dereferenceable-uris) * **Description**: Linked-Data style identifiers: `https://id.bdinetwork.org/legal-id/kvk/68750110`. The scheme URI serves documentation and validation rules when dereferenced. * **Pros**: Dereferenceable — scheme metadata, validation rules, and trust-level mappings can be served at the URI; aligns with Gaia-X / W3C practice. * **Cons**: Ties the identifier schema to ownership and uptime of a domain; longer strings; requires hosting infrastructure and content governance before the first identifier can be minted. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 3 — ISO 6523-anchored URN with BDI fallback namespace. ### URI Grammar [Section titled “URI Grammar”](#uri-grammar) ```plaintext urn:iso6523:[:] primary: scheme has an ISO 6523 ICD code urn:bdi:legal-id:[:] fallback: scheme has no ICD code ``` * `` — four-digit ISO 6523 International Code Designator. * `` — lowercase scheme token from the BDI-governed fallback vocabulary below. * `` — the identifier value, normalised per scheme (see normalisation rules). **Optional**: a URI without the value part denotes the identifier *scheme itself* — used in onboarding forms, query filters, and validation-source configuration. Persisted `LegalEntityIdentifier` records always carry the value. ### Scheme Mapping [Section titled “Scheme Mapping”](#scheme-mapping) Current `IdentifierType` enum values map as follows: | Type | Scheme URI | Example | | ---------------------------------------- | ----------------------------------- | ------------------------------------------ | | KVK (NL Chamber of Commerce) | `urn:iso6523:0106` | `urn:iso6523:0106:68750110` | | KBO (BE Crossroads Bank for Enterprises) | `urn:iso6523:0208` | `urn:iso6523:0208:0439291125` | | LEI (GLEIF) | `urn:iso6523:0199` | `urn:iso6523:0199:529900T8BM49AURSDO55` | | EUID (BRIS) | `urn:bdi:legal-id:euid` | `urn:bdi:legal-id:euid:NLNHR.68750110` | | EORI (EU customs) | `urn:bdi:legal-id:eori` | `urn:bdi:legal-id:eori:NL123456789` | | VAT (VIES format, country-prefixed) | `urn:bdi:legal-id:vat` | `urn:bdi:legal-id:vat:NL861785721B01` | | HRB (DE Handelsregister) | *no own scheme* — represent as EUID | `urn:bdi:legal-id:euid:DEK1101R.HRB116737` | A raw HRB number is ambiguous without its register court (the same `HRB 116737` exists at multiple Amtsgerichte). The EUID form encodes the court (`DEK1101R`), and the enrichment pipeline already produces this form, so German register entries are represented as EUID rather than getting a fallback token of their own. Extensibility examples — usable immediately, no vocabulary or code change needed: | Scheme | Scheme URI | | ------------------------- | ------------------ | | GS1 GLN (global) | `urn:iso6523:0088` | | DUNS (global) | `urn:iso6523:0060` | | Australian ABN | `urn:iso6523:0151` | | Japanese Corporate Number | `urn:iso6523:0188` | ### Normalisation Rules [Section titled “Normalisation Rules”](#normalisation-rules) To make exact-match lookup reliable, identifiers are normalised before persistence or comparison: * The URN prefix and scheme part are lowercase (`urn:iso6523:0106`, not `URN:ISO6523:0106`). * The value part keeps the casing the issuing scheme defines (LEI and VAT uppercase; EUID as issued by BRIS). * Whitespace and presentation separators are stripped from the value (KVK `687 501 10` → `68750110`). * Per-scheme syntax validation (KVK: 8 digits; LEI: 20 chars ISO 17442 incl. checksum; VAT: VIES country-prefixed format) is applied where a validator is available. ### Rationale [Section titled “Rationale”](#rationale) * **Business alignment**: CTN onboards across borders; the enrichment architecture already integrates Peppol, GLEIF, and BRIS. Anchoring on the same scheme registry those ecosystems use avoids a translation layer at every boundary. * **Global, not European**: ISO 6523 is frequently mistaken for a European registry because Peppol popularised it there. The ICD list covers global schemes (DUNS, GLN) and national schemes far beyond the EU (AU, JP, SG, MY); Peppol authorities on those markets keep extending it. Non-European onboarding does not require BDI action. * **Minimal governance surface**: BDI governs only the fallback token vocabulary (`euid`, `eori`, `vat` at present). Everything else is delegated to the ISO 6523 registration authority. * **Risk assessment**: the main risk is the unregistered `iso6523` URN namespace (see Negative consequences); mitigated by a documented, trivial mapping to the Peppol canonical form. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: * New identifier schemes are configuration/data, not code. * A single string is the identifier — APIs, tokens, logs, and audit records carry one self-describing value. * Zero-translation interop with Peppol participant identifiers: `urn:iso6523:0106:68750110` ↔ `iso6523-actorid-upis::0106:68750110` is a mechanical prefix swap. * **Negative**: * The `iso6523` URN namespace identifier (NID) is **not IANA-registered**. These URIs are well-formed URNs syntactically but not resolvable through IANA registries. Peppol’s own canonical participant-identifier form (`iso6523-actorid-upis:::`) is the registered interchange format; the bidirectional mapping is trivial and must be applied at Peppol boundaries. * Numeric ICD codes are opaque to humans; UI and documentation must render a display name (from a scheme metadata table) next to the raw URI. * Two namespaces (`urn:iso6523:` and `urn:bdi:legal-id:`) once a fallback scheme is involved; consumers must accept both. * **Neutral**: * A scheme metadata table (display name, validation pattern, issuing registry, validation source) is needed regardless of the chosen option; this ADR makes the scheme URI its natural key. * Should ICD codes be registered later for EORI or EUID, the fallback URIs can be migrated by a one-time data update; both forms can be temporarily accepted as aliases. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) *The model is greenfield (pre-release). Changes are with respect to setup documents in arc42. An exploratory implementation of Phase 1 exists on branch `feat/asr-panache-entities`.* 1. **Phase 1** — model shape *(decided; implemented exploratorily)*: * `LegalEntityIdentifier` drops the `IdentifierType` enum, the `identifier_value` column, **and** the `registry_name` column (the issuing registry is a property of the scheme, derivable from the scheme-metadata table — Phase 2 — not a per-row attribute). * It gains a single `identifier_uri` column: **`String`**, `NOT NULL`, **globally unique** (a business register identifier denotes exactly one legal entity). A richer value-object type is deferred — see Phase 2. * `validation_status` is kept as an enum (orthogonal to the scheme). `validation_source` is **knowingly kept as a code-bound enum** pending Phase 3. * Repository access patterns: `findByIdentifierUri` (exact, index-friendly) and `listByScheme` (prefix match on the scheme URI, e.g. `identifier_uri LIKE 'urn:iso6523:0106:%'`). * **Open (Phase 2 migration concern):** the global-uniqueness constraint interacts with soft-delete (`is_deleted`). Resolve by either a *partial unique index* (`WHERE is_deleted = false`) or by *tombstoning the URI* on soft-delete. Not decided here. * **Caveat:** with no parser/validator yet, the entity is a plain holder that trusts callers to supply an already-normalised URI. Normalisation/validation arrives in Phase 2. 2. **Phase 2**: Add a parser/builder helper (split scheme URI / value, normalise, validate per scheme) plus the scheme metadata table keyed by scheme URI (display name, validation pattern, issuing registry, validation source). The preferred eventual type for `identifier_uri` is a **value object** (e.g. a record mapped via `AttributeConverter`) that encapsulates this parsing — promoting the Phase-1 `String` once the grammar helper exists. 3. **Phase 3**: Apply the same pattern to `ValidationSource` (also flagged in `ISSUES.md`) — validation sources become URIs reusable across schemes; candidate for a follow-up ADR. Until then the `ValidationSource` enum is retained deliberately. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [ ] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [00004-naming-follows-glossary.md](00004-naming-follows-glossary.md) — scheme display names follow the glossary * [Identifier enrichment architecture](../05-building-blocks/ctn-identifier-enrichment-architecture.md) — EUID generation, Peppol/GLEIF/VIES integration * `asr-service/src/main/java/org/bdinetwork/asr/model/ISSUES.md` — model review items driving this decision * ISO/IEC 6523 — Structure for the identification of organizations and organization parts; ICD list * [Peppol Policy for use of Identifiers](https://docs.peppol.eu/edelivery/policies/PEPPOL-EDN-Policy-for-use-of-identifiers-4.3.0-2024-11-15.pdf) — EAS/ICD code usage, canonical participant identifier form * [EUID structure (BRIS)](https://e-justice.europa.eu/489/EN/business_registers__search_for_a_company_in_the_eu) — country + register + registration number ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ---------------- | | 2026-06-05 | Proposed | Initial proposal | # ADR-00012: OnboardingRequest Lifecycle Transition Seam ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-06-13 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | RdN Question: I don’t really understand in which cases this can occur. It is pending with the “aspirant-participant” and in that case the request can only be managed with him/her. Or he/she has submitted the request and than it is in the hands of the ASR-admin. In other words in my opinion we can’t have colliding states in which ASR admin and the adspirant-participant are stepping on each other feet. ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) `OnboardingRequest` is becoming a long-lived, stateful aggregate. Today its `status` is a bare JPA-mapped enum (`Submitted, UnderReview, Approved, Rejected`) on an entity with all-public mutable fields, and the only transition that exists is intake setting `status = Submitted` inside `OnboardingRequestService.save`. The roadmap (see CONTEXT.md *Onboarding lifecycle*, and the project memory `onboarding-future-lifecycle`) turns it into an aggregate mutated by **two human actors taking turns** — a requestor (authenticated “edit onboarding request” page) and an admin (validation, with multi-round “needs more data” back-and-forth) — plus a **system sweep** that auto-rejects abandoned requests. Three handlers (edit page, admin validation, expiry job) are about to land, each tempted to mutate `status` directly. Without a seam, the transition rules (which moves are legal, who may make them, what gates each) scatter across those handlers and the invariant “an illegal transition is unrepresentable” is enforced only by reviewer vigilance. The country-flow divergence (NL eHerkenning step-up, DE document upload) also needs a home; the settled intake design keeps intake DTOs uniform, so divergence must live **post-intake**, in this phase. We need a place that owns the lifecycle so that: * the legal set of state transitions is defined once and illegal moves cannot be expressed, * each transition records *who* performed it, * per-country enrichment requirements plug in as guards without leaking into the intake DTOs or the core, * the one-open-request-per-identifier intake invariant (ADR-scope of the duplicate guard) and the lifecycle stay consistent about which states are “open”. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Status quo — anaemic field, transitions in handlers [Section titled “Option 1: Status quo — anaemic field, transitions in handlers”](#option-1-status-quo--anaemic-field-transitions-in-handlers) * **Description**: Keep `public Status status` and let each handler assign it. * **Pros**: No new code; familiar. * **Cons**: Transition rules duplicated across three handlers; illegal moves (`Submitted → Approved`, mutating someone else’s request) representable and only caught by review; no single place for guards, actor recording, or the future audit hook. This is the smell the ADR exists to remove. ### Option 2: Rich-entity methods [Section titled “Option 2: Rich-entity methods”](#option-2-rich-entity-methods) * **Description**: Put `startReview()/sendBack()/approve()/…` on the `OnboardingRequest` entity; it guards its own status. * **Pros**: Behaviour with data; “tell, don’t ask”. * **Cons**: To make illegal moves unrepresentable the public `status` field must be locked, fighting the house anaemic-entity style; worse, contextual guards need injected collaborators (repositories, the per-country enrichment SPI, the acting Actor) — an `@Entity` reaching into CDI beans is the wrong dependency direction. ### Option 3: Typed state-machine library [Section titled “Option 3: Typed state-machine library”](#option-3-typed-state-machine-library) * **Description**: Model the lifecycle with a state-machine framework / explicit transition table as the runtime engine. * **Pros**: Strong “illegal edge absent from the table” guarantee. * **Cons**: Over-engineered for 5 states / 6 verbs; injectable guards still need a home outside the table; adds a dependency and a vocabulary the rest of the service doesn’t use. ### Option 4: Hybrid — pure transition core wrapped by a CDI shell *(chosen)* [Section titled “Option 4: Hybrid — pure transition core wrapped by a CDI shell (chosen)”](#option-4-hybrid--pure-transition-core-wrapped-by-a-cdi-shell-chosen) * **Description**: A dependency-free transition **core** owns the legal-edge topology `(from, verb) → to`; a CDI **shell** is the only writer of `status` and owns the contextual (I/O-dependent) guards, actor recording, and persistence. The entity’s `status` is locked behind a single `transitionTo(next)` mutator that delegates to the core. * **Pros**: Illegal transitions unrepresentable *even from inside the model package* (compiler + core, not review); pure core is a trivial unit-test/prototype target; guards that need injected deps live where injection works; fits the existing `@ApplicationScoped` / `@Transactional` style. * **Cons**: One more module than a single rich entity; the `status` field becomes the one non-public field on an otherwise public-field entity (mixed visibility). ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 4 — pure transition core + CDI shell + locked setter. ### State set [Section titled “State set”](#state-set) ```plaintext Submitted ──startReview──▶ UnderReview ──approve──▶ Approved (terminal) │ ▲ sendBack │ │ resubmit ▼ │ ChangesRequested UnderReview ──reject──▶ Rejected (terminal) Submitted | ChangesRequested ──expire──▶ Rejected ``` * `ChangesRequested` is a **distinct first-class state**, not a flag on `UnderReview` — it marks that control has passed back to the requestor. * Stale/abandoned requests are auto-rejected into **`Rejected`**; there is no separate `Expired` state (product treats abandonment as a rejection). * **Open states** (a request “in progress”) = `Submitted, UnderReview, ChangesRequested`. **Terminal** = `Approved, Rejected`. ### Lifecycle verbs [Section titled “Lifecycle verbs”](#lifecycle-verbs) | verb | edge | actor | | ------------- | ---------------------------------------- | ------------------- | | `startReview` | Submitted → UnderReview | admin (explicit) | | `sendBack` | UnderReview → ChangesRequested | admin (note req.) | | `resubmit` | ChangesRequested → UnderReview | requestor | | `approve` | UnderReview → Approved | admin | | `reject` | UnderReview → Rejected | admin (note req.) | | `expire` | Submitted \| ChangesRequested → Rejected | system service acct | * Creation into `Submitted` is **Intake**, not a verb (intake stays one-phase). * **Enrichment** (requestor adding/correcting documents and data) is a **separate guarded data-update**, *not* a transition; `resubmit` is the requestor’s explicit “I’m done, look again” move. `startReview` is explicit (an admin claim), never implicit on a list/read. ### Structure [Section titled “Structure”](#structure) * **Pure core** (in the `…onboarding` model/lifecycle package): the transition table, `transition(from, verb) → to`, and the derived `isTerminal(status)` = “no outgoing edge” / `isOpen` = its negation. No JPA, no CDI, no I/O. * **Entity**: `status` is non-public; the sole mutator `transitionTo(next)` delegates to the core for legality (throws `IllegalTransitionException` on an edge the table does not contain). JPA field-access hydration is unaffected. * **Shell** (CDI, `@Transactional`): loads the aggregate, runs the **contextual guards**, calls `entity.transitionTo(next)`, stamps actor/time, persists. It is the only writer of `status`. ### Guards and authorization [Section titled “Guards and authorization”](#guards-and-authorization) * **Topology** (is this edge legal at all) lives in the core. * **Contextual guards** (is this legal edge allowed *now*) live in the shell: ownership, per-country enrichment completeness, the expiry clock. * **Authorization is two-layer**: coarse role checks stay at the HTTP/resource layer (existing Quarkus security policy — admin vs authenticated requestor); the **data-dependent ownership guard** (`actor.sub == request.applicantUserId` for requestor verbs) lives in the shell, where the aggregate is loaded, so it cannot be bypassed by a new endpoint. * **Every verb carries a non-null Actor** (KC `sub`), used for both the ownership guard and the future audit record. `expire` runs under a **service-account Actor**, not as an absent actor (see the narrowed *Actor* glossary entry). Only Intake is actor-less. ### Per-country divergence [Section titled “Per-country divergence”](#per-country-divergence) Per-country enrichment requirements are a pluggable SPI, **`OnboardingEnrichmentPolicy`**, mirroring the existing `OnboardingProvider` SPI (ADR-00010 phase 1): CDI-discovered, `supports(country)` + a completeness check, resolved by **`intakeCountry`** (the server-stamped flow country, not the applicant’s address country). The shell invokes it as the guard on **`resubmit`** — a requestor cannot hand the ball back to review until systematic country requirements are met. Ad-hoc admin requests (“scan is unreadable”) are handled by the `sendBack` note and re-judged by the admin, not machine-guarded. ### Audit [Section titled “Audit”](#audit) A full transition history (per-round who/when/from→to/note) is **deferred** to a future auditing service; **until then transition history is not persisted**. The shell, as the single transition writer, is the **one place the future audit call goes** and is marked as an explicit (currently no-op) extension point. Retained as current state now: a single overwritten `lastDecisionNote` (the latest `sendBack`/ `reject` reason — functional, the requestor reads it) and the existing `reviewedAt`/`reviewedBy` stamps (denormalised “latest decision”). ### Open-states consistency (interaction with the duplicate-request invariant) [Section titled “Open-states consistency (interaction with the duplicate-request invariant)”](#open-states-consistency-interaction-with-the-duplicate-request-invariant) The intake duplicate guard (one open request per business register identifier) **keeps ownership of that invariant** but **sources the open-state definition from the lifecycle core** (`isOpen`) instead of a hand-maintained list. The DB partial unique index `uq_onboarding_open_identifier` and the status check constraint must be widened to include `ChangesRequested`; because SQL cannot derive the set, a test **pins the index’s status list to the core’s open set** so the two cannot drift. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: * Illegal transitions are unrepresentable; transition rules live once. * Three upcoming handlers call verbs, never `status =`. * Country divergence has a home that keeps intake DTOs uniform. * “Open vs terminal” derives from topology — adding a state can’t silently break the duplicate guard once the pin-test is in place. * **Negative**: * Transition history is lost until the auditing service exists (accepted; the seam is marked). * A DB migration is required (widen check constraint + partial index for `ChangesRequested`). * Mixed field visibility on the entity (`status` non-public, the rest public). * **Neutral**: * Assumes an authenticated requestor identity and a system service account exist — provisioning them is **out of scope** (see dependency below). ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1 — core + entity lock**: pure transition core (table + `isTerminal`/ `isOpen`), entity `transitionTo` + non-public `status`, `IllegalTransitionException`. Unit-test the topology (a prototype is a reasonable first pass). 2. **Phase 2 — shell + verbs**: CDI transition shell with the six verbs, non-null Actor parameter, ownership guard, `lastDecisionNote`, the marked (no-op) audit hook. Migrate intake’s duplicate guard to consume `isOpen`. 3. **Phase 3 — schema**: Liquibase migration widening the check constraint and the partial unique index for `ChangesRequested`; the index-vs-core pin test. 4. **Phase 4 — enrichment SPI**: `OnboardingEnrichmentPolicy` interface + the country implementations; wire it as the `resubmit` guard. Lands with the edit page. ## Dependencies and Out of Scope [Section titled “Dependencies and Out of Scope”](#dependencies-and-out-of-scope) * **KC account at intake** (ADR-00003 sibling, **deferred**): this ADR *assumes* the requestor has an authenticated Keycloak identity (Actor = `sub`) by the time they `resubmit`/enrich, and that a service-account identity exists for `expire`. Provisioning the requestor’s KC account at intake — a shift from today’s anonymous intake — and the expiry service account are a separate identity decision to be made when the edit page is built. Not decided here. **Decided in [ADR-00013](00013-applicant-keycloak-account-provisioned-at-intake.md)**: the account is provisioned (reuse-or-create by email) on intake; the expiry service account remains out of scope. * The “edit onboarding request” page and the expiry scheduler themselves are consumers of this seam, designed separately. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed (ownership guard, service-account scope) * [ ] Performance impact assessed * [ ] Integration impact evaluated * [ ] Documentation updated (CONTEXT.md lifecycle terms added) * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [00003-separate-users-and-participants.md](00003-separate-users-and-participants.md) — Keycloak owns identity; basis for the deferred KC-account-at-intake sibling * [00010-evolving-to-quarkus-extensions.md](00010-evolving-to-quarkus-extensions.md) — the `OnboardingProvider` SPI pattern the enrichment SPI mirrors * [00011-uri-based-legal-entity-identifier-scheme.md](00011-uri-based-legal-entity-identifier-scheme.md) / [TDR-0002](../../ai/adrs/TDR-0002-onboarding-stores-registry-identifier-as-identifier-uri.md) — the business register identifier the duplicate guard keys on * `docs/ai/CONTEXT.md` — *Onboarding lifecycle* and *Actors and auditing* terms * `onboarding-future-lifecycle` (project memory) — roadmap/source of intent ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ---------------- | | 2026-06-13 | Proposed | Initial proposal | # ADR-00013: Applicant Keycloak account provisioned at intake ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-06-15 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | RdN Question: I have two cases in my mind, for which I am not certain that the proposed option will work correctly. 1. Creating accounts with fake email accounts. In other projects I stumble every time across this issue. Bots which create accounts with fake email addresses. Email addresses which never are verified. In those projects I have chosen the route only to create IdP account after the email has been verified. In order not to pollute your IdP with fake accounts, you have to cleanup afterwards. Not sure if this is feasible with Keycloak and/or proposed option below. 2. How to deal with duplicates. We certainly come across the situation that for instance a DHL user will try to register himself with ASR (again). He is customer with multiple Inland Terminals. That there is now suddenly an Inlandbase Association which supersedes the Inland Terminals will be hard to digest, and certainly be forgotten by some in the long run. ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) The onboarding request workflow needs to be demonstrable end-to-end: an applicant submits an anonymous **Intake**, an admin **reviews and decides**, and — when the admin sends the request back — the applicant logs in to an authenticated **edit page** to enrich/correct the request and `resubmit` it. That authenticated edit page is the problem. ADR-00012 (the lifecycle seam) *assumes* the requestor already holds an authenticated Keycloak identity (Actor = `sub`) by the time they `resubmit`, and explicitly **deferred** the question of how that identity comes to exist: *“Provisioning the requestor’s KC account at intake — a shift from today’s anonymous intake — … a separate identity decision to be made when the edit page is built. Not decided here.”* The edit page is now being built, so the decision is due. Two settled constraints frame it: * **Intake stays anonymous and low-effort** (glossary *Intake*): no login, no registry calls, syntax + local duplicate check only. Adding a credentialing step to the intake form would erode this. * **ADR-00003 separates users (Keycloak) from participants (ASR)** and resolves the chicken-and-egg with *User-Identity-First*: a person registers in Keycloak, *then* onboards. Anonymous intake already departs from that ordering; this ADR must say how identity is reconciled for the onboarding flow without reintroducing the prototype’s anti-patterns (business data in Keycloak, read-only-profile dead-ends). `OnboardingRequest.applicantUserId` (a nullable Keycloak `sub`) already exists on the entity; what is undecided is **when and how it is populated**. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Silent mint at intake + set-password email *(chosen)* [Section titled “Option 1: Silent mint at intake + set-password email (chosen)”](#option-1-silent-mint-at-intake--set-password-email-chosen) * **Description**: On successful intake, asr-service **ensures** a Keycloak account for the applicant via the KC Admin API — **reuse-or-create keyed by email** — stores its `sub` in `applicantUserId`, and (only when it *created* the account) triggers Keycloak’s set-password / reset-credentials email. The intake form gains **no** password or username field. The submission response tells the applicant to expect the email and includes a link to the edit page for this request. * **Pros**: Intake stays minimal and anonymous-feeling (no credentialing step on the form); the request↔account link (`sub`) exists from submission, so every later requestor verb has a real Actor exactly as ADR-00012 assumes; account creation is a single-writer server-side side-effect (asr-service → KC Admin API), consistent with ADR-00007’s mutation-proxy direction. * **Cons**: Inverts ADR-00003’s *User-Identity-First* ordering for this flow (identity becomes a side-effect of intake); intake now has an external dependency (KC Admin API) and an email side-effect, so a KC outage degrades intake. ### Option 2: Self-service registration at intake [Section titled “Option 2: Self-service registration at intake”](#option-2-self-service-registration-at-intake) * **Description**: The intake form itself collects a password (or runs Keycloak’s registration), so the applicant leaves intake already credentialed. * **Pros**: Simplest credential flow; no later “ensure account” reconciliation. * **Cons**: Makes intake heavier and no longer anonymous/low-effort — a direct conflict with the settled *Intake* design; pushes a security step into the public, unauthenticated entry point. ### Option 3: Deferred — mint on first edit access [Section titled “Option 3: Deferred — mint on first edit access”](#option-3-deferred--mint-on-first-edit-access) * **Description**: Intake stays fully anonymous, `applicantUserId` null. The account is created later, when the admin sends the request back and the applicant follows a magic link that triggers Keycloak registration. * **Pros**: Keeps intake perfectly pure; no KC dependency at submission. * **Cons**: The request has **no Actor** until first edit, complicating the ownership guard and audit story ADR-00012 relies on; the magic-link-to-KC binding is the genuinely hard part and is merely postponed, not avoided. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 1 — silent reuse-or-create at intake, set-password email on create. This is a deliberately **simple first version** to make the workflow demonstrable. ### Rationale [Section titled “Rationale”](#rationale) * It satisfies the demo’s gating requirement (a login-protected applicant edit page) while preserving the minimal-intake invariant, because credentialing happens out-of-band via Keycloak’s own email rather than on the intake form. * **The stable request↔account link is the Keycloak `sub`, never the email.** Email is configured as a Keycloak **login alias** (`loginWithEmailAllowed`), and the Keycloak **username is system-generated** (internal, stable). The applicant logs in with their email; if the email later changes it is updated via the identity-mutation-proxy (ADR-00007) without touching the `sub` or the username — so “email changeable without changing the account” holds by construction. * **Reuse-or-create** handles the returning applicant (same person, second company / different identifier): provisioning looks the user up by email, reuses the existing `sub` if present (and sends **no** second reset email), otherwise creates and triggers the email. This avoids a Keycloak duplicate-email collision that “always create” would hit. * The ordering inversion vs ADR-00003 is accepted *for the onboarding flow only* and recorded here so a future reader does not “fix” it back to identity-first. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: Edit page is unblocked; every requestor lifecycle verb has a non-null Actor from submission onward; no business data is written to Keycloak (only identity), so the prototype anti-patterns are avoided. * **Negative**: Intake gains a synchronous dependency on the KC Admin API and an email side-effect; a Keycloak failure degrades intake (mitigation/atomicity is an implementation concern — provisioning failure must not silently produce a request with a null `applicantUserId` that can never be edited). * **Neutral**: The intake action itself **remains actor-less** — the *Actor* glossary entry is unchanged; the account is provisioned *as a result of* an anonymous action, the anonymous action does not acquire an Actor. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1 — provisioning**: asr-service “ensure KC account” step on intake (reuse-or-create by email; generated username; `loginWithEmailAllowed`; store `sub`); trigger set-password email on create only; **grant the `requestor` realm role to the ensured account on both paths** (idempotent top-up — see *Requestor role grant* below). 2. **Phase 2 — submission response**: change the intake response to instruct the applicant to expect the email and link to the edit page for the request. 3. **Phase 3 — applicant edit page**: login-protected; editable fields `legalName, applicantName, addressLine1, postalCode, city, countryCode`; read-only `identifierUri, applicantEmail, intakeCountry, status`; editable in `Submitted` and `ChangesRequested`; ends a round with `resubmit`. 4. **Phase 4 — admin review screen**: read all fields + drive `startReview/approve/reject/sendBack`; disabled placeholder controls for the future admin-side verification (trust-level assignment, DE manual-proof acceptance), no logic. The `resubmit` enrichment guard keeps the ADR-00012 `OnboardingEnrichmentPolicy` SPI seam, with a single **permissive default** registered for the demo (no completeness gate); per-country enrichment gating remains future work. ### Requestor role grant (issue #47) [Section titled “Requestor role grant (issue #47)”](#requestor-role-grant-issue-47) ADR-00012 gates the owner-scoped requestor endpoints with `@RolesAllowed("requestor")`. The provisioner is therefore the producer that must satisfy that gate: `ensureAccount` **explicitly grants the `requestor` realm role** (`Roles.REQUESTOR`) to the account via the admin client, on **both** the create and reuse paths. The grant is idempotent (Keycloak’s role-mapping add is a no-op when present), so the reuse path **tops up unconditionally** rather than assuming a prior grant — a returning or externally-created account is never left without the role. A grant failure **propagates** (intake fails loudly); a dangling KC account from a failed grant self-heals on retry via the reuse-path top-up. This **replaces a dev-only stopgap** where `requestor` sat in the realm’s `default-roles-quarkus` composite, so any user created without explicit roles (and the service accounts) silently inherited it. That composite membership is removed in the same change, so dev mirrors production: only provisioned applicants carry `requestor`. **Production contract** (no production realm IaC exists in this repo yet — not near prod; recorded here as the deployment requirement): the production Keycloak realm must define a **non-composite `requestor` realm role**, and the asr-service admin client’s service account must hold **`view-realm`** (to read the role representation) **and `manage-users`** (to add the role mapping). The grant code is realm-neutral; only these two facts must hold wherever it runs. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed (KC Admin API scope, reuse-or-create, no business data in KC) * [ ] Performance impact assessed (intake → KC Admin API synchronous call) * [ ] Integration impact evaluated (intake degraded on KC outage; provisioning atomicity) * [ ] Documentation updated (CONTEXT.md *Intake* / provisioning terms) * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [00003-separate-users-and-participants.md](00003-separate-users-and-participants.md) — Keycloak owns identity; the *User-Identity-First* ordering this ADR inverts for the onboarding flow * [00007-identity-mutation-proxy.md](00007-identity-mutation-proxy.md) — the path by which the applicant’s email (login alias) is later mutated without changing the account * [00012-onboarding-request-lifecycle-transition-seam.md](00012-onboarding-request-lifecycle-transition-seam.md) — the lifecycle seam that deferred this decision and assumes the Actor this ADR provisions * Anchor issue [#33](https://github.com/Basic-Data-Infrastructure/CNCL-BDI/issues/33) — driving requirement + acceptance criteria (implementation issues link here) * `docs/ai/CONTEXT.md` — *Onboarding* / *Actors* terms ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 2026-06-14 | Proposed | Initial proposal | | 2026-06-15 | Proposed | Folded in #47: provisioner grants `requestor` explicitly (both paths, idempotent top-up), retiring the `default-roles-quarkus` stopgap; production realm/scope contract recorded | # ADR-00014: Claim Verification Provider Seam and eHerkenning Control-Proof ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-06-17 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) An applicant must be able to **prove** the claims on an Onboarding Request — first that the company they are registering is real, and that **they are authorised to act for it**. The first authorised-representation mechanism is **eHerkenning** (the Dutch eID for businesses), surfaced as an authenticated step-up during onboarding. In the (hopefully not too distant future also European Business Wallets, eIDAS or known in NL as EDI-wallet). This is the first instance of a seam that will recur: future modules will poll **KvK**, **VIES**, **GLEIF**, **Handelsregister**, the Belgian **KBO**, and comparable institutions in BE/DE/elsewhere. The output of each is the same in kind — **validated claims coupled with an assurance level** — even though the mechanisms differ wildly (an interactive browser redirect vs a server-to-server register lookup). The seam should be **reusable across consortia**: multiple NL dataspaces may use the same eHerkenning module with their own onboarding flows, which points at the marketplace layer (ADR-00009, ADR-00010). Three prior decisions constrain this one: * **ADR-00003** — identity lives in Keycloak; participant business data, **verification records** live in ASR. No business data mirrored as Keycloak attributes. * **ADR-00006** (still open) — how participant context (including verified identifiers) is composed into tokens. Its prototype findings **M16** (trust level missing from tokens) and **M17** (identifier used as a mutable attribute key) are the cautionary tales here. * **ADR-00008** — Keycloak is the only IdP ASR talks to; eHerkenning federates **behind** Keycloak. ADR-00013 already provisions an applicant Keycloak account at intake. Please note that for Connected Trade Network we provide eHerkenning as an optional proof. In many cases the user is not aware how to use eHerkenning. Usually it is with his accountant. ### What eHerkenning actually returns (research, 2026-06-14) [Section titled “What eHerkenning actually returns (research, 2026-06-14)”](#what-eherkenning-actually-returns-research-2026-06-14) A premise correction that shapes the model. eHerkenning (technically the *Afsprakenstelsel Elektronische Toegangsdiensten*) is **SAML 2.0 only**; a contracted **Herkenningsmakelaar** (broker) is mandatory — one cannot join the federation directly. A successful login asserts: * **LegalSubjectID / EntityConcernedID** — the company, by **KvK number** (`urn:etoegang:1.9:EntityConcernedID:KvKnr`), RSIN, or vestigingsnummer (a ServiceRestriction). Encrypted per recipient. * **ActingSubjectID** — the person, as a **per-service-provider pseudonym** (`urn:etoegang:core:ActingSubjectID`), **not a name**. The real name/BSN is *not* delivered under normal representation, at any assurance level. * **Representation** + a **mandate** (machtiging): the Machtigingenregister asserts the acting person is **authorised to act for that company for this service**. * **Assurance level** — `urn:etoegang:core:assurance-class:loa3` (EH3 = eIDAS *Substantial*, the de-facto minimum) or `loa4` (EH4 = *High*). Assurance does **not** change whether a name is returned. * **No company name** in core eHerkenning (resolved out-of-band from KvK). A broker OIDC facade (a commercial eID broker) exposes a convenience `organisation` claim, but it is broker-derived, not an authoritative etoegang attribute. So eHerkenning proves **control/authority over an identifier**, plus a baseline that the identifier is a live, registered entity at login time. It does **not** prove the entity’s **attributes** (name, address, VAT, EUID) — those still require register enrichment (already built as `nl.ctn.asr.enrichment.*`). For cross-border (BE/DE), foreign companies arrive through the *same* broker via the eIDAS koppelpunt and yield a foreign legal identifier + legal name but no KvK — an origin-dependent claim shape. ### The central question [Section titled “The central question”](#the-central-question) How are these verified claims captured into ASR such that **nobody can easily tamper with them**, the mechanism is **reusable** across verification sources and consortia, and it respects the Keycloak/ASR ownership boundary? ## Considered Options [Section titled “Considered Options”](#considered-options) The contested, load-bearing choice is **where the verified claims live and how they get there**. (The seam shape and lifecycle choices follow from it and are recorded under Decision Outcome.) ### Option 1: Store eHerkenning claims as Keycloak user attributes, read via userinfo [Section titled “Option 1: Store eHerkenning claims as Keycloak user attributes, read via userinfo”](#option-1-store-eherkenning-claims-as-keycloak-user-attributes-read-via-userinfo) * **Description**: Persist KvK / assurance / mandate as attributes on the applicant’s Keycloak user; ASR reads them from the OIDC userinfo endpoint. * **Pros**: No ASR persistence to build; claims “follow” the account. * **Cons**: Violates ADR-00003 (business/verification data in Keycloak). Keycloak user attributes are **admin-mutable plaintext** — the exact tamper hole to avoid; Keycloak neither signs nor enforces them. Re-creates **M16/M17**. A company is **not** a property of a person’s account (one person may hold mandates for several companies). Userinfo returns *current* attribute state, not what was true at verification time. **Rejected.** ### Option 2: Transient signed token → ASR verifies → ASR persists an immutable attestation (chosen) [Section titled “Option 2: Transient signed token → ASR verifies → ASR persists an immutable attestation (chosen)”](#option-2-transient-signed-token--asr-verifies--asr-persists-an-immutable-attestation-chosen) * **Description**: Keycloak performs the (brokered) eHerkenning login and mints a **short-lived token carrying the claims as transient transport** (session-scoped, not durable attributes). ASR verifies signature/audience/freshness, extracts the claims, and writes a **write-once attestation** into its own store, with an evidence reference to the underlying assertion. * **Pros**: Claims live in ASR (ADR-00003 single source of truth). Tamper-resistance: append-only attestation + audit, no admin-editable verification state. Transport token is one-time and signature-verified. Identity facts that *are* account-bound (pseudonym, `idp`) legitimately stay in Keycloak. Achievable **config-only** in Keycloak on a broker OIDC facade (no plugin). * **Cons**: Requires ASR persistence + a Keycloak mapper chain. The attestation is point-in-time and can go stale (accepted — aging deferred). ### Option 3: Keep/pass the original eHerkenning assertion as the primary store [Section titled “Option 3: Keep/pass the original eHerkenning assertion as the primary store”](#option-3-keeppass-the-original-eherkenning-assertion-as-the-primary-store) * **Description**: Persist and re-parse the raw signed SAML assertion as the record of truth. * **Pros**: Cryptographically self-evident. * **Cons**: Forces every consumer to parse SAML; brittle; couples ASR to one source’s wire format. **Rejected as primary** — but a reference/hash of the assertion is kept as **evidence** on the Option-2 attestation. ### Option 4: Keep token minimal, ASR exposes a context endpoint fetched on demand [Section titled “Option 4: Keep token minimal, ASR exposes a context endpoint fetched on demand”](#option-4-keep-token-minimal-asr-exposes-a-context-endpoint-fetched-on-demand) * **Description**: ADR-00006 Option 4 style — no verification claims in the token; consumers call ASR when needed. * **Pros**: Clean boundary; small tokens. * **Cons**: Addresses *serving* claims to downstream consumers, not *capturing* a one-time interactive verification event. Orthogonal to this decision; left to ADR-00006. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: **Option 2** — a transient signed token carrying eHerkenning claims into ASR, which verifies and persists an immutable attestation; Option 3 retained only as evidence. Built behind a reusable seam: ### The seam [Section titled “The seam”](#the-seam) * **Claim Verification Provider** — a marketplace SPI in core `org.bdinetwork.asr.spi` that, given a subject (an Onboarding Request’s identifier + applicant), returns a **Verified Claim Set**: individually-sourced claims, each carrying its **Validation Source** and an **assurance level**, in a broker-neutral, eIDAS-aligned vocabulary (`LOW` / `SUBSTANTIAL` / `HIGH`). **Not** a fixed `{kvk, name}` record — the shape varies by source and by subject origin. * **Two sub-interfaces over the shared output**: `InteractiveClaimVerificationProvider` (`start()` → redirect, `complete()` → Verified Claim Set; user present, multi-round capable) and `BackendClaimVerificationProvider` (`verify()` → Verified Claim Set; no user). Forcing one signature onto both would leak one mode’s accident into the other. * **Two attestation kinds, kept distinct**: **control** (eHerkenning: “a mandate-holder controls KvK X”) vs **existence/attributes** (register enrichment: “KvK X is real and looks like this”). Control proof implies existence-at-login-time, but the two have independent revocation triggers, so they are separate attestations. ### eHerkenning specifics [Section titled “eHerkenning specifics”](#eherkenning-specifics) * **Keycloak-as-broker** (ADR-00008). **Broker independence is a requirement**: ASR always consumes a **broker-neutral** claim set; the broker-specific mapping is confined to the Keycloak IdP-mapper layer. The transport vs implementation choice — **Path A** (broker OIDC facade, e.g. a commercial eID broker; config-only Keycloak) vs **Path B** (the implementation partner `Extended-SAML-IDP` plugin; broker-agnostic) — is deferred to research ticket **KEYC-1192**. * The step-up is an **account-linked, same-person** login on the intake-provisioned account (ADR-00013): the applicant is already authenticated; eHerkenning is linked onto that **current session** via Keycloak’s Application-Initiated Action `idp_link` (`kc_action=idp_link:eherkenning-sim`), which requires the requestor to hold the `account` role `manage-account-links`. The link is **session-scoped** and matched by the live session, **not by email** — faithful to eHerkenning, which returns **no name and no email**. The **acting-subject pseudonym, representation flag and assurance are stored on the attestation** (the authoritative “who proved the mandate”); same-person is a **flow assumption, not a schema invariant**, so delegation is later additive. * **Persistence**: extend the existing `verification` table (`type=EHerkenning`, `category=Automatic`) with a nullable `onboarding_request_id` (no Participant exists until approval), a structured `claims` payload, and `assurance_level`. The **claim payload is write-once** (re-verification = a new row); the verification **status** (Pending / Approved / Rejected / Revoked) is the mutable lifecycle layered on top; status history lives in `audit_record`. * **Behaviour**: lives under **Enrichment** (the authenticated edit page) and does **not** auto-transition the request. On `complete()`: the eHerkenning KvK is compared to the request’s identifier — **match** → positive control attestation; **mismatch** → failure recorded **and flagged for admin attention** (fraud/mistake guard). Offered **NL-only** for MVP. The admin assigns the trust level at approval with the attestation as evidence. ### Simulated broker realm for the end-to-end demo (issue #45) [Section titled “Simulated broker realm for the end-to-end demo (issue #45)”](#simulated-broker-realm-for-the-end-to-end-demo-issue-45) * A second Keycloak realm, **`eherkenning-sim`** (`asr-service/src/dev-services/keycloak/eherkenning-sim-realm.json`), is imported alongside `quarkus-realm.json` (multi-value `realm-path`). It is a **faithful OIDC stand-in for a commercial eID broker’s eHerkenning OIDC facade** (Path A): a per-acting-person user (e.g. the seeded `arda` acting-person login, no email) whose token emits eHerkenning-shaped claims (a KvK number, an acting-subject pseudonym, an assurance-class LoA, and a representation flag) via **per-user attribute protocol mappers** (`oidc-usermodel-attribute-mapper`, #62) — each fact is read from the logged-in acting person’s own user attributes rather than a realm-global hardcoded value, so different acting persons can prove different KvK numbers. * The application realm federates it declaratively as the `eherkenning-sim` **OIDC broker IdP** (`identityProviders`), with `storeToken=false`. The interactive link is driven by the Application-Initiated Action `idp_link` (`kc_action=idp_link:eherkenning-sim`), which links the brokered identity onto the requestor’s **current session** — **not** matched by email (eHerkenning returns none) and never a duplicate (ADR-00013). The transient-claim chain is **IdP-mapper → user-session-note → protocol-mapper** entirely in the realm JSON: `oidc-user-session-note-idp-mapper`s carry each brokered claim into a session note (**no** `oidc-user-attribute-idp-mapper`, so nothing lands on a Keycloak user attribute on the requestor account — ADR-00003/00006), and `oidc-usersessionmodel-note-mapper`s on the interactive-login client emit each note into the issued token. * **Only the literal broker claim names are unverified guesses** at a commercial eID broker’s contract (documented inline in both realm JSONs). All broker-claim-name mapping lives in the IdP-mapper layer (ADR-00014/00008): a **real-broker swap is a configuration change confined to that layer**, not a code change — `EHerkenningTokenVerifier` reads the **neutral token claim names** `kvk` / `loa` / `acting_subject` / `represents`, which **default** to these literals but are themselves **devops-configurable** (see *Configurable neutral token claim contract* below, #72). > **Note (2026-06-15, spike — see PRD #61):** the *interactive* link mechanism settled as **Keycloak Application-Initiated Action `idp_link`** (`kc_action=idp_link:eherkenning-sim`), as the body above now describes. Two earlier candidates were rejected and are recorded here for history: a **verified-email `first broker login` match** (superseded — eHerkenning returns no email, so there is nothing to match on) and **Client-Initiated Account Linking** (`/broker/{idp}/link`, deprecated in KC26 and rejects the public client). The AIA links the brokered identity onto the **current** requestor session and mints a token carrying the claims; it requires the requestor to hold the `account` role `manage-account-links`. Brokered claims are **session-scoped** (they ride every token for the session lifetime, surviving refresh and repeat login under `syncMode=IMPORT`); the protection is the audience-restricted verification + write-once attestation, not token scarcity. The sim emits **no name**. Path A/B and the production key source remain KEYC-1192. ### Configurable neutral token claim contract (2026-06-17, #72) [Section titled “Configurable neutral token claim contract (2026-06-17, #72)”](#configurable-neutral-token-claim-contract-2026-06-17-72) The four token claim names the ASR verifier reads — and the LoA value vocabulary it maps to assurance levels — are **devops-configurable via `application.properties`**, each defaulting to today’s literal so an **unconfigured deployment is byte-for-byte unchanged**. * **Claim names** — under the existing `eherkenning.token.*` family (sibling to `expected-audience`, `allowed-skew-seconds`, `public-key-location`), exposed as a `@ConfigMapping`: * `eherkenning.token.claim.kvk` (default `kvk`) * `eherkenning.token.claim.loa` (default `loa`) * `eherkenning.token.claim.acting-subject` (default `acting_subject`) * `eherkenning.token.claim.represents` (default `represents`) * `EHerkenningTokenVerifier` reads each field by its configured name, and `TOKEN_CLAIM_NAMES` is **derived from the configured set**, not a hardcoded literal set. \[#73] * **LoA value vocabulary** — the broker LoA value strings accepted for each eIDAS level, as lists: * `eherkenning.token.loa.substantial` (default `loa3,EH3,urn:etoegang:core:assurance-class:loa3`) * `eherkenning.token.loa.high` (default `loa4,EH4,urn:etoegang:core:assurance-class:loa4`) * An unrecognised value still fails with the same rejection semantics as today. \[#74] * **Transport**: `application.properties` only — the `quarkus-config-yaml` extension is **out of scope** (maintainer call, 2026-06-17). **Boundary stance (relation to broker-neutrality).** This **preserves** the broker-neutral principle, it does not relax it. Raw broker attribute names — etoegang SAML URNs (`urn:etoegang:1.9:EntityConcernedID:KvKnr`, …) or a a commercial eID broker OIDC-facade’s claim names — continue to be mapped to neutral claims **only** in the Keycloak IdP-mapper layer; **no raw broker name enters ASR**. What becomes configurable is the **neutral token-claim contract**: the key names (and LoA value vocabulary) the verifier expects in the *already-verified, already-neutralised* token. Previously both ends — the Keycloak protocol-mapper output and the ASR verifier input — independently hardcoded the same four literals, an implicit coupling. Externalising the ASR end lets that contract be re-aligned with a re-configured mapper output (or a different broker facade) **without an ASR code change or redeploy** — the same config-only swap goal the IdP-mapper layer already serves. The relaxation is narrow and deliberate: claim **values** are still sourced only from the verified token (M16/M17 and ADR-00003 unaffected); only the *key names and LoA vocabulary* the verifier reads under are externalised. ### Rationale [Section titled “Rationale”](#rationale) * **Business alignment**: proves authorised representation — the strongest thing eHerkenning offers — while keeping the door open to reuse across consortia and across future sources. * **Technical**: honours ADR-00003 (claims in ASR, not Keycloak), avoids M16/M17, and is achievable config-only on a broker OIDC facade. The control/existence split matches the genuinely different things eHerkenning and register lookups attest. * **Risk**: the chief tamper risk (admin-editable verification state) is removed by keeping claims in an append-only ASR attestation; the transport token is one-time and verified. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: single source of truth for verification (ASR); tamper-resistant append-only attestations; one reusable seam for all present and future verification sources; Keycloak stays stock on the broker-OIDC path; broker-swappable by config. * **Negative**: attestations are **point-in-time and can go stale** — **aging / revocation / re-verification is explicitly out of scope** for now and must be designed before relying on attestation freshness. A Keycloak mapper chain (IdP-mapper → session note → protocol mapper) must be built and tested to keep claims transient (not persisted to user attributes). * **Neutral / deferred (tracked)**: * **Delegation / ketenmachtiging** (`eherkenning_intermediate_*`), **BE/DE-via-eIDAS**, and a numeric trust-level mapping are deferred. * **Convergence debt**: the existing `nl.ctn.asr.enrichment.*` cluster is the non-interactive provider in embryo but writes `legal_entity_identifier` directly rather than attestations. For MVP it is **left as-is**; folding it behind `BackendClaimVerificationProvider` (unifying `EnrichmentResult` with `VerifiedClaimSet`) is deferred. Two models coexist until then. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1 (MVP / demo)**: Define the SPI + `VerifiedClaimSet` in `org.bdinetwork.asr.spi`; implement the interactive eHerkenning provider in `org.bdinetwork.marketplace.asr.eherkenning`; wire Keycloak → broker (a commercial eID broker OIDC for the demo) with the transient-claim mapper chain; extend the `verification` table; surface match/mismatch on the admin review screen. NL-only. 2. **Phase 2**: Resolve **KEYC-1192** (Path A vs Path B; broker-neutral claim contract across HMs). Design attestation **aging / re-verification / revocation**. 3. **Phase 3**: Fold `nl.ctn.asr.enrichment.*` behind `BackendClaimVerificationProvider` (convergence). Add delegation / ketenmachtiging and BE/DE-via-eIDAS. Align with ADR-00006. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed (focus: transient-token verification, no claims in Keycloak attributes, mismatch/fraud handling, append-only attestation integrity) * [ ] Performance impact assessed * [ ] Integration impact evaluated (Keycloak mapper chain, broker dependency) * [ ] Documentation updated (CONTEXT.md glossary entries added) * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [00003-separate-users-and-participants.md](00003-separate-users-and-participants.md) — ownership boundary * [00006-token-claim-composition.md](00006-token-claim-composition.md) — token claims, M16/M17 (advances) * [00008-external-idp-federation-strategy.md](00008-external-idp-federation-strategy.md) — Keycloak-as-broker (refines) * [00009-layered-source-code-structure.md](00009-layered-source-code-structure.md), [00010-evolving-to-quarkus-extensions.md](00010-evolving-to-quarkus-extensions.md) — SPI / marketplace layering * [00011-uri-based-legal-entity-identifier-scheme.md](00011-uri-based-legal-entity-identifier-scheme.md) — Identifier URI matched against * [00012-onboarding-request-lifecycle-transition-seam.md](00012-onboarding-request-lifecycle-transition-seam.md) — Enrichment vs lifecycle verbs * [00013-applicant-keycloak-account-provisioned-at-intake.md](00013-applicant-keycloak-account-provisioned-at-intake.md) — the account linked at step-up * [ASR Data Model](../05-building-blocks/ctn-asr-data-model.md) — `verification`, `audit_record` * [Identifier Enrichment Architecture](../05-building-blocks/ctn-identifier-enrichment-architecture.md) — the non-interactive providers * [CONTEXT.md](../../ai/CONTEXT.md) — Claim Verification Provider, Verified Claim Set, Attestation * the implementation partner Jira **KEYC-1192** — broker-independent eHerkenning integration (Path A vs B) ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 2026-06-14 | Proposed | Initial proposal from grilling session | | 2026-06-15 | Proposed | Interactive link mechanism settled as AIA `idp_link` (CIAL deprecated in KC26; email-match superseded), spike-verified; see PRD #61 | | 2026-06-17 | Proposed | Neutral token-claim contract (the four claim names + LoA value vocabulary) made devops-configurable via `application.properties`, defaults preserve behaviour; broker-neutrality preserved (raw names stay in the IdP-mapper layer). See #72 / #73 / #74 | # ADR-00015: End-to-End Browser Test Tooling — Playwright + Chromium, Failsafe-bound ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-06-29 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | RdN: This ADR is about E2E testing. Playwright, chromium is fine for me. I would like to understand how the overall testing strategy will look like. For instance in the previous prototypes I have used testcases for API testing, which I would like to re-use and submit into testsuite. I also have E2E tests, obviously they all need to be altered a bit, so they will work with this solution. My goal is that we (product owner, end-users etc.) can add testcases, which will be added to the testsuite and which will be tested every time we deploy. Yup, old fashioned Test Driven Development, you never have enough test cases, regression issues will surface rather sooner than later dadadadidadida. ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) The eHerkenning control step-up (ADR-00014) is a browser-driven flow: the `asr-demo-frontend` `applicant-edit.html` page runs OIDC code + PKCE through keycloak-js, drives a Keycloak Application-Initiated-Action `idp_link` round-trip to the simulated eHerkenning broker realm, then POSTs the brokered token to the `complete-token` endpoint and reflects the recorded control attestation. The existing `EHerkenningStepUpBrokerIT` (issue #45) proves the broker + token chain at the **HTTP/token layer only**, via RestAssured and the Keycloak admin client — **no browser**. Nothing exercises the frontend’s OIDC-PKCE init, the AIA `idp_link` confirmation, or the page’s `complete-token` POST + reload. That gap let frontend/OIDC behaviour pass `mvnw verify` green-but-broken. Issue #67 closes it with one real-browser tracer-bullet test, which needs a browser-automation framework wired into the build. **Scope.** This decision is about **e2e test tooling only**. It is explicitly **not** a resolution of the open *frontend-stack* question for the ASR portals (tracked as an Open Question in [ctn-coding-standards.md](../08-crosscutting/ctn-coding-standards.md)). The `asr-demo-frontend` is a throwaway static demo (plain HTML + vanilla JS, no build step), not the portal stack. > **Relationship to other test ADRs.** This ADR covers only the **tactical, in-repo Java e2e tier** — one real-browser test bound to the backend’s Maven build. The **strategic UI-automation framework** for the real portal (Playwright + TypeScript, TAaaS-owned) is a separate decision: [ADR-000017](000017-test-automation-using-playwright-with-typescript.md). Both tiers sit under the overall [test strategy in ADR-00023](00023-test-strategy.md), which explains why the Java (here) and TypeScript (00017) toolchains are a deliberate per-layer split rather than a duplication. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Playwright for Java + Chromium, bound to Maven Failsafe [Section titled “Option 1: Playwright for Java + Chromium, bound to Maven Failsafe”](#option-1-playwright-for-java--chromium-bound-to-maven-failsafe) * **Description**: Add `com.microsoft.playwright:playwright` (test scope, pinned) and drive headless Chromium from a `@QuarkusTest` `*IT` running under Failsafe. * **Pros**: Single-language build (Java/Maven, no Node toolchain); auto-installs its bundled Chromium on first launch; first-class request route-interception (used here to serve keycloak-js offline); integrates with the existing DevServices-Keycloak + Docker-gated test path. * **Cons**: Pulls a Chromium binary on the first Docker-host run; pins ports the static demo hardcodes. ### Option 2: Cypress (or another Node/JS e2e framework) [Section titled “Option 2: Cypress (or another Node/JS e2e framework)”](#option-2-cypress-or-another-nodejs-e2e-framework) * **Description**: Run the e2e suite in a separate Node toolchain. * **Pros**: Popular in frontend ecosystems; rich DX for a real SPA. * **Cons**: Introduces a Node/npm build alongside Maven for a backend repo with no frontend build step; would not bind cleanly into the one Docker-gated `./mvnw verify` path; heavier for a single tracer-bullet test. ### Option 3: Selenium / WebDriver [Section titled “Option 3: Selenium / WebDriver”](#option-3-selenium--webdriver) * **Description**: Classic WebDriver-based browser automation from Java. * **Pros**: Mature, Java-native. * **Cons**: Driver/browser version management is fiddlier; no built-in request interception (needed to serve keycloak-js offline); slower, flakier API than Playwright’s auto-waiting locators. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 1 — **Playwright for Java + Chromium, bound to Maven Failsafe**, gated exactly like the database integration tests. ### Rationale [Section titled “Rationale”](#rationale) * Keeps the build single-toolchain (Maven only) for a backend repo whose only frontend is a buildless static demo. * Reuses the existing two-realm DevServices Keycloak harness (`EHerkenningStepUpBrokerProfile`) and the Docker gate, so the e2e test runs in the same `./mvnw verify` path as the `@Tag("requires-database")` suite and is skipped identically when Docker is absent. * Playwright’s request route-interception cleanly mitigates the one frontend dependency the test cannot otherwise satisfy offline (see Risks). ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: Real-browser coverage of OIDC-PKCE + AIA `idp_link` that the HTTP/token-layer IT cannot reach; one toolchain; deterministic Docker gating. * **Negative**: A Chromium binary is fetched on the first enabled run; the test pins fixed ports (the static demo hardcodes Keycloak/API coordinates). * **Neutral**: A second e2e framework could still be chosen later for the real portal once the frontend-stack Open Question is resolved; this decision does not constrain that. ### Build wiring and gating [Section titled “Build wiring and gating”](#build-wiring-and-gating) * `com.microsoft.playwright:playwright` is a **test-scoped, pinned** dependency (version in ``, not BOM-managed). * Each real-browser test class (`EHerkenningStepUpBrowserE2EIT`, `EHerkenningAdminAndStepUpBrowserE2EIT`, `BulkImportAdminBrowserE2EIT`) carries **two** JUnit5 tags: `@Tag("requires-database")` (it needs the DevServices Keycloak + Postgres stack) **and** the dedicated `@Tag("browser-e2e")` that marks it as a *real-browser* test. The browser-e2e tag — not a class-name glob — is the single discriminator every profile filters on, so adding or renaming a browser e2e class needs no build-config edit. They run via the Failsafe `database-integration-tests` execution. Because they bind Keycloak to a **fixed** port they must run exactly once; `requires-database` keeps them out of the default `*IT.java` execution (which would otherwise run every `*IT` a second time and collide on the pinned port). * **Not run in CI (excluded from the pipeline).** The real-browser tier is slow, fragile, and pins ports, so `asr-service-ci.yml` deliberately does **not** run it: the `build` job passes `-Pci-skip-browser-e2e`, which adds `browser-e2e` to the DB Failsafe execution, and there is **no dedicated `e2e` job** (an earlier one was removed). Run the suite locally on demand with `./mvnw -Pci-only-browser-e2e verify` (scopes Failsafe to the `browser-e2e` group in a single execution — one Keycloak boot, no pinned-port collision; do **not** pass `-Dit.test`). * On a Docker host (`-DskipITs=false`) a plain local `./mvnw verify` runs them alongside the DB ITs. A Docker-less run (`-DskipITs=true`) activates the `no-docker` profile, whose `excludedGroups=requires-database` excludes them — reported **SKIPPED**, not green, mirroring the DB/IT policy. ### Harness shape [Section titled “Harness shape”](#harness-shape) * `EHerkenningBrowserE2EProfile` extends `EHerkenningStepUpBrokerProfile`, trusting the **dev quarkus-realm RS256 key** (the browser posts the realm-minted token, not the self-signed token-path key), and pins the e2e Keycloak/API ports with `KC_HOSTNAME` on the full Keycloak URL (split-horizon). * `StaticFrontendTestResource` serves `asr-demo-frontend/` over a JDK `HttpServer`, rewriting the page’s hardcoded dev coordinates to the e2e ports at serve time (so a developer’s running `quarkus:dev` on the dev ports does not conflict) — **no frontend source is edited**. * The profile re-imports both realms from temp copies whose front-channel `:8888` is rewritten to the e2e Keycloak port (back-channel `:8080` untouched), so the browser broker redirect and the brokered token `iss` reach the test Keycloak. ### Risk: keycloak-js CDN dependency (the one frontend-coupling risk) [Section titled “Risk: keycloak-js CDN dependency (the one frontend-coupling risk)”](#risk-keycloak-js-cdn-dependency-the-one-frontend-coupling-risk) `applicant-edit.html` imports keycloak-js from `cdn.jsdelivr.net`. A sandboxed/offline CI cannot fetch it, so `keycloak.init` would never run. The harness **route-intercepts** the CDN URL via Playwright and fulfils it from a **vendored copy** bundled as a test resource (`e2e/keycloak-js-26.2.0.esm.js`). This keeps the test offline-safe and edits **no frontend file** — the interception lives entirely in the harness. If the page’s pinned keycloak-js version changes, refresh the vendored copy to match. ## Related Documents [Section titled “Related Documents”](#related-documents) * [ADR-00014: Claim Verification Provider Seam and eHerkenning](00014-claim-verification-provider-seam-and-eherkenning.md) * [ADR-00008: External IdP Federation Strategy](00008-external-idp-federation-strategy.md) * [Coding standards §7.3 End-to-End Tests (UI)](../08-crosscutting/ctn-coding-standards.md) * [Dev-mode demo walkthrough](../../dev-demo-walkthrough.md) ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | --------------------------------------------------------------------------------------------------------------------------------- | | 2026-06-15 | Proposed | Initial proposal (issue #67) | | 2026-06-29 | Proposed | Added dedicated `@Tag("browser-e2e")`; real-browser tier excluded from CI (dedicated `e2e` job removed), now local/on-demand only | # ADR-00016: Language Support and Documentation Policy ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-06-18 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | RdN Question : I would like to understand which tool will be used. I am familiar with POT-files and PO-editors, in which you can have you labels in your UI easily translated by (native) speakers. Nowadays it is often AI powered. E.g. integrates quite well with Claude.ai. I would prefer an external tool, so that the ASR Admin in future can have a look at the translations and override when needed. For now it will be the Product Owner together with his AI buddy who will be looking into the translations. We should also consider to add this to the Definition of Done. Changes of UI labels available in all supported languages. And translations approved by Product Owner. ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) We need a clear, consistent policy for language usage across documentation, stakeholder communication, user interfaces, and source code. Ambiguity leads to friction in collaboration, inconsistent terminology, and additional rework during reviews and translations. The policy must: * Define the canonical language for permanent technical design documentation. * Define the language for stakeholder-facing artefacts (e.g., product vision, epics, user stories, acceptance criteria). * Ensure all user interfaces are internationalizable with a sensible default locale. * Define language conventions for technical source code to ensure consistency with common tooling and libraries. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Single-language Dutch everywhere [Section titled “Option 1: Single-language Dutch everywhere”](#option-1-single-language-dutch-everywhere) * Description: All artefacts (docs, stories, UI, code) in Dutch. * Pros: One language for all stakeholders; easier stakeholder reviews. * Cons: Excludes/limits international contributors; conflicts with common programming/library conventions; harder to reuse OSS/i18n resources; increases friction in vendor collaboration. ### Option 2: Single-language English everywhere (one locale) [Section titled “Option 2: Single-language English everywhere (one locale)”](#option-2-single-language-english-everywhere-one-locale) * Description: All artefacts (docs, stories, UI, code) in one variant of English. * Pros: Aligns with tooling/OSS; simplifies collaboration across borders; reduces translation overhead. * Cons: Less accessible for Dutch-speaking business stakeholders; risks loss of nuance in policy/legal contexts; may reduce adoption/engagement of non-technical stakeholders. ### Option 3: Mixed policy by audience and artefact type (Chosen) [Section titled “Option 3: Mixed policy by audience and artefact type (Chosen)”](#option-3-mixed-policy-by-audience-and-artefact-type-chosen) * Description: Use different languages per audience/artefact with explicit locale rules. * Pros: Optimizes comprehension per audience; keeps code/tooling aligned with industry norms; supports internationalization for UI. * Cons: Requires discipline and templates; introduces bilingual processes; needs glossary and translation workflows for some artefacts. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) Chosen option: Option 3 — Mixed policy by audience and artefact type. * Permanent technical design documentation: English. * Stakeholder communication (user stories and higher): Dutch. * All UIs must support i18n, with default locale English (en-GB). * Technical source code: English (en-US). ### Rationale [Section titled “Rationale”](#rationale) * Business alignment: Business-facing planning and validation benefit from Dutch for clarity with local stakeholders. * Technical considerations: Code, APIs, and many dependencies use American English spellings; using en-US in code minimizes friction with linters, libraries, and common vocabulary. * Global collaboration: English technical documentation increases reach, improves onboarding of external contributors, and facilitates vendor collaboration. * UX/Localization: en-GB as the default UI locale aligns with many public-sector/user-facing conventions in the Netherlands while keeping the UI internationalizable. * Risk/Cost: Adds some translation and review overhead, but mitigated by templates, linters, and a shared glossary. ### Consequences [Section titled “Consequences”](#consequences) * Positive: * Clear guidance per artefact reduces review churn. * Better interoperability with tooling and OSS ecosystems. * UI prepared for multi-language audiences from day one. * Negative: * Bilingual process requires discipline and reviewer awareness. * Occasional translation needed between Dutch stakeholder artefacts and English technical designs. * Neutral: * Minor spelling differences (en-GB vs en-US) are constrained by artefact type to avoid confusion. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. Documentation and templates * Update ADR, arc42, and design templates to state “Technical documentation language: English”. * Ensure product/backlog templates state “Stakeholder artefacts language: Dutch”. 2. UI internationalization * Require i18n libraries/framework support in all UI modules; set default locale to en-GB. * Establish message key conventions and folder structure for locale resources. 3. Source code conventions * Enforce en-US spelling in identifiers, comments, and API schemas. * Configure linters/IDE inspections/spell-check dictionaries accordingly. 4. Glossary and translation workflow * Maintain a bilingual glossary for domain terms (Dutch ↔ English). * Define a lightweight process for translating user-facing copy and aligning terms across artefacts. 5. Governance * Add checklist items to PR/review templates to verify policy adherence. * Periodically audit for drift and update the glossary. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [ ] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * 00004-naming-follows-glossary.md * 00009-layered-source-code-structure.md * REQ/ADR templates in this directory * Any UI development guidelines (i18n) ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ---------------------------- | | 2026-06-18 | Proposed | Initial proposal and policy. | *** ADR format based on Michael Nygard’s template with CTN-specific enhancements # ADR-00018: Build-enforced Java formatting via Spotless + an Eclipse JDT profile ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-06-24 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) `asr-service` (Quarkus) has no machine-enforced Java formatting. Sources today follow IntelliJ IDEA’s default formatting by convention only — there is no shared config, no apply command, and no CI gate. Contributors may use IntelliJ (primary), Eclipse, or VS Code, so formatting can drift per editor and per person, producing noisy diffs and review friction. We want three properties: 1. A single command formats the whole module (`mvn spotless:apply`). 2. CI fails when any file is not formatted. 3. All three IDEs auto-format **identically to the build** from one shared config. > **Formatting is not linting.** Spotless is a *formatter*: it rewrites *layout* — whitespace, indentation, line breaks, import order — to one canonical style. It does **not** judge what the code *does*: unused variables, likely bugs, null-safety, complexity, and security anti-patterns are all out of scope. Those are the job of a *linter* / static-analysis tool (Error Prone, SpotBugs, PMD, Checkstyle, SonarQube). A formatter makes code *look the same*; a linter checks that code is *well-formed*. This ADR is strictly about formatting; linting is a separate, not-yet-taken decision — see *Follow-up: linting* below. The repository is a **monorepo** (`CNCL-BDI`) holding several components with deliberately separate builds: `asr-service` (`nl.bdi.asr`), `visibility-dashboard/backend` (`nl.bdi.cloudtower`), plus static/TS frontends and Keycloak assets. There is **no parent reactor POM**, and CI is **path-scoped per subtree** (see [TDR-0004](../../ai/adrs/TDR-0004-asr-service-ci-single-path-scoped-workflow.md)), whose first asr-service workflow reserves a `format`-gate placeholder. Any formatting decision must fit that monorepo shape rather than assume a single application. ## Considered Options [Section titled “Considered Options”](#considered-options) The core board-relevant choice is the **formatting engine and the style source of truth**, because it determines IDE parity, CI feasibility, and the size of the one-time reformat. ### Option 1: `spotless-maven-plugin` driving an Eclipse JDT formatter profile (chosen) [Section titled “Option 1: spotless-maven-plugin driving an Eclipse JDT formatter profile (chosen)”](#option-1-spotless-maven-plugin-driving-an-eclipse-jdt-formatter-profile-chosen) * **Description**: Use `com.diffplug.spotless:spotless-maven-plugin` with its `` step pointed at a committed `eclipse-formatter.xml`. Eclipse imports the XML natively; IntelliJ achieves true parity via the *Adapter for Eclipse Code Formatter* plugin (delegates to the real Eclipse JDT engine); VS Code reads it via `java.format.settings.url`. * **Pros**: One config consumed by all three IDEs; the Eclipse JDT formatter runs in-process and headless, so it is fast in CI; `spotless:check` binds to `verify` for a zero-wiring local gate. * **Cons**: The Eclipse JDT engine differs from IntelliJ’s native engine, so *native* IntelliJ XML import is only approximate — true parity requires the per-contributor Adapter plugin. ### Option 2: `googleJavaFormat` / `palantirJavaFormat` [Section titled “Option 2: googleJavaFormat / palantirJavaFormat”](#option-2-googlejavaformat--palantirjavaformat) * **Description**: Adopt an opinionated, fixed formatter as the Spotless step. * **Pros**: Zero style config to maintain; deterministic. * **Cons**: Opinionated styles (e.g. 2-space indent) diverge heavily from the current 4-space, IntelliJ-default code — maximal one-time churn — and still need a per-IDE plugin for live parity. Not consumable as a shared Eclipse/IntelliJ/VS Code config. ### Option 3: Spotless `` step [Section titled “Option 3: Spotless \ step”](#option-3-spotless-idea-step) * **Description**: Let Spotless invoke IntelliJ’s own formatter. * **Cons**: Requires a local IntelliJ binary and launches a separate IDEA process per file — unusable in CI and for non-IntelliJ contributors. Rejected outright. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 1 — `spotless-maven-plugin` with an Eclipse JDT profile — together with the following scoping and operational decisions that make it fit the monorepo: * **Scope: `asr-service` only.** Not `visibility-dashboard/backend`, not the TS frontends. This fills TDR-0004’s `format`-gate placeholder and avoids inventing shared formatting plumbing across builds that are deliberately separate — the same “no premature abstraction across notional consumers” reasoning TDR-0004 applied to CI. vis-dashboard can adopt the proven config later as its own installment. * **Config location: `asr-service/config/eclipse-formatter.xml`**, referenced via `${project.basedir}` — self-contained inside the one subtree its path-scoped CI watches; no reactor / `${maven.multiModuleProjectDirectory}` path tricks. * **Profile philosophy: clean canonical, not IntelliJ-mimicry.** The profile is *Eclipse built-in defaults with one deliberate override* — `tabulation.char=space` — at width 120, K\&R braces, indent 4 (all of which already match the code). We explicitly **do not** run the laborious key-by-key tuning loop to approximate IntelliJ’s engine; once we accept a one-time reformat, fighting an engine difference the Adapter plugin already resolves buys nothing. The XML is kept *complete* (all `org.eclipse.jdt.core.formatter.*` keys) so missing keys do not silently fall back to Eclipse’s tab-based defaults. * **Import order is normalised**, not preserved: `java, javax, jakarta, org, com, ` then `static` last (`java,javax,jakarta,org,com,\#`). Reshuffle churn is accepted in exchange for an explicit, documented order. Wildcard imports are forbidden (``; one `jakarta.persistence.*` is expanded), and IntelliJ’s `*`-collapse thresholds are raised to 999 to stop re-collapse drift. * **IDE parity via the Adapter for Eclipse Code Formatter plugin** (IntelliJ), and the **Spotless eclipse formatter version is pinned** to align with the plugin’s bundled engine. CI `spotless:check` is the authoritative gate; the plugin is a convenience so contributors rarely hit it. * **Deferred reformat to protect in-flight branches.** The 177-file reformat would force a massive rebase on every open branch, so it is split off in time (see Implementation Plan). ### Rationale [Section titled “Rationale”](#rationale) Option 1 is the only one that satisfies property 3 (one config, all three IDEs) while remaining CI-viable (property 2) and low-churn against the current 4-space IntelliJ-default code. The two honest costs — the per-contributor Adapter plugin and an irreducible engine difference under *native* import — are mitigated by making the Adapter plugin the documented path and treating CI as the source of truth: a contributor without the plugin simply gets a gate failure pointing at `mvn spotless:apply`, never silently-divergent code. Scoping to `asr-service` and keeping the config inside that subtree mirrors the established monorepo house pattern and TDR-0004’s path-scoped CI. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: One shared config across IDEs; fast headless CI gate; explicit import order and no wildcards; the config is self-contained in the `asr-service` subtree; the build stays green at every commit because the gate is armed only when the tree becomes compliant. * **Negative**: True IntelliJ parity depends on a per-contributor plugin install and a version pin that must be kept aligned; a one-time 177-file reformat commit; `visibility-dashboard` is left ungated for now (a future installment). Repo-root IntelliJ code style applies `asr-style` monorepo-wide (IntelliJ code style cannot be scoped to a subtree), so editing vis-dashboard Java in IntelliJ will nudge it toward `asr-style` on save — accepted as harmless eventual consistency, not a contract. * **Neutral**: vis-dashboard adoption, a shared pre-push hook, and TDR-0004’s two-layer CI split each remain separate future decisions. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Plumbing commit (now), on a branch off clean `main`, build stays green**: * `asr-service/pom.xml`: Spotless plugin `` only — java includes, `` step with pinned version + `${project.basedir}/config/eclipse-formatter.xml`, the ``, `removeUnusedImports`, `forbidWildcardImports`, `trimTrailingWhitespace`, `endWithNewline`. **No `` block** — `spotless:apply`/`check` are runnable on demand but nothing binds to `verify`, so the gate does not fail on still-unformatted code. * `asr-service/config/eclipse-formatter.xml` (the canonical profile, `name="asr-style"`). * Repo-root IDE files: `.editorconfig` (universal settings + `[*.java]` space/4), `.vscode/settings.json` (`java.format.settings.url` → `asr-service/config/eclipse-formatter.xml`, profile `asr-style`), `.idea/codeStyles/Project.xml` + `codeStyleConfig.xml` (un-ignore exactly these two in `.gitignore`). * `asr-service/CONTRIBUTING.md`: the two commands, config location, Adapter plugin + pinned version, wildcard-threshold note. 2. **Deferred reformat commit (later)**: once in-flight branches have merged, rebase this branch onto updated `main`, run `mvn spotless:apply`, commit the reformat **together with** adding the `` `check`→`verify` binding (gate armed exactly when the tree is compliant), then add `.git-blame-ignore-revs` carrying that commit’s SHA so `git blame` skips the mass reformat. Merge. 3. **CI wiring is owned by TDR-0004**: this ADR delivers pom + config + baseline only; TDR-0004’s `format` job calls `mvn spotless:check`, path-scoped to `asr-service/**`. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] `mvn spotless:apply` is idempotent (re-run produces no changes). * [ ] After the reformat commit, `mvn spotless:check` passes on the committed tree. * [ ] A deliberate mis-format makes `mvn verify` fail with a Spotless violation pointing to `apply`. * [ ] IntelliJ “Reformat Code” (Adapter plugin, pinned version) round-trips without re-dirtying files. * [ ] Wildcard imports do not reappear after IntelliJ optimize-imports. * [ ] VS Code format-on-save round-trips cleanly against the build. ## Follow-up: linting (separate, not-yet-taken decision) [Section titled “Follow-up: linting (separate, not-yet-taken decision)”](#follow-up-linting-separate-not-yet-taken-decision) Formatting (this ADR) and linting are complementary and both are cheap to enforce in the same path-scoped CI, so linting is a natural next installment. It is **not decided here**; the recommendation below is a starting point for that decision. **Recommended direction**, mirroring this ADR’s philosophy (build-enforced, `asr-service`-scoped, low-churn, introduced non-gating then armed once the baseline is clean, CI job owned by TDR-0004): 1. **Error Prone as the primary linter.** A javac plugin that catches real bugs at compile time (null dereferences, format-string mistakes, broken `equals`/`hashCode`, mutable-static leaks) with famously low false positives and near-zero runtime cost. Optionally pair it with **NullAway** for null-safety. Best value for the effort — gate on a curated ruleset. 2. **SpotBugs + FindSecBugs as a security pass.** Bytecode analysis for bug and security patterns. Because the ASR handles authentication, tokens, and PII, a security-focused linter earns its place. Start **report-only**, then gate on high-severity findings. 3. **Skip Checkstyle for formatting.** Spotless already owns layout, so Checkstyle would overlap and add noise. Only reconsider it later for *non-format* conventions (naming, Javadoc presence) if a concrete need appears. 4. **Treat SonarQube / SonarCloud as a later “quality dashboard” decision.** It aggregates linters and adds a quality gate plus security hotspots, but brings infrastructure. Worthwhile once there is appetite for a dashboard — not the first step. Rollout should follow the same pattern as the Spotless gate: introduce the tool non-gating, fix the baseline in a dedicated commit (with `.git-blame-ignore-revs` if it touches many files), then arm the CI gate exactly when the tree is clean. Config stays inside the `asr-service` subtree; the CI job is owned by [TDR-0004](../../ai/adrs/TDR-0004-asr-service-ci-single-path-scoped-workflow.md). When taken, this warrants its own ADR. ## Related Documents [Section titled “Related Documents”](#related-documents) * [TDR-0004](../../ai/adrs/TDR-0004-asr-service-ci-single-path-scoped-workflow.md) — asr-service CI is a single path-scoped workflow; owns the `format` job that runs this gate. * [ADR-00002](00002-define-a-preferred-stack.md) — preferred technology stack (Spotless is a candidate addition to its build-tooling row). * [ADR-00009](00009-layered-source-code-structure.md) — layered source structure / monorepo shape. * Handoff: `.scratch/spotless-formatting-handoff.md` (original plan; this ADR amends its single-module and minimal-churn-mimicry assumptions). ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ---------------- | | 2026-06-24 | Proposed | Initial proposal | *** *ADR format based on Michael Nygard’s template with CTN-specific enhancements* # ADR-00019: Bulk Import is a Distinct Onboarding Source with Orthogonal Validity and Admin-Owned Repair ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | ------------ | ----------- | | CTN Project Team | Proposed | 2026-06-25 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Not Required | — | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | RdN: Although we are developing this currently for CTN, this feature will also come in handy (a re-usable) for other initiatives. If BDI Conformance Team approves we can submit it as RFC for the BDI Framework. ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) The first bulk-import implementation (TDR-0003) reused the self-service intake path verbatim: `BulkImportService` looped each spreadsheet row through `OnboardingRequestService.save(req, adminSub)`, so every row got the same validate-or-reject, account provisioning **and** set-password mail as an anonymous self-service submission. The steer then was “map only what fits, add no new rules.” Operational reality has moved past that. Bulk import is now understood as a genuinely different workflow from self-service intake, not a thin wrapper over it: * **Failed rows must survive.** Today a row that fails mapping or validation is dropped into a transient `ImportResultResp.failed[]` and lost. The association admin needs those rows **persisted** so they can be fixed later — the whole point of an admin bulk channel is that the admin, not the applicant, repairs bad data. * **The applicant is not in the loop at creation.** A bulk row should provision the applicant’s Keycloak account but **not** email them yet; first contact (the set-password mail) is a deliberate admin action, sent when the admin is ready. * **The admin owns the row, not the requestor.** Bulk rows are edited and corrected by an admin acting *namens de aanvrager*; the self-service rule “only the requestor edits, only in `{Submitted, ChangesRequested}`” does not apply. * **Uploads must be separable.** Multiple bulk uploads need to be told apart. * **Duplicates will be grouped later.** Post-enrichment deduplication (COMP-19) will group requests; before enrichment, “duplicate” means *same Identifier URI*. The central design tension is how to represent a **failed/incomplete** row. The existing model fights it on two fronts: `identifierUri` is deliberately `nullable = false` (TDR-0002, ADR-00011), yet the most common failed row is one whose identifier could not be minted; and the lifecycle (ADR-00012) is a **pure flat edge table** whose `openStates()` is reused verbatim as the duplicate-guard scope, so any new “failed” state ripples into duplicate detection and the partial unique index. A future reader needs to know *why* `identifierUri` became nullable after being deliberately required, *why* there is no `Failed` status despite “failed rows,” and *why* bulk no longer shares the self-service create path. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: `Failed` as a new lifecycle Status (same entity, relaxed invariants) [Section titled “Option 1: Failed as a new lifecycle Status (same entity, relaxed invariants)”](#option-1-failed-as-a-new-lifecycle-status-same-entity-relaxed-invariants) * **Description**: Add `Failed` to `OnboardingRequest.Status`; relax `identifierUri` to nullable; wire `Failed` into the edge table (admin repairs it forward into the normal flow, discards it to `Rejected`). * **Pros**: One table, fix-in-place, no promotion step. * **Cons**: Encodes *validity* (a data-completeness property) as a *lifecycle position* — a category error. `Failed` is lifecycle-open, so it leaks into `openStates()` and thus the duplicate guard and the partial unique index, forcing those two notions to be split apart. Still loses the original un-mapped cell data. ### Option 2: Reuse `ChangesRequested` for incomplete bulk rows [Section titled “Option 2: Reuse ChangesRequested for incomplete bulk rows”](#option-2-reuse-changesrequested-for-incomplete-bulk-rows) * **Description**: Land incomplete bulk rows directly in the existing `ChangesRequested` state. * **Pros**: No new status at all. * **Cons**: Same category error as Option 1, merely hidden — valid rows would sit in `Submitted`, invalid ones in `ChangesRequested`, so the lifecycle position again encodes validity. Overloads `ChangesRequested`’s meaning (an admin *reviewed and bounced it back*, with a decision note the requestor reads) onto “born incomplete, never reviewed.” Still needs nullable identifier and admin-scoped editing. ### Option 3: Separate import-staging table, promote valid rows [Section titled “Option 3: Separate import-staging table, promote valid rows”](#option-3-separate-import-staging-table-promote-valid-rows) * **Description**: Failed (and raw) rows persist as staging records holding raw cells + per-row errors + batch id; the admin fixes them in staging; on success a fully-valid `OnboardingRequest` is *promoted* out. * **Pros**: `OnboardingRequest` stays well-formed; raw data preserved; batches and pre-enrichment dedup attach naturally to staging. * **Cons**: A second edit surface and a promotion/copy step; duplicated data; more moving parts than the demo warrants. ### Option 4: Orthogonal Validity axis (chosen) [Section titled “Option 4: Orthogonal Validity axis (chosen)”](#option-4-orthogonal-validity-axis-chosen) * **Description**: Keep the lifecycle topology **unchanged** (no `Failed`). Add a second, **orthogonal** axis — **Validity** (`COMPLETE` / `INCOMPLETE`) — recomputed on every edit. All bulk rows enter at `Submitted`; incomplete ones carry `INCOMPLETE`. Relax `identifierUri` to nullable. Guard `approve` by `COMPLETE` — the “validate before accept” gate. This is the user’s “hierarchical state machine” intuition expressed as **parallel regions**, the only form that does not reverse the pure-flat-core decision of ADR-00012. * **Pros**: No category error — validity and lifecycle position are independent, as they truly are. Lifecycle topology and `openStates()` are **untouched**, so the duplicate guard and partial unique index need no decoupling. An edit moves *validity*, never *status*, so the rule “Enrichment is not a Lifecycle verb” still holds. * **Cons**: Relaxes the deliberately-required `identifierUri` invariant for the whole population (mitigated by the `approve`-guard, which restores it before acceptance). Validity is a derived property that must be stored to be queryable. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: **Option 4** — bulk import becomes a distinct onboarding **Source** with an **orthogonal Validity axis** and **admin-owned repair**, sharing the lifecycle but not the create path of self-service intake. ### Rationale [Section titled “Rationale”](#rationale) Option 4 is the only one that does not commit the category error of encoding *data completeness* as a *lifecycle position*. Because validity is modelled as a parallel region rather than a status, the prized pure-flat lifecycle core (ADR-00012) and its duplicate-guard reuse survive completely unchanged — a large, error-prone area (`openStates()`, the soft warning, the partial unique index, the consistency test) is left untouched. Option 3 keeps invariants strongest but pays with a staging table and a promotion step disproportionate to the demo. Options 1 and 2 are cheaper to type but bake in the conflation this ADR exists to avoid. ### The decided package [Section titled “The decided package”](#the-decided-package) 1. **Source axis.** A persisted `enum Source { SELF_SERVICE, BULK_IMPORT }` on every Onboarding Request. Explicit, **not** inferred from `createdBy` presence (a future admin-keyed single entry would also carry a `createdBy`). It is the spine the divergent behaviour branches on. 2. **Orthogonal Validity.** `COMPLETE` / `INCOMPLETE`, recomputed on every edit; `identifierUri` becomes nullable; `approve` is guarded by `COMPLETE`. No `Failed` status. Self-service rows are always `COMPLETE` (intake rejects invalid input before persistence); only `BULK_IMPORT` rows can be `INCOMPLETE`. 3. **Distinct bulk create path.** Bulk no longer reuses `save()` unchanged. It (a) provisions the Keycloak account via `ensureAccount` but **withholds** `triggerSetPassword`; (b) **never rejects a row** — a row that fails mapping or validation is persisted with whatever fields mapped (identifier possibly null) and flagged `INCOMPLETE`; only a **whole-file** failure (non-`.xlsx`, missing header) hard-fails the upload (400, nothing persisted); (c) stamps `source = BULK_IMPORT`, `batchId`, `createdBy = adminSub`, `status = Submitted`; (d) treats an existing **active participant** for the identifier as a **per-row flag**, not a thrown error. The COMP-15 soft duplicate warning still applies when an identifier is present. 4. **Invitation.** The set-password mail for a bulk row is sent by a separate, **idempotent** admin action, allowed **only when `COMPLETE`**, recorded by an `invitedAt` timestamp. It is **not** a lifecycle verb (changes neither status nor the edge table). For MVP it is *activation-only* — it does not hand editing to the applicant; the admin keeps control. 5. **Admin-owned editing.** `BULK_IMPORT` rows are editable by **any admin** (not only the importer) in **any open state**, the repair target being `identifierUri` (which re-derives `intakeCountry` and the processor, per TDR-0003’s scheme→country mapping). `applicantEmail` is editable only **before** invitation; `status` stays behind the verbs; the supplied address country is **never auto-overwritten** — a country/identifier mismatch is a GUI advisory, not a hard rule. 6. **Separable uploads.** A `batchId` UUID stamped on every row of one upload (self-service rows leave it null); sort/group by it, with `createdAt` for time. **No** first-class batch entity for MVP and **no** batch-wide operations; metadata can attach to the id later. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: lifecycle core, duplicate guard and partial unique index untouched; validity and lifecycle stay independent; failed rows are no longer lost; bulk and self-service diverge cleanly along the `Source` axis; account-to-request link stays honest (email locked post-invite). * **Negative**: `identifierUri` is now nullable for the whole population (the deliberate TDR-0002/ADR-00011 invariant is restored only at the `approve` gate, not by the column); a second create path to maintain; validity must be stored to be queryable. * **Neutral**: the German court-code / scheme→country derivation remains the TDR-0003 shared lookup; raw un-mapped cells are **not** preserved for MVP (the admin cross-references the original spreadsheet by row number + `Bron/Bron-klantcode`). ### Deferred (explicitly out of scope) [Section titled “Deferred (explicitly out of scope)”](#deferred-explicitly-out-of-scope) * **Deduplication grouping** → COMP-19. Pre-enrichment rule: duplicate == same Identifier URI; grouped in the admin panel. Post-enrichment (KvK etc.) grouping is COMP-19’s concern. * **Validity storage mechanism** (PostgreSQL generated column vs maintained boolean) → follow-up TDR. * **Four-eyes split** (one admin enriches, another validates) — possible consortium requirement; the “any admin, any open state” rule is the MVP placeholder. * **Post-invite email editing** for bulk; **invitation-as-handoff** to the applicant; **true cross-country re-dispatch** when the identifier scheme changes; **raw-row provenance** snapshot. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1**: schema — `source`, `batch_id`, `invited_at`, validity marker; make `identifier_uri` nullable (Liquibase, TDR-0001). Distinct bulk create path on `OnboardingRequestService` / `BulkImportService` (withhold mail, persist-not-reject, compute validity, stamp markers, per-row active-participant flag). `approve` guard on `COMPLETE`. Admin Invitation action (`triggerSetPassword` + `invitedAt`, idempotent, COMPLETE-gated). Admin-scoped editing of bulk rows incl. `identifierUri`. Admin-panel: incomplete filter, batch grouping, country/identifier mismatch advisory. 2. **Phase 2**: validity-storage TDR; COMP-19 deduplication grouping. 3. **Phase 3**: four-eyes, post-invite email edit, invitation handoff, cross-country re-dispatch, raw-row provenance — as separate tickets if pulled in. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed (admin-gated import + invitation; email-edit locking) * [ ] Performance impact assessed * [ ] Integration impact evaluated (Keycloak provisioning timing; mail) * [ ] Documentation updated (`docs/ai/CONTEXT.md` glossary entries added) * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [ADR-00012](00012-onboarding-request-lifecycle-transition-seam.md) — onboarding lifecycle & the pure flat transition core this ADR deliberately leaves unchanged. *(Note: much code/doc text cites “ADR-00014” for the lifecycle — a known numbering drift; the lifecycle decision is file 00012.)* * [ADR-00013](00013-applicant-keycloak-account-provisioned-at-intake.md) — applicant account provisioning, whose mail this ADR defers for bulk. * [ADR-00011](00011-uri-based-legal-entity-identifier-scheme.md) — URI-based identifier scheme; the `identifierUri` invariant relaxed here. * [TDR-0002](../../ai/adrs/TDR-0002-onboarding-stores-registry-identifier-as-identifier-uri.md) — stores the identifier as an Identifier URI (the now-nullable field). * [TDR-0003](../../ai/adrs/TDR-0003-spreadsheet-bulk-import-maps-row-to-onboarding-request.md) — the prior bulk import (row→request, scheme→country) this ADR supersedes the create-path semantics of. * [REQ-20260625-COMP](REQ-20260625-COMP-bulk-import-distinct-workflow.md) — the driving requirement. * COMP-24, COMP-119 (existing bulk import), COMP-19 (deduplication) — Jira. ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ---------------------------------------------------------- | | 2026-06-25 | Proposed | Initial proposal, from the grill-with-docs design session. | *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`). A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* *** *ADR format based on Michael Nygard’s template with CTN-specific enhancements* # ADR-00020: Test Data Handling Strategy ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-06-15 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | RdN: EB please chime in, what has been used earlier as test data? ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) As the CTN implementation and with it, the BDI (Basic Data Infrastructure) ecosystem grows, we need a consistent strategy for handling test and mock data across different environments. We currently have a mix of real-world company data (CTN) and various synthetic entries. Using real company data in development and early testing phases can lead to privacy concerns, confusion with actual production entities, and difficulty in remembering specific test cases. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Use Real-world (CTN) Data Everywhere [Section titled “Option 1: Use Real-world (CTN) Data Everywhere”](#option-1-use-real-world-ctn-data-everywhere) * **Description**: Use a copy of the actual CTN company list in all environments (Dev, Test, Acceptance). * **Pros**: Data is realistic and covers all production edge cases. * **Cons**: Privacy concerns; risk of accidental actions against real entities; hard to manage specific test scenarios. ### Option 2: Use Fully Synthetic Data Everywhere [Section titled “Option 2: Use Fully Synthetic Data Everywhere”](#option-2-use-fully-synthetic-data-everywhere) * **Description**: Generate random or semi-random data for all environments. * **Pros**: No privacy risks; clean separation from production. * **Cons**: Might miss realistic data patterns; Acceptance testing feels “fake” to stakeholders. ### Option 3: Tiered Approach: Actual Data for Acceptance, Synthetic Personae for Dev/Test (Chosen) [Section titled “Option 3: Tiered Approach: Actual Data for Acceptance, Synthetic Personae for Dev/Test (Chosen)”](#option-3-tiered-approach-actual-data-for-acceptance-synthetic-personae-for-devtest-chosen) * **Description**: Use a static set of actual CTN companies for the Acceptance environment, and a neutral, synthetic set based on memorable personae for Dev, Test, and BDI contexts. * **Pros**: Acceptance remains realistic; Dev/Test is safe, neutral, and easy to navigate using memorable names. * **Cons**: Requires maintaining two sets of test data. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 3: Tiered Approach. ### Rationale [Section titled “Rationale”](#rationale) * **Business alignment**: Acceptance environments need to demonstrate value to stakeholders using familiar real-world entities (CTN companies). * **Technical considerations**: Development and automated testing benefit from predictable, memorable data. Using personae with alliterative names (e.g., “Anneke the Admin” for “Company A”) makes it easier for developers and testers to communicate and remember specific scenarios. * **Risk assessment**: Reduces the risk of using PII (Personally Identifiable Information) in non-production environments while maintaining a high-fidelity environment for final acceptance. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: Clearer communication during development; safer handling of data; realistic final validation in Acceptance. * **Negative**: Overhead in maintaining the persona-based data set and ensuring it stays synchronized with schema changes. * **Neutral**: Shift in mindset for developers to think in terms of personae rather than random strings. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1**: Define the standard set of alliterative personae and their associated companies/roles (e.g., Anneke the Admin, Barend the Buyer). 2. **Phase 2**: Update the `dev-seed` SQL scripts (`onboarding-demo-data.sql`) to use these new personae. 3. **Phase 3**: Establish a process for refreshing the Acceptance environment with a curated, static set of actual CTN companies. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [x] Integration impact evaluated * [ ] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [ADR-00011: URI-based Legal Entity Identifier Scheme](00011-uri-based-legal-entity-identifier-scheme.md) * [ADR-00014: Claim Verification Provider Seam and eHerkenning](00014-claim-verification-provider-seam-and-eherkenning.md) ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ---------------- | | 2026-06-15 | Proposed | Initial proposal | *** *ADR format based on Michael Nygard’s template with CTN-specific enhancements* # ADR-00021: Documents as a decoupled service behind an ObjectStorageProvider seam, Postgres-first ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | ------------ | ----------- | | CTN Project Team | Approved | 2026-06-25 | | CTN Technical Advisory Board | Approved | 2026-07-08 | | CTN Steering Committee | Not Required | 2026-07-08 | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) The platform needs to store MIME-typed documents — mostly PDFs, also scanned images — with upload metadata, addressable by a stable UUID so other entities can reference them (see [REQ-20260625: document store](REQ-20260625-document-store.md)). The first consumer is onboarding: a requestor uploads proof supporting an Onboarding Request (e.g. a registry extract for a business-register identifier, a signed acceptance of the consortium contract conditions). But the same store must serve later needs that have nothing to do with onboarding — periodic re-validation uploads on a Participant, and documents that validate no entity at all (pure storage). Two architectural questions must be answered together: 1. **Where do the bytes physically live?** The preferred stack ([ADR-00002](00002-define-a-preferred-stack.md)) names only PostgreSQL — there is no object store (MinIO/S3) provisioned, and deployment is Kubernetes-native. “Blob store” colloquially implies S3. 2. **How does a Document relate to the things that reference it?** The headline ask was “1:N from Onboarding Request,” but the real requirement is broader: many future entity *types* will reference documents, and the onboarding reference must survive the Onboarding Request → Participant transition at approval. The house precedent (`Verification`) carries two nullable typed FK columns (`onboarding_request_id`, `participant_id`). These are hard to reverse (a storage seam and a reference model that other code depends on), surprising (no S3; no owner column despite a stated 1:N), and the result of real trade-offs — hence this ADR. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Bytes directly in Postgres, no seam [Section titled “Option 1: Bytes directly in Postgres, no seam”](#option-1-bytes-directly-in-postgres-no-seam) * **Description**: A `bytea` column (or split payload table) on the document entity; commit to Postgres as the byte store. * **Pros**: Zero new infra; transactional with metadata; one backup/restore story; fits ADR-00002 exactly. * **Cons**: Hard-codes Postgres as the byte store; migrating to an object store later touches the entity and every reader; Postgres soft-degrades past tens of MB/row. ### Option 2: External object store now (MinIO/S3), metadata in Postgres [Section titled “Option 2: External object store now (MinIO/S3), metadata in Postgres”](#option-2-external-object-store-now-minios3-metadata-in-postgres) * **Description**: Provision an S3-compatible store immediately; Postgres holds only metadata + an object key. * **Pros**: Scales to large files; keeps the DB lean; matches the colloquial “blob store” expectation. * **Cons**: New infrastructure **not in the preferred stack** (amends ADR-00002 with real ops cost); a second consistency boundary (orphaned objects / dangling rows); new credentials and failure modes — all for files realistically in the hundreds-of-KB-to-a-few-MB range. ### Option 3: `ObjectStorageProvider` SPI, Postgres-first implementation *(chosen)* [Section titled “Option 3: ObjectStorageProvider SPI, Postgres-first implementation (chosen)”](#option-3-objectstorageprovider-spi-postgres-first-implementation-chosen) * **Description**: Core defines a streaming byte-storage SPI keyed by the Document’s UUID; the first implementation stores bytes in a `document_blob` Postgres table; a later implementation can target MinIO/S3 without touching the Document model or its references. Mirrors the existing onboarding / claim-verification provider seams. * **Pros**: No new infra now, yet no lock-in; swapping backends is a new impl, not a data-model migration; consistent with the platform’s SPI-seam idiom. * **Cons**: One layer of indirection before any object store exists; the Postgres impl materializes bytes in heap (acceptable under the size cap; true streaming arrives with an object-store impl). ### Reference-model sub-options (orthogonal to the substrate) [Section titled “Reference-model sub-options (orthogonal to the substrate)”](#reference-model-sub-options-orthogonal-to-the-substrate) * **R1 — Two nullable typed FKs on the Document** (`onboarding_request_id`, `participant_id`), matching `Verification`. *Rejected*: ties the general store to onboarding/participant concepts and needs a new column per future referrer type. * **R2 — Polymorphic `owner_type` + `owner_id` on the Document**. *Rejected*: still models “owned by one,” breaks FK integrity, and diverges from house style. * **R3 — Document carries no owner; the referrer holds the `document_id`** *(chosen)*. The Document knows nothing about who references it; supersession/migration is the referrer swapping which id it points at. A document↔identifier relationship, when a consumer needs one, is the consumer’s, keyed by the canonical [`IdentifierUri`](00011-uri-based-legal-entity-identifier-scheme.md) string (stable across the Request→Participant transition), not by a FK on the Document. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: Option 3 (storage seam, Postgres-first) + R3 (decoupled, referrer-holds-the-link). A general, standalone `DocumentService` over two tables: ```plaintext document -- metadata, core-owned, always Postgres document_id UUID PK document_type text (enum: RegistryExtract | ContractAcceptance | Other) content_type text (MIME) file_name text size_bytes bigint expiry_date date NULL -- per-type policy: forbidden/optional/required + MetaEntity (created_at/by, updated_at/by, is_deleted) document_blob -- the Postgres ObjectStorageProvider impl only document_id UUID PK/FK bytes bytea -- a future S3 impl drops this table entirely ``` Bytes are owned by the `ObjectStorageProvider` SPI (streaming `InputStream` in/out, keyed by `document_id`, no presigned/direct-URL operation — access is always API-mediated). The Postgres implementation backs the `document_blob` table. ### Rationale [Section titled “Rationale”](#rationale) * **Business alignment**: one trustworthy, auditable home for proof documents, shared across onboarding, re-validation and future consumers, with provenance and a business-validity (expiry) marker. * **Technical considerations**: realistic file sizes sit comfortably in Postgres, so Option 2’s infrastructure is unjustified today; the seam removes the lock-in that makes Option 1 risky. R3 keeps the store general — the explicit requirement — and makes the onboarding “migration” a non-event: the link names the URI / lives on the referrer, never the Document. * **Risk assessment**: the chief risk is shipping a service with no HTTP exercise (the upload endpoint rides with the first consumer); mitigated by tests driving `DocumentService` + the Postgres impl directly (store → read → byte round-trip, expiry-policy and allow-list/size enforcement). * **Cost/benefit**: a small indirection cost buys backend portability and keeps the Postgres-only stack intact until an object store is genuinely warranted. ### Key properties [Section titled “Key properties”](#key-properties) * **Document Type** is a closed enum carrying a per-constant **expiry policy** (`FORBIDDEN` / `OPTIONAL` / `REQUIRED`). Seed: `RegistryExtract` (OPTIONAL), `ContractAcceptance` (FORBIDDEN), `Other` (FORBIDDEN). Grown per consumer. * **Expiry = business-validity**, queryable, never a storage trigger — **no auto-delete**. Explicit deletion is a deliberately deferred, separate concern (`is_deleted` reserved for it). * **Write-once**: bytes and intrinsic metadata are set at creation; corrections / re-validations are new Documents. The only mutations are an **admin correction of `expiry_date` and `document_type`** and the soft-delete flag. * **Ingest guards**: configurable MIME allow-list (default `application/pdf`, `image/png`, `image/jpeg`) and a per-MIME-type size cap, enforced in `DocumentService`. Magic-byte content sniffing is a deferred hardening follow-up. * **Access**: authenticated upload; read restricted to the uploader (`created_by`) and `association-admin`; everything API-mediated, with an extension point for a future organization-scoped `OrgAdminRole`. ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: backend-portable byte storage; a general store the whole platform can reuse; provenance, expiry and write-once integrity out of the box; no new infra. * **Negative**: orphaned Documents are possible once a standalone upload path exists (mitigated for now: upload is one-step through a consumer endpoint, so the reference is created in the same transaction — no orphan); the Postgres impl does not truly stream; `REQUIRED` expiry has no seed type exercising it yet. * **Neutral**: extends ADR-00002 with a seam rather than amending the stack; a future object-store impl and an isolated-origin download path are anticipated, not built. ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1 (this slice)**: `document` + `document_blob` Liquibase changelog; `Document` entity + `DocumentType` enum with expiry policy; `ObjectStorageProvider` SPI + Postgres impl; `DocumentService` (store / read / metadata / admin correction); allow-list + per-MIME size config. Tests drive service + impl directly. **No HTTP endpoint.** 2. **Phase 2 (with first consumer)**: a one-step consumer-owned upload endpoint (e.g. onboarding) that delegates to `DocumentService` and creates the referencing entity in the same transaction; read/download endpoint (API-mediated). 3. **Phase 3 (later)**: object-store `ObjectStorageProvider` impl (MinIO/S3, true streaming); explicit deletion + orphan GC; magic-byte sniffing; isolated-origin download offload; `OrgAdminRole` read access. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed (PII at rest, access control, content sniffing) * [ ] Performance impact assessed (metadata reads do not load bytes) * [ ] Integration impact evaluated (first consumer wiring) * [ ] Documentation updated (CONTEXT.md glossary entries) * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [REQ-20260625: document store](REQ-20260625-document-store.md) — driving requirement * [ADR-00002](00002-define-a-preferred-stack.md) — preferred stack (extended here) * [ADR-00011](00011-uri-based-legal-entity-identifier-scheme.md) — `IdentifierUri` (the stable key for a future document↔identifier link) * [ADR-00014](00014-claim-verification-provider-seam-and-eherkenning.md) — the SPI-seam idiom this mirrors * [TDR-0001](../../ai/adrs/TDR-0001-liquibase-owns-schema.md) — Liquibase owns schema * `docs/ai/CONTEXT.md` — Documents glossary section ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ------------------------------------------------------ | | 2026-06-25 | Proposed | Initial proposal from a grill-with-docs design session | *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`).* # ADR-00022: Module seam pattern — in-process seam vs internal, with a REST-boundary zone ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-07-06 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | RdN Question: I don’t see the relevancy of this ADR, at least not for CTN and broader BDI initiatives. TBH isn’t this just about basic development hygiene and adhering to coding standards/best practises. ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) `asr-service` is a Quarkus modulith. Its top-level packages under `org.bdinetwork.asr` are a **hybrid**: business/domain packages (`audit`, `document`, `verification`, `onboarding`) sit beside layer-first packages (`api`, `model`, `repository`, `spi`, `error`, `config`). `model` and `repository` are prototype **parking packages** — layer-first staging expected to dissolve into their owning modules over time. Nothing today constrains what one module may reach in another. A sibling module can `@Inject` any public bean or import any type of another module, including its JPA entities and internal services. As the codebase grows this invites accidental coupling, cycles, and a persistence layer that leaks across module boundaries. Concrete example found in the audit module: the write service’s seam method `AuditRecorder.record(..., AuditRecord.Action, ...)` forced callers ( `DocumentService`, `VerificationStatusService`) to import a JPA `@Entity` (`AuditRecord`) just to name an action enum. We want each module to expose a small, explicit **seam** (what other in-process modules may reach) and hide everything else. This is a Quarkus project — there is **no Spring Modulith** — and Java’s own access control cannot express the boundary (see options below). ADR-00009 (layered source code structure) is **orthogonal**: it governs tenancy/reusability namespaces (`org.bdinetwork.*` core / `marketplace` / `nl.ctn.*`), not intra-module packaging. This ADR layers on top of it. ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: Status quo (hybrid, no enforcement) [Section titled “Option 1: Status quo (hybrid, no enforcement)”](#option-1-status-quo-hybrid-no-enforcement) * **Description**: keep the layer-first/domain-first mix; rely on convention. * **Pros**: no work. * **Cons**: no boundary; coupling and cycles accrete silently; parking packages never resolve; the entity-leak class of bug recurs. ### Option 2: Language-level enforcement (package-private / JPMS `module-info`) [Section titled “Option 2: Language-level enforcement (package-private / JPMS module-info)”](#option-2-language-level-enforcement-package-private--jpms-module-info) * **Description**: hide internals with `package-private`, or declare `exports` via the Java Platform Module System. * **Pros**: compiler-enforced. * **Cons**: `package-private` does **not** nest — it cannot hide `audit.internal.persistence` from `audit.rest`. CDI cross-package proxying and JAX-RS resources require **public** bean types, so package-private cannot hide beans that are injected or scanned. JPMS is heavyweight and disruptive to the Quarkus build. Language access cannot express ” module-internal across sub-packages.” ### Option 3: Package-placement seam + ArchUnit enforcement (chosen) [Section titled “Option 3: Package-placement seam + ArchUnit enforcement (chosen)”](#option-3-package-placement-seam--archunit-enforcement-chosen) * **Description**: define module visibility by **package placement**, enforced by an ArchUnit allow-list test. Types are `public` as CDI/JAX-RS require; ArchUnit — not the compiler — forbids cross-module access to non-seam packages. * **Pros**: works with CDI/JAX-RS; fits the repo’s existing ArchUnit usage; fail-safe default (anything not seam is hidden); one rule scales to all modules via a `MODULES` list. * **Cons**: enforcement is a test, not the compiler (a violation fails the build in CI, but the IDE won’t stop you typing it); requires discipline to keep the `MODULES` list current. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option: Option 3.** Adopt a repo-wide **module seam pattern**, enforced by ArchUnit, with **audit as the pilot module**. Each top-level business package under `org.bdinetwork.asr` is a module with four package zones: 1. **Seam** — the module **root** package plus whitelisted seam sub-packages `dto`, `exceptions`, and `entity` ( Amendment 1). The only surface other in-process modules may reach. Cross-module communication goes through the root \* *service interface*\* (+ any published events). Callers inject the interface, never the impl. 2. **`entity/`** — the module’s **published persistence types** (Amendment 1, see below): JPA entities that other modules may map `@ManyToOne` associations to. Part of the seam; created lazily, only when a cross-module association warrants it. 3. **`rest/`** — the module’s **HTTP boundary** (JAX-RS resource + its wire DTOs/mappers). A top-level **sibling of `internal`**, not part of the seam. Externally significant and visible at a glance, but **not injectable** by sibling modules. 4. **`internal/`** — hidden implementation: service impls, application services, `internal.persistence` (JPA entities + repositories), and any further hidden packages added freely under `internal..`. ### Deviation from the source spec (deliberate) [Section titled “Deviation from the source spec (deliberate)”](#deviation-from-the-source-spec-deliberate) The source pattern places the REST resource under `internal.rest`. We instead make **`rest/` a top-level third zone**, a sibling of `internal`. Rationale: a module’s HTTP surface is externally significant — knowing a module *has* an HTTP boundary, and what it is, matters when reading the module — so it should be visible at the top rather than buried in `internal`. This is safe because the ArchUnit rule is an **allow-list** (seam = root + `dto` + `exceptions`); anything *not* seam — including `rest/` — is already forbidden to other modules. So `rest/` and `internal/` have **identical cross-module enforcement**; the split is purely human-facing signalling of “this is the module’s published HTTP surface.” A resource freely calls its own module’s `internal` (same-module access is unrestricted). ### Rules of the pattern [Section titled “Rules of the pattern”](#rules-of-the-pattern) * **Seam = root + `dto` + `exceptions` + `entity`.** Nothing else is reachable from other modules. Create `dto`/ `exceptions`/`entity` **lazily** — only when a seam type needs them. * **`dto/` holds the service interface’s own types**, and (per the existing `ArchitectureTest.dtos_should_be_records`) each must be a `record`. Enums and other non-record seam value types go at the **root**, not `dto/`. * **HTTP-only wire types live under `rest/`** (e.g. `rest/dto`), never the seam `dto/`. * **Service impl is `public` and lives in `internal`.** ArchUnit hides it; callers inject the root interface. * **`internal` is a subtree.** Add hidden packages under `internal.*` freely (fail-safe). Never create hidden packages as seam siblings other than `rest/`. * **`model`/`repository` parking packages dissolve** into each module’s `internal.persistence` as modules migrate. * **Tests**: same-module tests may touch `internal`. Cross-module/e2e tests go through HTTP (RestAssured), not Java imports. Boundary rules exclude tests (`ImportOption.DoNotIncludeTests`). ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: explicit, enforced module boundaries; no accidental cross-module injection; persistence stays inside its module; the entity-leak bug class is designed out (seam exposes only seam types); one ArchUnit rule scales across modules; a clear migration target for the parking packages. * **Negative**: enforcement is a CI test, not the compiler; single-impl services carry interface/impl ceremony; migrating each module is real work; the `MODULES` list must be maintained. * **Neutral**: types remain `public` (CDI/JAX-RS demand it) — visibility is by package, not access modifier. Within a module, a JAX-RS resource may consume a JPA entity from `internal.persistence` (same-module, allowed). ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1 — Audit pilot (COMP-146).** Restructure `org.bdinetwork.asr.audit` to: ````plaintext audit/ AuditRecorder (seam: interface) Action (seam: enum — replaces the leaked AuditRecord.Action) rest/ AuditRecordResource dto/AuditRecordResp (+ from(entity) mapper) internal/ AuditRecorderImpl AuditRecordQueryService AuditPayloads (payload → JsonNode serializer, TDR-0005) persistence/ AuditRecord (JPA entity, uses seam Action) AuditRecordRepository ``` `AuditRecordQueryService` returns the entity; the resource maps to `AuditRecordResp` (with an in-code comment recording the choice and a reminder to introduce an application-level projection when the read side gains complexity). Add `package-info.java` (`@NullMarked`) to each new package. ```` 2. **Phase 2 — Enforcement.** Add `ModuleBoundaryTest` ( `@AnalyzeClasses(packages="org.bdinetwork.asr", DoNotIncludeTests)`) with `onlySeamReachableAcrossModules` ( allow-list: root + `dto` + `exceptions`) and `modulesAreCycleFree`. `MODULES = { "org.bdinetwork.asr.audit" }` to start. Drop the source spec’s buggy `internalsAreModulePrivate` rule (subsumed by the allow-list). 3. **Phase 3 — Roll out.** Migrate `document`, `verification`, `onboarding` opportunistically as they are touched, adding each to `MODULES`; dissolve `model`/`repository` per module into `internal.persistence`. ## Amendment 1 (2026-07-06): published-entity zone [Section titled “Amendment 1 (2026-07-06): published-entity zone”](#amendment-1-2026-07-06-published-entity-zone) ### Context [Section titled “Context”](#context) The COMP-146 severing slice (#154) applied the original pattern strictly: `LegalEntityIdentifier`’s `@ManyToOne OnboardingRequest` was replaced by a bare `UUID onboardingRequestId` so nothing outside the onboarding module could reference its hidden aggregate. Result on the ground: the schema’s `onboarding_request_id` column has **no foreign-key constraint** (the baseline deliberately created it as “a bare id, not an FK”), so the link now has neither object navigation nor database referential integrity — it is an orphanable stamp. And the coupling did not disappear: `EvidenceRepository` still traverses `legalEntityIdentifier.onboardingRequestId` in JPQL, just untyped. This over-shoots the goal. These modules are one modulith sharing one database; they are not microservice candidates whose persistence must be separable. Sacrificing referential integrity for import-level isolation is the wrong trade. What the original pattern lacked is a way for a module to keep its entities **at home** while letting sibling modules hold real, FK-backed `@ManyToOne` associations to a *chosen few* of them — without reopening the whole `internal.persistence` subtree or regressing to a global `model` basket. ### Decision [Section titled “Decision”](#decision) Add a fourth zone, **`entity/`** — the module’s **published persistence types** — as a whitelisted seam sub-package alongside `dto` and `exceptions` (one entry in `ModuleBoundaryTest.SEAM_SUBPACKAGES`). This is the modulith equivalent of Spring Modulith’s *named interfaces*: an explicit, opt-in widening of the seam, per type. Rules of the zone: 1. **Entity class only, never its repository.** The repository stays in `internal.persistence`. Sibling modules may hold references to a published entity — `@ManyToOne` fields, method signatures, lazy navigation (reads) — but every write to it and every query *for* it goes through the owning module’s seam service. Publishing an entity publishes a type, not a data-access path. 2. **A published entity has no outbound module references.** No association, field, or signature on a published entity may point at another module’s types (in particular: no inverse `@OneToMany` back across the boundary). Associations point *into* the `entity` zone, never out of it. This keeps the cross-module FK graph directed and keeps the owning module ignorant of its dependents. 3. **Every cross-module `@ManyToOne` is backed by a real Liquibase FK constraint** with explicitly decided delete semantics. Lifecycle claims (“evidence outlives onboarding”) must be expressed as an `ON DELETE` choice, not as absence of a constraint. Under the repo’s soft-delete convention (`is_deleted`), `RESTRICT` is the default and costs nothing. 4. **Two dependency graphs, each cycle-free, judged separately.** The *data graph* (dependencies on `M.entity..`) and the *behavior graph* (dependencies on module roots, `dto`, `exceptions`) are stratified: a module may be upstream in one and downstream in the other without that counting as a cycle. Example this amendment legalises: evidence-side entities reference `onboarding.entity.OnboardingRequest` (data), while onboarding services query evidence finders ( behavior). The Phase-2 cycle rule, when it lands, checks each graph independently; a cycle *within* either graph remains forbidden. What this amendment does **not** relax: * Entities do not move to the module root — the root stays the behavior seam (service interface, seam enums/value types). Mixing entities into it would blur which surface a caller is using. * `internal.persistence` remains the **default** home for entities. Publication is the exception and needs a cross-module association to justify it; an `entity/` package with no external dependents should be folded back into `internal.persistence`. * The bare-UUID + FK form stays available for links that are genuinely provenance-only (no navigation need) — but it must carry the FK constraint; rule 3 applies either way. ### Consequences [Section titled “Consequences”](#consequences-1) * **Positive**: cross-module many-to-one relations regain type safety, JPQL navigation, and DB referential integrity; entity ownership stays visible in the package tree; publication is explicit and reviewable (a type moving into `entity/` is a boundary decision, seen in the diff); the enforcement mechanism is unchanged — same allow-list, one more sub-package. * **Negative**: the seam is wider — a published entity’s whole public surface (fields, mutators) is reachable, so write-discipline (rule 1) rests on review plus a possible later ArchUnit rule against calling published-entity mutators cross-module; two-graph cycle checking is subtler than one global slice. * **Neutral**: JPA cascades across the boundary are effectively excluded by rule 2 (cascades follow outbound associations, which published entities don’t have). ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [x] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [ADR-00009: Layered source code structure](./00009-layered-source-code-structure.md) — orthogonal (tenancy namespaces, not intra-module layout) * [ADR-00021: Documents behind an ObjectStorageProvider seam](./00021-document-store-objectstorageprovider-seam.md) — related “seam” vocabulary * [TDR-0005: Audit payload serialization](../../ai/adrs/) — `AuditPayloads`, referenced by the pilot * GitHub issue #148 (admin audit-records read endpoint) and epic COMP-146 (improve audit logging) ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 2026-07-02 | Proposed | Initial proposal — audit as pilot module | | 2026-07-02 | Proposed | Onboarding enforced alongside the audit pilot (PRD #151): aggregate hidden behind the seam, boundary rule strengthened to dependency-checking, analyze scope widened to all production roots. (Briefly marked Accepted — premature, still on the PR branch.) | | 2026-07-06 | Proposed | Amendment 1: `entity/` published-entity zone — opt-in seam widening for cross-module `@ManyToOne` with mandatory DB FK; stratified data/behavior cycle graphs | | 2026-07-06 | Proposed | Phase 3 roll-out (COMP-146 #160–#166): `onboarding` and `evidence` migrated to the seam pattern; `Actor` promoted to `org.bdinetwork.asr.common`; behavior-graph cycle rule enforced, data-graph rule deferred until `model` dissolves (GitHub #159); TDR-0006 | # ADR-00023: CTN Test Strategy — layers, ownership, and CI gating ## Approval Status [Section titled “Approval Status”](#approval-status) *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`), or `Not Required`. A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* | Governance body | Status | Last update | | ---------------------------- | -------- | ----------- | | CTN Project Team | Proposed | 2026-07-08 | | CTN Technical Advisory Board | Pending | — | | CTN Steering Committee | Pending | — | | BDI Conformance Team | Pending | — | | BDI Framework Team | Pending | — | ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) Several decisions already touch testing, and two of them look alike at first glance: [ADR-00015](00015-e2e-playwright-chromium-failsafe.md) (in-repo Java browser tests) and [ADR-000017](000017-test-automation-using-playwright-with-typescript.md) (a strategic Playwright + TypeScript automation framework). They both pick Playwright over Cypress and Selenium, which makes them read as duplicates — but they operate at different layers and answer different questions. There was, until now, no single document that says how the whole test estate fits together: what is tested at which layer, which tooling is used where, who owns and contributes each layer, and what runs on every deploy. This ADR is that overview. It does not re-decide anything the layer-specific ADRs already settled; it places them in one picture and makes the boundaries explicit so the two Playwright ADRs no longer read as overlapping. The guiding goal (raised by the product owner) is old-fashioned test-driven development: a growing, shared suite where the product owner, testers, and end-users can add test cases; where prototype API test cases are carried forward rather than thrown away; and where the suite runs on every deploy so regressions surface early. ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) CTN adopts a layered test strategy. Each layer has a defined tool, a run trigger, and an owner. The layers are complementary; nothing below replaces a lower layer. ### Test layers [Section titled “Test layers”](#test-layers) | Layer | What it proves | Tooling | Runs | Owner | | ------------------------ | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- | ---------------------------------------------------- | | Unit | Logic of a single class/module in isolation | JUnit 5 | Every build (`./mvnw test`), every deploy | Developers | | Integration | A module against real infra (DB, Keycloak) | Quarkus `@QuarkusTest` `*IT` on Failsafe, DevServices Postgres + Keycloak, `@Tag("requires-database")`, Docker-gated | Every CI build where Docker is present; SKIPPED (not green) when absent | Developers | | API / contract | HTTP endpoints and token flows at the wire level | RestAssured + Keycloak admin client (e.g. `EHerkenningStepUpBrokerIT`) | Every CI build (same gate as integration) | Developers; prototype API cases carried forward here | | E2E — in-repo (tactical) | Real-browser flows of the buildless `asr-demo-frontend`, bound to the backend build | **Playwright for Java** on Maven Failsafe — see [ADR-00015](00015-e2e-playwright-chromium-failsafe.md) | Local / on-demand (`@Tag("browser-e2e")`, excluded from CI) | Developers | | E2E — portal (strategic) | Real-browser flows of the production portal UI | **Playwright + TypeScript** TAaaS framework (Page Object / Component / Service / Utility layers, SOLID/DRY) — see [ADR-000017](000017-test-automation-using-playwright-with-typescript.md) | Its own CI/CD pipeline, once the portal frontend stack is chosen | TAaaS + testers + product owner / end-users | | Test data | Consistent, compliant fixtures across all layers | See [ADR-00020](00020-test-data-handling-strategy.md) (tiered: real data for acceptance, synthetic elsewhere) | Underpins all layers | Developers + testers | ### Why two end-to-end tiers (and why that is not a contradiction) [Section titled “Why two end-to-end tiers (and why that is not a contradiction)”](#why-two-end-to-end-tiers-and-why-that-is-not-a-contradiction) The two Playwright ADRs deliberately choose different toolchains because they test different surfaces: * **In-repo Java tier (ADR-00015)** keeps a single Maven toolchain inside the backend repo, whose only frontend today is a throwaway static demo with no build step. Binding one real-browser tracer-bullet test to Failsafe — gated exactly like the database integration tests — buys real OIDC/PKCE coverage without dragging a Node build into a backend repo. ADR-00015 explicitly rejects a separate Node/JS framework *for this purpose*. * **Strategic TypeScript tier (ADR-000017)** is the standalone UI-automation framework for the *real portal*, owned by the TAaaS team and contributed to by testers and the product owner. It targets the portal’s frontend stack — an [open question](../08-crosscutting/ctn-coding-standards.md) still — and lives in its own repository and pipeline. They are **sequential and complementary**, not competing: the Java tier covers what exists now (the demo and backend contract flows); the TypeScript tier is the durable home for portal UI automation once the frontend stack lands. Whether the two toolchains eventually converge onto one is a future decision, taken when the portal stack is decided; this strategy does not force convergence and does not block it. ### Contribution and TDD model [Section titled “Contribution and TDD model”](#contribution-and-tdd-model) * Developers keep the unit, integration, API, and in-repo e2e layers green as part of normal development; these run on every deploy through CI. * The strategic TypeScript suite is the place where the product owner, testers, and end-users add and grow test cases — smoke tests, critical business workflows, and regression scenarios — so the suite expands with the product. * Prototype API test cases are migrated into the API/contract layer rather than discarded, preserving accumulated coverage. ## Consequences [Section titled “Consequences”](#consequences) * **Positive**: one shared picture of the test estate; the 00015 / 000017 overlap is resolved as a deliberate per-layer split; a clear path for non-developers to contribute tests; regressions surface on every deploy. * **Negative**: two Playwright toolchains (Java and TypeScript) must be maintained until (or unless) a convergence decision is taken; ownership spans developers and a separate TAaaS/test function. * **Neutral**: this ADR is an umbrella — changes to any single layer are still made in that layer’s own ADR, and this document is updated to keep the overview current. ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [ ] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * [ADR-00015: End-to-End Browser Test Tooling — Playwright + Chromium, Failsafe-bound](00015-e2e-playwright-chromium-failsafe.md) — the in-repo Java e2e tier * [ADR-000017: Test Automation Using Playwright with TypeScript](000017-test-automation-using-playwright-with-typescript.md) — the strategic portal UI-automation framework * [ADR-00020: Test Data Handling Strategy](00020-test-data-handling-strategy.md) — fixtures across all layers * [Coding standards §7 Testing](../08-crosscutting/ctn-coding-standards.md) — layer conventions and the open frontend-stack question ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | 2026-07-08 | Proposed | Initial umbrella strategy tying together ADR-00015 (in-repo Java e2e) and ADR-000017 (strategic TypeScript framework), plus ADR-00020 (test data) | # ADR Template *Copy this template for new Architecture Decision Records* ```yaml --- status: Proposed lastUpdate: YYYY-MM-DD --- ``` # ADR-\[NNNNN]: \[Decision Title] [Section titled “ADR-\[NNNNN\]: \[Decision Title\]”](#adr-nnnnn-decision-title) *Date: YYYY-MM-DD*\ *Status: \[Proposed | Accepted | Reviewed | Approved | Rejected | Superseded by NNNNN]*\ *Deciders: \[List of people involved in the decision]* ## Context and Problem Statement [Section titled “Context and Problem Statement”](#context-and-problem-statement) \[Describe the architectural problem or challenge that needs to be addressed. Include business context, technical constraints, and stakeholder requirements.] ## Considered Options [Section titled “Considered Options”](#considered-options) ### Option 1: \[Option Name] [Section titled “Option 1: \[Option Name\]”](#option-1-option-name) * **Description**: \[Brief description] * **Pros**: \[List advantages] * **Cons**: \[List disadvantages] ### Option 2: \[Option Name] [Section titled “Option 2: \[Option Name\]”](#option-2-option-name) * **Description**: \[Brief description] * **Pros**: \[List advantages] * **Cons**: \[List disadvantages] ### Option 3: \[Option Name] [Section titled “Option 3: \[Option Name\]”](#option-3-option-name) * **Description**: \[Brief description] * **Pros**: \[List advantages] * **Cons**: \[List disadvantages] ## Decision Outcome [Section titled “Decision Outcome”](#decision-outcome) **Chosen option**: \[Selected option and brief rationale] ### Rationale [Section titled “Rationale”](#rationale) \[Detailed explanation of why this option was selected, including:] * Business alignment * Technical considerations * Risk assessment * Cost/benefit analysis ### Consequences [Section titled “Consequences”](#consequences) * **Positive**: \[Expected benefits] * **Negative**: \[Expected drawbacks or risks] * **Neutral**: \[Other implications] ## Implementation Plan [Section titled “Implementation Plan”](#implementation-plan) 1. **Phase 1**: \[First implementation steps] 2. **Phase 2**: \[Follow-up actions] 3. **Phase 3**: \[Future considerations] ## Compliance and Validation [Section titled “Compliance and Validation”](#compliance-and-validation) * [ ] Security review completed * [ ] Performance impact assessed * [ ] Integration impact evaluated * [ ] Documentation updated * [ ] Stakeholder approval obtained ## Related Documents [Section titled “Related Documents”](#related-documents) * \[Link to related ADRs] * \[Link to relevant architecture documentation] * \[Link to implementation guides] ## Status History [Section titled “Status History”](#status-history) | Date | Status | Notes | | ---------- | --------- | ----------------------- | | YYYY-MM-DD | Proposed | Initial proposal | | YYYY-MM-DD | \[Status] | \[Status change reason] | *Status flow: `Proposed → Accepted → Reviewed → Approved` (or `Rejected`). A superseded ADR keeps its file; only its status changes to `Superseded by NNNNN`.* *** *ADR format based on Michael Nygard’s template with CTN-specific enhancements* # 9. Architecture Decisions (ADRs) This chapter holds the Architecture Decision Records that capture the choices made during the design and evolution of the Connected Trade Network. ## Two decision tiers [Section titled “Two decision tiers”](#two-decision-tiers) Decisions are split into two tiers by audience: * **ADR — Architecture Decision Record** (*this directory*): board-level, large-scale choices an architecture board would own (stack, structure, identity, federation). Five-digit sequential ids (`00001`, …). * **TDR — Technical Decision Record** ([`docs/ai/adrs/`](../../ai/adrs/README.md)): low-level, purely technical choices of concern only to builders (schema tooling, intake field mapping). Separate four-digit sequence (`TDR-0001`, …). A decision belongs in the ADR tier when it is large-scale and board-relevant; otherwise it is a TDR. Each tier has its own numbering sequence and index; cross-references between tiers are by id. ## Existing ADRs [Section titled “Existing ADRs”](#existing-adrs) | Ref | Title | | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | [00001](00001-record-architecture-decisions.md) | Record architecture decisions | | [00002](00002-define-a-preferred-stack.md) | Define a preferred technology stack | | [00003](00003-separate-users-and-participants.md) | Separate users and participants | | [00004](00004-naming-follows-glossary.md) | Naming follows the glossary | | [00005](00005-use-agents-md-for-ai-guidelines.md) | Use `AGENTS.md` for AI guidelines | | [00006](00006-token-claim-composition.md) | Token claim composition | | [00007](00007-identity-mutation-proxy.md) | Identity mutation proxy | | [00008](00008-external-idp-federation-strategy.md) | External IdP federation strategy | | [00009](00009-layered-source-code-structure.md) | Layered source code structure | | [00010](00010-evolving-to-quarkus-extensions.md) | Evolving from Package-based Plugins to Quarkus Extensions | | [00011](00011-uri-based-legal-entity-identifier-scheme.md) | URI-Based Legal Entity Identifier Scheme | | [00012](00012-onboarding-request-lifecycle-transition-seam.md) | OnboardingRequest Lifecycle Transition Seam | | [00013](00013-applicant-keycloak-account-provisioned-at-intake.md) | Applicant Keycloak account provisioned at intake | | [00014](00014-claim-verification-provider-seam-and-eherkenning.md) | Claim Verification Provider seam and eHerkenning control-proof | | [00015](00015-e2e-playwright-chromium-failsafe.md) | End-to-End Browser Test Tooling — Playwright + Chromium | | [00016](00016-language-support-and-documentation-policy.md) | Language Support and Documentation Policy | | [00017](000017-test-automation-using-playwright-with-typescript.md) | Test Automation Using Playwright with TypeScript (strategic portal UI framework) | | [00018](00018-build-enforced-java-formatting-via-spotless.md) | Build-enforced Java formatting via Spotless + an Eclipse JDT profile | | [00019](00019-bulk-import-distinct-source-with-orthogonal-validity.md) | Bulk import is a distinct onboarding Source with orthogonal Validity and admin-owned repair | | [00020](00020-test-data-handling-strategy.md) | Test Data Handling Strategy | | [00021](00021-document-store-objectstorageprovider-seam.md) | Documents as a decoupled service behind an ObjectStorageProvider seam, Postgres-first | | [00022](00022-module-seam-pattern.md) | Module seam pattern — in-process seam vs internal, with a REST-boundary zone (audit pilot) | | [00023](00023-test-strategy.md) | CTN Test Strategy — layers, ownership, and CI gating (umbrella over 00015 / 00017 / 00020) | (Two early decisions were reclassified as technical decisions and moved to the TDR tier — see [`docs/ai/adrs/`](../../ai/adrs/README.md). Numbering caveats: `00015` was used twice (Test Data Handling Strategy and the Playwright e2e tooling ADR); ADR-00017 is present in-repo but its file is named with an extra leading zero (`000017-…`); `00019` is currently unused. A separate ADR-renumbering branch (not yet merged) fixes these collisions/gaps — reconcile `00021`’s number against that branch on merge.) ## Templates [Section titled “Templates”](#templates) * [`ADR-TEMPLATE.md`](https://github.com/Basic-Data-Infrastructure/CNCL-BDI/blob/main/docs/arc42/09-decisions/ADR-TEMPLATE.md) — MADR-style template for new architecture decisions. * [`REQ-TEMPLATE.md`](https://github.com/Basic-Data-Infrastructure/CNCL-BDI/blob/main/docs/arc42/09-decisions/REQ-TEMPLATE.md) — Template for capturing requirements that drive an ADR. ## Conventions [Section titled “Conventions”](#conventions) * Numbering is sequential (`00001`, `00002`, …). Pick the next free number. * Status moves through `Proposed` → `Accepted` → `Reviewed` → `Approved` (or `Rejected`); a superseded ADR keeps its file but its status changes to `Superseded by 0000N`. * Every ADR has Context, Decision and Consequences sections (see the template). # REQ-20260625-COMP: Bulk Import as a Distinct, Admin-Owned Onboarding Workflow *Date: 2026-06-25* *Requestor: the implementation partner (CTN / BDI)* *Priority: High* *Status: Under Review* ## Request Summary [Section titled “Request Summary”](#request-summary) The current bulk upload of Onboarding Requests is internally just the manual (self-service) intake path run per row. This must be split into its own workflow: bulk rows must survive validation failure (persisted for admin repair), provision a Keycloak account without emailing the applicant yet, be edited by an admin rather than the requesting entity, be classified by an explicit `source`, be separable per upload, and later be deduplicated/grouped in the admin stage. ## Business Context [Section titled “Business Context”](#business-context) ### Business Driver [Section titled “Business Driver”](#business-driver) The association admin onboards many participants from a spreadsheet *namens de aanvrager*. The demo-era implementation (reuse the anonymous intake path) cannot support the real operational flow: bad rows vanish, applicants get emailed prematurely, and the applicant — not the admin — would own correction. The admin needs a controlled, auditable batch workflow they drive end-to-end. ### Stakeholders Affected [Section titled “Stakeholders Affected”](#stakeholders-affected) * **Primary**: association admins running bulk onboarding. * **Secondary**: prospective participants (applicants) whose accounts are pre-provisioned and who are invited on the admin’s schedule; the consortium (potential four-eyes rule). ### Business Impact [Section titled “Business Impact”](#business-impact) * **If Implemented**: a usable bulk onboarding channel — failed rows are fixable, first contact is deliberate, uploads are traceable, duplicates are surfaced. * **If Not Implemented**: bulk import stays a demo wrapper — lost rows, premature mails, wrong ownership of correction. ## Technical Requirements [Section titled “Technical Requirements”](#technical-requirements) ### Functional Requirements [Section titled “Functional Requirements”](#functional-requirements) 1. Every Onboarding Request carries an explicit `source` of `SELF_SERVICE` or `BULK_IMPORT`. 2. A bulk row that fails mapping/validation is **persisted** (not dropped), flagged incomplete, and repairable by an admin; only whole-file failures hard-fail the upload. 3. A bulk row provisions the applicant’s Keycloak account but does **not** send the set-password mail at creation. 4. An admin can later send that mail as an idempotent action, only once the row is complete; the event is recorded. 5. Bulk rows are editable by an admin (any admin), not by the requesting entity. 6. A request cannot be approved until it is complete (“validate before accept”). 7. Multiple uploads are separable. 8. Duplicates are detectable; before enrichment “duplicate” means *same Identifier URI*; the admin panel groups them (full grouping = COMP-19). ### Non-Functional Requirements [Section titled “Non-Functional Requirements”](#non-functional-requirements) * **Security**: import and invitation are admin-gated; the account↔request link stays consistent (no email change after invitation in MVP). * **Reliability**: per-row independence (one bad row cannot abort the batch). * **Maintainability**: must not erode the pure-flat onboarding lifecycle core (ADR-00012) or its duplicate-guard reuse. ### Integration Requirements [Section titled “Integration Requirements”](#integration-requirements) * Keycloak account provisioning (ADR-00013) with deferred set-password mail. * The TDR-0003 scheme→country / court-code derivation for identifier repair. ## Current State Analysis [Section titled “Current State Analysis”](#current-state-analysis) ### Existing Capabilities [Section titled “Existing Capabilities”](#existing-capabilities) * `BulkImportService` parses `.xlsx` and loops rows through `OnboardingRequestService.save(req, adminSub)` (TDR-0003), which validates-or-rejects, provisions the account **and** sends the set-password mail. * Failed rows are returned transiently in `ImportResultResp.failed[]` and lost. * `OnboardingRequest.identifierUri` is `nullable = false`; the lifecycle is a pure flat edge table whose `openStates()` also scopes the duplicate guard. ### Gaps Identified [Section titled “Gaps Identified”](#gaps-identified) * No persistence of failed rows; no admin repair path. * No way to provision without emailing; no separate invitation action. * No `source` classification; no per-upload identity; no completeness gate on approval. ### Impact on Current Architecture [Section titled “Impact on Current Architecture”](#impact-on-current-architecture) * A nullable `identifierUri`; new `source` / `batchId` / `invitedAt` / validity columns; a distinct bulk create path; admin-scoped editing rules; an approval guard. ## Proposed Approach [Section titled “Proposed Approach”](#proposed-approach) See [ADR-00019](00019-bulk-import-distinct-source-with-orthogonal-validity.md) for the decided design (Source axis + orthogonal Validity, distinct create path, deferred Invitation, admin-owned repair, `batchId` separability). ## Success Criteria [Section titled “Success Criteria”](#success-criteria) ### Acceptance Criteria [Section titled “Acceptance Criteria”](#acceptance-criteria) * [ ] A failed bulk row is persisted, visibly incomplete, and repairable by an admin. * [ ] Bulk creation provisions the account but sends no mail until the admin invites. * [ ] Invitation is admin-only, idempotent, allowed only when complete, and recorded. * [ ] An incomplete request cannot be approved. * [ ] Each request exposes its `source`; bulk rows are admin-editable, self-service rows are not. * [ ] Uploads are distinguishable and groupable; same-Identifier-URI duplicates surface grouped. ## Related Documents [Section titled “Related Documents”](#related-documents) * [ADR-00019](00019-bulk-import-distinct-source-with-orthogonal-validity.md) — the decision. * [TDR-0003](../../ai/adrs/TDR-0003-spreadsheet-bulk-import-maps-row-to-onboarding-request.md) — prior bulk import. * COMP-24, COMP-119, COMP-19 (Jira). ## Status History [Section titled “Status History”](#status-history) | Date | Status | Reviewer | Notes | | ---------- | ------------ | -------------------------- | ------------------------------------------------- | | 2026-06-25 | Submitted | the implementation partner | Captured from the grill-with-docs design session. | | 2026-06-25 | Under Review | the implementation partner | Drives ADR-00019 (Proposed). | *** *Requests help capture and track stakeholder requirements in a structured way* # REQ-20260625: General document store for MIME-typed uploads *Date: 2026-06-25*\ *Requestor: the implementation partner (on behalf of the CTN platform)*\ *Priority: High*\ *Status: Submitted* ## Request Summary [Section titled “Request Summary”](#request-summary) A general, reusable capability to store MIME-typed documents (mostly PDF, also scanned images) with their upload metadata, addressable by a stable UUID so that various platform entities can reference them. The first consumer is the onboarding flow (a requestor uploading proof supporting an Onboarding Request), but the store itself must not be onboarding-specific. ## Business Context [Section titled “Business Context”](#business-context) ### Business Driver [Section titled “Business Driver”](#business-driver) Participants and applicants must be able to supply supporting documents — registry extracts proving a business-register identifier, a signed acceptance of the consortium contract conditions, and (later) periodic re-validation uploads. The platform needs one trustworthy place to keep these, with provenance (who uploaded, when) and a business-validity (expiry) marker, rather than a per-feature bolt-on. ### Stakeholders Affected [Section titled “Stakeholders Affected”](#stakeholders-affected) * **Primary**: prospective participants / requestors (upload proof); association back-office admins (review proof). * **Secondary**: future organization admins (`OrgAdminRole`); future automated re-validation; any later entity that needs to attach documents. ### Business Impact [Section titled “Business Impact”](#business-impact) * **If Implemented**: documents become first-class, referenceable, auditable; the onboarding verification flow and future re-validation gain a shared foundation. * **If Not Implemented**: each feature invents its own file handling, with inconsistent metadata, provenance and access control. ## Technical Requirements [Section titled “Technical Requirements”](#technical-requirements) ### Functional Requirements [Section titled “Functional Requirements”](#functional-requirements) 1. Store a MIME-typed document and return a stable UUID identity. 2. Persist upload metadata: MIME type, file name, size, business kind (Document Type), optional business-validity expiry, and uploader/timestamps. 3. Read a stored document’s bytes and metadata back, API-mediated only. 4. Let other entities reference a document by its UUID (the referrer holds the link). 5. Govern, per Document Type, whether an expiry date is forbidden / optional / required. ### Non-Functional Requirements [Section titled “Non-Functional Requirements”](#non-functional-requirements) * **Performance**: metadata listing must not load byte payloads. * **Security**: only authenticated users may upload; read restricted to the uploader and association admins; no direct/presigned object access — everything via the API; the design must not preclude later serving user content from an isolated origin. * **Scalability**: the byte-storage backend must be swappable (Postgres now, an object store such as MinIO/S3 later) without changing the document model or its references. * **Reliability**: stored documents are write-once; corrections are new documents (admins may correct expiry and document type in place). ### Integration Requirements [Section titled “Integration Requirements”](#integration-requirements) * Keycloak-issued identities for uploader provenance (`sub`) and role checks. * The onboarding flow as first consumer (delegating upload to the document service). ## Current State Analysis [Section titled “Current State Analysis”](#current-state-analysis) ### Existing Capabilities [Section titled “Existing Capabilities”](#existing-capabilities) * Quarkus + PostgreSQL + Keycloak stack (ADR-00002); JPA entities on `MetaEntity` (created/updated by/at, soft-delete); Liquibase owns schema (TDR-0001). * Established SPI-seam pattern (onboarding provider, claim-verification provider). * Multipart upload precedent (`BulkImportResource`). * `Verification` precedent for an entity that hangs off an Onboarding Request via a bare id and later relates to a Participant. ### Gaps Identified [Section titled “Gaps Identified”](#gaps-identified) * No document/blob storage of any kind. * No object store in the preferred stack (Postgres only). * No concept of a stored file’s identity, metadata, or expiry. ### Impact on Current Architecture [Section titled “Impact on Current Architecture”](#impact-on-current-architecture) * Adds a new core service + entity + a storage SPI; extends ADR-00002’s Postgres-only stack with a seam that anticipates an object store later. ## Proposed Approach [Section titled “Proposed Approach”](#proposed-approach) See ADR-00021. In short: a decoupled `DocumentService` over `document` + `document_blob` tables; bytes behind an `ObjectStorageProvider` SPI with a Postgres-first implementation; referrers hold the `document_id`. ## Success Criteria [Section titled “Success Criteria”](#success-criteria) ### Acceptance Criteria [Section titled “Acceptance Criteria”](#acceptance-criteria) * [ ] Storing a document returns a UUID; metadata persisted; bytes round-trip via the Postgres `ObjectStorageProvider`. * [ ] Per-Document-Type expiry policy enforced at store time. * [ ] MIME allow-list and per-MIME size cap enforced. * [ ] Metadata reads do not load byte payloads. * [ ] Read authorized only for uploader and association admin. ## Related Documents [Section titled “Related Documents”](#related-documents) * [ADR-00021](00021-document-store-objectstorageprovider-seam.md) * [ADR-00002](00002-define-a-preferred-stack.md) — preferred stack (extended here) * [TDR-0001](../../ai/adrs/TDR-0001-liquibase-owns-schema.md) — Liquibase owns schema ## Status History [Section titled “Status History”](#status-history) | Date | Status | Reviewer | Notes | | ---------- | --------- | -------------------------- | ------------------------------------ | | 2026-06-25 | Submitted | the implementation partner | Initial submission, drives ADR-00021 | # Request Template *Copy this template for architectural requests and requirements* # REQ-\[YYYYMMDD]-\[Source]: \[Request Title] [Section titled “REQ-\[YYYYMMDD\]-\[Source\]: \[Request Title\]”](#req-yyyymmdd-source-request-title) *Date: YYYY-MM-DD*\ *Requestor: \[Name and Organization]*\ *Priority: \[Low | Medium | High | Critical]*\ *Status: \[Submitted | Under Review | Approved | Rejected | Implemented]* ## Request Summary [Section titled “Request Summary”](#request-summary) \[Brief description of the architectural request or requirement] ## Business Context [Section titled “Business Context”](#business-context) ### Business Driver [Section titled “Business Driver”](#business-driver) \[Why is this request important for the business?] ### Stakeholders Affected [Section titled “Stakeholders Affected”](#stakeholders-affected) * **Primary**: \[Who directly benefits] * **Secondary**: \[Who is indirectly affected] ### Business Impact [Section titled “Business Impact”](#business-impact) * **If Implemented**: \[Positive business outcomes] * **If Not Implemented**: \[Business risks or missed opportunities] ## Technical Requirements [Section titled “Technical Requirements”](#technical-requirements) ### Functional Requirements [Section titled “Functional Requirements”](#functional-requirements) 1. \[Requirement 1] 2. \[Requirement 2] 3. \[Requirement 3] ### Non-Functional Requirements [Section titled “Non-Functional Requirements”](#non-functional-requirements) * **Performance**: \[Performance requirements] * **Security**: \[Security requirements] * **Scalability**: \[Scalability requirements] * **Reliability**: \[Reliability requirements] ### Integration Requirements [Section titled “Integration Requirements”](#integration-requirements) * \[Integration point 1] * \[Integration point 2] ## Current State Analysis [Section titled “Current State Analysis”](#current-state-analysis) ### Existing Capabilities [Section titled “Existing Capabilities”](#existing-capabilities) \[What currently exists that relates to this request] ### Gaps Identified [Section titled “Gaps Identified”](#gaps-identified) * \[Gap 1] * \[Gap 2] * \[Gap 3] ### Impact on Current Architecture [Section titled “Impact on Current Architecture”](#impact-on-current-architecture) \[How this request affects existing systems and components] ## Proposed Approach [Section titled “Proposed Approach”](#proposed-approach) ### High-Level Solution [Section titled “High-Level Solution”](#high-level-solution) \[Suggested approach to address the request] ### Architecture Changes Required [Section titled “Architecture Changes Required”](#architecture-changes-required) * **New Components**: \[Components to be added] * **Modified Components**: \[Components to be changed] * **Deprecated Components**: \[Components to be removed] ### Implementation Options [Section titled “Implementation Options”](#implementation-options) 1. **Option 1**: \[Description, pros, cons] 2. **Option 2**: \[Description, pros, cons] 3. **Option 3**: \[Description, pros, cons] ## Resource Requirements [Section titled “Resource Requirements”](#resource-requirements) ### Development Effort [Section titled “Development Effort”](#development-effort) * **Estimated Timeline**: \[Time estimate] * **Team Resources**: \[Team requirements] * **External Dependencies**: \[External resources needed] ### Infrastructure Requirements [Section titled “Infrastructure Requirements”](#infrastructure-requirements) * \[Infrastructure need 1] * \[Infrastructure need 2] ## Risk Assessment [Section titled “Risk Assessment”](#risk-assessment) ### Technical Risks [Section titled “Technical Risks”](#technical-risks) * **Risk 1**: \[Description and mitigation] * **Risk 2**: \[Description and mitigation] ### Business Risks [Section titled “Business Risks”](#business-risks) * **Risk 1**: \[Description and mitigation] * **Risk 2**: \[Description and mitigation] ## Success Criteria [Section titled “Success Criteria”](#success-criteria) ### Acceptance Criteria [Section titled “Acceptance Criteria”](#acceptance-criteria) * [ ] \[Criteria 1] * [ ] \[Criteria 2] * [ ] \[Criteria 3] ### Success Metrics [Section titled “Success Metrics”](#success-metrics) * **Metric 1**: \[How to measure success] * **Metric 2**: \[How to measure success] ## Review and Approval [Section titled “Review and Approval”](#review-and-approval) ### Technical Review [Section titled “Technical Review”](#technical-review) * [ ] Architecture impact assessed * [ ] Security implications reviewed * [ ] Performance impact evaluated * [ ] Integration complexity analyzed ### Business Review [Section titled “Business Review”](#business-review) * [ ] Business value validated * [ ] Stakeholder approval obtained * [ ] Resource allocation confirmed * [ ] Timeline approved ## Related Documents [Section titled “Related Documents”](#related-documents) * \[Link to related requests] * \[Link to relevant architecture documentation] * \[Link to business requirements] ## Status History [Section titled “Status History”](#status-history) | Date | Status | Reviewer | Notes | | ---------- | --------- | -------- | ----------------------- | | YYYY-MM-DD | Submitted | \[Name] | Initial submission | | YYYY-MM-DD | \[Status] | \[Name] | \[Status change reason] | *** *Requests help capture and track stakeholder requirements in a structured way* # Quality Requirements *Quality goals, scenarios, and measures for the Connected Trade Network* ## Quality Goals Overview [Section titled “Quality Goals Overview”](#quality-goals-overview) The Connected Trade Network must meet stringent quality requirements to ensure secure, reliable, and efficient operations in the container transport ecosystem. ### Top 5 Quality Goals [Section titled “Top 5 Quality Goals”](#top-5-quality-goals) | Priority | Quality Goal | Description | Key Metric | | -------- | -------------------- | ------------------------------------------------ | -------------------------------- | | 1 | **Security** | Zero-trust data access control | No unauthorized access incidents | | 2 | **Performance** | Fast response times for real-time operations | API response <500ms (P95) | | 3 | **Reliability** | Consistent availability for transport operations | 99.9% uptime | | 4 | **Scalability** | Handle growing network participants | Support 10,000+ active parties | | 5 | **Interoperability** | Seamless integration with diverse systems | 100% standards compliance | ## Quality Tree [Section titled “Quality Tree”](#quality-tree) ```plaintext Connected Trade Network Quality ├── Security │ ├── Authentication (OAuth 2.0, mTLS) │ ├── Authorisation (VAD validated locally at the data provider) │ ├── Encryption (TLS 1.3, AES-256) │ └── Audit (Complete trail) │ ├── Performance │ ├── Response Time (<500ms P95) │ ├── Throughput (1000 req/s) │ ├── Token Generation (<100ms) │ └── Cache Hit Ratio (>90%) │ ├── Reliability │ ├── Availability (99.9%) │ ├── Fault Tolerance (Multi-region) │ ├── Recovery (RTO 1hr, RPO 15min) │ └── Data Integrity (ACID) │ ├── Scalability │ ├── Horizontal (Auto-scaling) │ ├── Data Volume (Millions of records) │ ├── Concurrent Users (10,000+) │ └── Event Processing (10K/s) │ └── Usability ├── API Design (REST/OpenAPI) ├── Documentation (Comprehensive) ├── Error Messages (Clear) └── Integration (SDK/Examples) ``` ## Quality Scenarios [Section titled “Quality Scenarios”](#quality-scenarios) ### Security Scenarios [Section titled “Security Scenarios”](#security-scenarios) #### SC-1: Unauthorized Access Attempt [Section titled “SC-1: Unauthorized Access Attempt”](#sc-1-unauthorized-access-attempt) | Aspect | Description | | --------------- | --------------------------------------------------------------- | | **Stimulus** | Attacker attempts to access container data without valid tokens | | **Source** | External malicious actor | | **Environment** | Production system under normal load | | **Response** | System denies access, logs attempt, alerts security team | | **Measure** | 100% of unauthorized attempts blocked within 50ms | #### SC-2: Token Forgery [Section titled “SC-2: Token Forgery”](#sc-2-token-forgery) | Aspect | Description | | --------------- | --------------------------------------------------------------- | | **Stimulus** | Forged VAD token presented for data access | | **Source** | Compromised party system | | **Environment** | Production API gateway | | **Response** | Token signature validation fails (JWKS mismatch), access denied | | **Measure** | Forged tokens detected in <10ms with 100% accuracy | #### SC-3: Data Breach Prevention [Section titled “SC-3: Data Breach Prevention”](#sc-3-data-breach-prevention) | Aspect | Description | | --------------- | ----------------------------------------------------- | | **Stimulus** | SQL injection attempt on the ASR API | | **Source** | External attacker | | **Environment** | Public-facing API | | **Response** | Input sanitization prevents injection, attempt logged | | **Measure** | Zero successful injection attacks | ### Performance Scenarios [Section titled “Performance Scenarios”](#performance-scenarios) #### PE-1: Peak Load Handling [Section titled “PE-1: Peak Load Handling”](#pe-1-peak-load-handling) | Aspect | Description | | --------------- | ----------------------------------------------------- | | **Stimulus** | 5000 concurrent API requests during port congestion | | **Source** | Multiple transport parties | | **Environment** | Production system, business hours | | **Response** | System scales automatically, maintains response times | | **Measure** | P95 response time <500ms, P99 <1000ms | #### PE-2: Bulk Participant Onboarding [Section titled “PE-2: Bulk Participant Onboarding”](#pe-2-bulk-participant-onboarding) | Aspect | Description | | --------------- | ---------------------------------------------------------- | | **Stimulus** | Excel file with 200 participants uploaded | | **Source** | Association administrator | | **Environment** | Normal system load | | **Response** | File processed, participants onboarded, notifications sent | | **Measure** | Complete processing within 5 minutes | #### PE-3: Token Generation Load [Section titled “PE-3: Token Generation Load”](#pe-3-token-generation-load) | Aspect | Description | | --------------- | ------------------------------------ | | **Stimulus** | 1000 simultaneous VAD token requests | | **Source** | Participants at shift change | | **Environment** | Peak morning hours | | **Response** | Tokens generated without queuing | | **Measure** | All tokens generated within 100ms | ### Reliability Scenarios [Section titled “Reliability Scenarios”](#reliability-scenarios) #### RE-1: Database Failure [Section titled “RE-1: Database Failure”](#re-1-database-failure) | Aspect | Description | | --------------- | -------------------------------------------------- | | **Stimulus** | Primary PostgreSQL instance fails | | **Source** | Hardware failure | | **Environment** | Production operations | | **Response** | Automatic failover to replica, cache serves reads | | **Measure** | Service restored within 60 seconds, zero data loss | #### RE-2: Region Outage [Section titled “RE-2: Region Outage”](#re-2-region-outage) | Aspect | Description | | --------------- | ------------------------------------------------------------- | | **Stimulus** | Complete West Europe region outage | | **Source** | Major datacenter incident | | **Environment** | Active transport operations | | **Response** | Traffic routed to North Europe, degraded mode activated | | **Measure** | Service available within 5 minutes, core functions maintained | #### RE-3: Event Grid Failure [Section titled “RE-3: Event Grid Failure”](#re-3-event-grid-failure) | Aspect | Description | | --------------- | --------------------------------------------------- | | **Stimulus** | Event Grid service becomes unavailable | | **Source** | Azure service issue | | **Environment** | Normal operations | | **Response** | Events queued locally, batch delivery when restored | | **Measure** | No events lost, delivery delay <15 minutes | ### Scalability Scenarios [Section titled “Scalability Scenarios”](#scalability-scenarios) #### SA-1: Network Growth [Section titled “SA-1: Network Growth”](#sa-1-network-growth) | Aspect | Description | | --------------- | ---------------------------------------------------- | | **Stimulus** | Network grows from 1000 to 10000 active participants | | **Source** | Business expansion | | **Environment** | Production over 6 months | | **Response** | System scales infrastructure automatically | | **Measure** | Performance maintained, no degradation | #### SA-2: Data Volume Increase [Section titled “SA-2: Data Volume Increase”](#sa-2-data-volume-increase) | Aspect | Description | | --------------- | ---------------------------------------------------------- | | **Stimulus** | Participant, endpoint and audit records grow to 10 million | | **Source** | Accumulated network activity | | **Environment** | Production database | | **Response** | Partitioning activated, archival process engaged | | **Measure** | Query performance <100ms maintained | #### SA-3: Event Storm [Section titled “SA-3: Event Storm”](#sa-3-event-storm) | Aspect | Description | | --------------- | -------------------------------------------- | | **Stimulus** | 50000 events generated in 1 minute | | **Source** | Major port system update | | **Environment** | Event processing system | | **Response** | Events buffered, processed in priority order | | **Measure** | All events processed within 5 minutes | ### Usability Scenarios [Section titled “Usability Scenarios”](#usability-scenarios) #### US-1: API Integration [Section titled “US-1: API Integration”](#us-1-api-integration) | Aspect | Description | | --------------- | ------------------------------------- | | **Stimulus** | New developer integrates with CTN API | | **Source** | Third-party system developer | | **Environment** | Development environment | | **Response** | Developer uses documentation and SDK | | **Measure** | Working integration within 2 days | #### US-2: Error Resolution [Section titled “US-2: Error Resolution”](#us-2-error-resolution) | Aspect | Description | | --------------- | ---------------------------------------------- | | **Stimulus** | API returns validation error | | **Source** | Client application | | **Environment** | Production API | | **Response** | Clear error message with resolution guidance | | **Measure** | Issue resolved without support in 90% of cases | ## Performance Requirements [Section titled “Performance Requirements”](#performance-requirements) ### Response Time Requirements [Section titled “Response Time Requirements”](#response-time-requirements) | Operation | Target (P50) | Target (P95) | Target (P99) | Current | | --------------------------------- | ------------ | ------------ | ------------ | ------- | | Token Validation | 20ms | 50ms | 100ms | 35ms | | Participant Lookup | 30ms | 100ms | 200ms | 85ms | | Authorization Decision | 50ms | 150ms | 300ms | 120ms | | Endpoint Registration | 200ms | 500ms | 1000ms | 420ms | | Bulk Onboarding (per participant) | 100ms | 300ms | 500ms | 250ms | | Event Delivery | 200ms | 1000ms | 2000ms | 750ms | ### Throughput Requirements [Section titled “Throughput Requirements”](#throughput-requirements) | Component | Requirement | Current Capacity | Headroom | | ---------------- | ----------- | ---------------- | -------- | | API Gateway | 5000 req/s | 8000 req/s | 60% | | Token Generation | 1000/s | 1500/s | 50% | | Event Processing | 10000/s | 15000/s | 50% | | Database Writes | 500/s | 800/s | 60% | | Cache Operations | 50000/s | 75000/s | 50% | ### Resource Utilization Targets [Section titled “Resource Utilization Targets”](#resource-utilization-targets) | Resource | Normal | Peak | Alert | Critical | | ----------- | ------ | ---- | ----- | -------- | | CPU | 40% | 70% | 80% | 90% | | Memory | 50% | 75% | 85% | 95% | | Disk I/O | 30% | 60% | 75% | 90% | | Network | 40% | 70% | 80% | 95% | | Queue Depth | 100 | 1000 | 5000 | 10000 | ## Reliability Requirements [Section titled “Reliability Requirements”](#reliability-requirements) ### Availability Targets [Section titled “Availability Targets”](#availability-targets) | Service Level | Target | Measurement Period | Allowed Downtime | | -------------- | ------ | ------------------ | ---------------- | | Core Services | 99.9% | Monthly | 43 minutes | | API Gateway | 99.95% | Monthly | 22 minutes | | Data Access | 99.9% | Monthly | 43 minutes | | Event Delivery | 99.5% | Monthly | 3.6 hours | | Portal Access | 99.0% | Monthly | 7.2 hours | ### Failure Recovery [Section titled “Failure Recovery”](#failure-recovery) | Scenario | RTO | RPO | Method | | ----------------- | ---------- | ---------- | --------------------- | | Service Crash | 30 seconds | 0 | Auto-restart | | Database Failure | 1 minute | 0 | Replica failover | | Cache Failure | 5 minutes | N/A | Rebuild from DB | | Region Failure | 15 minutes | 15 minutes | Cross-region failover | | Complete Disaster | 1 hour | 15 minutes | Full restoration | ## Security Requirements [Section titled “Security Requirements”](#security-requirements) ### Authentication & Authorization [Section titled “Authentication & Authorization”](#authentication--authorization) | Requirement | Implementation | Validation | | ------------------------------- | ----------------------------------------- | ------------------- | | Multi-factor authentication | Keycloak MFA (enforced on admin surfaces) | Audit logs | | Token expiration | Short-lived VAD (1 hour) | Token validation | | Privilege escalation prevention | Role-based access control | Penetration testing | | API rate limiting | 1000 requests/minute per client | API Management | ### Encryption Standards [Section titled “Encryption Standards”](#encryption-standards) | Data State | Standard | Key Management | | ---------- | --------------- | -------------------- | | In Transit | TLS 1.3 minimum | Managed certificates | | At Rest | AES-256-GCM | Azure Key Vault | | Tokens | RS256 signing | HSM-backed keys | | Backups | AES-256 | Azure managed | ### Compliance Requirements [Section titled “Compliance Requirements”](#compliance-requirements) | Standard | Requirement | Evidence | | -------------- | --------------------------------- | ------------------------------------- | | GDPR | Data protection, right to erasure | Audit trails, data retention policies | | SOC 2 Type II | Security controls | Annual audit report | | ISO 27001 | Information security management | Certification | | NIS2 Directive | Cybersecurity measures | Compliance report | ## Monitoring and Alerting [Section titled “Monitoring and Alerting”](#monitoring-and-alerting) ### SLA Monitoring [Section titled “SLA Monitoring”](#sla-monitoring) ```yaml Metrics: - name: api_availability target: 99.9% window: 5m alert: below 99.5% - name: response_time_p95 target: 500ms window: 1m alert: above 750ms - name: error_rate target: <1% window: 5m alert: above 2% - name: token_generation_success target: 99.9% window: 5m alert: below 99% ``` ### Alert Priority Matrix [Section titled “Alert Priority Matrix”](#alert-priority-matrix) | Severity | Response Time | Examples | | -------- | ----------------- | --------------------------------------- | | Critical | 15 minutes | System down, data breach, total failure | | High | 1 hour | Performance degradation, partial outage | | Medium | 4 hours | Non-critical errors, high latency | | Low | Next business day | Minor issues, documentation | ## Testing Requirements [Section titled “Testing Requirements”](#testing-requirements) ### Test Coverage Targets [Section titled “Test Coverage Targets”](#test-coverage-targets) | Test Type | Coverage Target | Current | Gap | | ----------------- | ------------------- | ------- | --- | | Unit Tests | 80% | 75% | 5% | | Integration Tests | 70% | 65% | 5% | | API Tests | 100% | 95% | 5% | | Security Tests | 100% critical paths | 90% | 10% | | Performance Tests | All key scenarios | 80% | 20% | ### Load Testing Scenarios [Section titled “Load Testing Scenarios”](#load-testing-scenarios) | Scenario | Users | Duration | Success Criteria | | ----------- | ----- | ---------- | ------------------ | | Normal Load | 1000 | 2 hours | <500ms P95 | | Peak Load | 5000 | 30 minutes | <1000ms P95 | | Stress Test | 10000 | 15 minutes | No crashes | | Endurance | 2000 | 24 hours | Stable performance | ## Quality Assurance Process [Section titled “Quality Assurance Process”](#quality-assurance-process) ### Code Quality Gates [Section titled “Code Quality Gates”](#code-quality-gates) ```yaml Pull Request Checks: - Code Coverage: minimum 80% - Security Scan: no high/critical issues - Code Review: 2 approvals required - Build Success: all targets - Test Success: all tests passing - Performance: no regression >10% ``` ### Deployment Quality Checks [Section titled “Deployment Quality Checks”](#deployment-quality-checks) 1. **Pre-deployment**: Automated tests, security scan 2. **Deployment**: Blue-green deployment, health checks 3. **Post-deployment**: Smoke tests, monitoring validation 4. **Rollback Criteria**: Error rate >5%, response time >2x *** *These quality requirements ensure the Connected Trade Network meets operational excellence standards.* # Risks and Technical Debt *Identified risks, mitigation strategies, and technical debt for the Connected Trade Network* ## Risk Register Overview [Section titled “Risk Register Overview”](#risk-register-overview) This section documents identified risks, their potential impact, and mitigation strategies. Risks are categorized by type and probability/impact assessment. ## Risk Assessment Matrix [Section titled “Risk Assessment Matrix”](#risk-assessment-matrix) ```plaintext Impact → ↓ Low Medium High Critical P ┌─────────────────────────────────────────────┐ r │ │ o │ High │ R-7 │ R-4 │ R-1 │ R-2 │ b │ │ R-11 │ R-5 │ R-3 │ │ a │────────┼────────┼────────┼────────┼────────│ b │ Medium │ R-12 │ R-8 │ R-6 │ R-9 │ i │ │ │ R-13 │ R-10 │ │ l │────────┼────────┼────────┼────────┼────────│ i │ Low │ │ R-14 │ R-15 │ R-16 │ t │ │ │ │ │ │ y └─────────────────────────────────────────────┘ ``` ## Critical and High Risks [Section titled “Critical and High Risks”](#critical-and-high-risks) ### R-1: Non-Member Access Policy Ambiguity 🔴 [Section titled “R-1: Non-Member Access Policy Ambiguity 🔴”](#r-1-non-member-access-policy-ambiguity-) **Category**: Business\ **Probability**: High\ **Impact**: High\ **Status**: ⚠️ UNRESOLVED **Description**: Lack of clear business rules for granting data access to non-member subcontractors creates operational and legal uncertainty. **Potential Impact**: * Legal liability for unauthorized data sharing * Operational delays in transport chains * Inconsistent access decisions * Compliance violations **Mitigation Strategy**: * Stakeholder workshop to define non-member access policy (planned) * Legal review of liability framework * Pilot program with selected participants * Implement tiered access model as interim solution **Contingency Plan**: * Default to restrictive access * Manual approval workflow * Escalation to legal team *** ### R-2: Scalability Bottlenecks 🔴 [Section titled “R-2: Scalability Bottlenecks 🔴”](#r-2-scalability-bottlenecks-) **Category**: Technical\ **Probability**: High\ **Impact**: Critical **Description**: System may not scale to handle 10,000+ active participants with millions of records. **Current Bottlenecks**: * Database write throughput limits * Event Grid capacity constraints * Token generation performance * Cache synchronization overhead **Mitigation Strategy**: * Implement database sharding * Deploy read replicas * Optimize query patterns * Enhance caching strategy **Monitoring**: ```yaml Metrics to Track: - Database CPU utilization - Query response times - Event processing lag - Cache miss rate - API throttling incidents ``` *** ### R-3: Security Breach 🔴 [Section titled “R-3: Security Breach 🔴”](#r-3-security-breach-) **Category**: Security\ **Probability**: High\ **Impact**: High **Description**: Potential for unauthorized access to sensitive transport data through various attack vectors. **Attack Vectors**: * Token forgery or theft * API vulnerabilities * Insider threats * Supply chain attacks **Mitigation Strategy**: * Regular security audits * Penetration testing quarterly * Security training for developers * Implement zero-trust architecture * Enhanced monitoring and alerting **Incident Response Plan**: 1. Immediate containment 2. Impact assessment 3. Stakeholder notification 4. Forensic analysis 5. Recovery and remediation ## Medium Risks [Section titled “Medium Risks”](#medium-risks) ### R-4: Integration Complexity 🟡 [Section titled “R-4: Integration Complexity 🟡”](#r-4-integration-complexity-) **Category**: Technical\ **Probability**: High\ **Impact**: Medium **Description**: Complex integration requirements may delay adoption by transport parties. **Challenges**: * Diverse existing systems * Varying technical capabilities * Legacy system constraints * Multiple data formats **Mitigation**: * Provide comprehensive SDKs * Create integration templates * Offer professional services * Phased rollout approach *** ### R-5: Regulatory Compliance Changes 🟡 [Section titled “R-5: Regulatory Compliance Changes 🟡”](#r-5-regulatory-compliance-changes-) **Category**: Compliance\ **Probability**: High\ **Impact**: Medium **Description**: Evolving regulations (GDPR, NIS2, sector-specific) may require system changes. **Mitigation**: * Regular compliance reviews * Flexible architecture design * Legal team engagement * Audit trail capabilities * Maintain explicit mapping to current obligations: see [NIS2 compliance mapping](../08-crosscutting/ctn-nis2-compliance.md) **Related risk**: R-17 covers the specific NIS2 supply-chain liability exposure. *** ### R-6: Vendor Lock-in 🟡 [Section titled “R-6: Vendor Lock-in 🟡”](#r-6-vendor-lock-in-) **Category**: Technical\ **Probability**: Medium\ **Impact**: High **Description**: Heavy reliance on Azure-specific services creates vendor dependency. **Dependencies**: * Azure Kubernetes Service (AKS) * API Management * Document Intelligence **Mitigation**: * Abstract Azure services behind interfaces * Document Azure-agnostic alternatives * Maintain portability assessment * Regular vendor review *** ### R-7: Performance Degradation 🟡 [Section titled “R-7: Performance Degradation 🟡”](#r-7-performance-degradation-) **Category**: Technical\ **Probability**: High\ **Impact**: Low **Description**: System performance may degrade under unexpected load patterns. **Mitigation**: * Continuous performance testing * Auto-scaling configuration * Performance monitoring * Capacity planning *** ### R-8: Data Quality Issues 🟡 [Section titled “R-8: Data Quality Issues 🟡”](#r-8-data-quality-issues-) **Category**: Data\ **Probability**: Medium\ **Impact**: Medium **Description**: Poor data quality from Excel imports and manual entry affects system reliability. **Mitigation**: * Input validation rules * Data quality monitoring * User training programs * Automated data cleaning *** ### R-17: NIS2 Supply-Chain Liability Exposure 🟡 [Section titled “R-17: NIS2 Supply-Chain Liability Exposure 🟡”](#r-17-nis2-supply-chain-liability-exposure-) **Category**: Compliance / Commercial\ **Probability**: High\ **Impact**: High\ **Status**: ⚠️ ACTIVE — gap analysis published **Description**: Most CTN participants are themselves NIS2-obliged entities (Transport sector). Under Article 21(2)(d) they must perform supply-chain security assessments on their critical providers — including CTN. If CTN cannot demonstrate adequate security controls (ISO 27001, pentest reports, SBOM, CVD policy, incident response capability), participants face two equally bad options: include CTN in their attestations regardless and absorb the residual risk, or exclude CTN as a critical provider and reduce their reliance on it. Either outcome harms CTN adoption. A separate question is whether DIL itself qualifies as a NIS2 entity under “Digital Infrastructure” or “ICT service management”. If yes, DIL faces direct supervision by RDI, mandatory registration, 24h/72h/1M incident reporting, and management-body liability with personal exposure for directors. Penalties reach €10M / 2% of worldwide turnover. **Potential Impact**: * Participant churn or refusal to onboard * Direct regulatory exposure for DIL with personal director liability * Reputational damage from a publicly-reported security incident without an established response process * Penalties up to €10M / 2% of worldwide turnover **Mitigation Strategy**: * Maintain the [NIS2 compliance mapping](../08-crosscutting/ctn-nis2-compliance.md) as a living document * Pursue ISO 27001 certification for DIL operations * Schedule annual external penetration test * Publish vulnerability disclosure policy (`security.txt`) on all CTN portals * Establish SBOM generation in CI/CD * Document and rehearse 24h / 72h / 1M incident reporting flow * Provide NIS2 awareness training for the DIL management body * Obtain legal opinion on whether DIL qualifies as a NIS2 entity **Contingency Plan**: * If a participant’s NIS2 audit identifies CTN as inadequate: targeted remediation plan with documented timeline * If RDI determines DIL is in scope: registration, board training, and rapid-track ISO 27001 program * Maintain pre-drafted CSIRT-DSP notification templates so the 24h clock can be met from day one **Related**: R-5 (general regulatory change risk). ## Technical Debt Register [Section titled “Technical Debt Register”](#technical-debt-register) ### TD-1: Excel Format Dependencies 🔧 [Section titled “TD-1: Excel Format Dependencies 🔧”](#td-1-excel-format-dependencies-) **Severity**: High\ **Effort**: Medium\ **Priority**: P1 **Description**: System heavily depends on specific Excel formats for bulk participant onboarding. **Impact**: * Brittle processing logic * Version compatibility issues * Limited format flexibility **Resolution Plan**: * Implement format detection * Support multiple formats * Create format converter * Add validation layer **Estimated Effort**: 3 sprints *** ### TD-2: Token Versioning Strategy 🔧 [Section titled “TD-2: Token Versioning Strategy 🔧”](#td-2-token-versioning-strategy-) **Severity**: High\ **Effort**: High\ **Priority**: P1 **Description**: No clear versioning strategy for VAD token evolution. **Impact**: * Breaking changes risk * Backward compatibility issues * Integration challenges **Resolution Plan**: * Design versioning scheme * Implement version negotiation * Create migration tools * Document upgrade paths **Estimated Effort**: 4 sprints *** ### TD-3: Monitoring Gaps 🔧 [Section titled “TD-3: Monitoring Gaps 🔧”](#td-3-monitoring-gaps-) **Severity**: Medium\ **Effort**: Medium\ **Priority**: P2 **Description**: Incomplete observability for critical system components. **Gaps**: * Cache performance metrics * Detailed event tracking * Business KPI dashboards * End-to-end tracing **Resolution Plan**: * Implement distributed tracing * Enhance metric collection * Create custom dashboards * Add synthetic monitoring **Estimated Effort**: 2 sprints *** ### TD-4: Manual Certificate Management 🔧 [Section titled “TD-4: Manual Certificate Management 🔧”](#td-4-manual-certificate-management-) **Severity**: Medium\ **Effort**: Low\ **Priority**: P2 **Description**: Certificate rotation and management requires manual intervention. **Resolution Plan**: * Implement ACME automation * Create rotation scripts * Add expiration monitoring * Document procedures **Estimated Effort**: 1 sprint *** ### TD-5: Test Coverage Gaps 🔧 [Section titled “TD-5: Test Coverage Gaps 🔧”](#td-5-test-coverage-gaps-) **Severity**: Medium\ **Effort**: High\ **Priority**: P3 **Description**: Insufficient test coverage for critical paths. **Current Coverage**: * Unit tests: 75% (target: 80%) * Integration: 65% (target: 70%) * Security: 90% (target: 100%) **Resolution Plan**: * Identify critical paths * Write missing tests * Automate test execution * Enforce coverage gates **Estimated Effort**: Ongoing, 20% of development time ## Risk Mitigation Roadmap [Section titled “Risk Mitigation Roadmap”](#risk-mitigation-roadmap) The detailed quarterly mitigation backlog (originally drafted for 2024) has been folded into the phased plan in the [Implementation Roadmap](../13-roadmap/ctn-roadmap.md). Open items currently tracked: * [ ] Resolve non-member access policy (R-1) * [ ] Address scalability bottlenecks before production scale-out (R-2) * [ ] Annual external security audit / pentest (R-3, R-17) * [ ] Token versioning strategy (TD-2) * [ ] Automate certificate management (TD-4) * [ ] Close test-coverage gaps (TD-5) * [ ] Excel format flexibility (TD-1) * [ ] Disaster-recovery rehearsal ## Risk Monitoring Dashboard [Section titled “Risk Monitoring Dashboard”](#risk-monitoring-dashboard) ### Key Risk Indicators (KRIs) [Section titled “Key Risk Indicators (KRIs)”](#key-risk-indicators-kris) | Indicator | Current | Target | Trend | Action | | -------------------- | ------- | ------ | ----- | --------------- | | Security incidents | 0 | 0 | → | Monitor | | API error rate | 1.2% | <1% | ↑ | Investigate | | Scalability headroom | 60% | >50% | ↓ | Plan capacity | | Integration success | 75% | >90% | ↑ | Support needed | | Compliance score | 85% | 100% | → | Review required | ### Risk Review Process [Section titled “Risk Review Process”](#risk-review-process) **Monthly Review**: * Update risk probability and impact * Review mitigation effectiveness * Identify new risks * Update technical debt **Quarterly Assessment**: * Executive risk briefing * Budget allocation review * Strategy adjustment * Stakeholder communication ## Contingency Plans [Section titled “Contingency Plans”](#contingency-plans) ### System Failure Contingency [Section titled “System Failure Contingency”](#system-failure-contingency) ```yaml Trigger: Complete system outage Response: 1. Activate incident response team 2. Switch to manual processes 3. Enable read-only mode 4. Communicate with stakeholders 5. Initiate recovery procedures Recovery Time: 1 hour maximum ``` ### Data Breach Contingency [Section titled “Data Breach Contingency”](#data-breach-contingency) ```yaml Trigger: Confirmed unauthorized access Response: 1. Isolate affected systems 2. Revoke compromised tokens 3. Notify data protection officer 4. Begin forensic investigation 5. Prepare regulatory notifications 6. Implement additional controls Notification Time: Within 72 hours (GDPR) ``` ### Vendor Service Outage [Section titled “Vendor Service Outage”](#vendor-service-outage) ```yaml Trigger: Azure service unavailable Response: 1. Activate multi-region failover 2. Use cached data where possible 3. Queue non-critical operations 4. Monitor service restoration 5. Process queued operations Degraded Operation: Maintain core functions ``` ## Technical Debt Prioritization [Section titled “Technical Debt Prioritization”](#technical-debt-prioritization) ### Debt Quadrants [Section titled “Debt Quadrants”](#debt-quadrants) ```plaintext Impact → Low High ┌─────────────────────────┐ │ Quick Wins │ Priority │ Urgent │ TD-4 │ TD-1 │ ↑ │ │ TD-2 │ │ ├─────────────┼───────────┤ │ │ Nice to │ Important │ │ Have │ TD-3 │ │ │ TD-5 │ └─────────────────────────┘ Easy ← Effort → Hard ``` ### Debt Reduction Strategy [Section titled “Debt Reduction Strategy”](#debt-reduction-strategy) 1. Allocate 20% of sprint capacity to debt reduction 2. Prioritize security and performance debt 3. Combine debt work with feature development 4. Track debt metrics and trends 5. Regular debt review sessions *** *This risk and technical debt register is reviewed monthly and updated as conditions change.* # DIL / CTN Glossary ## Colophon [Section titled “Colophon”](#colophon) | | | | ----------------------- | ----------------------------------- | | **Purpose** | Glossary | | **Author** | Ramon de Noronha & Léarco Kooman | | **Target Audience** | Program Team CTN | | **Status** | Approved | | **Date of approval** | | | **Name of approver** | Program Team CTN | | **Version** | 1.2 | | **Version explanation** | NA | | **Rights** | Shared externally with stakeholders | ## Glossary [Section titled “Glossary”](#glossary) This is a glossary with the terms, acronyms, and concepts used in the Connected Trade Network (CTN) architecture by the CTN Project Team. **Important Terminology Note:** This glossary distinguishes between *“Connected Trade Network (CTN)”* (the digital infrastructure initiative/system) and *“Container Transport”* (the physical business process being improved). **Audience and Scope.** This glossary is written for two overlapping audiences. First, the CTN Stakeholders and delivery-party onboarding: senior people from outside the logistics field who need to follow CTN steering-committee conversations — for them, the CTN Business Terms and CTN Technical Terms sections are the relevant parts. Second, anyone joining the programme who wants a working understanding of the surrounding container-logistics vocabulary — for them, the Logistics Terms section is the entry point. The Dutch Maritime Heritage section is included as light context; skip it if you are in a hurry. Where terms overlap with the BDI reference architecture (e.g. Association, Association Register, Custodian, Authority), the definitions in this glossary aim to be consistent with the BDI gitbook; where they differ, the BDI gitbook is leading. ## A Brief History on Dutch Maritime Heritage [Section titled “A Brief History on Dutch Maritime Heritage”](#a-brief-history-on-dutch-maritime-heritage) Pick up any modern container-logistics document — a Hong Kong terminal’s operating manual, a Long Beach trucker’s schedule, an Italian forwarder’s rate sheet — and you will find Dutch words on every page. They have been thoroughly anglicised, sometimes for so long that English-speaking professionals no longer recognize them as foreign at all, but they are recognizably Dutch. Bill of Lading. Skipper. Freight. Deck. Dock. Yacht. Sloop. Cruise. Cargo. Hoist. Keel. The vocabulary of moving things by water is, to a remarkable degree, the vocabulary of the Low Countries. This is not an accident. It is the linguistic residue of about four hundred years during which the Dutch dominated, and then helped to professionalize, long-distance maritime trade. The story is worth a short detour, both for fun and because it explains why a glossary like this one — written in English, in Rotterdam, in 2026, by a Dutch program team — reads the way it does. ### The Hanseatic League (12th–17th century) [Section titled “The Hanseatic League (12th–17th century)”](#the-hanseatic-league-12th17th-century) The first big wave of Dutch / Low-German maritime vocabulary reached English through the Hanseatic League, a confederation of trading cities that ran the North Sea and Baltic economy for roughly four centuries. London had a Hansa “factory” (the Steelyard) from the 13th century until 1598; English merchants, dockworkers, and shipwrights worked alongside Hansa skippers daily for generations. The Hansa’s working language was Middle Low German, very closely related to Middle Dutch. Words such as “skipper” (schipper), “freight” (vrecht / vracht), “keel” (kiel), “deck” (dek), and “hoy” (heude) enter English in this period and never really leave. ### The Dutch Golden Age & the VOC (1602–1799) [Section titled “The Dutch Golden Age & the VOC (1602–1799)”](#the-dutch-golden-age--the-voc-16021799) The second, much larger wave comes during the Dutch Golden Age, when Amsterdam was the financial capital of Europe and the VOC (Vereenigde Oostindische Compagnie, the Dutch East India Company) was the world’s largest commercial enterprise. The VOC operated for almost two centuries (1602–1799), at its peak ran some 4,700 ships, employed roughly a million Europeans on Asia voyages, and shipped some 2.5 million tonnes of Asian goods to Europe. Its sister organization, the WIC (West-Indische Compagnie, 1621–1791), did the same for the Atlantic and the Americas. The VOC effectively invented or perfected several practices that the rest of the world later copied: the publicly traded joint-stock company; standardised cargo manifests and bills of lading; insurance and average-loss conventions; a centralised admiralty bookkeeping system; standardised ship designs (the fluyt) optimised for cargo capacity rather than warship duty; and the use of professional captains and supercargoes under written instructions from a board. Each of these practices came with its own vocabulary — and when the English and others adopted the practices, they tended to import the words too. Anglo-Dutch relations in this period were a tense mixture of cooperation and rivalry — four Anglo-Dutch wars in 130 years, a Dutch invasion of London (the Medway raid of 1667), and finally a Dutch stadtholder (William III) on the English throne. Throughout it all, English shipwrights, sailors, and merchants worked with Dutch counterparts, copied their ships, and absorbed their terminology. By the end of the 17th century the vocabulary was effectively shared. ### A non-exhaustive list of Dutch words still in working English [Section titled “A non-exhaustive list of Dutch words still in working English”](#a-non-exhaustive-list-of-dutch-words-still-in-working-english) Roughly grouped — many of these are now so anglicised that their origin is invisible. NL form in parentheses. * **Ships & rigging** — yacht (jacht), sloop (sloep), schooner (schoener — via Dutch from English “to scoon”), smack (smak), hoy (heude), pink (pink), iceboat (ijsboot), brig (likely from Dutch brik), cruiser (kruiser), frigate (fregat — ultimately Italian, but the modern naval sense reached English via Dutch usage). * **Ship parts** — deck (dek), keel (kiel), bow (boeg), aft (achter), hull (hol — cognate), mast (mast), boom (boom, “tree, pole”), bowsprit (boegspriet), avast (houd vast, “hold fast”), caboose (kombuis, ship’s galley). * **Cargo & handling** — lading (laden, to load), freight (vrecht / vracht), cargo (via Spanish from Latin, but reinforced by Dutch usage), hoist (hijsen), pump (pomp), buoy (boei), boom (boom, also “barrier”), bulwark (bolwerk), dredge (dreggen). * **Ports & infrastructure** — dock (dok), quay (kaai — via French but reinforced by Dutch usage in English ports), dam (dam), dike (dijk), polder (polder), sluice (sluis), lock (sluis / lok), wharf (werf), bollard (bolder). * **People & roles** — skipper (schipper), boss (baas), freebooter (vrijbuiter), filibuster (vrijbuiter, via Spanish filibustero), boodle (boedel, “estate”), wagon (wagen). * **Trade, finance & bookkeeping** — stock (stok / stokken, share register), guilder (gulden), dollar (daalder, via Low German), groove (groef), tariff (via French and Italian, but the Dutch were the major standardisers in early modern Europe), bond (band, in the financial-instrument sense). * **Weather & navigation** — cruise (kruisen, “to cross”), maelstrom (maalstroom), iceberg (ijsberg), reef (rif), shoal (schol — cognate), landscape (landschap — not maritime, but emblematic). * **Less obvious** — blunderbuss (donderbus, “thunder gun”), brandy (brandewijn, “burnt wine”), coleslaw (koolsla, “cabbage salad”), cookie (koekje), cruller (kruller), gin (jenever — via genever), Santa Claus (Sinterklaas), Yankee (Jan Kees, allegedly — debated). ### Why so many? [Section titled “Why so many?”](#why-so-many) Three reinforcing reasons. First, the languages were already cousins: English and Dutch are both West Germanic, so a Dutch loanword tends to sit comfortably in English phonology and never feel foreign. Second, the Dutch dominated the practices — from joint-stock finance to standardised ship designs — so when those practices spread, the vocabulary spread with them. Third, the maritime world is small and shared: the same dockyards in London, Hamburg, and Amsterdam employed the same kinds of people doing the same kinds of work, and the working vocabulary cross-pollinated continuously for centuries. The pattern continues in the modern era. Containerisation itself was an American invention (Malcolm McLean, 1956), but Rotterdam built the first dedicated container terminal in continental Europe (1967), the first ECT terminal on Maasvlakte opened in 1984, and the Dutch have been disproportionately present in international logistics standardisation ever since — including, now, in BDI and CTN. The vocabulary keeps coming. The terms below are organised by theme rather than alphabetically. They are not exhaustive — for a broader reference, Crowley’s public glossary (crowley.com/resources/glossary) is a useful companion — but they cover the vocabulary you will hear in any CTN conversation about container flows, terminals, and hinterland transport. ### Container Basics [Section titled “Container Basics”](#container-basics) * **FCL / LCL** — Full Container Load vs. Less than Container Load. FCL means one shipper fills an entire container; LCL is consolidated cargo from multiple shippers sharing one container, handled at a CFS (Container Freight Station). * **TEU / FEU** — Twenty-foot Equivalent Unit and Forty-foot Equivalent Unit. The standard capacity measure for container ships, terminals, and trade statistics. A 40’ container counts as 2 TEU. * **Reefer** — Refrigerated container with its own cooling unit. Used for perishables, pharmaceuticals, and temperature-sensitive cargo. Requires power (genset, ship reefer plug, or terminal plug-in) throughout the journey. * **Tare / Payload / VGM** — Tare is the empty weight of the container; payload is the cargo weight; VGM (Verified Gross Mass) is the SOLAS-mandated total weight (tare + cargo) that must be declared by the shipper before loading on a sea vessel. * **ISO Container Sizes** — Standard sizes are 20’, 40’, and 45’ long, in standard (8’6”) and high-cube (9’6”) heights. Identified per ISO 6346 with a four-letter owner/equipment code plus a six-digit serial and check digit. ### Locations [Section titled “Locations”](#locations) * **CY (Container Yard)** — The stacking area at a terminal where containers wait between vessel and onward transport. CY-to-CY service means the carrier’s responsibility is from origin yard to destination yard. * **CFS (Container Freight Station)** — Facility where LCL cargo is consolidated into containers (stuffing) or de-consolidated out of them (stripping). Typically operated by the carrier or a logistics provider. * **ICD (Inland Container Depot)** — An inland “dry port” with customs facilities, container handling, and rail or barge connections to a seaport. Allows customs clearance away from congested seaport terminals. * **Bonded Warehouse** — A customs-supervised facility where imported goods can be stored without duties being paid, until released for free circulation, re-exported, or moved under transit. Known in NL as a douane-entrepot. * **Empty Depot** — Off-terminal storage for empty containers. Carriers reposition empties here for the next export booking; truckers pick up and drop off empties to avoid using scarce terminal slots. ### Movements [Section titled “Movements”](#movements) * **Gate-in / Gate-out** — The physical entry or exit of a container at a terminal gate. Triggers CODECO messages and starts/stops detention or demurrage clocks. The exact gate-in moment is the most common dispute point between carriers and customers. * **LOLO / RORO** — Lift On – Lift Off (containers lifted by crane) versus Roll On – Roll Off (wheeled cargo driven on and off). Most container traffic is LOLO; RORO is used for cars, trucks, and project cargo. * **Stripping / Stuffing** — Unloading cargo out of a container (stripping) or loading cargo into one (stuffing). Done at a CFS, warehouse, or directly at the consignee’s site. * **Transhipment** — Moving a container from one vessel to another at an intermediate port, without customs clearance. Critical for hub-and-spoke networks; transhipment-heavy ports (e.g. Algeciras, Tangier Med) compete on crane productivity. * **Pre-/On-/End-carriage** — Three legs of an intermodal journey: pre-carriage (origin to load port), on-carriage / main-carriage (the sea or rail trunk leg), and end-carriage (discharge port to destination). The carrier may handle one, two, or all three. * **Modal Shift** — Strategic shift of freight from one mode to another — usually road to rail or inland waterway — for emissions, cost, or congestion reasons. A core policy driver for CTN and Dutch hinterland strategy. ### Time & Cost [Section titled “Time & Cost”](#time--cost) * **Demurrage** — Charge levied by the carrier when a container stays on the terminal beyond its free time, waiting to be picked up. Paid by the consignee. The clock runs from gate-in (import) or container availability. * **Detention** — Charge levied by the carrier when a container stays off the terminal (at the customer’s site) beyond its free time, before being returned empty. Paid by the consignee. Often combined with demurrage as “D\&D.” * **Dwell Time** — The total time a container spends on a terminal between arrival and departure. A key terminal performance KPI; long dwell times signal congestion or customs delays. * **Free Time** — The window granted by the carrier or terminal before demurrage / detention starts. Typically 3–7 days; negotiable for large shippers. * **THC (Terminal Handling Charge)** — Fee charged by the carrier (and ultimately the terminal) for loading or discharging the container at origin and destination ports. Itemised on the freight invoice. * **BAF / CAF** — Bunker Adjustment Factor (fuel surcharge) and Currency Adjustment Factor (FX surcharge). Surcharges added to the base sea freight rate; volatility passes through to shippers. ### Documents [Section titled “Documents”](#documents) * **B/L (Bill of Lading)** — The classic sea transport contract, cargo receipt, and document of title issued by the carrier. Three originals are common; whoever holds an endorsed original can claim the cargo. See also the Business Terms section for variants. * **Sea Waybill** — Non-negotiable transport document. Faster than a B/L (no original needs to be couriered to the consignee) but cannot be used to transfer title in transit. Common in trusted shipper-consignee relationships. * **HBL / MBL** — House Bill of Lading (issued by a forwarder or NVOCC to the actual shipper) vs. Master Bill of Lading (issued by the carrier to the forwarder/NVOCC). Both cover the same physical cargo. * **CMR** — International road consignment note under the CMR Convention. The default freight document for cross-border trucking in Europe; required for customs and proof of delivery. * **MRN (Movement Reference Number)** — Unique customs reference for a declaration (import, export, or transit). Links physical cargo to its customs status; the terminal will not release a container without a valid MRN. * **T1 / T2** — EU customs transit documents. T1 covers non-Union goods moving under customs supervision (e.g. from a seaport to an inland clearance point); T2 covers Union goods crossing a non-EU territory. * **ENS (Entry Summary Declaration)** — Pre-arrival security declaration that must be filed with the customs authority of the first EU port of entry, before the vessel arrives. Filed by the carrier or a representative. ### EDI Messages (CTN-relevant) [Section titled “EDI Messages (CTN-relevant)”](#edi-messages-ctn-relevant) * **CODECO** — EDIFACT message for container gate-in and gate-out events at a terminal. Sent from terminal operator to carrier (and increasingly to platforms like CTN) to confirm physical container movement. * **COPRAR** — Container discharge / load order. Sent by the carrier to the terminal listing which containers to load on or discharge from a specific vessel call. * **COARRI** — Container discharge / load report. Sent by the terminal back to the carrier confirming which containers were actually loaded or discharged, with stowage positions. * **COPINO** — Container pre-notification. Sent by a trucker or barge operator to the terminal announcing an upcoming visit and the containers involved. * **BAPLIE** — Bayplan / Stowage Plan Occupied and Empty Locations. Standardised stowage plan for a container vessel, exchanged between carriers, terminals, and stowage planners. * **IFTMIN / IFTSTA** — IFTMIN is a transport instruction (booking, pickup, delivery order); IFTSTA is a transport status report. Together they cover most carrier-to-shipper status messaging. ### Parties [Section titled “Parties”](#parties) * **Shipper / Consignee / Notify Party** — Shipper is the contractual sender of the cargo; consignee is the party entitled to receive it; notify party is whoever the carrier alerts on arrival (often the customs broker). All three are named on the B/L. * **Forwarder** — Freight forwarder — arranges transport on behalf of the shipper or consignee, typically without operating ships, planes, or trucks themselves. The English word comes via Dutch trade vocabulary. * **Carrier** — The party that contractually performs the transport. For sea cargo, usually a shipping line operating its own or chartered vessels. See also “Carrier” in the Business Terms section. * **NVOCC (Non-Vessel-Operating Common Carrier)** — A party that issues its own bills of lading and acts as a carrier to the shipper, but does not own or operate vessels — it buys slots from actual carriers. Common in the US and Asia trades. * **Terminal Operator** — Operates the physical terminal (cranes, yard, gates). Major Rotterdam examples: ECT, APM Terminals Maasvlakte II, Rotterdam World Gateway (RWG). * **BCO (Beneficial Cargo Owner)** — The actual owner of the goods being shipped — not the forwarder, not the carrier, not the consignee on paper, but the commercial entity that owns the cargo and bears the trade risk. * **3PL / 4PL** — Third-Party Logistics provider executes logistics (warehousing, transport) for a shipper. A Fourth-Party Logistics provider orchestrates and manages multiple 3PLs and supply-chain partners on the shipper’s behalf. ### Inland Navigation & Hinterland (NL focus) [Section titled “Inland Navigation & Hinterland (NL focus)”](#inland-navigation--hinterland-nl-focus) * **Inland Terminal / Barge Terminal** — A terminal in the hinterland served by inland waterways (barges) and / or rail, connecting to one or more seaports. Examples: Contargo, ITG, Van Berkel, BCTN. The backbone of Dutch and German container hinterland. * **Hinterland** — The inland region served by a seaport — where its containers come from and go to. Rotterdam’s hinterland reaches deep into Germany, Switzerland, and Austria via the Rhine corridor. * **Call Size** — Number of containers a barge or feeder vessel exchanges (load + discharge) at a single seaport terminal call. Small call sizes are inefficient for terminals and a structural source of friction in Rotterdam’s inland network. * **Bundling** — Combining containers from multiple barge operators or shippers into larger, more efficient terminal calls. A central tactic for reducing waiting time at deep-sea terminals. * **Corridor** — A fixed inland transport route with regular service, often jointly marketed by multiple terminals and operators (e.g. the Rotterdam–Duisburg or Rotterdam–Basel corridors). ### Customs & Compliance [Section titled “Customs & Compliance”](#customs--compliance) * **AEO (Authorised Economic Operator)** — EU customs status granted to traders that meet specific reliability, solvency, and security standards. Brings benefits such as fewer physical inspections and simplified declarations. * **HS Code** — Harmonised System code — the international six-digit goods classification used for tariffs, statistics, and trade controls. EU adds two further digits (CN code) and member states sometimes more. * **Incoterms** — International Commercial Terms (ICC) defining where in the journey the seller’s responsibility ends and the buyer’s begins. Common ones: FOB, CIF, DAP, DDP, EXW. Determines who arranges and pays for transport, insurance, and clearance. * **Customs Broker** — Licensed party that files import, export, and transit declarations on behalf of the trader. In Dutch: douane-expediteur. ### Data & Platforms (CTN World) [Section titled “Data & Platforms (CTN World)”](#data--platforms-ctn-world) * **Port Community System (PCS)** — Neutral platform that exchanges information between all parties in a port community (carriers, terminals, customs, forwarders, truckers). The Dutch PCS is Portbase; Antwerp’s is NxtPort. * **Track & Trace / ETA / ETD** — Container or shipment visibility services. ETA = Estimated Time of Arrival; ETD = Estimated Time of Departure. CTN aims to replace fragmented tracking with federated, source-of-truth events. * **Data Sovereignty** — The principle that the party producing or holding data controls who may see and use it, under what conditions. Central to BDI and CTN architecture; contrasts with central-platform models. * **Federated Data Sharing** — Architectural pattern where data stays at the source (the custodian) and is accessed on request via authorised, audited queries — rather than being copied into a central database. The technical heart of CTN. ## CTN Business Terms [Section titled “CTN Business Terms”](#ctn-business-terms) ### A [Section titled “A”](#a) * **Association** — Legal entity that serves as trust anchor for both federated trust/authentication and local onboarding. In CTN, the Association will be structured as a *“Coöperatieve Vereniging”*: it has no legal members but rather participants who are onboarded to it and is governed by its founding members on the board. Participation is not limited to industry — regulators such as ILT and RWS can also join as participants. * **Association Register (ASR)** — Register of onboarded participants of a particular BDI Association instance. Performs compliance checks and issues VAD tokens that prove participant status and trustworthiness. * **Authority** — Regulatory bodies (customs, port authorities, safety inspectors) that need access to transport data in order to perform regulatory oversight, ensure compliance, and support security operations. ### B [Section titled “B”](#b) * **BDI (Basic Data Infrastructure)** — The foundational framework that defines principles, standards, and protocols for secure, sovereign data exchange between organizations. BDI has seven core principles: 1. **Connecting Physical and Digital Processes** — Digital data flows mirror physical operations 2. **Event-Driven Architecture** — Systems communicate through asynchronous events 3. **Zero Trust** — Every interaction requires explicit verification 4. **Dynamic Data** — Information reflects current operational state 5. **Data at the Source** — Data remains with its owner/custodian 6. **Local Decision-Making** — Distributed control, no central authority 7. **Coherent Security** — Comprehensive security framework across all layers CTN is a specific implementation of the BDI framework for container transport. * **BDI Connector** — A software component that implements BDI protocols, enabling organizations to participate in the data exchange network. * **Beneficial Cargo Owner (BCO)** — The ultimate owner of the goods being transported. Decides on the booking approach, pays transport charges, bears responsibility for compliance with applicable regulations, and receives the cargo at destination. May orchestrate container movements directly, or delegate orchestration to a freight forwarder. * **Bill of Lading (B/L or BOL)** — Transport contract and cargo receipt issued by the carrier; includes Shipper, Consignee, and Notify Party(ies). Variants include Master B/L (carrier-issued) and House B/L (forwarder/NVOCC-issued). Serves as proof of ownership and is required for cargo release at destination. * **BRIS (Business Registers Interconnection System)** — European system enabling the identification of companies and their branches across EU Member States by interconnecting national business registers. Used to verify EUID identifiers. ### C [Section titled “C”](#c) * **Cargo Director (PCS Secure Chain)** — is the ‘director of the cargo’ and can orchestrate the inland transport or appoint someone else as cargo director. The party designated as Release-to-Party automatically also assumes the role of Cargo Director. This role was introduced as part of the trusted supply chain concept. Multiple parties may act as Cargo Director throughout the chain, but at any given moment there is only one active Cargo Director. The Cargo Director can transfer this role to another party. This role can be fulfilled by importers, forwarders, and in some cases inland terminals and/or carriers. * **Carrier (Ocean Carrier / Line / Agent)** — Issues the Bill of Lading or electronic B/L, operates the ocean service, manages release conditions and destination charges. May operate through local agents at various ports. * **Carrier’s Agent** — Local representative of the carrier at destination ports, handling arrival notices, delivery orders, and coordination with terminal operators. * *Carrier (PCS Secure Chain)* — also known as Shipping Company. Responsible for releasing the container to a release-to-party at the terminal. This party has agreements with the terminal and pays the handling costs. The interests of the shipping company may be represented by a local agent that performs the administrative activities on its behalf. In such cases, this local party submits the release in Portbase on behalf of the carrier. * **Chamber of Commerce (CoC)** — National business registry that registers companies and maintains legal entity information. Examples include Kamer van Koophandel (KvK) in the Netherlands, Handelsregister (HRB) in Germany. Provides verification services for company legitimacy and business information. * **Charter (PCS Secure Chain)** — Within MCA Road, it is possible to designate an executing carrier in the pre-notification. This party does not perform any actions within MCA but is responsible for physically collecting the container. This role is therefore purely operational and can only be assumed by road carriers. * **CIM/SMGS (Rail Consignment Note)** — International rail consignment documentation used for cross-border rail transport. * **CMR (Road Consignment Note)** — Proof of contract for international road transport under the CMR convention. * **Commercial Invoice & Packing List** — Seller’s invoice and detailed cargo packaging information used for customs valuation and compliance verification. * **Consignee** — Party entitled to take delivery of cargo at destination; named on the Bill of Lading. Can be the BCO or another entity authorized to receive the goods. * **Container** — Standard shipping container (20ft, 40ft, or 45ft) used in intermodal transport. The primary unit of cargo being tracked and coordinated in container transport operations. * **Container Number (ISO 6346)** — Standardized container identifier consisting of owner code, serial number, and check digit, used for unique identification throughout the transport chain. * **Custodian** — An organization that holds operational data (terminals, carriers, logistics providers) and controls access to it based on authorization policies. * **Customs (Douane)** — Government authority responsible for import, export, transit, and security filings. Manages compliance with trade regulations and collects duties. * **Customs Broker** — Licensed agent who files import declarations on behalf of BCO/Consignee; links MRN (Movement Reference Number) to B/L and coordinates with terminal processes. * **Container Transport** — The business process of moving containers from origin to destination, involving multiple parties (carriers, terminals, truckers) and modes of transport (ship, rail, truck). This is the *problem domain* that CTN addresses. * **CTN (Connected Trade Network)** — The programme that defines and delivers shared, federated data infrastructure for container transport. CTN is a *programme* (not a single solution): it coordinates participants, agreements, and technical components that together enable secure data sharing across the chain. The 2026 scope is the Association Register, endpoint registration, and the use case *Visibility Achterland Containerlogistiek*; later phases are tracked in the [Roadmap](../13-roadmap/ctn-roadmap.md). ### D [Section titled “D”](#d) * **Data at the Source** — BDI principle that data primarily remains with its owner/custodian and is accessed on demand through controlled interfaces, rather than being copied into a central store. In practice this is a design preference rather than an absolute — specific datasets may be cached or replicated under explicit agreements. * **Delivery Order (DO)** — Authorization from carrier or carrier’s agent to terminal operator for container release and pickup. In the Netherlands, typically issued via Portbase Secure Chain system. * **Dossier** *(out of scope for 2026 — see [Roadmap](../13-roadmap/ctn-roadmap.md))* — A collection of information about a transport order, including involved parties, their roles, and relationships. Managed by the Orchestration Register in a later phase. * **Dual-Token Pattern** *(out of scope for 2026 — see [Roadmap](../13-roadmap/ctn-roadmap.md))* — Authorisation pattern that combines a VAD (participation) and a VOD (involvement) token. Planned for the phase that introduces the Orchestration Register. ### E [Section titled “E”](#e) * **Electronic B/L (eBL)** — Digital version of the Bill of Lading, providing the same legal status as paper B/L while enabling faster, more secure transfer of ownership and cargo release. * **ENS (Entry Summary Declaration)** — Pre-arrival security filing required under EU Import Control System 2 (ICS2) for risk assessment before cargo enters EU territory. * **Enrollment** — The process of registering parties as participants in a specific transport dossier, giving them access rights to relevant data. * **EORI (Economic Operators Registration and Identification)** — Number required by the EU for all parties involved in customs activities (import, export, transit). * **Establishment (Vestiging)** — A physical location of a legal entity (rechtspersoon) with its own address but no separate LEI/VAT. Identified by a location-level number — **vestigingsnummer** (NL, 12 digits), **SIRET** (FR), or equivalent — distinct from the entity-level identifiers (KVK, HRB, UID) it shares with its parent legal entity. See [Identifier Enrichment Architecture](../05-building-blocks/ctn-identifier-enrichment-architecture.md#5-two-level-identity-model-legal-entity-vs-establishment). * **EUID (European Unique Identifier)** — European standard for company identification introduced by the European Commission in 2017. Combines country code, trade register identifier, and company registration number to enable identification of companies and their branches across EU Member States through BRIS (Business Registers Interconnection System). Format example: NLHR12345678 (NL = Netherlands, HR = Handelsregister, followed by KvK number) or DEK1101R.HRB116737 (DE = Germany, K1101R = Hamburg trade register, HRB116737 = company number). * **Event-Driven** — Architectural pattern where systems communicate through asynchronous events rather than direct API calls. * **Evidence Token** — A signed token (in 2026: VAD) that provides evidence of a claim but is not itself an access token. Used by data providers when making authorisation decisions. ### F [Section titled “F”](#f) * **Forwarder** — Freight forwarding company that arranges transport and documentation for the BCO (cargo owner). May issue a house B/L, handles customs clearance and Port Community System (Portbase) actions. May act as a secondary orchestrator in the transport chain. * *Forwarder (PCS Secure Chain)* — is the inland operator who ‘forwarded’ the transport assignment to another inland operator. A single party may therefore have fulfilled both roles (inland operator and forwarder) at different points in the chain. ### G [Section titled “G”](#g) * **GLEIF (Global Legal Entity Identifier Foundation)** — International organization that manages the global LEI (Legal Entity Identifier) system. Provides LEI verification services and maintains the global LEI database, enabling unique identification of legal entities participating in financial and commercial transactions worldwide. ### H [Section titled “H”](#h) * **Hinterland Carrier (Truck / Barge / Rail)** — Transport operator performing port-to-inland or inland-to-port movements. Coordinates prenotifications and time slots with terminals for efficient gate operations. ### I [Section titled “I”](#i) * **Inland Operator (PCS Secure Chain)** — is authorized to execute the inland transport. This party is nominated by the Cargo Director to perform the transport. The Inland Operator role can be fulfilled by an Inland Terminal or an inland transport party. * **Inland Terminal / Depot** — Facility that receives containers via barge, rail, or road; handles gate movements, storage, and empty container returns. Serves as consolidation point for hinterland distribution. * **Inspection Agencies (e.g., NVWA)** — Regulatory bodies that can impose veterinary, phytosanitary, or other compliance inspections, potentially resulting in cargo holds or additional checks. * **Intermodal Transport** — Movement of containers using multiple modes of transport (ship, rail, truck) without handling the cargo itself. * **Involvement Type** — The role a party plays in a transport order (e.g., CARRIER, TERMINAL, TRUCKER, CUSTOMS\_AGENT). ### J [Section titled “J”](#j) * **JWT (JSON Web Token)** — Standard token format used for the VAD token in 2026 (VOD in later phases). Self-contained, cryptographically signed tokens containing claims. * **JWS (JSON Web Signature)** — The specific JWT variant used in CTN, where tokens are digitally signed but not encrypted. * **JWKS (JSON Web Key Set)** — Endpoint that publishes public keys used to verify JWT signatures. ### K [Section titled “K”](#k) * **Keycloak** — Open-source identity and access management solution (OAuth 2.0, OpenID Connect, SAML 2.0). **The chosen IdP for CTN** (see [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md)). One Keycloak realm hosts both human users (with outward federation to Entra ID, eHerkenning or local accounts per [ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md)) and M2M clients (client-credentials grant). Keycloak Organizations (26.6) provide per-participant user grouping and per-participant IdP linking. * **KvK (Kamer van Koophandel)** — Dutch Chamber of Commerce - the official business registry in the Netherlands. Registers all companies and legal entities, assigns unique 8-digit KvK numbers, and provides verification services for company legitimacy. CTN Association Register integrates with KvK API for automated verification of Dutch legal entities. ### L [Section titled “L”](#l) * **LEI (Legal Entity Identifier)** — 20-character alphanumeric code that uniquely identifies legal entities participating in financial and commercial transactions globally. Issued and managed by GLEIF. Used in CTN for high-assurance identity verification of association participants. Format: 20 uppercase alphanumeric characters (e.g., 529900T8BM49AURSDO55). * **LINDAS** — The Swiss federal Linked Data Service (`lindas.admin.ch`). Publishes the Zefix commercial-register data as open, queryable Linked Data (SPARQL), enabling Swiss UID lookup by company name **without** a Zefix account. ### M [Section titled “M”](#m) * **Member (Founding)** — Organizations that founded the CTN Association and form its board. Currently consists of Contargo, Inland Terminals Group, and Van Berkel Logistics. The board decides upon the terms for onboarding participants and determines which members and participants of other associations will be trusted. * *Note:* The CTN Association is structured as a *Coöperatieve Vereniging*. It has no general “members” in the legal sense; it is governed by its founding members on the board, while all organizations that join after founding are called **Participants** (see separate entry). * **MRN (Movement Reference Number)** — Unique reference number issued by customs upon validation of a declaration (import, transit, or ENS). Used to track and link cargo movements through customs systems. * **Multi-Modal** — Transport involving multiple containers that may be combined on the same vessel or vehicle for parts of their journey. ### N [Section titled “N”](#n) * **Non-Participant Party** — A transport party that is not an association participant but is involved in a transport through subcontracting relationships. * **Notify Party / First & Second Notify Party** — Contact person(s) to be informed upon cargo arrival and release status. Listed on booking/shipping instructions and Bill of Lading to ensure timely notification of cargo availability. ### O [Section titled “O”](#o) * **OAuth 2.0** — Industry-standard protocol used for service-to-service authentication in CTN (client credentials grant for M2M). * **OffeneRegister.de** — Open-data publication of the German commercial register (Handelsregister) by OKFN / OpenCorporates: 5M+ companies as bulk JSONL / SQLite under CC-BY. Used as a no-rate-limit source for German HRB/HRA + court code (and hence EUID) in batch enrichment. * **Open Location Code (Plus Code)** — Open geocoding standard that encodes a \~3×3 m location as a short alphanumeric code (e.g. `8FVC9G8F+6X`). Used as a universal per-location identifier when no register identifier is available; an open alternative to what3words. * **Orchestration** — The process of coordinating multiple parties involved in fulfilling a transport order. * **Orchestration Register** *(out of scope for 2026 — see [Roadmap](../13-roadmap/ctn-roadmap.md))* — Planned later-phase component that would manage transport dossiers, party enrolment and issue VOD tokens. Not part of the 2026 delivery; design notes parked under `docs/_out-of-scope-2026/`. * **Orchestrator** — The party responsible for coordinating a transport order (typically BCO or forwarder). ### P [Section titled “P”](#p) * **Participant** — Any organization that has been successfully onboarded to the CTN Association and has accepted its terms and conditions. Participants can consume and provide data to other participants and founding members within the association’s network. Since the CTN Association is structured as a *Coöperatieve Vereniging* governed by its founding members on the board, all organizations that join the association (after its founding) are called participants rather than members. Participants receive VAD tokens after passing compliance checks. * **Party** — Any organization involved in the transport of containers, regardless of their membership or participation status in an association. This includes carriers, terminals, truckers, forwarders, customs brokers, and other logistics service providers. * **PDP (Policy Decision Point)** — Component that makes authorisation decisions based on policies and provided evidence. In 2026 the PDP is the data provider itself; no central CTN PDP exists. * **PIP (Policy Information Point)** — Component that provides information needed for authorisation decisions (e.g. party participation status). * **Port Authority / Harbour Master** — Entity responsible for managing port operations, vessel traffic, nautical safety, and port infrastructure in major ports (e.g., Rotterdam, Amsterdam). * **Portbase (PCS)** — Dutch Port Community System used for Notification Import Documentation (MID) and Hinterland Container Notification (HCN). Facilitates digital information exchange between port stakeholders. * **Principal** — The party that takes responsibility (including potential billing) for another party’s data access. ### R [Section titled “R”](#r) * **Release-to-Party (PCS Secure Chain)** — is the party to whom the cargo is released. This is a contractual agreement between the shipping company (carrier) and the party that booked the transport. For example, the cargo may belong to Tesla, but Tesla may have directly registered DHL as the Release-to-Party in the Port Community System. In that case, DHL is the Release-to-Party. ### S [Section titled “S”](#s) * **Sea Waybill (SWB)** — Non-negotiable transport document where no original bills are required. Carrier releases cargo directly to the named consignee without requiring presentation of physical documents. * **SIREN** — French 9-digit identifier of a legal entity (entreprise), issued by INSEE. The entity-level French identifier (basis for the EUID `FR.SIREN.…`). * **SIRET** — French 14-digit identifier of a specific **establishment** (établissement) = SIREN + 5-digit NIC. The French establishment-level identifier; retrievable per location from the free `recherche-entreprises` API. * **Shipper** — Party contracting the carriage at origin; named on the Bill of Lading. May be the BCO (cargo owner) or a forwarder/NVOCC acting on behalf of the BCO. * **Subcontractor** — A party brought into a transport operation by another party, often not directly contracted by the orchestrator. ### T [Section titled “T”](#t) * **Telex Release / Express Release** — Release instruction allowing cargo pickup without presenting original Bill of Lading documents. Shipper surrenders originals at origin or carrier issues electronic release authorization. * **Tenant** — In multi-tenant architecture, an isolated instance serving a specific orchestrator or organization. * **Terminal Operator (Deepsea)** — Operator managing container discharge, storage, and gate-in/gate-out operations at seaport terminals. Coordinates with carriers, hauliers, and customs for efficient cargo flow. * **Tiered Access** — Different levels of data access based on participation status and business relationships. The exact tier model is policy-owned by each data provider in 2026. * **Transport Order** — The business transaction that initiates container movement from origin to destination. * **Trust Chain** — The hierarchy of trust relationships from association participation through orchestration to data access. ### U [Section titled “U”](#u) * **UBO (Ultimate Beneficial Owner)** — The natural person who ultimately owns or controls an organization. Verified during compliance checks. * **UID (Unternehmens-Identifikationsnummer)** — Swiss business identification number, format `CHE-###.###.###`. The Swiss legal-entity identifier, retrievable from Zefix / LINDAS. * **UN/LOCODE** — United Nations Code for Trade and Transport Locations - standardized location codes for ports, terminals, and inland facilities (e.g., NLRTM for Rotterdam). * **User** — The human person who interacts with CTN systems to consume data, register endpoints, manage organization profiles, and perform other operational activities. Users represent their organisation (founding member, participant, or party) within the CTN network. Users live in Keycloak; the linkage between a user and the participant(s) they act for is handled by Keycloak Organizations (see [ADR-00003](../09-decisions/00003-separate-users-and-participants.md)). ### W [Section titled “W”](#w) * **Wild-Card VOD** *(out of scope for 2026 — see [Roadmap](../13-roadmap/ctn-roadmap.md))* — Concept for a special VOD that would give authorities oversight access across multiple dossiers. Tied to the (deferred) Orchestration Register. ### Z [Section titled “Z”](#z) * **Zefix (Zentraler Firmenindex)** — The central index of the Swiss commercial register. Offers a web search and a REST API (`ZefixPublicREST`, free but requires a registered account via `zefix@bj.admin.ch`); the same data is openly queryable via [LINDAS](#l) SPARQL and opendata.swiss. * **Zero Trust** — Security principle where no implicit trust exists between parties; every interaction must be verified. ## CTN Technical Terms [Section titled “CTN Technical Terms”](#ctn-technical-terms) ### A [Section titled “A”](#a-1) * **ACME Protocol** — Automated Certificate Management Environment - protocol for automated certificate issuance and renewal. * **AES-256** — Advanced Encryption Standard with 256-bit keys, used for data encryption at rest. * **API Gateway** — Entry point for all API requests, handling authentication, rate limiting, and routing. * **Asymmetric Cryptography** — Encryption using public/private key pairs, used for token signing. ### C [Section titled “C”](#c-1) * **Cache** — High-speed data storage (Redis) used to improve performance for frequently accessed data. * **Circuit Breaker** — Pattern that prevents cascading failures by stopping calls to failing services. * **CloudEvents** — Specification for describing event data in a common format. * **Correlation ID** — Unique identifier that tracks a request across multiple services. ### E [Section titled “E”](#e-1) * **Event Grid** — Azure service for event routing and delivery at scale. * **Eventual Consistency** — Data consistency model where updates propagate asynchronously. ### H [Section titled “H”](#h-1) * **HATEOAS** — Hypermedia as the Engine of Application State - REST maturity level with discoverable APIs. * **HSM (Hardware Security Module)** — Dedicated hardware for secure key storage and cryptographic operations. ### I [Section titled “I”](#i-1) * **Idempotency** — Property where repeated operations produce the same result, important for retry logic. ### M [Section titled “M”](#m-1) * **Managed Identity** — Azure feature for service-to-service authentication without credentials. * **mTLS (Mutual TLS)** — Transport Layer Security where both client and server authenticate using certificates. ### O [Section titled “O”](#o-1) * **OpenAPI** — Specification format for describing REST APIs (formerly Swagger). ### P [Section titled “P”](#p-1) * **PostgreSQL** — Open-source relational database used for transactional data storage. * **Private Endpoint** — Azure networking feature that provides private IP addresses for services. ### R [Section titled “R”](#r-1) * **Rate Limiting** — Controlling the rate of requests to prevent abuse and ensure fair usage. * **RBAC (Role-Based Access Control)** — Access control based on user roles rather than individual permissions. * **Redis** — In-memory data structure store used for caching and real-time operations. * **REST (Representational State Transfer)** — Architectural style for designing networked applications. * **Retry Policy** — Strategy for retrying failed operations with backoff and limits. * **RS256** — RSA signature with SHA-256, algorithm used for JWT signing. * **RTO (Recovery Time Objective)** — Target time to restore service after a failure. * **RPO (Recovery Point Objective)** — Maximum acceptable data loss measured in time. ### S [Section titled “S”](#s-1) * **Saga Pattern** — Pattern for managing distributed transactions across multiple services. * **Service Bus** — Azure messaging service for reliable async communication. * **SLA (Service Level Agreement)** — Formal agreement defining expected service performance. ### T [Section titled “T”](#t-1) * **TLS (Transport Layer Security)** — Cryptographic protocol for secure communications. * **TTL (Time To Live)** — Duration for which data remains valid in cache or storage. ### V [Section titled “V”](#v) * **VAD (Verifiable Association Data)** — A JWT token issued by the Association Register that proves an organization’s participant status, compliance verification, and trustworthiness. Contains static claims about the organization. * **VAT (Value Added Tax)** — Consumption tax levied on goods and services in the EU. Each EU member state issues VAT numbers to registered businesses for tax identification and cross-border trade. VAT numbers are verified through VIES system. * **VIES (VAT Information Exchange System)** — European Commission system for validating VAT numbers across the EU. Provides a single access point to query and validate VAT numbers from individual EU member state registries, helping businesses verify trading partners and ensure correct VAT treatment. Each EU country has its own VAT number format, but VIES provides centralized validation. API: * **VNet (Virtual Network)** — Azure networking construct for isolating and connecting resources. * **VOD (Verifiable Orchestration Data)** *(out of scope for 2026 — see [Roadmap](../13-roadmap/ctn-roadmap.md))* — Planned later-phase JWT token, to be issued by the Orchestration Register, that would prove a party’s involvement in a specific transport order. Not part of the 2026 delivery. ### W [Section titled “W”](#w-1) * **WAF (Web Application Firewall)** — Security layer protecting web applications from common attacks. * **Webhook** — HTTP callback for event notifications. ### X [Section titled “X”](#x) * **X.509** — Standard for public key certificates used in TLS and signing. ## ASR Bounded Context [Section titled “ASR Bounded Context”](#asr-bounded-context) The Association Register (ASR) bounded context manages the registration, verification, and lifecycle of participants within the CTN dataspace. * **Participant** — Root entity: a registered CTN participant. Holds lifecycle status, membership tier, trust level, and the link to its identity representation. ## Acronym Reference [Section titled “Acronym Reference”](#acronym-reference) | Acronym | Full Form | | --------- | --------------------------------------------------------------------- | | ACME | Automated Certificate Management Environment | | ADR | Architecture Decision Record | | API | Application Programming Interface | | ASR | Association Register | | B/L | Bill of Lading | | BCO | Beneficial Cargo Owner | | BDI | Basic Data Infrastructure | | BRIS | Business Registers Interconnection System | | VAD | BDI Verifiable Assurance Document | | VOD | BDI Verifiable Orchestration Document *(later-phase, see roadmap)* | | CDN | Content Delivery Network | | CIM/SMGS | Rail Consignment Note (International) | | CMR | Road Consignment Note (Convention Marchandises Routières) | | CoC | Chamber of Commerce | | CTN | Connected Trade Network | | DAR | Dynamic Authorization Register | | DCSA | Digital Container Shipping Association | | DO | Delivery Order | | eBL | Electronic Bill of Lading | | ENS | Entry Summary Declaration | | EORI | Economic Operators Registration and Identification | | EUID | European Unique Identifier | | GDPR | General Data Protection Regulation | | GLEIF | Global Legal Entity Identifier Foundation | | HRB | Handelsregister (German Trade Register) | | HSM | Hardware Security Module | | ICS2 | Import Control System 2 | | ISO 6346 | Container Number Standard | | JWT | JSON Web Token | | JWK | JSON Web Key | | JWKS | JSON Web Key Set | | JWS | JSON Web Signature | | KPI | Key Performance Indicator | | KRI | Key Risk Indicator | | KvK | Kamer van Koophandel (Dutch Chamber of Commerce) | | LEI | Legal Entity Identifier | | MFA | Multi-Factor Authentication | | MRN | Movement Reference Number | | mTLS | Mutual Transport Layer Security | | NIS2 | Network and Information Security Directive 2 | | NSG | Network Security Group | | NVOCC | Non-Vessel Operating Common Carrier | | NVWA | Nederlandse Voedsel- en Warenautoriteit (Dutch Food Safety Authority) | | OAuth | Open Authorization | | OIDC | OpenID Connect | | OPA | Open Policy Agent *(later-phase, see roadmap)* | | P95/P99 | 95th/99th Percentile | | PaaS | Platform as a Service | | PCS | Port Community System | | PDP | Policy Decision Point | | PEP | Policy Enforcement Point | | PIP | Policy Information Point | | RBAC | Role-Based Access Control | | REST | Representational State Transfer | | RPO | Recovery Point Objective | | RTO | Recovery Time Objective | | SAML | Security Assertion Markup Language | | SDK | Software Development Kit | | SIEM | Security Information and Event Management | | SLA | Service Level Agreement | | SOC | Service Organization Control | | SPA | Single Page Application | | SQL | Structured Query Language | | SWB | Sea Waybill | | TLS | Transport Layer Security | | TMS | Transport Management System | | TTL | Time To Live | | UAT | User Acceptance Testing | | UBO | Ultimate Beneficial Owner | | UN/LOCODE | United Nations Code for Trade and Transport Locations | | URI | Uniform Resource Identifier | | URL | Uniform Resource Locator | | UUID | Universally Unique Identifier | | VAT | Value Added Tax | | VIES | VAT Information Exchange System | | WAF | Web Application Firewall | | XML | Extensible Markup Language | *** *This glossary is maintained by the architecture team and updated as new terms are introduced.* # CTN Implementation Roadmap *Phased approach for Connected Trade Network deployment* > **Status note (2026-05-29):** the roadmap below reflects the agreed phasing. Phase 1 (CTN Core — ASR + endpoint registration + Visibility) is in progress. Per-component delivery status is tracked outside this document. > > **This is the only document in the architecture set where Orchestration Register, VOD, dual-token validation, OPA, and 2R+ are described.** All other chapters scope themselves to the 2026 delivery only. The design notes for the deferred components are parked in `docs/_out-of-scope-2026/` (gitignored) and will be brought back into the repository when the corresponding phase starts. ## Overview [Section titled “Overview”](#overview) The Connected Trade Network follows the BDI Reference Architecture roadmap with a pragmatic, phased approach. CTN prioritizes simplicity and rapid deployment, starting with hosted OAuth services to avoid initial X.509 certificate complexity. ## Release Strategy [Section titled “Release Strategy”](#release-strategy) CTN is delivered as a phased rollout. The 2026 baseline is deliberately narrow — it covers the Association Register, endpoint registration and the first use case (“Visibility Achterland Containerlogistiek”) so participants can start exchanging operational data without waiting for heavier components. Later phases add the Orchestration Register, dossier-level authorisation and full PKI federation. * **Phase 1 (v1.0) — ASR baseline (2026)**: Association Register + endpoint registration + VAD-only local authorisation at the data provider. * **Phase 2 (v1.1) — Orchestration Register + dual-token (opt-in)**: Orchestration Register, VOD, OPA-based PDP, dual-token validation. * **Phase 3 (v2.x)**: enhanced trust and compliance, broader provider integration. * **Future (v3.0)**: full PKI, federation across associations. *** ## Phase 1: CTN v1.0 — ASR baseline (2026) [Section titled “Phase 1: CTN v1.0 — ASR baseline (2026)”](#phase-1-ctn-v10--asr-baseline-2026) *Target: Operational MVP on the VAD-only baseline.* ### Scope [Section titled “Scope”](#scope) Establish the minimum infrastructure that lets participants exchange public-data flows (CODECO gate-in/gate-out events, container numbers, ETAs, master B/L metadata) under verifiable participation. **No Orchestration Register and no VOD in this phase.** ### Components [Section titled “Components”](#components) #### Association Register (ASR) [Section titled “Association Register (ASR)”](#association-register-asr) * **Authentication**: Self-hosted Keycloak with outward federation (Entra ID / eHerkenning / local accounts) — see [ADR-00002](../09-decisions/00002-define-a-preferred-stack.md), [ADR-00008](../09-decisions/00008-external-idp-federation-strategy.md) * **Onboarding**: Manual process with form-based registration * **Verification**: Basic KvK/LEI lookup (manual review) * **Token Generation**: VAD tokens via OAuth provider * **Data Store**: PostgreSQL with participant records #### Local authorisation at the data provider [Section titled “Local authorisation at the data provider”](#local-authorisation-at-the-data-provider) * **Authorisation**: each data provider validates VAD claims and applies its own policy; no central PDP in 2026. * **Token Validation**: VAD signature + claims. * **Data Access**: provider-defined data sets per requester tier. #### BDI Connector [Section titled “BDI Connector”](#bdi-connector) * **Function**: Basic API gateway/proxy * **Authentication**: OAuth M2M client credentials * **Communication**: TLS 1.3 only * **Integration**: REST APIs with OpenAPI specs ### Simplifications for v1.0 [Section titled “Simplifications for v1.0”](#simplifications-for-v10) ✅ **What’s Included:** * Hosted OAuth provider (no certificate management) * VAD-only authorisation * Whitelist-based policy * Manual onboarding process * Email-based notifications * Single association support ❌ **What’s Deferred to Phase 2 (see `docs/_out-of-scope-2026/`):** * Orchestration Register * VOD token generation * Dual-token validation (VAD + VOD) * Per-dossier party enrolment * Delegation chains ❌ **What’s Deferred to later phases:** * X.509 certificate generation, ACME protocol * Automated compliance checks * Event broker integration (Event Grid / EventBridge / Pulsar) * Multi-association support * OPA policy engine * DNS validation ### Success Criteria [Section titled “Success Criteria”](#success-criteria) * 10+ organisations onboarded * VAD-only flow live for at least one public-data use case (e.g. CODECO subscription) * <1s authorisation decisions * 99% uptime for core services *** ## Phase 2: CTN v1.1 — Add Orchestration Register + Dual-Token (opt-in) [Section titled “Phase 2: CTN v1.1 — Add Orchestration Register + Dual-Token (opt-in)”](#phase-2-ctn-v11--add-orchestration-register--dual-token-opt-in) *Target: Add the Orchestration Register and dual-token validation for sensitive transports — opt-in per data provider and per dossier. Brings back the design notes currently parked in `docs/_out-of-scope-2026/`.* ### Enhancements [Section titled “Enhancements”](#enhancements) #### Orchestration Register (new component, opt-in) [Section titled “Orchestration Register (new component, opt-in)”](#orchestration-register-new-component-opt-in) * Dossier management for participating orchestrators * Excel-driven party enrolment (50+ parties per dossier) * VOD issuance (no delegation chains yet) * Subcontractor delegation pattern (limited) * Bring back `_out-of-scope-2026/ctn-portal-query-federation.md` and the fuzzy-search component for party discovery within dossiers. #### Event-Driven Architecture [Section titled “Event-Driven Architecture”](#event-driven-architecture) * Event-broker integration (Azure Event Grid as reference; AWS EventBridge / Apache Pulsar / Kafka as alternatives) * Webhook notifications * Basic event replay capability #### Enhanced Authorisation (Local Policy Engine + OPA) [Section titled “Enhanced Authorisation (Local Policy Engine + OPA)”](#enhanced-authorisation-local-policy-engine--opa) * Re-introduce the Local Policy Engine and OPA-based six-flow model from `_out-of-scope-2026/ctn-authorization-model.md` and `_out-of-scope-2026/ctn-open-policy-agent-implementation.md`. * Dual-token (VAD + VOD) validation path, active for opt-in providers / dossiers. * Re-introduce `_out-of-scope-2026/ctn-token-architecture.md` (rewritten with VOD details) and `_out-of-scope-2026/ctn-three-tier-authentication.md` for the delegation tiers. * Role-based access patterns; basic data-minimisation rules. #### Operational Improvements [Section titled “Operational Improvements”](#operational-improvements) * Automated monitoring and alerting * Self-service participant portal * API rate limiting and throttling ### New Capabilities [Section titled “New Capabilities”](#new-capabilities) * DNS validation as alternative authentication * Multiple orchestrator support * Basic semantic mapping * Standardized data sets *** ## Phase 3: CTN v2.0 - Trust & Compliance (Q1-Q2 2027) [Section titled “Phase 3: CTN v2.0 - Trust & Compliance (Q1-Q2 2027)”](#phase-3-ctn-v20---trust--compliance-q1-q2-2027) *Target: Enterprise-grade with regulatory compliance* ### Major Additions [Section titled “Major Additions”](#major-additions) #### Component Registry [Section titled “Component Registry”](#component-registry) * Centralized service discovery * Trust list synchronization * Provider validation #### Enhanced Security [Section titled “Enhanced Security”](#enhanced-security) * Business assurance tokens with claims * Signed orchestration evidence * JWKS endpoints for key distribution #### Regulatory Integration [Section titled “Regulatory Integration”](#regulatory-integration) * Authority access patterns * Compliance data sets * Audit trail requirements * GDPR automation #### Advanced Features [Section titled “Advanced Features”](#advanced-features) * Caching strategies * Performance optimization * Multi-region deployment * Disaster recovery ### Trust Framework [Section titled “Trust Framework”](#trust-framework) * Association trust lists * Component verification * Cross-association interoperability *** ## Phase 4: CTN v2.1 - Federation (Q3-Q4 2027) [Section titled “Phase 4: CTN v2.1 - Federation (Q3-Q4 2027)”](#phase-4-ctn-v21---federation-q3-q4-2027) *Target: Federated operations across associations* ### Capabilities [Section titled “Capabilities”](#capabilities) * Multi-association membership * Federated trust model * Enhanced token validation * Cross-border operations ### Security Enhancements [Section titled “Security Enhancements”](#security-enhancements) * Protection against social engineering * Advanced threat detection * Automated compliance verification *** ## Future: CTN v3.0 - Full PKI (2027+) [Section titled “Future: CTN v3.0 - Full PKI (2027+)”](#future-ctn-v30---full-pki-2027) *Target: Complete self-sovereignty with PKI* ### Full PKI Implementation [Section titled “Full PKI Implementation”](#full-pki-implementation) * BDI X.509 certificate authority * ACME protocol support * Automated key rotation * Self-signed certificate ecosystem ### Advanced Capabilities [Section titled “Advanced Capabilities”](#advanced-capabilities) * Event broker integration (Pulsar/Kafka) * Blockchain evidence trails * AI-powered authorization * Predictive analytics *** ## Implementation Principles [Section titled “Implementation Principles”](#implementation-principles) ### For v1.0 Release [Section titled “For v1.0 Release”](#for-v10-release) 1. **Simplicity First**: Use hosted services where possible 2. **Manual Processes OK**: Automation can come later 3. **Proven Technology**: OAuth over X.509 initially 4. **Minimal Viable Security**: Whitelists before complex policies 5. **Quick Wins**: Focus on immediate business value ### Technical Decisions by Version [Section titled “Technical Decisions by Version”](#technical-decisions-by-version) #### v1.0 - Foundational Decisions [Section titled “v1.0 - Foundational Decisions”](#v10---foundational-decisions) | Component | Complex Option | v1.0 Simple Choice | | -------------- | ---------------------------------- | --------------------------------------- | | Authentication | X.509 certificates per participant | Self-hosted Keycloak (OAuth 2.0 / OIDC) | | Authorization | OPA with Rego | Simple whitelists | | Events | Event Grid + Webhooks | Email notifications | | Onboarding | Automated verification | Manual review | | Certificates | ACME protocol | OAuth tokens only | | Delegation | Chain validation | Single level only | | Trust Model | Business assurance tokens | Identity-based whitelisting | ### Migration Path [Section titled “Migration Path”](#migration-path) Each phase builds on the previous: ```plaintext v1.0 (Foundation) → v1.1 (Multi-Association) → v2.0 (Trust Lists) → v2.1 (Federation) → v2.2 (Event Brokers) → v3.0 (Full PKI) ``` No breaking changes between versions - only additive improvements. *** ## Risk Mitigation for Simplified v1.0 [Section titled “Risk Mitigation for Simplified v1.0”](#risk-mitigation-for-simplified-v10) | Risk | Mitigation | | ---------------------------- | ----------------------------------- | | OAuth provider dependency | Choose enterprise provider with SLA | | Manual processes don’t scale | Plan automation for v1.1 | | Limited security model | Sufficient for pilot phase | | No delegation support | Document workarounds | | Email notification delays | Acceptable for initial scale | *** ## Success Metrics by Phase [Section titled “Success Metrics by Phase”](#success-metrics-by-phase) ### v1.0 Metrics (Foundation) [Section titled “v1.0 Metrics (Foundation)”](#v10-metrics-foundation) * Time to onboard: <24 hours * Authorization latency: <1 second * System availability: 99% * Participant satisfaction: >80% ### v1.1 Metrics (Scaling) [Section titled “v1.1 Metrics (Scaling)”](#v11-metrics-scaling) * Parties per dossier: 50+ * Events processed: 1000/minute * API calls: 100/second * Automated processes: 50% ### v2.0 Metrics (Trust) [Section titled “v2.0 Metrics (Trust)”](#v20-metrics-trust) * Compliance automation: 90% * Cross-association ops: Enabled * Security incidents: <1/month * Audit compliance: 100% ### v2.2 Metrics (Event Brokers) [Section titled “v2.2 Metrics (Event Brokers)”](#v22-metrics-event-brokers) * Event throughput: 10,000/second * Message delivery: 99.99% * Event replay time: <5 minutes * Broker availability: 99.95% *** # CTN - Architecture Documentation ## Overview [Section titled “Overview”](#overview) The **Connected Trade Network (CTN)** is a digital infrastructure initiative that implements the BDI (Basic Data Infrastructure) reference architecture framework for secure data sharing in **container transport operations**. This documentation follows the arc42 template to provide comprehensive architectural guidance. > **Important Terminology**: Throughout this documentation, we distinguish between **“CTN”** (the digital infrastructure initiative/system) and **“container transport”** (the physical business process being improved). CTN is one of several initiatives using the BDI framework as its foundation. While BDI defines the core principles and patterns, CTN applies these specifically to the container logistics domain with the Hinterland Terminals and Portbase. ### Scope for 2026 [Section titled “Scope for 2026”](#scope-for-2026) The 2026 delivery is deliberately narrow: 1. **Association Register (ASR)** — production-ready: participant onboarding, eHerkenning-based authentication of authorised representatives, periodic re-verification against KvK / KBO / VIES / GLEIF, M2M-credential issuance. 2. **Endpoint registration** — data-providing parties register their endpoints (OpenAPI YAML upload, validation, discovery); data-consuming parties register their applications and obtain OAuth client credentials. 3. **Use case “Visibility Achterland Containerlogistiek”** — first practical application: combining Portbase data with inland-terminal operational data into a dashboard for import containers moving over barge into the hinterland. Authorisation remains **local at the data provider**: each data owner decides which registered party may receive which data on the basis of the conditions they configure themselves. VAD-only flow. Later-phase concerns (Orchestration Register, dual-token VAD+VOD validation, OPA-based policy engine, full 2R+ federation) are explicitly out of scope for 2026 and are tracked in the [Roadmap](13-roadmap/ctn-roadmap.md). Design notes for those components are parked in `docs/_out-of-scope-2026/` (gitignored) and will be brought back when the corresponding phase starts. ## Document Structure [Section titled “Document Structure”](#document-structure) ### 📚 Core Architecture Documentation [Section titled “📚 Core Architecture Documentation”](#-core-architecture-documentation) | Section | Description | | ----------------------------------------------- | ---------------------------------------------------- | | [01. Introduction and Goals](01-introduction/) | Business context, requirements, stakeholders | | [02. Architecture Constraints](02-constraints/) | Technical, organizational, and political constraints | | [03. System Scope and Context](03-context/) | System boundaries and external interfaces | | [04. Solution Strategy](04-solution-strategy/) | Core architectural decisions and approaches | | [05. Building Block View](05-building-blocks/) | Component structure and detailed designs | | [06. Runtime View](06-runtime/) | Dynamic behavior and interaction scenarios | | [07. Deployment View](07-deployment/) | Infrastructure and deployment architecture | | [08. Cross-Cutting Concepts](08-crosscutting/) | Technical concepts applied across components | | [09. Architecture Decisions](09-decisions/) | Key decisions and their rationales (ADRs) | | [10. Quality Requirements](10-quality/) | Quality goals, scenarios, and metrics | | [11. Risks and Technical Debt](11-risks/) | Risk register and debt management | | [12. Glossary](12-glossary/) | Terms, acronyms, and definitions | | [13. Implementation Roadmap](13-roadmap/) | Phased deployment approach | ### Quick Navigation [Section titled “Quick Navigation”](#quick-navigation) **For Business Stakeholders:** * Start with [Introduction and Goals](01-introduction/) * Review [CTN Initiative Overview](01-introduction/ctn-initiative-overview.md) * Understand [Solution Strategy](04-solution-strategy/) **For Architects:** * Review [Architecture Constraints](02-constraints/) * Study [Building Blocks](05-building-blocks/) * Check [Architecture Decisions](09-decisions/) **For Developers:** * Start with [Building Blocks](05-building-blocks/) * Review [Runtime Scenarios](06-runtime/) * Check [Cross-Cutting Concepts](08-crosscutting/) **For Operations:** * Focus on [Deployment View](07-deployment/) * Review [Quality Requirements](10-quality/) * Monitor [Risks and Technical Debt](11-risks/) * Track [NIS2 Compliance](08-crosscutting/ctn-nis2-compliance.md) ## Key Architectural Principles [Section titled “Key Architectural Principles”](#key-architectural-principles) ### BDI Framework Foundation [Section titled “BDI Framework Foundation”](#bdi-framework-foundation) CTN implements the seven core BDI principles: 1. **Connecting Physical and Digital** - Bridging real-world logistics with digital data 2. **Event-Driven** - Asynchronous communication via an event broker (e.g. Azure Event Grid, AWS EventBridge, Apache Pulsar) 3. **Zero Trust** - No implicit trust between participants 4. **Dynamic Data** - Real-time, time-sensitive information 5. **Data at the Source** - Custodians maintain data control 6. **Local Decision-Making** - Distributed authorisation 7. **Coherent Security** - Integrated security framework ## Governance [Section titled “Governance”](#governance) **Technical Advisory Board:** Reviews and approves architectural decisions\ **Update Frequency:** Monthly review, quarterly major updates\ **Change Process:** Pull requests with architecture board review ## *** # Export Documentation to Word > How to convert the markdown documentation to Microsoft Word (DOCX) using the CTN/DIL Word template This guide explains how to convert the markdown documentation to Microsoft Word (DOCX) format, styled with the **CTN/DIL Word template** in `word-template/`. > This is an **authoring/offline utility**, not part of the website build. The live site is built with Astro (`npm run build`); Word export is a separate, on-demand step you run locally when you need `.docx` deliverables. ## Overview [Section titled “Overview”](#overview) The export walks every markdown file under `docs/` and produces a professionally formatted Word document per file, including: * CTN/DIL logo and branding from the template * A colophon page (document metadata) taken from the template * Heading hierarchy (H1–H4) mapped to the template’s styles * Tables with bold headers * Bullet/numbered lists * Code elements in monospace (Courier New) * Mermaid diagrams rendered to PNG (requires mermaid-cli — see below) > ℹ️ **The `docs/arc42` content is synced from the CNCL-BDI source repo.** Run `npm run sync-arc42` **before** exporting if you want the Word output to reflect the latest arc42 docs. See [How To: Update Arc42 Documentation](./update-arc42-docs.md). ## Prerequisites [Section titled “Prerequisites”](#prerequisites) 1. **Node.js ≥ 22.12.0** (the repo standard; required anyway for the Astro build) ```bash node --version ``` 2. **Pandoc 3.x** — the core markdown → DOCX converter ```bash pandoc --version # macOS: brew install pandoc ``` 3. **Python 3 with `python-docx`** — used to merge the template colophon/header ```bash pip3 install --break-system-packages python-docx ``` 4. **NPM dependencies** (installs `gray-matter`, used to read frontmatter) ```bash npm install ``` 5. **(Optional) Mermaid CLI** — only needed to render Mermaid diagrams as images. The export downloads it on demand via `npx`, but that pulls a headless Chromium (large/slow on first run). To avoid the per-run download, add it once: ```bash npm install --save-dev @mermaid-js/mermaid-cli ``` Without it, the export still completes — diagrams are left as fenced code blocks. ## How to Export [Section titled “How to Export”](#how-to-export) ```bash # (optional) refresh arc42 from the source repo first npm run sync-arc42 # convert all markdown under docs/ to Word npm run export-word # (identical alias: npm run md-to-docx) ``` * **Output location:** `Export/` (gitignored — not committed) * **Output naming:** `YYYYMMDD-DIL-CTN-.docx` * Date comes from the `lastUpdate` frontmatter field (falls back to today’s date) * Example: `20260531-DIL-CTN-ctn-connector.docx` The export mirrors the `docs/` directory structure into `Export/`. With the current content set (`docs/arc42` + `docs/how-to` + `docs/index.md`) this is \~50 files and takes a couple of minutes (longer on the first run if Mermaid CLI is downloaded). > **Note:** the export **appends** to `Export/`; it does not wipe it first. Old runs (including content from before any folder was removed) can linger there. Clean it with `rm -rf Export/` if you want a fresh set. ## The Word template [Section titled “The Word template”](#the-word-template) All styling comes from a single reference document: ```plaintext word-template/20251111_DIL_CTN_file_name.docx ``` The export script (`scripts/md-to-docx.cjs`) passes this file to Pandoc via `--reference-doc` and then merges in the colophon/header with `scripts/merge-docx-template-v3.py`. If the template is missing the export aborts with a clear error. **To change the styling** (colors, fonts, header, logo, colophon): 1. Edit `word-template/20251111_DIL_CTN_file_name.docx` in Word. 2. Save it (keep the same filename, or update `TEMPLATE_FILE` in `scripts/md-to-docx.cjs`). 3. Re-run `npm run export-word` — all documents pick up the new styling. ## How it works [Section titled “How it works”](#how-it-works) Two-step conversion per file: 1. **Pandoc** converts the markdown to DOCX using the template as `--reference-doc`, preserving heading styles, tables and lists. Mermaid blocks are pre-rendered to PNG (via `mmdc`) and embedded as images. 2. **Template merge** (`merge-docx-template-v3.py`) prepends the colophon page, copies the header/logo, replaces the `<>` placeholder, bolds table headers, and applies Courier New to code-like spans. Temporary artifacts (`*.temp.md`, `*.temp.docx`, `.temp-mermaid/`) are created during conversion and cleaned up automatically. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### `Missing script: "export-word"` [Section titled “Missing script: "export-word"”](#missing-script-export-word) You’re on an old checkout. The scripts `export-word`/`md-to-docx` are defined in `package.json` and run `scripts/md-to-docx.cjs`. Pull the latest and `npm install`. ### `require is not defined` when running the script directly [Section titled “require is not defined when running the script directly”](#require-is-not-defined-when-running-the-script-directly) Run it through npm (`npm run export-word`) or call the `.cjs` file (`node scripts/md-to-docx.cjs`). The script is CommonJS; the package is `"type": "module"`, so a plain `.js` extension would be treated as ESM and fail. ### Pandoc not found [Section titled “Pandoc not found”](#pandoc-not-found) Install Pandoc: `brew install pandoc` (macOS), then `pandoc --version`. ### `ModuleNotFoundError: No module named 'docx'` [Section titled “ModuleNotFoundError: No module named 'docx'”](#modulenotfounderror-no-module-named-docx) `pip3 install --break-system-packages python-docx` ### Mermaid diagram conversion fails (`could not determine executable to run`) [Section titled “Mermaid diagram conversion fails (could not determine executable to run)”](#mermaid-diagram-conversion-fails-could-not-determine-executable-to-run) mermaid-cli isn’t available. Either install it (`npm install --save-dev @mermaid-js/mermaid-cli`) or ignore it — the export still completes and leaves those diagrams as code blocks (non-fatal). ### Word shows “unknown content” / offers to repair on open [Section titled “Word shows “unknown content” / offers to repair on open”](#word-shows-unknown-content--offers-to-repair-on-open) Click **Yes**; Word auto-repairs without data loss. This is a minor DOCX XML quirk. ## Related files [Section titled “Related files”](#related-files) * `scripts/md-to-docx.cjs` — conversion orchestrator (template path, naming, Pandoc + merge) * `scripts/merge-docx-template-v3.py` — colophon/header merge and formatting * `word-template/20251111_DIL_CTN_file_name.docx` — the reference template ## Summary [Section titled “Summary”](#summary) ```bash npm run sync-arc42 # (optional) refresh arc42 from CNCL-BDI npm run export-word # convert docs/ -> Export/*.docx using the template open Export/ # view generated documents rm -rf Export/ # clean output ``` # Architecture Diagram **Last Updated:** October 20, 2025 **Status:** Living Document *** ## System Architecture Overview [Section titled “System Architecture Overview”](#system-architecture-overview) This interactive architecture diagram provides a comprehensive view of the CTN Association Register system components, their relationships, and data flows. The diagram is maintained in IcePanel and represents the current state of the system architecture. *** ## Interactive Diagram [Section titled “Interactive Diagram”](#interactive-diagram) *** ## How to Use [Section titled “How to Use”](#how-to-use) * **Zoom**: Use the zoom controls or mouse wheel to zoom in/out * **Pan**: Click and drag to navigate the diagram * **Interact**: Click on components to see details and relationships * **Fullscreen**: Use the fullscreen button for a larger view *** ## Related Documentation [Section titled “Related Documentation”](#related-documentation) * [arc42 Architecture Documentation](/arc42/ctn-arc42-index/) * [Building Blocks](/arc42/05-building-blocks/ctn-building-blocks/) * [Deployment View](/arc42/07-deployment/ctn-deployment/) * [System Context](/arc42/03-context/ctn-context/) # Query the arc42 Documentation with AI Tools > How team members can connect the CTN arc42 documentation (GitHub) to Claude, ChatGPT and Google Gemini to ask questions about the architecture This guide explains how anyone in the team can ask questions about the CTN arc42 architecture documentation using their own AI assistant — including **Claude**, **ChatGPT**, **Google Gemini**, **GitHub Copilot**, **Perplexity**, **Cursor** and **JetBrains Junie** — by connecting it to the GitHub repository where the documentation lives. The tools fall into three groups: assistants that connect to GitHub directly (Claude, ChatGPT, Copilot, Gemini), AI code editors that read the project after you open it (Cursor, JetBrains Junie), and tools that work from a public URL (Perplexity). Pick whichever you already use. ## Overview [Section titled “Overview”](#overview) The arc42 documentation is the single source of truth for the CTN architecture. The source markdown files live in the public GitHub repository: * **Repository:** [`Basic-Data-Infrastructure/CNCL-BDI`](https://github.com/Basic-Data-Infrastructure/CNCL-BDI) * **Folder:** [`docs/arc42`](https://github.com/Basic-Data-Infrastructure/CNCL-BDI/tree/main/docs/arc42) Because the repository is public, every team member can point their AI tool at it and ask questions such as *“How does association onboarding work?”* or *“Which ADR describes the token claim composition?”* — and get answers grounded in the actual documentation, with references back to the source files. > The AI provider interfaces below change regularly and most of these connectors are still in beta. If a menu item looks slightly different, search the tool’s help for “GitHub” — the overall flow is the same: connect GitHub, select this repository, then ask your question. ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * An account for the AI tool you want to use (Claude, ChatGPT or Gemini). * Access to the repository. It is public, so no special permissions are needed for reading. If it is ever made private, you will need to authorize the connector with a GitHub account that has access (and possibly GitHub-organization admin approval). *** ## Option 1 — Claude (claude.ai) [Section titled “Option 1 — Claude (claude.ai)”](#option-1--claude-claudeai) Claude has a built-in GitHub integration that lets you attach repository files directly to a conversation. 1. Open a new chat at [claude.ai](https://claude.ai). 2. Click the **+** button in the lower-left corner of the message box. 3. Choose **Add from GitHub**. 4. If this is your first time, you will be redirected to GitHub to **authenticate / authorize** Claude. Approve access. 5. In the file browser, navigate to `Basic-Data-Infrastructure/CNCL-BDI` and select the **`docs/arc42`** folder (or specific files you care about). 6. Send your question. Claude reads the selected content and answers based on it. **Tips** * You can add multiple folders or repositories as long as they fit in Claude’s context window. For broad questions, selecting the whole `docs/arc42` folder works well. * When the documentation is updated, click **Sync now** to pull the latest changes into your chat. *** ## Option 2 — ChatGPT (chatgpt.com) [Section titled “Option 2 — ChatGPT (chatgpt.com)”](#option-2--chatgpt-chatgptcom) ChatGPT connects to GitHub as an “app” / connector, after which it can read code, README and documentation files from the repositories you grant access to. 1. In ChatGPT, open **Settings → Apps** (also called Connectors). 2. Find **GitHub** in the app directory and click to connect. 3. You are sent to GitHub to **install and authorize** the ChatGPT app. Grant it access to the `CNCL-BDI` repository (or all repositories you are allowed to share). 4. Wait a few minutes — there is typically a **\~5-minute delay** before a repository shows up as available in ChatGPT. 5. Start a chat, enable the **GitHub** connector (or use **Deep research → GitHub** for thorough, cited answers), and ask your question about the arc42 docs. **Tips** * If the repository does not appear, open **Settings → Apps → GitHub → Choose repositories** and make sure `CNCL-BDI` is selected. * For a brand-new or private repository that will not show up, you can nudge GitHub’s index by running a search on GitHub itself in the form `repo:Basic-Data-Infrastructure/CNCL-BDI`. *** ## Option 3 — Google Gemini (gemini.google.com) [Section titled “Option 3 — Google Gemini (gemini.google.com)”](#option-3--google-gemini-geminigooglecom) Google’s general-purpose AI assistant is **Gemini**. The Gemini web app can import a GitHub repository directly into a prompt. 1. Open [gemini.google.com](https://gemini.google.com). 2. In the prompt box, click **Add file** (the attach/+ control). 3. Choose **Import code**. 4. Paste the repository or branch URL: `https://github.com/Basic-Data-Infrastructure/CNCL-BDI` (you can also point at the `docs/arc42` path or a specific branch). 5. Click **Import**, then ask your question about the architecture. ### Recommended alternative: NotebookLM [Section titled “Recommended alternative: NotebookLM”](#recommended-alternative-notebooklm) For document-style question-and-answer work, Google’s **[NotebookLM](https://notebooklm.google.com)** is often a better fit than the Gemini chat app. It gives source-grounded, citation-backed answers from exactly the documents you add. 1. Open NotebookLM and create a new notebook. 2. Add the arc42 documents as sources — the simplest approach is to paste the GitHub file/folder URLs, or download the documents and upload the markdown files. 3. Ask questions; NotebookLM answers only from the sources you provided and cites them. *** ## Option 4 — GitHub Copilot [Section titled “Option 4 — GitHub Copilot”](#option-4--github-copilot) If you already use **GitHub Copilot**, you can ask questions about the repository without any extra setup — Copilot Chat works directly on GitHub when you are in the context of the repository. 1. Open the repository on GitHub: [`Basic-Data-Infrastructure/CNCL-BDI`](https://github.com/Basic-Data-Infrastructure/CNCL-BDI). 2. Click the **Copilot icon** in the top navigation bar. A chat panel opens on the page you are viewing. 3. Ask your question. Because you are in the repository’s context, Copilot automatically indexes it and answers based on its contents — e.g. *“Summarise the arc42 deployment chapter in docs/arc42.”* 4. As you browse to specific files, pull requests or issues, they are attached as context, so you can narrow your questions to, for example, `docs/arc42/09-decisions`. **Tips** * The same Copilot Chat is available in your **IDE** (VS Code, Visual Studio, JetBrains) and in **GitHub Mobile** — handy if you prefer to work where the code is. * A GitHub Copilot license is required (individual or provided through your organization). *** ## Option 5 — Perplexity [Section titled “Option 5 — Perplexity”](#option-5--perplexity) Perplexity has no GitHub connector, but it reads public web pages — which is all you need, because the repository is public. 1. Open [perplexity.ai](https://www.perplexity.ai). 2. Paste a public URL into the prompt and ask your question. Two good choices: * A specific file or folder in the repo, e.g. a chapter under [`docs/arc42`](https://github.com/Basic-Data-Infrastructure/CNCL-BDI/tree/main/docs/arc42). * The published **`llms-full.txt`** (the entire documentation in one file — see *Alternative* below), which gives Perplexity the full context in a single link. 3. Perplexity answers with citations back to the source. **Tip:** For repeated use, create a **Space** in Perplexity and add the URLs as sources, so every question in that Space is grounded in the arc42 documentation. *** ## Option 6 — Cursor [Section titled “Option 6 — Cursor”](#option-6--cursor) Cursor is an AI code editor. It is a good fit for team members who already work in an IDE, because it answers from the project you have open. 1. Clone the repository (or open a copy you already have): ```bash git clone https://github.com/Basic-Data-Infrastructure/CNCL-BDI ``` 2. Open the folder in Cursor. 3. Open the chat (`Cmd/Ctrl + L`), reference the docs with `@` — for example `@docs/arc42` or `@Codebase` — and ask your question. Cursor indexes the project and answers from it. *** ## Option 7 — JetBrains Junie [Section titled “Option 7 — JetBrains Junie”](#option-7--jetbrains-junie) **Junie** is JetBrains’ AI coding agent (it creates the `.junie` folder in a project). This is the setup used at Conclusion. 1. Open the `CNCL-BDI` project in a JetBrains IDE (IntelliJ IDEA, PyCharm, etc.) with the Junie plugin — or run the Junie CLI from inside the project folder. 2. In the Junie chat, reference files or folders with `@` (e.g. `@docs/arc42`) and ask your question. Junie reads the referenced documents and answers from them. 3. Project context for Junie lives in **`.junie/guidelines.md`** (or `AGENTS.md`) in the repository root. Because it is committed to the repo, the whole team shares the same context and gets consistent answers. The arc42 documents themselves are in `docs/arc42`. *** ## What you can ask [Section titled “What you can ask”](#what-you-can-ask) Once connected, useful questions include: * “Summarise chapter 5 (Building Blocks) of the CTN arc42 documentation.” * “Which architecture decision (ADR) covers external IdP federation, and what was decided?” * “Explain the difference between users and members in the CTN context.” * “What are the main risks listed in the arc42 documentation?” * “Define the glossary terms used in the association onboarding flow.” ## Keeping answers up to date [Section titled “Keeping answers up to date”](#keeping-answers-up-to-date) These tools read a snapshot of the repository at the time you connect or sync. After the documentation is updated on GitHub, use the tool’s refresh/sync option (or re-import) so the AI sees the latest version. ## Alternative: the published documentation site [Section titled “Alternative: the published documentation site”](#alternative-the-published-documentation-site) If you would rather not connect GitHub at all, the CTN documentation portal automatically publishes machine-readable versions of the docs for AI tools: * **Compact index:** <https://docs.preview.inlandbase.com/llms.txt> * **Full documentation (single file):** <https://docs.preview.inlandbase.com/llms-full.txt> Paste either URL into any AI tool that accepts a link (Perplexity, Claude, ChatGPT, Gemini, …) and ask your questions from there. The `llms-full.txt` file contains the complete documentation, so it gives the AI the most context in a single link. ## References [Section titled “References”](#references) * Claude — [Use the GitHub integration](https://support.claude.com/en/articles/10167454-use-the-github-integration) * ChatGPT — [Connecting GitHub to ChatGPT](https://help.openai.com/en/articles/11145903-connecting-github-to-chatgpt) * Gemini — [Import a GitHub repository & ask about it in the Gemini web app](https://support.google.com/gemini/answer/16176929) * GitHub Copilot — [Asking GitHub Copilot questions in GitHub](https://docs.github.com/en/copilot/how-tos/chat-with-copilot/chat-in-github) * Perplexity — [perplexity.ai](https://www.perplexity.ai) * Cursor — [Cursor documentation](https://docs.cursor.com) * JetBrains Junie — [Getting started with Junie](https://junie.jetbrains.com/docs/) · [Guidelines and memory](https://junie.jetbrains.com/docs/guidelines-and-memory.html) # How To: Update Arc42 Documentation **Last Updated:** May 31, 2026 **Difficulty:** Beginner **Estimated Time:** 10–15 minutes *** ## Overview [Section titled “Overview”](#overview) This guide explains how to update or add arc42 architecture documentation **and** publish it to the CTN Documentation Portal. The most important thing to understand first: **there are two repositories.** | Repository | Where | What it is | | --------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------ | | **CNCL-BDI** (source) | GitHub: `Basic-Data-Infrastructure/CNCL-BDI` | The **single source of truth** for all arc42 docs. You edit content **here**. | | **CTN-Documentation** (publishing site) | Azure DevOps: `DeNoronhaConsulting/CTN Documentation` | The **Astro + Starlight** website that publishes the docs. It does **not** author content. | > ⚠️ **Do not edit `docs/arc42/` inside the CTN-Documentation repo.** That folder is generated by the sync step and will be **wiped and overwritten** the next time anyone runs `npm run sync-arc42`. Any edit you make there directly will be lost. Edit the docs in **CNCL-BDI** instead. How content reaches the live site: ``` graph LR A[Edit docs in CNCL-BDI/docs/arc42] --> B[Push to GitHub] B --> C[npm run sync-arc42 in CTN-Documentation] C --> D[npm run build -> dist/] D --> E[Commit docs/arc42 + push to Azure DevOps] E --> F[Azure Pipeline builds & deploys] F --> G[Live site] ``` > **Note:** This how-to guide itself (and everything under `docs/how-to/`) **is** authored directly in the CTN-Documentation repo — only `docs/arc42/` is synced from CNCL-BDI. *** ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * ✅ Git access to **both** repos: CNCL-BDI (GitHub) and CTN-Documentation (Azure DevOps) * ✅ The two repos checked out **as siblings** in the same parent folder: ```plaintext <parent>/ ├── CNCL-BDI/ # source content └── CTN-Documentation/ # publishing site ``` (The sync script looks for `../CNCL-BDI/docs/arc42` by default; override with the `ARC42_SOURCE` env var.) * ✅ **Node.js ≥ 22.12.0** — required by Astro 6. Node 20 will fail with *“not supported by Astro!”*. Use `nvm use 22` or a Node ≥ 22 install. * ✅ A text/markdown editor (VS Code, etc.) *** ## Document Status Workflow [Section titled “Document Status Workflow”](#document-status-workflow) All arc42 documents use status metadata in frontmatter to track their lifecycle: | Status | Description | When to use | | ------------- | ------------------ | ----------------------------------- | | **Draft** | Initial work | Being written, not ready for review | | **Review** | Ready for feedback | Complete enough for team review | | **Approved** | Finalized | Reviewed and approved | | **Published** | Active version | Live and in use | | **Archived** | Old version | Superseded or no longer relevant | ```yaml --- status: Published lastUpdate: 2026-05-31 --- ``` *** ## Quick Reference [Section titled “Quick Reference”](#quick-reference) **TL;DR — edit in the source repo, sync, build, publish:** ```bash # 1. Edit content in the SOURCE repo (CNCL-BDI) cd ../CNCL-BDI code docs/arc42/05-building-blocks/ctn-connector.md git add docs/arc42 && git commit -m "docs: update connector" && git push # to GitHub # 2. Sync + build + publish from the PUBLISHING repo (CTN-Documentation) cd ../CTN-Documentation nvm use 22 # ensure Node >= 22.12.0 npm run sync-arc42 # copy CNCL-BDI/docs/arc42 -> docs/arc42 (wipe + replace) npm run build # prepare-content + astro build -> dist/ (also validates links) git add docs/arc42 && git commit -m "docs: sync arc42 from CNCL-BDI" && git push # to Azure DevOps # 3. Done — the Azure pipeline builds and deploys automatically ``` *** ## Step-by-Step Process [Section titled “Step-by-Step Process”](#step-by-step-process) ### Step 1: Edit the content in CNCL-BDI (source of truth) [Section titled “Step 1: Edit the content in CNCL-BDI (source of truth)”](#step-1-edit-the-content-in-cncl-bdi-source-of-truth) All arc42 documents live in the **CNCL-BDI** repo under `docs/arc42/`: ```plaintext CNCL-BDI/docs/arc42/ ├── ctn-arc42-index.md ├── 00-guidelines/ctn-documentation-guidelines.md ├── 01-introduction/ │ ├── ctn-initiative-overview.md │ ├── bdi-framework-principles.md │ └── ctn-terminology-clarification.md ├── 02-constraints/02-constraints.md ├── 03-context/ctn-context.md ├── 04-solution-strategy/ctn-solution-strategy.md ├── 05-building-blocks/ │ ├── ctn-building-blocks.md │ ├── ctn-connector.md │ ├── ctn-association-register.md │ └── ... ├── 06-runtime/ ... 07-deployment/ ... 08-crosscutting/ ... ├── 09-decisions/ # ADRs (00001-…), ADR-TEMPLATE.md, README.md ├── 10-quality/ ... 11-risks/ ... 12-glossary/ ... 13-roadmap/ ``` **File naming:** `descriptive-title.md`, lowercase-with-hyphens, **no date prefixes**. Place each file in the correct numbered chapter directory — the website sidebar is **autogenerated** from these directories. **Editing an existing document:** 1. Open the file in CNCL-BDI, e.g. `docs/arc42/05-building-blocks/ctn-connector.md`. 2. Make your changes in Markdown. 3. Update the frontmatter `status` and `lastUpdate` as appropriate. 4. Save. **Creating a new document:** ```markdown --- status: Draft lastUpdate: 2026-05-31 --- # New Component Architecture ## Overview Your content here... ``` Drop it in the relevant chapter directory — it appears in the sidebar automatically after the next build. ### Step 2: Commit & push the source repo (GitHub) [Section titled “Step 2: Commit & push the source repo (GitHub)”](#step-2-commit--push-the-source-repo-github) ```bash cd ../CNCL-BDI git add docs/arc42 git commit -m "docs: add connector retry semantics" git push # pushes to GitHub (Basic-Data-Infrastructure/CNCL-BDI) ``` This makes your edit the canonical version. (It is also what feeds GitHub Discussions–based comments on the site, via giscus.) ### Step 3: Sync into the publishing repo [Section titled “Step 3: Sync into the publishing repo”](#step-3-sync-into-the-publishing-repo) From the **CTN-Documentation** repo, copy the latest arc42 content in: ```bash cd ../CTN-Documentation nvm use 22 # Astro requires Node >= 22.12.0 npm install # first time only / after dependency changes npm run sync-arc42 ``` `sync-arc42` runs `scripts/sync-arc42.cjs`, which: * copies `../CNCL-BDI/docs/arc42` → `docs/arc42`, **wiping the destination first** so deleted/renamed upstream files don’t leave stale copies; * can be pointed elsewhere via the `ARC42_SOURCE` env var; * **auto-skips** when the source isn’t present (e.g. in CI) and uses the already-committed `docs/arc42`. You should see `✅ Synced N markdown file(s) into docs/arc42`. ### Step 4: Build and preview locally [Section titled “Step 4: Build and preview locally”](#step-4-build-and-preview-locally) ```bash # One-off production build (also validates all internal links) npm run build # …or run a live dev server with hot reload npm run dev # then open the printed http://localhost:4321 URL # …or preview the built output npm run preview ``` `npm run build` runs `prepare-content` (stages `docs/arc42` + `docs/how-to` into `src/content/docs/`, builds the splash homepage and “recently updated” cards) and then `astro build`, producing static HTML in `dist/`. A healthy build ends with something like: ```plaintext [build] N page(s) built in … · All internal links are valid. · [build] Complete! ``` If the link validator reports broken links, fix them in **CNCL-BDI**, then re-run `sync-arc42` and `build`. ### Step 5: Commit the synced docs & push (publish) [Section titled “Step 5: Commit the synced docs & push (publish)”](#step-5-commit-the-synced-docs--push-publish) The Azure pipeline only checks out the CTN-Documentation repo, so the synced `docs/arc42` content **must be committed here** for the live site to update. ```bash git add docs/arc42 git commit -m "docs: sync arc42 from CNCL-BDI" git push # pushes to Azure DevOps (DeNoronhaConsulting/CTN Documentation) ``` ### Step 6: Automatic deployment [Section titled “Step 6: Automatic deployment”](#step-6-automatic-deployment) After the push to `main`, the Azure DevOps pipeline (`azure-pipelines.yml`) automatically: 1. ✅ Installs Node `22.x` and dependencies 2. ✅ Runs `npm run build` (in CI the sync is skipped — it builds from the committed `docs/arc42`) 3. ✅ Copies `staticwebapp.config.json` into `dist/` 4. ✅ Deploys `dist/` to the Azure Static Web App **Verify deployment (\~4–5 minutes):** 1. Check the build at the pipeline (link below). 2. Visit the portal: <https://docs.preview.inlandbase.com> 3. Confirm your page appears in the sidebar and at its URL. Starlight URLs are lowercase, directory-style, with **no `.html`**, e.g. `https://docs.preview.inlandbase.com/arc42/05-building-blocks/ctn-connector/` *** ## Markdown Best Practices [Section titled “Markdown Best Practices”](#markdown-best-practices) ### Headings [Section titled “Headings”](#headings) ```markdown # H1 — Page title (use once per document) ## H2 — Major section ### H3 — Subsection ``` ### Links [Section titled “Links”](#links) Prefer relative links between docs; the build validates internal links. ```markdown [Link text](https://example.com) [Another arc42 page](../05-building-blocks/ctn-connector.md) ``` ### Code blocks [Section titled “Code blocks”](#code-blocks) ````markdown ```typescript interface User { id: string; name: string; } ``` ```` ### Tables [Section titled “Tables”](#tables) ```markdown | Feature | Status | Priority | |---------|--------|----------| | Auth | ✅ Done | High | ``` ### Mermaid diagrams [Section titled “Mermaid diagrams”](#mermaid-diagrams) Mermaid is rendered at **build time** via the `astro-mermaid` integration — just use a fenced `mermaid` block: ````markdown ```mermaid graph LR A[Client] --> B[API Gateway] --> C[Service] --> D[Database] ``` ```` *** ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Build fails with “Node.js … is not supported by Astro!” [Section titled “Build fails with “Node.js … is not supported by Astro!””](#build-fails-with-nodejs--is-not-supported-by-astro) Astro 6 requires **Node ≥ 22.12.0**. Run `node --version`; if it’s older, `nvm use 22` (or install Node 22+). ### `sync-arc42` reports “arc42 source not found” [Section titled “sync-arc42 reports “arc42 source not found””](#sync-arc42-reports-arc42-source-not-found) The CNCL-BDI repo isn’t a sibling of CTN-Documentation, or is checked out elsewhere. Either place it at `../CNCL-BDI`, or set the source explicitly: ```bash ARC42_SOURCE=/absolute/path/to/CNCL-BDI/docs/arc42 npm run sync-arc42 ``` ### My edit to `docs/arc42` in CTN-Documentation disappeared [Section titled “My edit to docs/arc42 in CTN-Documentation disappeared”](#my-edit-to-docsarc42-in-ctn-documentation-disappeared) Expected — that folder is overwritten by `sync-arc42`. Make the edit in **CNCL-BDI** and re-sync. ### Document doesn’t appear in the sidebar [Section titled “Document doesn’t appear in the sidebar”](#document-doesnt-appear-in-the-sidebar) 1. Confirm the file is in the correct numbered chapter directory under CNCL-BDI’s `docs/arc42/` (the sidebar autogenerates per directory). 2. Confirm you ran `npm run sync-arc42` and then `npm run build`. 3. Check the build output for errors. ### Build reports broken internal links [Section titled “Build reports broken internal links”](#build-reports-broken-internal-links) The `starlight-links-validator` plugin fails the build on broken internal links. Fix the link in **CNCL-BDI**, re-sync, rebuild. ### Build fails in Azure DevOps [Section titled “Build fails in Azure DevOps”](#build-fails-in-azure-devops) 1. Check the pipeline logs (link below). 2. Reproduce locally with `npm run build`. 3. Look for Markdown/link errors or missing dependencies. *** ## Quick Tips [Section titled “Quick Tips”](#quick-tips) ### DO ✅ [Section titled “DO ✅”](#do-) * Edit arc42 content in **CNCL-BDI**, never in CTN-Documentation’s `docs/arc42` * Keep the two repos as **siblings** * Use Node ≥ 22.12.0 * Run `sync-arc42` → `build` and check links before committing * Use `lowercase-with-hyphens.md` filenames in the right chapter directory * Keep `status` and `lastUpdate` frontmatter current ### DON’T ❌ [Section titled “DON’T ❌”](#dont-) * Don’t edit the synced `docs/arc42` in the publishing repo * Don’t use date prefixes in filenames * Don’t push to Azure DevOps without first syncing the latest source * Don’t expect content changes to publish from a CNCL-BDI push alone — you must sync + commit + push in CTN-Documentation *** ## Related Documentation [Section titled “Related Documentation”](#related-documentation) * **[arc42 Template](https://arc42.org/overview)** — the architecture documentation standard * **[Markdown Guide](https://www.markdownguide.org/)** — Markdown syntax reference * **[Mermaid Documentation](https://mermaid.js.org/)** — diagram syntax * **[Starlight](https://starlight.astro.build/)** — the docs framework this site is built on * **Azure DevOps build pipeline** — <https://dev.azure.com/DeNoronhaConsulting/CTN%20Documentation/_build> *** ## Summary [Section titled “Summary”](#summary) 1. ✏️ **Edit** arc42 docs in **CNCL-BDI** (`docs/arc42/`) and push to GitHub 2. 🔄 **Sync** into CTN-Documentation: `npm run sync-arc42` 3. 🧪 **Build & preview**: `npm run build` (or `npm run dev`) — fix any broken links 4. 💾 **Commit & push** the synced `docs/arc42` to Azure DevOps 5. ⏱️ **Wait \~4–5 min** for the pipeline to build and deploy 6. ✅ **Verify** on <https://docs.preview.inlandbase.com>