---------------------------------------------------------------------- This is the API documentation for the splam library. ---------------------------------------------------------------------- ## Functions Utility functions add_highlight_keyword(keyword: str, css_class: str) -> None Persist a custom keyword/phrase to highlight class mapping. add_user(username: str, password: str, role: str = 'admin') -> None Add or replace a user's salted password hash and role. attempt_login(username: str, password: str) -> tuple[bool, str] Verify credentials subject to failed-attempt lockout. Returns `(success, outcome)`, where `outcome` is the string recorded in the audit trail. After `MAX_FAILED_LOGINS` consecutive failures the account is refused for `LOCKOUT_SECONDS` even when given the right password; a successful login clears the counter. build_client(services: list[str]) Return a chatlas client seeded with the admin-task notes for `services`. Raises ImportError if the `chat` extra isn't installed, and ValueError for an unrecognized `SPLAM_CHAT_PROVIDER`. chat_available() -> bool Return True when the optional `chat` extra is installed. doc_columns(width_px: float) -> int Return the man page width in columns that fills `width_px` of screen. Rounded down to a multiple of five so dragging the window edge re-renders a handful of times rather than once per pixel, and clamped because a man page set much below 60 columns loses its indentation and much above 220 is too wide a line to read. doc_references(service: str) -> list[str] Return the URIs listed in the unit's `Documentation=` property. get_all_service_info() -> dict[str, dict] Return built-in SERVICE_INFO merged with user-saved overrides (overrides win). get_documentation(service: str, columns: int = 80) -> str Return `systemctl help `: the man pages named by Documentation=. The pager is forced to `cat` because man pipes through `$PAGER` whenever it's set, which would leave this call waiting on a pager that has no terminal to draw on. get_logs(service: str, lines: int) -> str Return the last `lines` journalctl entries for `service`, newest first. get_role(username: str) -> str Return a user's role, or `DEFAULT_ROLE` for records saved before roles existed. get_service_info(service: str) -> dict Return admin-task notes for a service, falling back to generic guidance. get_status(service: str) -> str Return combined stdout/stderr of `systemctl status `. highlight(text: str) -> str Escape `text` and wrap known (and user-added) keywords in colored spans. highlight_term(text: str, term: str) -> str Escape `text` and wrap case-insensitive matches of `term` in . list_services() -> list[str] Return sorted service names known to systemd that have admin-task notes. load_users() -> dict Load the credentials file, or an empty dict if it doesn't exist yet. lockout_remaining(username: str) -> int Return whole seconds left on `username`'s lockout, or 0 if it isn't locked. record_action(username: str, service: str, action: str, reason: str, result: str) -> None Append a timestamped audit entry for a service control action. Called for rejected attempts as well as executed ones, so the trail shows what was tried, not only what succeeded. record_auth(username: str, action: str, result: str) -> None Append a timestamped audit entry for a login or logout attempt. Auth entries carry no service, so they surface regardless of which service the Audit Trail tab is filtered to. read_audit_log(service: str | None = None, limit: int = 200) -> list[dict] Return the most recent audit entries, newest first, optionally filtered by service. read_only_tools() -> list Return the read-only functions a chat client may call. `systemctl_action` is deliberately absent and will not be added: every service action in this app carries a reason typed by a person, and an action issued by the model would produce an audit entry justified by the thing being audited. redact(line: str) -> str Replace anything secret-shaped in a shell command line with `[REDACTED]`. save_service_info(service: str, info: dict) -> None Persist a user-edited admin-task entry for `service`, overriding any built-in entry. system_prompt(services: list[str]) -> str Build a system prompt covering every service discussed so far. Takes a list rather than one name so that selecting a second service adds its notes instead of replacing the first, which is what makes comparing two units in one conversation work. systemctl_action(action: str, service: str) -> subprocess.CompletedProcess Run `systemctl ` and return the completed process. verify_user(username: str, password: str) -> bool Check a username/password pair against the stored hash. This is the credential check alone and ignores lockout state; the interactive login path goes through `attempt_login`. ---------------------------------------------------------------------- This is the User Guide documentation for the package. ---------------------------------------------------------------------- ## Tutorial ### Getting Started This tutorial walks through installing splam, creating an account, and using the app end to end against `cron.service`. Follow the steps in order and by the end you'll have 1) logged in, 2) restarted a service with a documented reason, and 3) seen that action appear in the audit trail. ## 1. Install `splam` installs into a virtual environment. On Debian-based systems (Pop!\_OS, Ubuntu, Debian) this isn't optional because the system Python is marked externally managed under [PEP 668](https://peps.python.org/pep-0668/), and installing against it fails with `externally-managed-environment`. First clone the repo, then create the `.venv` and install: ```bash git clone https://github.com/mjfrigaard/splam.git cd splam python3 -m venv .venv .venv/bin/pip install -e . ``` That installs Shiny alongside `splam` and places `splam-users` and `shiny` in `.venv/bin/`. Now, activate the environment: ```bash source .venv/bin/activate ``` The remaining steps assume the virtual environment isactivated. If you'd rather not activate, prefix each command with `.venv/bin/` instead. ## 2. Create an account The app has no default account, so create one with the bundled CLI: ```bash splam-users add admin ``` Enter a password twice when prompted. This creates an `admin` account (which is what the rest of the tutorial assumes). Read-only `auditor` accounts are covered in the [how-to guide](01.01-how-to-logins.qmd). ## 3. Start the app ```bash shiny run splam.app:app --reload ``` ```{verbatim} INFO: Will watch for changes in these directories: ['/path/to/splam'] INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) INFO: Started reloader process [11105] using WatchFiles INFO: Started server process [11154] INFO: Waiting for application startup. INFO: Application startup complete. ``` Open the printed URL (typically ) in your browser. ![Screenshot of the login page](images/ui-login.png){width='100%'} ## 4. Log in Enter the username and password you created in step 2 and select **Log in**. ## 5. Select `cron.service` Use the **Service** dropdown in the sidebar to choose `cron.service`. The dropdown lists every `.service` unit systemd found on this host, so it's a long list. ![](images/ui-login-cron.png){width='100%'} Six tabs are available to an admin account (an auditor account sees the same set without **Actions**): - **Status**: live `systemctl status` output - **Actions**: the Reason field and the Start/Stop/Restart buttons - **Tasks**: admin-task notes for the selected service - **Doc**: the service's own documentation, from the unit file - **Logs**: recent `journalctl` entries, newest first - **Audit Trail**: a record of actions taken on this service through the app Open the **Tasks** tab and read through cron's purpose, checks, coommon issues, and compliance note. Then open **Doc**, which prints `cron(8)` because `cron.service` names it in `Documentation=`. ## 6. Restart it with a reason Switch to the **Actions** tab and click **Restart** without filling in the **Reason** field. The action is blocked and the Status tab shows `[rejected: reason required]`. You will see this in the **Audit Trail** tab as well. ![](images/ui-action-reason-required-cron.png){width='100%'} Now type a reason (e.g. `tutorial test restart`) into **Reason** and click **Restart** again. You'll be asked to authenticate. ![](images/ui-action-tutorial-reason-auth-cron.png){width='100%'} Switch back to the **Status** tab to see the result. ![](images/ui-status-restart-ok-cron.png){width='100%'} ## 7. Confirm it landed in the audit trail Switch to the **Audit Trail** tab. You should see a new entry with your username, the action, the reason you gave, and a timestamp. ![](images/ui-audit-trail-reason-cron.png){width='100%'} ## 8. Search the logs Switch to the **Logs** tab, set **Lines** if you want more history, and type `CRON` into **Search**. The output narrows to lines containing that term (case-insensitive). Notice that words like `error`, `started`, and `stopped` are already highlighted in the output (no search needed to spot them). ![](images/ui-logs-review-cron.png){width='100%'} ## Next steps - [How-To Guides](01.01-how-to-logins.qmd) for specific tasks (managing accounts, extending service info, adding highlight keywords, rebuilding the docs). - [Explanation](02.01-explanation-login-sessions.qmd) for the reasoning behind the login model, the audit trail, and the app's structure. ## How-To Guides ### Manage Logins Task-oriented recipes. Each section assumes you already have `splam` installed and running. See the [Tutorial](00-tutorial-getting-started.qmd) if not. ## Manage login accounts Add or update an account: ```bash splam-users add ``` List existing accounts and their roles: ```bash splam-users list ``` ## Give an account a role Every account has one of two roles: | Role | Can do | |------|--------| | `admin` | Everything: Start/Stop/Restart, Configure, and all read-only tabs | | `auditor` | Read-only: Status, Tasks, Doc, Logs, and Audit Trail | `admin` is the default, so `splam-users add ` with no `--role` creates an `admin` account. Users can add a read-only account with: ```bash splam-users add qa-reviewer --role auditor ``` An auditor sees no **Actions** tab and no **Configure** button. This role is also enforced server-side, so an action submitted by a hand-crafted client is rejected and recorded in the audit trail with the result `rejected: role not permitted`. Change an existing account's role by adding it again with the role you want (`add` replaces the whole record, so you'll be prompted for the password again). ## Unlock an account After five consecutive failed logins the account is refused for 15 minutes (even if the correct password is then supplied). The login form shows the remaining time, and each attempt is recorded in the audit trail. Users can adjust the number of failed login attempts and the lockout duration with `SPLAM_MAX_FAILED_LOGINS` and `SPLAM_LOCKOUT_SECONDS`. Lockout state is held in memory by the running process, so restarting the app clears every lockout. That restart is the manual unlock path. ## Where credentials are stored Accounts are stored as salted `PBKDF2` hashes in `credentials.json`, in the app's data directory (`~/.local/share/splam/` by default). Set `SPLAM_DATA_DIR` to relocate that directory: * `/var/lib/splam` is the conventional choice for a system deployment - or `SPLAM_CREDENTIALS` to point at this one file. The directory is created mode `0700` and the file mode `0600`, so it is readable only by the account running the app (i.e., treat it like `/etc/shadow`). ### Sudo Privileges Task-oriented recipes. Each section assumes you already have `splam` installed and running. See the [Tutorial](00-tutorial-getting-started.qmd) if not. ## Run the app so it can actually control services `systemctl start/stop/restart` require privilege. Running `shiny run` as an unprivileged user will authenticate fine but every service action will fail with a permission error in the **Status** tab. Either run the app as a user with the necessary rights (`root`, or a service account granted control over specific units via `polkit`/`sudoers`), or accept that Start/Stop/Restart will no-op with an error until that's configured. **Status** and **Logs** work regardless of privilege. ### Service Actions Task-oriented recipes. Each section assumes you already have `splam` installed and running. See the [Tutorial](00-tutorial-getting-started.qmd) if not. ## Require & check reasons before service actions The **Reason** field on the **Actions** tab is required for Start/Stop/ Restart. An empty reason blocks the action and shows `[rejected: reason required]` in the **Status** tab. No `systemctl` command is run, though the rejected attempt is still recorded in the audit trail. This is enforced server-side per session, not just in the UI. ## Restrict actions to admin accounts The **Actions** tab only appears for accounts with the `admin` role; an `auditor` account gets the read-only tabs and a note in the sidebar. This ensures that an action cannot be performed without a valid reason, preventing unauthorized or arbitrary service modifications. The check is repeated server-side before any `systemctl` command runs, so an action submitted outside the UI is rejected with `rejected: role not permitted` and recorded in the audit trail. See [Manage Logins](01.01-how-to-logins.qmd) for assigning roles. ### Admin-Task Notes Task-oriented recipes. Each section assumes you already have `splam` installed and running. See the [Tutorial](00-tutorial-getting-started.qmd) if not. ## Add or edit admin-task notes from the app Select a service in the sidebar (or type a new systemd unit name), click **Configure**, and fill in the **Admin-task notes** form: Purpose, Checks and Common issues (one per line), Suggested log search terms (comma-separated), and Compliance note. Click **Save admin-task notes**. This writes to `service_info_overrides.json` in the app's data directory (override the location with `SPLAM_SERVICE_INFO_OVERRIDES`) and takes effect immediately, with no redeploy needed. An override for an existing unit name (e.g. `cron.service`) replaces the built-in entry. The **Service** dropdown lists every `.service` unit in systemd's search path, so a unit installed on the host is already selectable before anyone writes notes for it. Notes saved against a name that isn't installed are added to the dropdown anyway, which is what keeps a note from disappearing when its package is removed. ## Add admin-task notes in code instead For notes that should ship with the package rather than live in a per-deployment override file, edit `src/splam/service_info.py` and add an entry to `SERVICE_INFO` keyed by the exact systemd unit name: ```python "my-service.service": { "purpose": "...", "checks": ["...", "..."], "common_issues": ["...", "..."], "log_patterns": ["error", "..."], "compliance_note": "...", }, ``` If the same guidance applies to multiple unit names (e.g. a service that's named differently across distros), define the dict once and point several keys at it. See `_SSH_INFO` (`ssh.service` / `sshd.service`) or `_TIME_SYNC_INFO` for the pattern. Any service without an entry falls back to `GENERIC_INFO`. ### Service Documentation Task-oriented recipes. Each section assumes you already have `splam` installed and running. See the [Tutorial](00-tutorial-getting-started.qmd) if not. Every `systemd` unit file can name its own documentation in a `Documentation=` line, and most packaged units do. The **Doc** tab reads that line for the selected service and prints what it points at, so the upstream manual is one tab away from the status output you are trying to interpret. ## Read a service's own documentation Select a service in the sidebar and open **Doc**. You get two things: 1. The `Documentation=` references the unit file declares, listed exactly as they are written. 2. The man pages those references name, rendered as plain text. For `cron.service` on Ubuntu, that is one reference, `man:cron(8)`, followed by the whole of `cron(8)`. Nothing is fetched over the network and nothing is cached: the tab shells out each time you land on it, so it always reflects the unit file currently on disk. Man hard-wraps its output to a fixed number of columns rather than leaving the browser to reflow it, so the tab measures itself and asks for a width that fits, between 60 and 220 columns. Widen the window and the page is re-rendered wider; make it narrow and the lines shorten rather than scrolling sideways. Both roles see this tab. Reading a manual changes nothing, so `auditor` accounts get it on the same terms as `admin`, like **Status**, **Tasks**, **Logs**, and **Audit Trail**. ## Understand the reference types `Documentation=` takes a space-separated list of URIs, in the maintainer's order of relevance, and a unit may mix schemes. Only `man:` entries are rendered as text; the rest are named and left for you to follow: | Scheme | Example | What the tab does | |-------------|-----------------------------|------------------------------| | `man:` | `man:cron(8)` | Renders the whole page inline. Several `man:` entries render one after another | | `https:` or `http:` | `https://gitlab.com/apparmor/apparmor/wikis/home/` | Lists it as a clickable link, and repeats it under `Additional documentation:` | | `info:` | `info:coreutils` | Lists it. Reading it means running `info coreutils` yourself | | `file:` | `file:/usr/share/doc/foo/README` | Lists the path, does not read the file | `systemd-journald.service` shows the first row doing its job: two `man:` entries, so the tab prints `systemd-journald.service(8)` and `journald.conf(5)` in full, around 700 lines. `apparmor.service` shows a mixed list, one man page plus a wiki URL. ## When there is nothing to show Plenty of units declare no documentation at all. The tab says so: ```verbatim Documentation for accounts-daemon.service not known. ``` That is systemd reporting an empty `Documentation=`, not an error in `splam`. Three things to try, in order: 1. Read the unit file itself with `systemctl cat accounts-daemon.service`. The `ExecStart=` line names the binary, and the binary usually has a man page of its own even when the unit doesn't name it. 2. Check the package: `dpkg -S $(which accounts-daemon)` then `dpkg -L | grep man`. 3. Write your own notes for it on the **Tasks** tab, which is what that tab is for. See [How-To: Admin-Task Notes](01.04-how-to-admin-task-notes.qmd). ## Add documentation to a unit you own For units your site ships, put the reference in the unit file: ```ini [Unit] Description=Nightly LIMS export Documentation=man:lims-export(1) https://wiki.example.org/lims/export ``` For a packaged unit you don't want to edit, use a drop-in so the change survives a package upgrade: ```bash sudo systemctl edit my-service.service ``` That opens an empty drop-in at `/etc/systemd/system/my-service.service.d/override.conf`. Add the `[Unit]` section above, save, then: ```bash sudo systemctl daemon-reload ``` The **Doc** tab picks it up on the next render. A drop-in `Documentation=` appends to whatever the packaged unit already declared; to replace the list instead, set `Documentation=` to an empty value on its own line first. ## Tell it apart from the Tasks tab Both tabs describe the same service and they answer different questions: | | **Doc** | **Tasks** | |-------------|------------------------------|------------------------------| | Written by | The package maintainer, upstream | Your administrators | | Source | `Documentation=` in the unit file | `SERVICE_INFO` and `service_info_overrides.json` | | Changes when | You upgrade the package | Someone clicks **Configure** | | Answers | What this service is and what its options mean | What we check here, what breaks here, what compliance expects | On a regulated host the second column is the one an auditor will ask about, because it records local decisions. The first is reference material that arrived with the package. See [Explanation: Config Overrides](02.05-explanation-config-overrides.qmd) for where each is stored. ## Do it from a terminal instead The tab runs the same commands you would: ```bash systemctl show --property=Documentation --value cron.service systemctl help cron.service ``` The first prints the raw references, the second renders them. The app changes two things about the second, both in the environment it runs man in: the pager is forced to `cat`, since there is no terminal for a pager to draw on, and `MANWIDTH` is set from the width of the tab. To read the tab's exact output in a terminal, set both yourself: ```bash PAGER=cat MANWIDTH=120 systemctl help cron.service ``` `systemctl cat cron.service` is the companion worth knowing: it prints the unit file itself, drop-ins included, which is how you confirm a `Documentation=` line landed where you meant it to. ## Further reading - [systemd.unit(5)](https://www.man7.org/linux/man-pages/man5/systemd.unit.5.html), which defines `Documentation=` and the unit search path the **Service** dropdown is built from - [systemctl(1)](https://www.man7.org/linux/man-pages/man1/systemctl.1.html), for `help`, `show`, and `cat` ### Log Review Task-oriented recipes. Each section assumes you already have `splam` installed and running. See the [Tutorial](00-tutorial-getting-started.qmd) if not. ## Search the logs for a keyword Open the **Logs** tab, set **Lines** to how far back you want to look, then type a term into **Search** (e.g. `error`, `fatal`, `denied`). The match is a case-insensitive substring check applied to the lines already fetched. Searches don't requery `journalctl`, so widen **Lines** first if the term you want might be further back. ## Read highlighted keywords without searching Some known keywords are colored inline as the log renders, so users can often spot what matters without typing a search term: * error/failure terms (`error`, `failed`, `denied`, `fatal`, `invalid`, `dead`) in red * in-progress terms (`warning`, `activating`, `deactivating`) in amber * success terms (`ok`, `accepted`, `active (running)`, `success`/`successfully`) in cyan * action terms (`start`/`started`/`starting`, `stop`/`stopped`/`stopping`, `restart`/`restarted`/`restarting`, `CRON`) in pink To add your own keyword, see [Configure Highlighting](01.10-how-to-configure-highlighting.qmd). ### Audit Trail Task-oriented recipes. Each section assumes you already have `splam` installed and running. See the [Tutorial](00-tutorial-getting-started.qmd) if not. ## Review the audit trail for a service Select the service in the sidebar, then open the **Audit Trail** tab. Entries are newest first and show timestamp, user, service, action, result, and the reason given. Entries are stored as JSON lines in `audit_log.jsonl` in the app's data directory (override with `SPLAM_AUDIT_LOG`), one file per app instance, written mode `0600`. The tab shows actions for the selected service, plus every login and logout attempt. Auth entries carry no service, so they stay visible whichever service a user has selected, and show a `-` in the service column. The action (start/stop/restart) and result (e.g. `restart ok`, or an error from `systemctl`) are highlighted the same way as the Logs and Status tabs. See [Configure Highlighting](01.10-how-to-configure-highlighting.qmd) to add your own keywords. ### Set Up Ollama [Ollama](https://ollama.com) runs a language model as a local service on port `11434`. It is the default provider for `splam`'s [Chat panel](01.09-how-to-chatbot.qmd), and the reason that panel can be the default at all: nothing you type and nothing the model reads leaves the host. This page covers Ubuntu 24.04 and the distributions built on it, including Pop!_OS. Commands assume `sudo` and a `systemd` init, which is what you already need for `splam` itself. ## Check the machine can run it A local model is the one part of this setup with hardware requirements. Three commands tell you where you stand: ```bash free -h # RAM: the "available" column, not "total" df -h /usr # disk: models land under /usr/share/ollama lspci | grep -iE "vga|3d" # graphics ``` Size the model against available RAM. A working rule is that you want roughly twice the download size free while the model is loaded, leaving room for the context window. | Model | Download | Comfortable in | |--------------------|--------------|------------------------| | `llama3.2` (3B) | about 2 GB | 8 GB RAM | | `llama3.1` (8B) | about 4.7 GB | 16 GB RAM | | `qwen2.5:14b` | about 9 GB | 32 GB RAM | Graphics decides speed, not whether it works. Ollama accelerates on NVIDIA through CUDA and on AMD through ROCm. Everything else, including Intel integrated graphics and the NPU on recent Intel laptop chips, runs on the CPU: ```bash nvidia-smi # NVIDIA present if this prints a table ls /dev/kfd # AMD ROCm present if this exists lscpu | grep -o avx2 # CPU path wants AVX2, which any recent chip has ``` Neither of the first two existing means CPU inference, which is supported and correct, just slower. On CPU prefer the 3B model. The 8B models answer better but you wait long enough per reply that the panel stops getting used. ## Install it The official script is the supported path on Ubuntu. There is no `apt` repository: ```bash curl -fsSL https://ollama.com/install.sh | sh ``` Read it first if that matters to you, and it should on a regulated host: ```bash curl -fsSL https://ollama.com/install.sh | less ``` The script needs root and makes four changes worth knowing about before you run it on a machine you have to account for: | Change | Where | |----------------------------|-----------------------------------------| | The `ollama` binary | `/usr/local/bin/ollama` | | A system user and group named `ollama` | `/etc/passwd`, `/etc/group` | | A `systemd` unit enabled at boot | `/etc/systemd/system/ollama.service` | | Downloaded models | `/usr/share/ollama/.ollama/models` | That last one is why the disk check above looks at `/usr` rather than `$HOME`. Models are owned by the `ollama` user, not by you. ## Confirm the service is up The unit is started and enabled by the installer: ```bash systemctl status ollama ``` Ask the service itself, which is the check that matters: ```bash curl http://127.0.0.1:11434 ``` It answers `Ollama is running`. If the unit is masked or you would rather not have a boot service, `ollama serve` runs the same server in the foreground and the rest of this page is unchanged. It binds to `127.0.0.1` by default, so nothing off the host can reach it. Leave it that way. If some other tool has already claimed `11434`, find it with `ss -tlnp | grep 11434` before changing anything. ## Pull a model `llama3.2` is what `splam` asks for unless told otherwise: ```bash ollama pull llama3.2 ``` Confirm what you have, and check the size against what you measured: ```bash ollama list ``` Then talk to it directly, before involving `splam` at all. This separates a model problem from an app problem: ```bash ollama run llama3.2 "reply with the single word: ready" ``` An answer here means the service, the model, and the hardware are all fine, and anything that goes wrong afterward is configuration. ## Point splam at it Nothing to configure. `ollama` is already the default provider and `llama3.2` the default model, so with the `chat` extra installed the panel finds it: ```bash .venv/bin/pip install -e ".[chat]" shiny run splam.app:app ``` Open the **Chat** tab and ask something only your own admin-task notes can answer, such as "what's the compliance note for this service?". An answer in your own wording proves the whole path. See [How-To: Chatbot](01.09-how-to-chatbot.qmd) for what the panel can and cannot do once it is talking. To use a different local model, pull it and name it: ```bash ollama pull qwen2.5:14b SPLAM_CHAT_MODEL=qwen2.5:14b shiny run splam.app:app ``` ## Keep an eye on disk Models accumulate. Every `pull` is a few GB and nothing removes them: ```bash du -sh /usr/share/ollama/.ollama/models ``` ```bash ollama rm llama3.1 ``` To keep them somewhere other than `/usr`, override the service environment rather than editing the unit file, which an update will overwrite: ```bash sudo systemctl edit ollama ``` Add: ```ini [Service] Environment="OLLAMA_MODELS=/srv/ollama/models" ``` Then create the directory, give it to the `ollama` user, and restart: ```bash sudo mkdir -p /srv/ollama/models ``` ```bash sudo chown -R ollama:ollama /srv/ollama ``` ```bash sudo systemctl restart ollama ``` ## Update it Re-running the install script upgrades in place and keeps your models: ```bash curl -fsSL https://ollama.com/install.sh | sh ``` ```bash systemctl restart ollama ``` ```bash ollama --version ``` ## Remove it Uninstalling is four steps, and none of them is `apt remove`: ```bash sudo systemctl disable --now ollama ``` ```bash sudo rm /etc/systemd/system/ollama.service /usr/local/bin/ollama ``` ```bash sudo rm -r /usr/share/ollama ``` ```bash sudo userdel ollama && sudo groupdel ollama ``` With the service gone, `splam` shows the same setup help it showed before the install. The app keeps working; only the **Chat** tab changes. To take the chat code path out entirely, uninstall the extra instead, as described in [Turn it off for everyone](01.09-how-to-chatbot.qmd#turn-it-off-for-everyone). ## When it doesn't work | What you see | What it means | |---------------------------|-------------------------------------| | `Can't find locally running ollama.` in the Chat tab | The service isn't running. `systemctl start ollama` | | `Unit ollama.service could not be found` | The install script didn't finish. Re-run it and read the output | | `model "llama3.2" not found` | Installed but nothing pulled. `ollama pull llama3.2` | | `ollama: command not found` after installing | `/usr/local/bin` isn't on this shell's `PATH`. Start a new shell | | Replies take minutes | CPU inference on too large a model. Pull `llama3.2` and set `SPLAM_CHAT_MODEL` | | The service dies partway through a reply | Out of memory. Check `journalctl -u ollama -n 50` for the OOM kill, then use a smaller model | The service writes to the journal like anything else on the box, so the same **Logs** tab you use for the rest of the system works here: ```bash journalctl -u ollama -n 100 --no-pager ``` ## Further reading - [Ollama's model library](https://ollama.com/library), for models beyond the three named here - [How-To: Chatbot](01.09-how-to-chatbot.qmd), for the panel this feeds - [Explanation: Chatbot Scope](02.04-explanation-chatbot-scope.qmd), for why local is the default ### Chatbot Task-oriented recipes. Each section assumes you already have `splam` installed and running. See the [Tutorial](00-tutorial-getting-started.qmd) if not. Read [Explanation: Chatbot Scope](02.04-explanation-chatbot-scope.qmd) alongside this page before enabling anything here, especially if your deployment handles regulated data. ## Add the Chat panel The chatbot is optional and `splam` never imports it on its own: ```bash .venv/bin/pip install -e ".[chat]" ``` That pulls in `chatlas`, and it's the only new package involved. The chat widget itself is `shinychat`, which `shiny` already depends on, so it's in your environment whether you use it or not: ```bash .venv/bin/pip show shinychat | grep Required-by ``` Restart the app and a **Chat** tab appears after **Audit Trail**, in the same tab strip as **Status**, **Tasks**, and **Logs**. Skip the extra and there's no Chat tab, no network access, and no API key anywhere on the host. If the extra is installed but no model provider can be reached, the tab still appears. Instead of a chat box it shows what went wrong and how to finish the setup, and the app prints the same thing on startup: ``` splam: Chat not configured (Can't find locally running ollama.) ``` Whether that check happens at startup depends on the provider. Ollama, OpenAI, and Google all verify themselves when the app starts, so the panel reports the problem before you type anything. Anthropic doesn't: it accepts a missing or wrong key at startup, so you get a working-looking chat box and the first message is what fails. ## Point it at a model `chatlas` talks to several providers. Two environment variables choose which, both read once when the app starts: | Setting | Default | Behavior | |-----------------------|-----------------------|--------------------------| | `SPLAM_CHAT_PROVIDER` | `ollama` | One of `ollama`, `anthropic`, `openai`, `google` | | `SPLAM_CHAT_MODEL` | `llama3.2` for `ollama`, otherwise the provider's own default | The model name passed straight through | There are two ways to satisfy that, set out in full below. Pick one: - [Option A: a local model](#option-a-a-local-model-with-ollama) keeps every log line on this host. It needs a model server installed and a few GB of disk, and answers are only as good as a model you can run locally. - [Option B: a hosted provider](#option-b-a-hosted-provider) needs no local install and gives better answers. It also sends whatever the panel sees to a third party, which on a regulated host is a decision with a paper trail attached. The default is Option A on purpose. Read [what leaves the machine](02.04-explanation-chatbot-scope.qmd#what-leaves-the-machine) before choosing Option B on anything that isn't your own workstation. ### Option A: a local model with Ollama [Ollama](https://ollama.com) serves models from `localhost`. Nothing the panel sends reaches the network. Installing it has hardware requirements and puts a service, a system user, and a few GB of models on the machine, so it gets its own page: [How-To: Set Up Ollama](01.08-how-to-set-up-ollama.qmd). Once it's running, there is nothing here to configure. `ollama` is already the default provider and `llama3.2` the default model: ```bash shiny run splam.app:app ``` No further pip install is required for this path. `chatlas` reaches Ollama through the `openai` client library, which it already depends on, so the `chat` extra alone is enough. ### Option B: a hosted provider Each hosted provider needs its client library and an API key. The key is read by the provider's own SDK, not by `splam`: | `SPLAM_CHAT_PROVIDER` | Install | API key variable | |----------------|--------------------------|--------------------------| | `anthropic` | `pip install "chatlas[anthropic]"` | `ANTHROPIC_API_KEY` | | `openai` | already installed with `chatlas` | `OPENAI_API_KEY` | | `google` | `pip install "chatlas[google]"` | `GOOGLE_API_KEY` | Install the client, export the key, and start the app with the provider named: ```bash .venv/bin/pip install "chatlas[anthropic]" export ANTHROPIC_API_KEY=... SPLAM_CHAT_PROVIDER=anthropic shiny run splam.app:app ``` Add `SPLAM_CHAT_MODEL` to pin a specific model rather than the provider's default: ```bash SPLAM_CHAT_PROVIDER=anthropic SPLAM_CHAT_MODEL=claude-sonnet-4-5 shiny run splam.app:app ``` Keep the key out of the shell history and out of the repository. A key exported in an interactive shell is visible to anything else running as that user, so on a shared host prefer a systemd unit's `EnvironmentFile=` pointing at a `0600` file. ### Confirm it worked A working provider gives you a chat box in the **Chat** tab. A broken one gives you the same tab showing the error and the two setup options, and prints the reason on startup prefixed `splam: Chat not configured`: | What the tab says | What it means | |----------------------------|-----------------------------------------| | `Can't find locally running ollama.` | Option A, but the Ollama service isn't running. Start it with `systemctl start ollama` | | `Missing credentials` (OpenAI) or `No API key was provided` (Google) | Option B, but the key variable wasn't set in the environment the app was started from | | `unknown chat provider: ...` | `SPLAM_CHAT_PROVIDER` is outside the four names above | | No **Chat** tab at all | The `chat` extra isn't installed. See [Add the Chat panel](#add-the-chat-panel) | | A chat box, then the first message errors | Anthropic with a missing or invalid `ANTHROPIC_API_KEY`. It's the one provider that doesn't check at startup | The quickest way to prove the whole path end to end is to select a service, open **Chat**, and ask something only the notes can answer, such as "what's the compliance note for this service?" An answer that quotes your own wording means the system prompt, the provider, and the streaming are all working. Both checks happen at startup, in this order, which is why a missing extra and a missing key look nothing alike: ```{mermaid} %%| eval: true %%| fig-width: 7.25 %%| fig-height: 10 %%| fig-align: 'center' %%| fig-caption: 'Startup checks' %%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"20px", "darkMode":true}}}%% flowchart TD Start(["shiny run splam.app:app"]) Extra{"chat extra
installed?"} NoTab["no Chat tab,
no model code runs"] Build[["build client
for the provider"]] Reach{"provider
answers?"} Help["Chat tab shows
setup help"] Box["Chat tab shows
a chat box"] Start --> Extra Extra -->|"no"| NoTab Extra ==>|"yes"| Build Build --> Reach Reach -->|"no"| Help Reach ==>|"yes"| Box style Start fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Extra fill:#4CBB9D,color:#FFFFFF style Reach fill:#4CBB9D,color:#FFFFFF style Build fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Box fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style NoTab fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 style Help fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 ``` Anthropic is the exception to the second diamond. It answers "yes" at startup whether or not the key is valid, and fails on the first message instead. ## Ask it about the selected service Open **Chat**, type a question, get a streamed answer. The panel is seeded with a system prompt built from the admin-task notes for whichever unit the sidebar's **Service** dropdown is on, the same entries the **Tasks** tab shows. Ask about `cron.service` and you get an answer shaped by the compliance note you wrote, not generic advice about cron. Changing the **Service** dropdown mid-conversation does not clear the thread. The notes for the newly selected unit are added to the context, so you can compare two services in one conversation. Every message carries the same three things. The fourth path opens only when `SPLAM_CHAT_TOOLS=1`, described in [Let it read logs on its own](#let-it-read-logs-on-its-own): ```{mermaid} %%| eval: true %%| fig-width: 8 %%| fig-height: 5.5 %%| fig-align: 'center' %%| fig-caption: 'What every message carries' %%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"20px", "darkMode":true}}}%% flowchart TD Notes[("admin-task notes
for selected services")] Question[/"what you typed"/] History[("this session's
earlier messages")] Model[["the model"]] Tools[["read-only tools:
get_status, get_logs,
get_service_info,
read_audit_log"]] Answer[\"streamed answer"\] Notes ==> Model Question ==> Model History ==> Model Model ==> Answer Model <-.->|"tools on"| Tools style Notes fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Question fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style History fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Model fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Answer fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Tools fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 ``` Nothing else is in there. The panel does not read the **Logs** tab you happen to have open, and it does not see other users' sessions. ## Know who can see it The **Chat** tab is visible to both `admin` and `auditor` accounts, unlike **Actions**. Reading and asking questions is what an auditor account is for, and nothing in the panel can change a service. See [Manage Logins](01.01-how-to-logins.qmd) for assigning roles. Typing in the panel counts as session activity, so a long conversation won't trip the idle timeout described in [Explanation: Login Sessions](02.01-explanation-login-sessions.qmd). ## Let it read logs on its own By default the model sees the selected service's notes and whatever you type. Turn on tools and it can fetch for itself: ```bash SPLAM_CHAT_TOOLS=1 shiny run splam.app:app ``` That registers four read-only functions: `get_status`, `get_logs`, `get_service_info`, and `read_audit_log`. Ask "what's in the last hundred lines for sshd" and it calls `get_logs` rather than waiting for you to paste. Calls are not individually approved. Setting the variable is the approval, granted once for the session, and each call and its result are shown in the conversation as they happen. Leave the variable unset and the model reads only what you type, which is the right default if you're pointed at a hosted provider. `systemctl_action` is not on that list and will not be added. The model cannot start, stop, or restart anything. [Why](02.04-explanation-chatbot-scope.qmd#why-it-reads-but-never-acts). ## Keep a transcript **Export conversation** in the panel writes the whole thread to Markdown, via `chatlas`'s `.export()`. Useful when the conversation informed a change you'll have to justify later. The transcript is a file you chose to save. It is not part of the audit trail, and a chat session on its own leaves no entry in `audit_log.jsonl`, because nothing was done to a service. See [Explanation: Audit Trail Scope](02.03-explanation-audit-trail-scope.qmd). ## Use it from a terminal instead The same conversation is available without a browser: ```bash splam-chat ``` You get a prompt. Type a question, get an answer, repeat. Type `exit` or press `Ctrl+C` to leave. This is `chatlas`'s own `.console()`, so its behavior is documented [upstream](https://posit-dev.github.io/chatlas/get-started/chatbots.html) rather than here. It reads the same two environment variables, and `--service` picks the unit whose notes get loaded: ```bash splam-chat --service cron.service ``` If no provider is reachable it reports the same problem the panel does, rather than a traceback: ``` splam-chat: chat is not configured (Can't find locally running ollama.) ``` Reach for it when there's no browser on the host, or when you're already in a terminal and don't want to log in for one question. Otherwise prefer the panel: the console has no login step, so it's bounded by the file permissions of whoever runs it rather than by a `splam` role, and it doesn't know which service you have selected. On a host where the `admin` and `auditor` split matters, don't treat the console as a read-only account. [Why the panel came first](02.04-explanation-chatbot-scope.qmd#why-a-panel-rather-than-a-terminal). ## Turn it off for everyone Don't install the extra. With `chatlas` absent there's no **Chat** tab, no `splam-chat` on the `PATH`, and no code path that reaches a model provider. For a locked-down deployment that's the whole control: leave it out of the install, and confirm it with one `pip list`. ## Further reading - [Chatbots in Shiny for Python](https://shiny.posit.co/py/docs/genai-chatbots.html), which the panel is built from - [Chatbots in chatlas](https://posit-dev.github.io/chatlas/get-started/chatbots.html), which documents the console the `splam-chat` command wraps - [Ollama](https://ollama.com) and its [model library](https://ollama.com/library), for local models beyond the two named here ### Configure Highlighting Task-oriented recipes. Each section assumes you already have `splam` installed and running. See the [Tutorial](00-tutorial-getting-started.qmd) if not. ## Add a custom highlight keyword Click **Configure** in the sidebar, scroll to **Highlight keyword**, enter a **Keyword or phrase**, choose a category under **Highlight as** (Error/red, Warning/amber, OK-success/cyan, Action tag/pink), and click **Save keyword**. The new keyword takes effect immediately on the Status, Logs, and Audit Trail tabs, with no redeploy needed. It's matched as a whole word (or exact phrase), case-insensitively, the same way the built-in keywords are. Saved keywords are stored in `highlight_overrides.json` in the app's data directory (override the location with `SPLAM_HIGHLIGHT_OVERRIDES`). ### Shell Audit Setup Task-oriented recipes. Each section assumes you already have `splam` installed and running. See the [Tutorial](00-tutorial-getting-started.qmd) if not. This page wires up real-time capture of `sudo` commands into incident markdown files. See [Shell Audit Feedback Loop](02.08-construction-shell-audit-feedback-loop.qmd) for why it's built this way. Everything here needs `auditd` and root, and none of it needs a pip extra: `redact`, `incidents`, and `shell_notify` ship with `splam` and use only the standard library. ## Add the audit rule ```bash sudo tee /etc/audit/rules.d/splam-shell.rules <<'EOF' -a always,exit -F arch=b64 -S execve -F euid=0 -F auid!=unset -k splam-shell EOF sudo augenrules --load ``` Confirm it loaded: ```bash sudo auditctl -l | grep splam-shell ``` ## Find the installed notify script `shell_notify.py` ships inside the `splam` package itself, next to `redact.py`. Find its real path once, from whatever environment runs `splam`: ```bash .venv/bin/python3 -c "import os, splam; print(os.path.join(os.path.dirname(splam.__file__), 'shell_notify.py'))" ``` Make it executable; it already has a `#!/usr/bin/env python3` shebang: ```bash sudo chmod +x /path/from/the/command/above/shell_notify.py ``` ## Wire up real-time dispatch Recent `auditd` (3.0+) reads this straight from `/etc/audit/plugins.d/`. Older systems route it through `audispd`, configured from `/etc/audit/auditd.conf`, but the plugin file is the same either way. Use the path the previous step printed: ```bash sudo tee /etc/audit/plugins.d/splam-shell.conf <<'EOF' active = yes direction = out path = /path/from/the/previous/step/shell_notify.py type = always format = string EOF sudo systemctl restart auditd ``` `shell_notify.py` finds `redact.py` by looking next to itself, which is enough as long as both stay inside the installed `splam` package where pip put them. If something ever separates them, `shell_notify.py` also checks `SPLAM_REDACT_MODULE`, an environment variable it reads with `os.environ`, not a command-line flag. The plugin config above has no way to set one, so point the plugin at a two-line wrapper instead of at `shell_notify.py` directly: ```bash sudo tee /usr/local/libexec/splam-shell-notify <-.md` is sitting in `incidents/`, with the command you just ran and the `journalctl` lines around it. Do this again after any change to the rule or the dispatch config; it's the whole checklist, and neither `auditd` nor `systemd` will tell you if you got a path wrong. ## Flag an incident for review For the ones you already know deserve a closer look before anyone treats them as final: ```bash .venv/bin/splam-incidents close "$(logname)" --review ``` Lands in `incidents/pending/` instead of `incidents/`. Move it (or edit it, then move it) once you've checked it: ```bash mv incidents/pending/2026-08-16T1430-sshd.service.md incidents/ ``` ## Change the idle-gap threshold Default is 600 seconds. It's read by `splam-incidents sweep`, so it belongs on the timer's service, not on `splam` itself: ```bash sudo systemctl edit splam-incidents-sweep.service ``` Add: ```ini [Service] Environment="SPLAM_INCIDENT_IDLE_GAP=900" ``` ```bash sudo systemctl daemon-reload ``` ## When it doesn't work | What you see | What it means | |---|---| | Buffer file never appears after a `sudo` command | Rule didn't load (`auditctl -l`) or the plugin path is wrong (`journalctl -u auditd`) | | `RuntimeError: could not locate redact.py` | `shell_notify.py` and `redact.py` aren't next to each other; set `SPLAM_REDACT_MODULE` | | Buffer filename doesn't match the account you expect | It's keyed on `auid`, the original login uid; if you `su` or `sudo -i` between sessions, `auid` follows the first login, not the current shell | | `incidents/` file exists but the `## Logs` section says `-- No entries --` | `journalctl` returned nothing for that service in that window; check the service name `correlate_and_write` detected matches the unit's real name | | Everything under `.buffer/` for the same user, always | `splam` itself is running interactively under that account rather than as its own systemd service; see the deployment note in [Shell Audit Feedback Loop](02.08-construction-shell-audit-feedback-loop.qmd#setting-it-up) | | `splam-incidents: command not found` | Reinstall with `.venv/bin/pip install -e .` so the `[project.scripts]` entry point registers | ## Further reading - [Shell Audit Feedback Loop](02.08-construction-shell-audit-feedback-loop.qmd), for why this is built this way - [How-To: Audit Trail](01.07-how-to-audit-trail.qmd), for what `splam` already records on its own - [How-To: Sudo Privileges](01.02-how-to-sudo-privileges.qmd), for the privilege `splam` itself needs to run service actions ## Explanation ### Login Sessions Background and reasoning behind design decisions in splam. This page won't tell you how to do something. See the How-To Guides for that. ## Why login doesn't persist across a refresh Authentication state (`authenticated`, `current_user`, `current_role`) lives in a per-session `reactive.value`, not a signed cookie. Refreshing the browser tab logs you out. This was a deliberate simplicity trade-off: a cookie-based session needs Starlette ASGI middleware and a secret key to sign it, which is more moving parts than a small internal admin tool needs to start. If the app grows beyond a handful of trusted operators, that's the first piece worth revisiting. ## Why the idle timeout is enforced on the server An unattended browser left on the admin panel is an unattended root console. After `SPLAM_IDLE_TIMEOUT` seconds without activity (15 minutes by default) the session is ended and the login form returns. The countdown runs in a `reactive.effect` that reschedules itself every 30 seconds with `reactive.invalidate_later`, and activity is recorded server-side by every button press and by changing the service, log length, or log search. A JavaScript timer in the page would have been less code. But anything the browser owns can be disabled by the browser. For the timeout to mean anything, the server has to hold the clock. The 30 second tick is why a logout can land up to 30 seconds after the timeout strictly elapses. The timeout is recorded in the audit trail as a logout with the result `idle timeout`, which distinguishes it from someone deliberately logging out. Getting in and staying in are two separate mechanisms, and the diagrams below 1) successful logins, 2) already locked, and 3) failed verification. ### Path 1: Successful Login ```{mermaid} %%| eval: true %%| fig-width: 4.5 %%| fig-height: 12.75 %%| fig-align: 'center' %%| fig-caption: 'Logins' %%| echo: false %%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"20px", "darkMode":true}}}%% flowchart TD subgraph Login["Login Attempt: Success"] Sub[/"username +
password"/] Locked{"locked?
(in-memory dict,
keyed on
username)"} Verify{"password
verified?"} Reset("fail counter reset") Begin(["session begins"]) end Trail[("audit_log.jsonl")] Sub ==> Locked Locked ==>|"no"| Verify Verify ==>|"yes"| Reset Reset ==> Begin Begin --> Trail style Login fill:#FFFFFF,color:#000000,rx:5,ry:5 style Sub fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Locked fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Verify fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Reset fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Begin fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Trail fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 ``` ### Path 2: Already Locked ```{mermaid} %%| eval: true %%| fig-width: 4.5 %%| fig-height: 9.25 %%| fig-align: 'center' %%| fig-caption: 'Logins' %%| echo: false %%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"20px", "darkMode":true}}}%% flowchart TD subgraph Login["Login Attempt: Locked"] Sub[/"username +
password"/] Locked{"locked?
(in-memory dict,
keyed on
username)"} Reject["rejected
password not checked"] end Trail[("audit_log.jsonl")] Sub ==> Locked Locked -->|"yes"| Reject Reject --> Trail style Login fill:#FFFFFF,color:#000000,rx:5,ry:5 style Sub fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Locked fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Trail fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Reject fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 ``` ### Path 3: Failed Verification ```{mermaid} %%| eval: true %%| fig-width: 5.75 %%| fig-height: 17.75 %%| fig-align: 'center' %%| fig-caption: 'Logins' %%| echo: false %%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"20px", "darkMode":true}}}%% flowchart TD subgraph Login["Login Attempt: Failure"] Sub[/"username +
password"/] Locked{"locked?
(in-memory dict,
keyed on
username)"} Verify{"password
verified?"} Incr("fail counter + 1") Max{"failures >=
MAX_FAILED_LOGINS
(default 5)"} Lock["account locked for
LOCKOUT_SECONDS
(default 15 min)"] end Trail[("audit_log.jsonl")] Sub ==> Locked Locked ==>|"no"| Verify Verify -->|"no"| Incr Incr --> Max Max -->|"yes"| Lock Max -.->|"no"| Sub Lock --> Trail style Login fill:#FFFFFF,color:#000000,rx:5,ry:5 style Sub fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Locked fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Verify fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Trail fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Incr fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 style Max fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 style Lock fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 ``` The lockout check comes first, before the password is verified. A locked account is rejected whether or not the password was right, which is what makes the lock worth having. ## Why lockouts live in memory Five consecutive failed logins lock an account for 15 minutes (`SPLAM_MAX_FAILED_LOGINS`, `SPLAM_LOCKOUT_SECONDS`). The counter is a module-level dict in `auth`, shared across browser sessions but not across restarts. Persisting it would mean another file on disk that the login path has to write on every failure. That's an unauthenticated write, which is exactly what you don't want on the login path. The audit log already records every failed attempt durably, so the persistent evidence exists either way; what's lost on restart is only the enforcement window. Locking is keyed on the submitted username, so a wrong username locks out just as a wrong password does and the form gives an attacker no signal about which accounts exist. ## Why there are only two roles `admin` and `auditor`, and nothing in between. The distinction that actually matters for an internal tool is whether an account can change the state of a service. Anything finer (per-service permissions, an approver role) implies a permissions model, a UI to manage it, and a review process: worth building when someone asks for it, not before. Accounts created before roles existed have no `role` field and are read as `admin`, so adding this didn't lock anyone out. ### Reason Required Background and reasoning behind design decisions in splam. This page won't tell you how to do something. See the How-To Guides for that. ## Why a Reason is required for service actions splam assumes its operator works in a regulated (pharma/biotech) environment, where changes to systems that touch validated data need to be traceable: who did what, when, and why. Requiring a reason before Start/Stop/Restart, and recording it alongside the result in the audit trail, gives you a durable record you can correlate with a change ticket without relying on operators remembering to document things separately. ### Audit Trail Scope Background and reasoning behind design decisions in splam. This page won't tell you how to do something. See the How-To Guides for that. ## What the audit trail does and doesn't capture The audit trail (`audit_log.jsonl`) records every Start/Stop/Restart *attempted* through this app: timestamp, user, service, action, reason, and result. Attempts rejected by the app (an unknown service, a missing reason, or an auditor account attempting an action) are recorded too, with a `rejected:` result, so the trail shows what was tried and not only what succeeded. Login and logout attempts are recorded the same way: failed logins carry the username that was submitted, lockouts are recorded once an account has failed too many times, and a session ended by the idle timeout is recorded as a logout with the result `idle timeout`. The boundary is the app itself, not the machine. Everything that reaches `audit_log.jsonl` passed through a form in this app; everything that didn't, didn't. ```{mermaid} %%| eval: true %%| fig-width: 6 %%| fig-height: 10.5 %%| fig-align: 'center' %%| fig-caption: 'Audit trails' %%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"20px", "darkMode":true}}}%% flowchart LR subgraph Cap["Captured"] Svc["Start / Stop / Restart
(rejected or not)"] Login["login attempts
success, failure, lockout"] Out["logout
deliberate or idle timeout"] end Log[("audit_log.jsonl")] subgraph Not["Not captured"] Shell["systemctl from a shell,
other tools"] Conf["config edits: crontab,
sshd_config, firewall rules"] end Svc ==> Log Login ==> Log Out ==> Log Shell --x Log Conf --x Log style Svc fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Login fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Out fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Log fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Shell fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 style Conf fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 style Cap fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:10,ry:10 style Not fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:10,ry:10 ``` Of the two blocked paths, configuration changes are the one worth naming twice. Several `SERVICE_INFO` entries call it out explicitly, because it's the gap admins are most likely to assume is covered when it isn't. Treat it as a record of *this app's* actions, not a general-purpose system audit log. That's what `auditd` and centralized syslog forwarding are for (see the `auditd.service` and `rsyslog.service` entries on the Tasks tab). ### Chatbot Scope Background and reasoning behind design decisions in `splam`. This page won't tell you how to do something. See the How-To Guides for that. ## Why a chatbot at all The **Tasks** tab already answers "what does this service do and what should I check." It answers it the same way every time, because it's a static document. What it can't do is answer "why is *this* status output saying *that*," which is the question an admin actually has at 2am with a failed unit in front of them. That's the gap. A model holding the deployment's own admin-task notes can read a specific `journalctl` dump and say something useful about it. Everything else on this page is about keeping that capability inside a boundary the deployment can live with. ## Why a panel rather than a terminal A chat panel and a terminal chatbot cost about the same to build, and the panel was still the right one to build first. Dependencies don't separate them. The chat widget is `shinychat`, which `shiny` already requires, so the panel adds `chatlas` and nothing else. A console adds `chatlas` and nothing else. Neither is meaningfully more code than the other, because `shinychat`'s `Chat` accepts a `chatlas` client directly and wires up streaming and history itself. What separates them is what each one already knows. The panel runs inside a session that has authenticated a user, resolved their role, and has a service selected in the sidebar. A console starts with none of that and can't get it: there's no login step, so it's bounded by the file permissions of whoever ran it. In a deployment that bothered to distinguish `admin` from `auditor`, a second entry point that ignores the distinction is a gap rather than a feature. The console remains worth shipping as a secondary interface, for a host with no browser or an admin who lives in a terminal. It just isn't the one to design around. ## Why it's off by default `chatlas` is an optional dependency and `splam` never imports it unless you installed it. That isn't tidiness. It's the only control a validated environment can verify cheaply: if the package isn't in the environment, there's no code path from `splam` to a model provider, and confirming that takes one `pip list` rather than a code review. Every other guard on this page depends on the app behaving as designed. This one doesn't. Installing the extra and configuring a provider are separate steps, and the panel distinguishes them. With `chatlas` absent there's no tab, because the deployment chose not to have the feature. With `chatlas` present but no provider reachable, the tab appears and explains what's missing. An earlier version hid the tab in both cases, which was wrong: someone who had just run the install had no way to tell a deliberate omission from a broken setup, and a missing tab reads as a bug rather than as a message. ## What leaves the machine ```{mermaid} %%| eval: true %%| fig-width: 7.75 %%| fig-height: 4.75 %%| fig-align: 'center' %%| fig-caption: 'What leaves the machine' %%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"20px", "darkMode":true}}}%% flowchart TD Q[/"your question
selected service"/] Notes[("admin-task notes")] Cmd[["Chat panel"]] Local[["local model
via ollama"]] Hosted[["hosted model"]] Act["start, stop, restart"] Q --> Cmd Notes --> Cmd Cmd ==>|"default:
stays on this host"| Local Cmd -.->|"opted in:
leaves this host"| Hosted Cmd --x|"never"| Act style Q fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Notes fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Cmd fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Local fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style Hosted fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 style Act fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 ``` Whatever the panel sends the model, the model's operator receives. With `SPLAM_CHAT_PROVIDER=ollama` that operator is the host itself and the question is closed. With a hosted provider it isn't, and the material at stake is worse than it first looks: `journalctl` output carries hostnames, usernames, internal IPs, file paths, batch identifiers, and occasionally a secret that got logged by accident. Sending a hundred lines of it to a hosted model is a disclosure of all of that, made by an admin at a text box, without a ticket. So `ollama` is the default. Not because local models answer better, they often don't, but because the default should be the setting that can't surprise you. Choosing a hosted provider takes an explicit environment variable and an API key, which is roughly the friction the decision deserves. Putting the chat inside the app raises the stakes here rather than lowering them. A panel sitting next to the **Logs** tab makes pasting log output the obvious next gesture, and with `SPLAM_CHAT_TOOLS` set the model fetches it without being asked. That's the point of the feature and also its main risk, which is why tools are off unless someone deliberately turns them on, and why that switch belongs to whoever starts the app rather than to whoever is typing in the panel. Per-call approval was considered and dropped. Approving a tool call from the UI means blocking mid-stream on a round trip to the browser, and the honest version of that prompt would appear several times in a row for a question that needs status and logs together. Operators would learn to click through it, which is worse than no prompt at all because it looks like a control. Enabling tools at startup is one decision, made once, by the person who also chose the provider. ## Why it reads but never acts `chatlas` can register any Python function as a tool, and `splam` already has `systemctl_action` sitting right there. Wiring it up would take one line. It stays unwired, and the four tools offered are all read-only: `get_status`, `get_logs`, `get_service_info`, and `read_audit_log`. The reason isn't that models are unreliable, though they are. It's that every service action in this app carries a **Reason** typed by a person, and that requirement is the point of the audit trail rather than a form validation quirk. See [Explanation: Reason Required](02.02-explanation-reason-required.qmd). A model-issued restart would produce an audit entry whose reason was written by the thing being audited. The record would look complete and mean nothing, which is worse than an obvious gap. The panel makes this line easy to hold, because **Actions** is a separate tab with its own buttons and its own required field. Nothing about a chat box invites you to expect a restart from it. ## Why a chat session isn't audited Using the **Chat** panel writes nothing to `audit_log.jsonl`. Reading isn't an audited event anywhere else in this app either: viewing the **Logs** tab leaves no entry, and neither does opening the audit trail itself. The log records attempts to change something, and a conversation changes nothing. **Export conversation** exists for the other case, when the conversation informed a change you'll have to justify later. That's a transcript you chose to keep, filed wherever your change control lives. It's deliberately not the same artifact as the audit trail, which is the app's own record of what it did. See [Explanation: Audit Trail Scope](02.03-explanation-audit-trail-scope.qmd). ## What the panel inherits Building inside the session means the chat gets four things for free that a console would have to invent: the authenticated username, the role, the selected service, and the idle timeout. The last one matters more than it sounds. Sessions are ended server-side after an idle period, and a chat reply that takes ninety seconds to stream shouldn't count as idleness, so typing in the panel marks the session active the same way changing the **Service** dropdown does. See [Explanation: Login Sessions](02.01-explanation-login-sessions.qmd). That inheritance is the whole argument for the panel. It's also the reason the panel can be shown to `auditor` accounts without a second thought: the tools are read-only, the role is already known, and an auditor asking questions about logs is the job rather than a loophole. ### Config Overrides Background and reasoning behind design decisions in splam. This page won't tell you how to do something. See the How-To Guides for that. ## Why admin-task notes and highlight keywords are a code + override pair `SERVICE_INFO` in `service_info.py` and `_PATTERNS` in `highlight.py` are plain Python data structures. That keeps the app dependency-free (no database, no migration), makes the shipped defaults reviewable as an ordinary code change, and lets a `SERVICE_INFO` entry be shared across multiple unit-name aliases (`ssh.service` / `sshd.service`) without duplication. Changes made through the **Configure** button don't touch that code. They're written to two JSON files in the app's data directory (`service_info_overrides.json` and `highlight_overrides.json`), which are merged on top of the built-in data at read time (override wins on a key collision). This gives operators a way to add or correct guidance and highlighting per deployment, without a code change or redeploy, while keeping the reviewed defaults in version control. The tradeoff: overrides live on the instance that made them and aren't automatically shared across deployments. Promote a useful override into the code (`SERVICE_INFO` or `_PATTERNS`) if it should ship by default. Nothing is merged ahead of time. The two halves stay separate on disk and come together on each read, which is why an override takes effect without a restart. ```{mermaid} %%| eval: true %%| fig-width: 13.5 %%| fig-height: 7.75 %%| fig-align: 'center' %%| fig-caption: 'Code + override pair' %%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"18px", "darkMode":true}}}%% flowchart TD subgraph Code["Code"] SI["SERVICE_INFO
service_info.py"] PT["_PATTERNS
highlight.py"] end subgraph Ovr["Overrides"] SIJ[("service_info_overrides.json")] PTJ[("highlight_overrides.json")] end Btn(["Configure button"]) Merge{"merge at read time
override wins"} OutA[\"Tasks tab guidance"\] OutB[\"highlighted Status /
Logs text"\] Btn --> SIJ Btn --> PTJ SI ==> Merge PT ==> Merge SIJ ==> Merge PTJ ==> Merge Merge ==> OutA Merge ==> OutB style SI fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5,font-family:monospace style PT fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5,font-family:monospace style Merge fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style OutA fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style OutB fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5 style SIJ fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5,font-family:monospace style PTJ fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5,font-family:monospace style Btn fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5 style Code fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:10,ry:10 style Ovr fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:10,ry:10 ``` ## Where the override files live The data directory resolves from `SPLAM_DATA_DIR`, else `XDG_DATA_HOME`, else `~/.local/share/splam` (see `paths.py`). Everything the app writes at runtime lands there: ``` / ├── credentials.json ├── audit_log.jsonl ├── service_info_overrides.json └── highlight_overrides.json ``` The two overrides files only exist once **Configure** has been used at least once. They aren't created at install time, so a deployment that never opened the dialog has a data directory with two files in it and behaves entirely off the shipped defaults. ### Package Layout Background and reasoning behind design decisions in splam. This page won't tell you how to do something. See the How-To Guides for that. ## Package layout The app is packaged with a `src/splam/` layout rather than flat scripts, which was necessary for `great-docs` (and any other tool doing static analysis) to treat it as an importable package rather than loose files. ``` src/splam/ ├── __init__.py ├── app.py ├── auth.py ├── audit.py ├── chat.py ├── highlight.py ├── manage_users.py ├── paths.py ├── service_info.py └── styles.css ``` The module split follows single responsibility: - `app.py`: Shiny UI/server wiring, plus the `systemctl`/`journalctl` wrapper functions. - `auth.py`: password hashing, verification, and login attempt/lockout state. - `audit.py`: the append-only action log. - `service_info.py`: the per-service guidance data (`SERVICE_INFO`) and its overrides. - `highlight.py`: keyword highlighting for the Status/Logs/Audit Trail output (`_PATTERNS`), and its overrides. - `chat.py`: chatbot client construction, system prompt, and the read-only tool set (`build_client`, `chat_available`, `read_only_tools`, `system_prompt`). See [Explanation: Chatbot Scope](02.04-explanation-chatbot-scope.qmd). - `manage_users.py`: the `splam-users` CLI entry point (`add`/`list` subcommands) for managing accounts in `credentials.json` outside the running app. - `paths.py`: resolves the data directory (`data_dir()`: `SPLAM_DATA_DIR`, else `XDG_DATA_HOME`, else `~/.local/share/splam`) and enforces owner-only permissions on it (`ensure_parent()`). ## The `highlight` name shadowing quirk `__init__.py` re-exports a function called `highlight` from `highlight.py`: ```python from .highlight import add_highlight_keyword, highlight, highlight_term ``` Because the re-exported name is identical to the submodule's own name, this line rebinds the `splam.highlight` attribute from the submodule to the function. In other words, after `import splam`, `splam.highlight` is ``, not the module. The module is still reachable (e.g. via `importlib.import_module("splam.highlight")`, `sys.modules`, or `from splam import highlight as _; import splam.highlight as highlight_mod`), but plain attribute access no longer finds it. This is intentional/accepted, not a bug to fix. `highlight` is the natural public name for the function, and the tests already work around it (`tests/conftest.py`, `tests/test_highlight.py` both import the submodule via `importlib.import_module("splam.highlight")` with a comment explaining why). Just be aware of it if you add new tests or tooling that touches `splam.highlight`. It also affects `great-docs`: when `great-docs.yml`'s API reference config resolves the bare name `highlight`, dynamic introspection resolves it to the *submodule* (not the function) and, under `dynamic: true`, fails to filter out the module's own imports (`data_dir`, `ensure_parent` from `paths.py`) as aliases. Left alone, this leaks spurious pages into the API reference (`highlight.data_dir`, `highlight.ensure_parent`, plus the module's constants) and prints a "Dynamic introspection failed" warning during `great-docs build`. The fix lives in `great-docs.yml`, which overrides `dynamic: false` for just that one reference entry so static analysis (which correctly identifies the imports as aliases) is used instead: ```yaml reference: - title: Functions contents: - ... - name: highlight dynamic: false - highlight_term - ... ``` If you rename or restructure `highlight.py`, or add another submodule whose name collides with one of its own re-exported members, check whether this override is still needed (or needs to move). ### Terminal Look Background and reasoning behind design decisions in splam. This page won't tell you how to do something. See the How-To Guides for that. ## Why the terminal look The terminal chrome (`styles.css`, loaded via `ui.include_css`) is a presentation layer on top of Shiny's default Bootstrap styling. It doesn't change any behavior, only how the existing inputs, buttons, and output panes are rendered. The palette is a dark, near-black background with a purple foreground and a pink accent, a Dracula-style pairing rather than a straight green terminal. Status/Logs/Audit Trail output goes further: `highlight.py` escapes the raw text and wraps recognized keywords in colored ``s (error terms in red, warnings in amber, success terms in cyan, action terms in pink) before it's rendered as HTML. That's why those three tabs use `render.ui` with a hand-built `
` instead of Shiny's plain `render.text`. It's still just styling applied to the same underlying command output.



## Under Construction

### Shell Audit Feedback Loop

Background and reasoning behind design decisions in splam. This page won't tell you how to do something. See [How-To: Shell Audit Setup](01.13-how-to-shell-audit-setup.qmd) for that.

## The loop splam doesn't close

Almost every service incident an admin handles follows the same three steps: 1) diagnose (`grep`, `journalctl`, `systemctl status`), 2) fix (a shell command, almost always run with `sudo`), 3) confirm (restart or reload, then re-check the logs and status). [Explanation: Audit Trail Scope](02.03-explanation-audit-trail-scope.qmd) already draws the line around what `splam`'s own trail covers, and "`systemctl` from a shell" is explicitly on the *not captured* side of that line. *That's the fix step, and it's also the step worth keeping.*

The diagnose step leaves a trail already, in `journalctl` and whatever the Logs tab shows. The confirm step is usually re-running the same commands from the diagnosis. The fix is the one piece of the loop that's pure signal (i.e., *the specific command that turned a broken state into a working one*), and right now it lives nowhere but a shell history file (per-user, unstructured, not tied to the incident) or the admin's memory. 


`redact()`, `incidents.py`, `manage_incidents.py`, and `shell_notify.py` capture the fix step reliably, link it with its diagnose/confirm counterparts, and turn all three into something readable. That's a working-to-learning feedback loop instead of a one-time fix that has to be rediscovered next time.

## Candidate sources for shell activity

Three places on the OS actually see what ran; they're not equivalent.

### sudo logfiles

`sudo`'s own logfile (`Defaults logfile="/var/log/sudo.log"` in sudoers) is the simplest: one plain-text line per invocation, already the format we want. Its blind spot is anything that isn't a direct `sudo `: drop into `sudo -i` or an existing root shell and only the first line gets logged, not what's typed after. It's also self-defeating as a control: the same root access it's meant to record is enough to turn it off.

### sudo I/O logging

`sudo` I/O logging (`Defaults log_input,log_output` plus an `iolog_dir`) records the full terminal session and can be replayed with `sudoreplay`. It sees everything the logfile misses, but that's also the problem: a fix command sits in a transcript along with every `ls`, typo, and page through `less`. Useful for a full session replay if one incident ever needs it; too noisy to be the primary source for a curated record.

### asciinema

`asciinema rec` is the same idea as `sudo` I/O logging, a full PTY transcript, in different packaging, and it inherits the same noise problem. It also loses ground on three points that matter here. It's manually invoked, with no built-in way to force it for every session; the closest the community gets is wrapping a login shell, which is openly documented as bypassable by switching shells (the same reasoning that already put `auditd` ahead of the `sudo` logfile). Its only redaction is a live mute hotkey the admin has to remember to press before typing something sensitive, the opposite of not depending on anyone remembering anything. And its output, asciicast, is JSON Lines with raw terminal escape sequences embedded in each event, not plain text and not markdown; it needs a player to be readable at all, which was the original problem this whole page exists to solve. It adds a third-party tool for a capability `sudo` I/O logging already covers, and covers less reliably.

### auditd

`auditd`, watching `execve` at the kernel level (`-a always,exit -F arch=b64 -S execve -F euid=0 -k splam-shell`), sees every process a `root`-privileged user execs, regardless of whether it went through `sudo`, `su`, or a shell that was already `root`. It's also the one of the three that's already load-bearing in `splam`: the **Tasks** tab's own `auditd.service` notes call it out as *the technical control usually satisfying 21 CFR Part 11 audit-trail requirements at the OS level, and say changes to its rules should go through change control rather than a self-review.* That's the tamper-resistance argument for using it here too, because disabling `auditd` is itself the kind of event that's supposed to get noticed. The downside is that `audit.log` is dense and keyed for machines, not admins, so it needs `ausearch -k splam-shell` (or direct parsing) before a human (or a parser) can make sense of it.

`auditd` is the source of record. The `sudo` logfile is worth turning on too, as a second, much simpler line to cross-check against. `sudo` I/O logging and `asciinema` both stay off; if full session replay is ever actually needed, `sudo` I/O logging is the one to reach for, since it's already the lower-friction option of the two.

```{mermaid}
%%| eval: false
%%| fig-width: 11.25
%%| fig-height: 15.5
%%| fig-align: 'center'
%%| fig-caption: 'Diagnose, fix, confirm'
%%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"18px", "darkMode":true}}}%%

flowchart TB
 subgraph Diag["Diagnose"]
        Look(["logs +
systemctl status"])
  end
 subgraph Fix["Fix"]
        Shell("shell command
(usually sudo)")
  end
 subgraph Conf["Confirm"]
        Restart("restart / reload")
        Recheck{"logs +
systemctl status
clean?"}
  end
    Look --> Journal[("journalctl")]
    Look ==> Shell
    Shell ==> Audit[("auditd
(execve, key-tagged)")] & Restart
    Shell -.-> SudoLog[("sudo logfile
(cross-check)")]
    Restart -.-> AppTrail[("audit_log.jsonl
(app-mediated only)")]
    Restart ==> Recheck
    Recheck -. "no" .-> Shell
    Recheck --> Journal
    Audit ==> Parser[["Parser
manage_incidents.py"]]
    Journal ==> Parser
    SudoLog -.-> Parser
    AppTrail -.-> Parser
    Parser == "correlate + write" ===> Out[/"markdown
incident
file"/]

    style Look fill:#FFCDD2,color:#000000,stroke:#D50000
    style Shell fill:#FFE0B2,color:#000000,rx:5,ry:5,stroke:#FF6D00
    style Restart fill:#C8E6C9,color:#000000,rx:5,ry:5,stroke:#00C853
    style Recheck fill:#C8E6C9,color:#000000,rx:5,ry:5,stroke:#00C853
    style Journal fill:#11a4e3,color:#FFFFFF,rx:5,ry:5,font-size:18px,text-align:left,stroke:#757575
    style Audit fill:#11a4e3,color:#FFFFFF,rx:5,ry:5,stroke:#757575
    style SudoLog fill:lightgrey,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5
    style AppTrail fill:lightgrey,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5
    style Parser fill:#FFE0B2,color:#000000,rx:5,ry:5,font-size:18px,stroke:#000000
    style Out fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Diag fill:#424242,color:#ffffff,stroke:#FFFFFF,stroke-width:1px,rx:10,ry:10
    style Fix fill:#424242,color:#ffffff,stroke:#FFFFFF,stroke-width:1px,rx:10,ry:10
    style Conf fill:#424242,color:#ffffff,stroke:#FFFFFF,stroke-width:1px,rx:10,ry:10

```

![Diagnose, fix, confirm](images/shell-audit-feedback-loop-diagnose-fix-confim.png)

The dotted lines mark what's optional or secondary: the `sudo` logfile is a cross-check, not the source of record; `audit_log.jsonl` only has something to contribute when the confirm step happened to go through `splam` itself rather than a bare `systemctl` on the shell.

## What runs, and when

Reading `audit.log` after the fact (`ausearch -k splam-shell`) would work, but it means polling for something that already happened. `auditd` can do better than that: it supports dispatching matched events to a script in real time as they occur, which turns capture from a periodic sweep into something that happens the moment a tagged command runs. That splits the "parser" from the diagram above into two pieces with very different jobs: something tiny that fires on every event (capture), and something slower that decides when a run of events is done and writes it up (correlate-and-write, referred to as "the parser" for the rest of this page).

The two flowcharts below each cover one piece in isolation. What they don't show is the timing between them, an admin's commands arriving one at a time while the buffer just sits there, until something (a timer, or the admin) decides the window is over:

```{mermaid}
%%| eval: false
%%| fig-width: 14.5
%%| fig-height: 9.25
%%| fig-align: 'center'
%%| fig-caption: 'Capture and flush over time'
%%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"20px", "darkMode":true}}}%%

sequenceDiagram
    actor Admin
    participant Audit as auditd
    participant Notify as notify
script
    participant Buffer as buffer
file
    participant Sweep as sweep
timer
    participant Parser as correlate +
write

    loop each privileged
command
        Admin->>Audit: runs command
(execve)
        Audit->>Notify: dispatch,
key=splam-shell
        Notify->>Buffer: append redacted
line
        Notify->>Notify: reset idle
timer
    end

    alt idle gap
elapses
        Sweep->>Buffer: check idle
time
        Sweep->>Parser: flush
    else admin closes
early
        Admin->>Parser: splam-incidents
close
    end

    Parser->>Parser: correlate against
journalctl +
audit_log.jsonl
    Parser-->>Buffer: clear
    Parser-->>Admin: markdown incident
file written
```

![Capture and flush over time](images/shell-audit-feedback-loop-seq-diag.png){width="100%"}

Everything above the `alt` can repeat any number of times, for any number of commands, before either branch fires exactly once to close the window.

### Capturing in real time

Every `execve` that matches the `splam-shell` rule gets dispatched to `shell_notify.py` the moment it happens. `auditd` spawns it once, at daemon startup, and streams matching records to its `stdin` for the life of the process (it isn't re-invoked per event). `shell_notify.py` has one job and stays small: append the record to a per-user buffer file, and reset an idle timer for that user. Nothing in the fast path resolves a service, reads `journalctl`, or writes markdown. `auditd`'s dispatch is synchronous, and a slow script in that path backs up event delivery for every rule, not just this one.

`auditd` splits one `execve` into a `SYSCALL` record, carrying `auid` and `key`, and an `EXECVE` record, carrying the argv, sharing a `msg=audit(timestamp:serial):` id but arriving as separate lines in no guaranteed order. `shell_notify.py`'s `Correlator` holds each id's pieces in memory until both have arrived, drops anything whose `SYSCALL` record isn't tagged `key="splam-shell"` as soon as that's known, and ages out whatever's left after 5 seconds so an event whose other half never shows up doesn't sit there for the life of the process.

Each line is `` tab ``. Nothing fancier: plain text stays greppable if someone opens the buffer directly, and it's the same shape correlate-and-write parses back apart later.

"Per-user" here means auditd's `auid`, the login uid that started the session, not `euid` or `uid`. Everything running under `sudo` has `euid=0` regardless of who ran it, so keying on `euid` would merge every admin's commands into one indistinguishable bucket. `auid` survives `sudo`, which is the only reason `.buffer/.log` can mean anything.

```{mermaid}
%%| eval: false
%%| fig-width: 6.25
%%| fig-height: 10
%%| fig-align: 'center'
%%| fig-caption: 'Real-time capture'
%%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"20px", "darkMode":true}}}%%

flowchart TD
    Cmd(["privileged
command
runs"])
    Rule[("auditd rule
(execve, euid=0,
login only)")]
    Plugin[["real-time dispatch
plugins.d / audispd"]]
    Script[["notify script
(tiny, synchronous)"]]
    Buffer[("buffer file
(.buffer/<user>.log)")]
    Reset(["idle timer
reset"])

    Cmd ==> Rule
    Rule ==> Plugin
    Plugin ==> Script
    Script ==> Buffer
    Script --> Reset

    style Cmd fill:#FFCDD2,color:#000000,stroke:#D50000
    style Plugin fill:#FFE0B2,color:#000000,rx:5,ry:5,stroke:#FF6D00
    style Script fill:#C8E6C9,color:#000000,rx:5,ry:5,stroke:#00C853
    style Buffer fill:#11a4e3,color:#FFFFFF,rx:5,ry:5,font-size:18px,stroke:#757575,text-align:center
    style Rule fill:#11a4e3,color:#FFFFFF,rx:5,ry:5,stroke:#757575,text-align:center
    style Reset fill:#4CBB9D,color:#FFFFFF

```

![Real-time capture](images/shell-audit-feedback-loop-real-time-capture.png){width="70%" fig-align='center'}

### Deciding when an incident is done

Real-time capture only answers *when a command gets recorded*, not *when a run of them becomes one incident*. That boundary decision is a hybrid: idle-gap by default, with an explicit marker to cut a window short on demand.

A periodic sweep (`splam-incidents sweep`, on a short systemd timer) checks each user's buffer against `SPLAM_INCIDENT_IDLE_GAP`: quiet that long, and the buffer is closed and correlated. No admin has to do anything for this to work, which matters, because most fixes end with a restart and a log check, not a deliberate "I'm done now" gesture. But the boundary is still a guess: a slow investigation can look like two incidents, and two quick unrelated ones close together can look like one.

`splam-incidents close ` is the escape hatch: an explicit, on-demand flush that doesn't wait for the idle gap, for the times the guess would be wrong and an admin knows the exact moment the fix ended.

```{mermaid}
%%| eval: false
%%| fig-width: 14.25
%%| fig-height: 12
%%| fig-align: 'center'
%%| fig-caption: 'Closing the window'
%%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"18px", "darkMode":true}}}%%

flowchart TD
    Buffer[("buffer
file")]
    Sweep("splam-incidents
sweep timer,
every 60s")
    Gap{"buffer idle
> threshold?"}
    Close[/"splam-incidents
close explicit,
on demand"/]
    Parser[["correlate +
write"]]
    Journal[("journalctl")]
    AppTrail[("audit_log.jsonl")]
    Out[\"markdown
incident
file"\]
    Cleared(["buffer
cleared"])

    Buffer --> Sweep
    Sweep --> Gap
    Gap -.->|"no"| Buffer
    Gap ==>|"yes"| Parser
    Close ==> Parser
    Journal -.-> Parser
    AppTrail -.-> Parser
    Parser ==> Out
    Parser --> Cleared

    style Gap fill:#FFCDD2,color:#000000,stroke:#D50000
    style Close fill:#C8E6C9,color:#000000,rx:5,ry:5,stroke:#00C853
    style Parser fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Out fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Buffer fill:#11a4e3,color:#FFFFFF,rx:5,ry:5,stroke:#757575
    style Sweep fill:#FFE0B2,color:#000000,rx:5,ry:5,stroke:#FF6D00
    style Journal fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5
    style AppTrail fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5
    style Cleared fill:#4CBB9D,color:#FFFFFF
    style Journal fill:#11a4e3,color:#FFFFFF,rx:5,ry:5,text-align:left,stroke:#757575
    style AppTrail fill:lightgrey,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5
    style Parser fill:#FFE0B2,color:#000000,rx:5,ry:5,stroke:#000000
    style Out fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5

```

![Closing the Window](images/shell-audit-feedback-loop-closing-window.png){width='100%' fig-align='center'}

Either path lands in the same place: correlate the buffered commands against `journalctl` for the affected service and against splam's own `audit_log.jsonl` in that window, write one markdown file (service, time range, commands run, the log lines immediately around them), and clear the buffer. Nothing here needs a database: `manage_incidents.py`'s `_fetch_journal` shells out to `journalctl -u  --since --until` for that window (skipped entirely when `detect_service` came back `unknown`), and `_fetch_audit_entries` filters `audit_log.jsonl` down to the same window by timestamp. `splam-incidents sweep` is a timer firing periodically, not a process that sits running.

The buffer is per-user, not per-service, so nothing hands correlate-and-write the service name directly. It scans the buffered commands for the last `systemctl  .service` invocation and uses that, on the assumption that the command closest to the confirm step is the one that names what got fixed; a buffer with no `systemctl` call in it at all gets filed under `unknown`. That's a heuristic, not a guarantee, and it's wrong for a fix that never calls `systemctl` directly, e.g. a config edit someone else restarts later from a different session.

## Setting it up

Three pieces: the audit rule, real-time dispatch (`plugins.d` on recent `auditd`, `audispd` on older systems), and a `splam-incidents-sweep.timer` running `splam-incidents sweep` every 60 seconds. See [How-To: Shell Audit Setup](01.13-how-to-shell-audit-setup.qmd) for the actual commands and unit files; what follows here is why two details in that setup matter.

`-F auid!=unset` on the audit rule matters more than it looks. `systemctl_action`, `get_status`, and `get_logs` in `app.py` all shell out with no `sudo` prefix (`splam` has to already run with the privilege it needs), so every Status check, Logs fetch, and Actions-tab restart is *also* an `execve` at `euid=0`. Without this filter, splam's own routine calls would match the rule too. Run `splam` as a proper systemd service (`User=splam`, no interactive login) and those calls carry no login session, so `auid` comes back unset and the rule excludes them at the source; only a real `sudo` session from an actual login gets tagged. This is one more reason to prefer the systemd-service deployment over running `splam` interactively from an admin's own shell: do that instead, and the admin's own clicks in the web UI inherit their real `auid` and end up mixed into their own SSH-session buffer along with their actual fixes.

`shell_notify.py` ships inside the `splam` package itself, next to `redact.py`, and the dispatch plugin's `path` points straight at that installed file rather than a separate copy somewhere like `/usr/local/libexec/`. That's not just convenience: `load_redact()` finds `redact.py` by looking next to itself on disk, so keeping the two files together, wherever pip happens to install them, is what makes that lookup work without either file needing to know the other's absolute path in advance.

## Redacting before anything is written

Deferring redaction to the correlate-and-write step, which an earlier draft of this page assumed, means the buffer sits on disk holding whatever ran, unredacted, for however long the window stays open. That's backwards. The cheapest place to strip a secret is the same place the command is first seen: the notify script, on the way into `.buffer/.log`, rather than later in the parser. One place for this to go wrong instead of two, and the buffer never holds anything the markdown file wouldn't also be allowed to show.

The rules are a denylist of shapes, not a smart parser: a table of `(name, regex)` pairs, the same shape as `_PATTERNS` in `highlight.py`. A flag and the value that follows it (`-p`, `--password`, `--token`, `--api-key`, `-u user:pass`), an env-var-style assignment (`\w*(SECRET|TOKEN|PASSWORD|KEY)\w*=\S+`), credentials embedded in a URL (`://user:pass@`). Each match becomes `[REDACTED]` in place; nothing else in the line changes, so a redacted command is still readable as a command, just missing the part that shouldn't be repeated.

The two ways this rule set can be wrong don't cost the same. A false positive (redacting a UUID or a commit SHA that happened to look secret-shaped) makes one line less useful. A false negative leaves a real credential sitting in a file meant to be read casually. When a pattern is close, it should redact. That bias belongs in how the rules are written, not in a comment explaining it afterward.

Same code-and-override split as `SERVICE_INFO` and `_PATTERNS`: a default pattern set ships in code, and `redaction_overrides.json` in the data directory lets a deployment add its own (an internal token format, an in-house secret naming convention) without a code change. See [Explanation: Config Overrides](02.05-explanation-config-overrides.qmd) for why that split exists.

`redact.py` imports nothing beyond the standard library, on purpose: `load_redact()` loads it by file path rather than `import`, precisely so the notify script's one `redact()` call per correlated command never pulls in the rest of `splam` (`shiny`, `chatlas`, everything the app needs to run). Importing the package normally would blow the tiny, synchronous budget from [Capturing in real time](#capturing-in-real-time) before the regex ran.

### A review gate for the ones that matter

No pattern list is exhaustive, and this one won't be either: it catches known shapes, not a secret that doesn't look like one. `splam-incidents close  --review` is the second layer, for an admin who already knows a fix touched something the patterns might miss: it writes to `incidents/pending/` instead of `incidents/`, and the file only moves into the reviewed directory once someone confirms or edits it by hand. Nothing about this is required, which is the same tradeoff the explicit marker already makes (exact, but only if used), so it stays additive rather than something the whole feature depends on.

`--review` would also be the natural place for a full session transcript to earn a spot, on the same opt-in terms: an admin who already knows the redacted command list isn't enough could ask for an `asciinema` recording of that one window. That's not built. [asciinema](#asciinema) was ruled out as the general-purpose capture mechanism, not as a tool worth never touching, and `--review` is the boundary a narrower, opt-in use of it would have to stay inside if it's ever added.

## Where the records live

Same data directory as everything else `splam` already writes (`paths.py`: `SPLAM_DATA_DIR`, else `XDG_DATA_HOME`, else `~/.local/share/splam`), in subdirectories so the incident files don't mix with the JSON the app itself owns. `SPLAM_INCIDENTS_DIR` and `SPLAM_BUFFER_DIR` override the two locations individually, the same convention as `SPLAM_AUDIT_LOG` and `SPLAM_HIGHLIGHT_OVERRIDES`:

```bash
/
├── credentials.json
├── audit_log.jsonl
├── service_info_overrides.json
├── highlight_overrides.json
├── redaction_overrides.json
├── incidents/
│   ├── 2026-08-15T1430-sshd.service.md
│   └── pending/
│       └── 2026-08-15T1512-nginx.service.md
└── .buffer/
    └── mfrigaard.log
```

`.buffer/` holds one file per user with an open, not-yet-closed window; it never has more than one file per active user at a time, and each file is deleted the moment its incident is written. `incidents/pending/` only exists once `--review` has been used at least once, same as the two existing overrides files only exist once **Configure** has been used.

Whether those files stay local operational data or get promoted somewhere durable (a runbook repo, a wiki) is the same open question [Explanation: Config Overrides](02.05-explanation-config-overrides.qmd) already raises about overrides: this app can write the file, but it can't decide for an organization whether that file belongs in version control.

## What this solves and doesn't

It puts the fix step next to the diagnose and confirm steps that already have a trail, so the next admin who hits the same failure has the actual command instead of a description of one. It does not replace `auditd` as the system of record. The markdown file is a curated derivative of `audit.log`, not a substitute for it, and `audit.log` stays the thing an auditor asks for.

Redaction runs at capture time now, in the notify script, so `.buffer/.log` and the markdown file never hold the raw form of anything the pattern list recognizes (see [Redacting before anything is written](#redacting-before-anything-is-written)). `audit.log` itself stays untouched; that's `auditd`'s own record, already under separate access control, and redacting it isn't this feature's call to make. What's left is narrower than "a raw secret in a casually-read file": a secret shape the pattern list doesn't recognize, slipping through into one. That's what the `--review` gate is for: not a promise the pattern list is complete, an admission that it can't be.

## Testing this

What's on this page splits cleanly into two kinds of thing to test, and they don't belong in the same suite.

### What's unit-tested

`redact()`, `is_idle`, `correlate_and_write`, and the pure pieces of `shell_notify.py` are all tested without a live `auditd`, a real `sudo` session, or a timer:

* `tests/test_redact.py`: table-driven, one case per pattern (`-p secret123` → `-p [REDACTED]`, `AWS_SECRET_ACCESS_KEY=xyz` → `AWS_SECRET_ACCESS_KEY=[REDACTED]`, `https://user:pw@host` → `https://[REDACTED]@host`) plus a set of things that survive untouched (`--port 8080`, `systemctl restart nginx`, a bare UUID), so the false-positive bias from [Redacting before anything is written](#redacting-before-anything-is-written) stays a test, not just a sentence on this page.
* `tests/test_incidents.py`: `is_idle` against an injected clock, no `sleep()` in sight; `correlate_and_write` against a fixture buffer, fixture `journalctl` lines, and a fixture `audit_log.jsonl`, asserting the markdown file's service, time range, and sections; `detect_service`, `parse_buffer`, `buffer_path`, `list_buffered_users`, and `clear_buffer` each get their own direct tests.
* `tests/test_shell_notify.py`: `event_id`, `field`, and `parse_argv` against realistic `SYSCALL`/`EXECVE` line fixtures; `Correlator.feed` fed those same lines in both orders, with a non-matching `key` and with unrelated interleaved events, plus a clock-injected test for the 5-second prune; `append_to_buffer` against a temp directory; `load_redact` against the real, on-disk `redact.py`.

This is also where most of the coverage sits. It's cheap to run, it's where a subtle bug is likeliest (a regex that almost matches, an off-by-one on the idle window, a `SYSCALL`/`EXECVE` correlation edge case), and none of it needs anything CI doesn't already have.

### What needs a live system

The `auditd` rule, the `plugins.d`/`audispd` dispatch wiring, and `shell_notify.py` actually being invoked by a real privileged command aren't things `pytest` can exercise without a rooted host running `auditd`, which most CI containers don't have and shouldn't need for this feature to ship. That path gets the smoke-test checklist in [How-To: Shell Audit Setup](01.13-how-to-shell-audit-setup.qmd#confirm-its-working) instead of an automated suite: run a known `sudo` command, confirm the buffer has one redacted line; close the window and confirm the markdown file lands with the right content and the buffer is gone. Worth doing by hand once during setup and again after any change to the rule or the dispatch config.



## Deploy

### AWS

`splam` controls `systemctl`/`journalctl` on the host it runs on, so it has to run directly on the machine it's managing (not in a container or behind Lambda). On AWS that means an EC2 instance, run as a `systemd` service with a reverse proxy in front for TLS.

This page assumes no prior AWS experience. If you already know EC2, VPCs, and security groups, skip to [step 2](#2-install-splam).

## 1. Launch the instance

### Where the instance lives

An EC2 instance is a virtual machine that runs inside a **VPC** (a private network you own within AWS) and one of its **subnets** (a smaller address range carved out of that VPC, tied to one Availability Zone). Every new AWS account already has a default VPC with a public subnet in each Availability Zone, and that default is fine for a single `splam` instance. Launch into it unless your organization has its own VPC design you're expected to follow.

A **public subnet** is one whose route table sends `0.0.0.0/0` traffic to an internet gateway, which is what gives an instance in it a reachable public IP. `splam` itself should never be reachable directly (see security groups below), so if your account has a private subnet with a NAT gateway or a VPN/Direct Connect path back to your network, prefer that instead. You'll SSH in through a bastion host or VPN rather than the public internet.

### Choosing an instance type and AMI

Any recent Amazon Linux 2023 or Ubuntu 22.04+ AMI works; `splam` needs Python 3.11 or newer, which both ship with. A `t3.micro` or `t3.small` is plenty. The app itself is lightweight, and the audit trail/incident files are small text and JSON.

### Storage

The default 8 GB root EBS volume (`gp3`) is enough for the OS, the `splam` install, and normal audit-trail growth. If you're also using the shell-audit feature (see [How-To: Shell Audit Setup](01.13-how-to-shell-audit-setup.qmd)), `auditd`'s own logs in `/var/log/audit/` can grow faster under heavy `sudo` use, and 20 GB gives more headroom before you need to think about log rotation.

### Security group

A security group is a stateful firewall attached to the instance (stateful meaning a reply to an allowed outbound or inbound request doesn't need its own separate rule). Lock it down to two inbound rules:

* `22/tcp` (SSH) from your admin CIDR/VPN only, never `0.0.0.0/0`
* `443/tcp` (HTTPS) from wherever operators connect, ideally also restricted to a known CIDR range rather than the whole internet

Don't open `8000/tcp` (or whatever port `splam` binds) to the internet. It stays behind the reverse proxy in step 5, reachable only from `localhost` or from the load balancer's own security group. Outbound rules can stay at the default "allow all", since the instance needs outbound access to pull the `splam` package and any OS updates.

```{mermaid}
%%| eval: true
%%| fig-width: 6.6
%%| fig-height: 4.4
%%| fig-align: 'center'
%%| fig-caption: 'Security group rules for the EC2 instance'
%%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"22px", "darkMode":true}}}%%

flowchart LR
    subgraph Internet["Internet"]
        Admin(["Admin
SSH client"])
        Operator(["Operator
Browser"])
        Other(["Anyone else"])
    end

    subgraph SG["EC2 Security Group"]
        EC2["splam
127.0.0.1:8000"]
    end

    Admin ==>|"22/tcp
admin CIDR/VPN only"| EC2
    Operator ==>|"443/tcp
HTTPS"| EC2
    Other --x|"8000/tcp
never opened"| EC2

    style Internet fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:10,ry:10
    style SG fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:10,ry:10
    style Admin fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Operator fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Other fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style EC2 fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5,font-family:monospace

```

### DNS

If operators will reach `splam` at a real hostname rather than an IP, create an `A` (or `ALIAS`) record in Route 53 (or whatever DNS provider you use) pointing at the instance's Elastic IP, or at the load balancer if you're using one (step 5). An Elastic IP is a static public IP you allocate and associate with the instance, so it survives a stop/start. Without one, a restarted instance gets a new IP and DNS breaks.

## 2. Install splam

Same steps as the [Tutorial](00-tutorial-getting-started.qmd), run as whatever user will own the service. SSH in first:

```bash
ssh -i your-key.pem ec2-user@
```

(`ec2-user` on Amazon Linux, `ubuntu` on Ubuntu AMIs.)

Install system dependencies. `splam` itself only needs Python and `git`, but both AMI families ship Python without `venv`/`pip` fully wired up by default on minimal images, so install explicitly:

```bash
# Amazon Linux 2023
sudo dnf install -y python3 python3-pip git

# Ubuntu
sudo apt update && sudo apt install -y python3 python3-venv python3-pip git
```

Clone the repo:

```bash
sudo git clone https://github.com/mjfrigaard/splam.git /opt/splam
```

Change directories:

```bash
cd /opt/splam
```

Create the Python virtual environment. `splam` and its dependencies (Shiny for Python and its transitive packages, pulled from PyPI) are installed into this venv rather than the system Python, which is what `pip` would otherwise refuse to touch under [PEP 668](https://peps.python.org/pep-0668/):

```bash
sudo python3 -m venv .venv
```

Install dependencies:

```bash
sudo .venv/bin/pip install -e .
```

This needs outbound internet access to reach PyPI (`pypi.org`), so if the instance is in a private subnet, confirm the NAT gateway or VPC endpoint path allows it before this step.

Add admin:

```bash
sudo .venv/bin/splam-users add admin
```

## 3. Grant systemctl privilege

The service account needs rights to control the units it manages. See [How-To: Sudo Privileges](01.02-how-to-sudo-privileges.qmd). Running as `root` is simplest for a first deploy, and a scoped `polkit`/`sudoers` rule is the tighter option once you know which units operators need.

## 4. Run splam as a systemd unit

`systemd` is the service manager already running on both Amazon Linux and Ubuntu; a unit file tells it how to start, restart, and supervise a process, which is what keeps `splam` running across reboots and crashes without a separate process manager.

Bind to `127.0.0.1` (not `0.0.0.0`) and let the reverse proxy handle external traffic. Create `/etc/systemd/system/splam.service`:

```ini
[Unit]
Description=splam
After=network.target

[Service]
Type=simple
WorkingDirectory=/opt/splam
Environment=SPLAM_DATA_DIR=/var/lib/splam
ExecStart=/opt/splam/.venv/bin/shiny run splam.app:app --host 127.0.0.1 --port 8000
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

`SPLAM_DATA_DIR=/var/lib/splam` moves credentials and audit logs out of `/opt/splam` and into the conventional system-deployment location (see [How-To: Manage Logins](01.01-how-to-logins.qmd)). Add `SPLAM_MAX_FAILED_LOGINS`, `SPLAM_LOCKOUT_SECONDS`, `SPLAM_IDLE_TIMEOUT`, or `SPLAM_AUDIT_LOG` as further `Environment=` lines if the defaults don't fit.

```bash
sudo mkdir -p /var/lib/splam
```

```bash
sudo systemctl daemon-reload
```

```bash
sudo systemctl enable --now splam
```

`enable` makes the unit start on boot; `--now` also starts it immediately. Confirm it's actually up before moving on:

```bash
sudo systemctl status splam
curl -I http://127.0.0.1:8000
```

The `curl` should return an HTTP response (even a redirect or login page), confirming `splam` is listening on `127.0.0.1:8000` as expected. If `systemctl status` shows `failed`, check `journalctl -u splam -n 50` for the reason before continuing. A common cause at this stage is the venv path in `ExecStart` not matching where step 2 actually installed it.

## 5. Put a reverse proxy in front

`splam` only listens on `localhost:8000`; something else has to accept the public `443/tcp` connection, terminate TLS, and forward the request. Two options:

```{mermaid}
%%| eval: true
%%| fig-width: 2.6
%%| fig-height: 6.6
%%| fig-align: 'center'
%%| fig-caption: 'Request path from operator to splam'
%%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"20px", "darkMode":true}}}%%

flowchart TD
    Operator(["Operator
Browser"]) ==>|"HTTPS :443"| Proxy["Reverse Proxy
nginx or ALB"]
    Proxy ==>|"HTTP :8000"| Splam["splam
127.0.0.1:8000"]
    Splam -.->|"controls"| Systemd[("systemd
+ journald")]

    style Operator fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Proxy fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Splam fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Systemd fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px
```

**Option A: nginx on the instance itself.** Install it, request a certificate (e.g. via [Certbot](https://certbot.eff.org/)/Let's Encrypt, which needs `80/tcp` briefly open during issuance), and proxy to `splam`:

```bash
sudo dnf install -y nginx   # or: sudo apt install -y nginx
```

`/etc/nginx/conf.d/splam.conf`:

```nginx
server {
    listen 443 ssl;
    server_name splam.example.com;

    ssl_certificate     /etc/letsencrypt/live/splam.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/splam.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }
}
```

The `Upgrade`/`Connection` headers matter because Shiny for Python holds its UI connection open over a WebSocket, not plain request/response HTTP; without them the proxy will accept the initial page load but the app will never become interactive.

**Option B: an Application Load Balancer (ALB).** AWS terminates TLS for you using a free certificate from ACM (AWS Certificate Manager), and forwards plain HTTP to the instance. This is less to maintain (no certificate renewal on the box, no nginx config) but costs more than a bare instance and is one more AWS resource to understand:

1. Request a certificate for your domain in ACM and validate it (DNS validation via Route 53 is the easiest path).
2. Create a target group of type "instance", protocol HTTP, port `8000`, and register the EC2 instance in it. Enable WebSocket support is the ALB default, so no extra flag is needed, but confirm the target group's health check path returns a non-error response from `splam` (the login page at `/` works).
3. Create the ALB itself in the public subnet(s), listener on `443` using the ACM certificate, forwarding to that target group.
4. Point DNS at the ALB's own DNS name (a `CNAME`/`ALIAS`), not at the instance.

With an ALB, the instance's security group should only accept `8000/tcp` from the ALB's security group (reference it by security-group ID, not by CIDR), and `443/tcp` is opened on the ALB's security group instead of the instance's.



## Dev-Guide

### Run the Tests

Unlike the other how-to guides, this one assumes a checkout of the repository rather than a running app: the tests ship in the repo, not in the installed package. See the [Tutorial](00-tutorial-getting-started.qmd) if you don't have one yet.

## Run the suite

The tests live in `tests/` and need `pytest`, which comes with the `dev` extra:

```bash
.venv/bin/pip install -e ".[dev]"
.venv/bin/pytest
```

`testpaths` is set in `pyproject.toml`, so plain `pytest` from the repo root collects the right directory. The whole suite runs in about three seconds; most of that is PBKDF2, which is slow on purpose.

Run one file, or one test, the usual way:

```bash
.venv/bin/pytest tests/test_auth.py
.venv/bin/pytest tests/test_auth.py::test_lockout_expires
.venv/bin/pytest -k lockout
```

## How the suite stays out of your real data

Running the tests never touches your real accounts or audit trail. Here's why.

```{mermaid}
%%| eval: true
%%| fig-width: 7
%%| fig-height: 11
%%| fig-align: 'center'
%%| fig-caption: 'How the suite stays isolated'
%%{init: {'theme': 'dark', 'themeVariables': { 'fontFamily': 'monospace', "fontSize":"20px", "darkMode":true}}}%%

flowchart TD
    subgraph S1["1. At import time"]
        Env[/"SPLAM_DATA_DIR
XDG_DATA_HOME"/]
        Resolve[["data_dir()"]]
        Const["four file locations,
remembered"]
        Env --> Resolve
        Resolve --> Const
    end

    subgraph S2["2. Before every test"]
        Patch["swap in temporary paths"]
        Clear["clear the lockout counters"]
    end

    subgraph S3["3. During the test"]
        Code[["add_user()
record_action()
save_service_info()"]]
    end

    Tmp[("temporary directory
deleted afterwards")]
    Real[("~/.local/share/splam/
your real data")]

    Const --> Patch
    Patch --> Code
    Clear --> Code
    Code ==> Tmp
    Code --x Real

    style S1 fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:10,ry:10
    style S2 fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:10,ry:10
    style S3 fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:10,ry:10
    style Env fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Resolve fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Const fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Patch fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Clear fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Code fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Tmp fill:#4CBB9D,color:#FFFFFF,rx:5,ry:5
    style Real fill:#FFFFFF,color:#000000,stroke:#333,stroke-width:1px,rx:5,ry:5
```

Each splam module works out where its file belongs once, the first time it's imported, and then remembers the answer. `auth` remembers where `credentials.json` is, `audit` remembers `audit_log.jsonl`, and the two override files work the same way.

Before every single test, `tests/conftest.py` swaps those remembered locations for paths inside a fresh temporary directory. Pytest deletes that directory when the test finishes. The code under test then saves and loads files exactly as it always does. It's simply doing it somewhere disposable.

The same setup step also resets the failed-login counters. Those counters are one dictionary belonging to the `auth` module, shared by everything in the process, so an account locked out by one test would still be locked when the next test started. Clearing them keeps each test independent.

The fixture rewrites those paths in memory rather than setting `SPLAM_DATA_DIR`, and the reason is timing. The modules read that variable at import, which happens before any test runs, so changing it later would arrive too late to matter.

## What the tests cover

The suite is split across six files, one per module. The tables below list all 72 tests and what each one pins down.

### `tests/test_auth.py` (18)

Password hashing, roles, and the failed-login lockout.

| Test | What it pins down |
|------|-------------------|
| `test_add_and_verify` | A password saved by `add_user` verifies against the stored hash |
| `test_wrong_password` | A wrong password is rejected for a user that does exist |
| `test_unknown_user` | An unknown username returns `False` rather than raising |
| `test_password_is_not_stored` | The plaintext password appears nowhere in `credentials.json` |
| `test_salt_is_per_user` | Two users with the same password get different salts and different hashes |
| `test_file_and_directory_permissions` | `credentials.json` is written `0600` inside a `0700` directory |
| `test_role_defaults_to_admin` | `add_user` with no role produces an admin |
| `test_role_is_stored` | A role passed to `add_user` is what `get_role` returns |
| `test_legacy_record_without_role_is_admin` | A record with no `role` field reads as admin and still verifies |
| `test_unknown_role_rejected` | An unrecognized role raises `ValueError` and writes nothing at all |
| `test_add_user_replaces_role` | Re-adding a user replaces the previous role instead of keeping it |
| `test_lockout_after_max_failures` | Failures below the threshold report `failed`; the one that hits it locks the account |
| `test_lockout_blocks_correct_password` | Once locked, the correct password is refused too |
| `test_lockout_expires` | Past the lockout window the account works again (the clock is stubbed through `_now`) |
| `test_success_resets_failure_counter` | A success clears accumulated failures, so the next failure counts as the first |
| `test_lockout_is_per_username` | Locking one account leaves every other account usable |
| `test_unknown_user_locks_out_too` | Attempts against a nonexistent username lock as well, so the form leaks no account list |
| `test_credentials_file_is_json` | Each record holds exactly `salt`, `hash`, and `role` |

### `tests/test_audit.py` (10)

The audit trail: what gets written, and what comes back out.

| Test | What it pins down |
|------|-------------------|
| `test_record_action_fields` | Every field is written, with a UTC timestamp |
| `test_record_auth_has_no_service` | Login and logout entries carry `service: null` |
| `test_rejected_actions_are_recorded` | A rejected attempt is recorded like any other, not dropped |
| `test_file_and_directory_permissions` | `audit_log.jsonl` is `0600` inside a `0700` directory |
| `test_empty_log_reads_as_empty_list` | A missing log reads as `[]` instead of raising |
| `test_newest_first` | Entries come back most recent first |
| `test_limit_keeps_most_recent` | `limit` truncates the old end, not the new one |
| `test_service_filter` | Filtering by service excludes other services |
| `test_service_filter_keeps_auth_entries` | Logins stay visible no matter which service is selected |
| `test_appends_rather_than_overwrites` | A second write appends instead of truncating the file |

### `tests/test_app.py` (23)

The `systemctl`, `journalctl`, and `systemd-analyze` wrappers, plus role-dependent UI. Every subprocess call is faked and the unit search path is pointed at a temporary directory, so no test touches real services.

| Test | What it pins down |
|------|-------------------|
| `test_unit_search_paths_asks_systemd` | The search path comes from `systemd-analyze unit-paths`, not a hardcoded list |
| `test_list_services_reads_every_search_path` | Units are collected from all search directories, and non-service units ignored |
| `test_list_services_skips_template_units` | Template units (`getty@.service`) stay out of the dropdown |
| `test_list_services_keeps_documented_units_without_a_unit_file` | A service with notes is selectable even with no unit file on disk |
| `test_list_services_deduplicates_across_paths` | A unit shadowed in two directories appears once |
| `test_list_services_survives_a_missing_search_path` | A search directory that doesn't exist is skipped, not raised on |
| `test_doc_references_splits_quoted_and_bare_uris` | Quoted `man:` entries and bare URLs both parse out of `Documentation=` |
| `test_doc_references_empty_when_none_declared` | A unit with no `Documentation=` yields an empty list |
| `test_get_documentation_runs_systemctl_help` | The Doc tab shells out to `systemctl help ` |
| `test_get_documentation_disables_the_pager` | `PAGER` and `MANPAGER` are forced to `cat`, so man never blocks on a pager |
| `test_get_documentation_wraps_at_the_requested_width` | The requested column count reaches man as `MANWIDTH` |
| `test_get_documentation_falls_back_to_eighty_columns` | With no width reported, man gets its own default of 80 |
| `test_doc_columns_grow_with_the_window` | A wider tab asks man for more columns |
| `test_doc_columns_round_to_five` | Column counts snap to a multiple of five, so a resize drag re-renders a few times, not per pixel |
| `test_doc_columns_stay_within_readable_bounds` | A very narrow or very wide tab is clamped to 60 and 220 columns |
| `test_get_documentation_reports_a_unit_without_docs` | systemd's "not known" message reaches the tab instead of being swallowed |
| `test_systemctl_action_command` | The exact `systemctl  ` argv is built |
| `test_get_status_combines_streams` | Status output is stdout followed by stderr |
| `test_get_logs_requests_reversed_lines` | `journalctl` is called with the line count and `-r` for newest first |
| `test_admin_panel_shows_actions` | An admin panel contains the Actions tab and the Configure button |
| `test_auditor_panel_hides_actions_and_configure` | An auditor panel contains neither, and says the role is read-only |
| `test_auditor_panel_keeps_read_only_tabs` | An auditor keeps Status, Tasks, Doc, Logs, and Audit Trail |
| `test_idle_timeout_default_is_fifteen_minutes` | The shipped idle timeout is 900 seconds |

### `tests/test_highlight.py` (9)

Keyword highlighting, including the escaping that makes it safe to render log lines as HTML.

| Test | What it pins down |
|------|-------------------|
| `test_escapes_html` | Markup in a log line is escaped before anything is wrapped |
| `test_wraps_error_keyword` | `error` is wrapped as `hl-error` |
| `test_matching_is_case_insensitive` | `ERROR` matches the same rule as `error` |
| `test_rejected_is_a_warning` | `rejected` is wrapped as `hl-warn` |
| `test_lockout_and_timeout_are_warnings` | `locked out` and `timeout` are wrapped as `hl-warn` |
| `test_custom_keyword_override` | A keyword saved through Configure gets the class it was given |
| `test_custom_keyword_wins_over_builtin` | Custom patterns are matched ahead of the built-in ones |
| `test_unknown_class_rejected` | An unknown highlight class raises and saves no override file |
| `test_unmatched_text_passes_through` | Text with no keywords is returned unchanged |

### `tests/test_paths.py` (6)

Where app data lands, and the permissions it lands with.

| Test | What it pins down |
|------|-------------------|
| `test_explicit_override_wins` | `SPLAM_DATA_DIR` takes precedence over `XDG_DATA_HOME` |
| `test_follows_xdg_data_home` | With no override, data goes under `$XDG_DATA_HOME/splam` |
| `test_defaults_under_home` | With neither set, data goes under `~/.local/share/splam` |
| `test_ensure_parent_creates_owner_only_directory` | Missing parent directories are created `0700` |
| `test_ensure_parent_is_idempotent` | Calling it against an existing directory leaves the contents alone |
| `test_import_has_no_side_effects` | `import splam` in a fresh subprocess creates no directory |

That last one matters more than it looks. The docs build imports the package to introspect it, and CI runs as a different user in a different home directory. A package that created `~/.local/share/splam/` on import would do it there too.

### `tests/test_service_info.py` (6)

Built-in admin-task notes and the overrides that replace them.

| Test | What it pins down |
|------|-------------------|
| `test_builtin_entry` | The shipped `cron.service` notes are returned |
| `test_unknown_service_falls_back_to_generic` | An undocumented service gets `GENERIC_INFO` |
| `test_override_wins_over_builtin` | A saved override replaces the built-in entry for that service |
| `test_override_adds_new_service` | An override can document a service with no built-in entry |
| `test_overrides_persist_to_the_data_directory` | Overrides are written to the app data directory |
| `test_builtins_survive_an_override` | Saving one override leaves the other built-ins in place |

## What the tests do not cover

The Shiny server function is not unit tested. Login wiring, the idle countdown firing, and the server-side role check on a submitted action all live inside `reactive.effect` closures that need a live session to run, so `tests/test_app.py` covers the pure helpers and the panel markup instead.

Those paths were verified by driving a real browser session over the Shiny websocket protocol: logging in as each role, confirming an auditor gets no Actions tab, watching a session get logged out by the idle timeout, and confirming a locked account is refused its own correct password. If you change the login effect or the idle timeout, that's the check to repeat, because the suite won't catch it.

## What CI runs on every push

`.github/workflows/tests.yml` runs the suite on every push to `main` and on every pull request, against Python 3.11, 3.12, and 3.13. The matrix doesn't stop at the first failure, so a break in one version shows up as exactly that.

Locally, `.venv/bin/pytest` before you push covers the same ground on whichever version your virtual environment is built against, which is the only part CI can't tell you sooner.


### Build Docs

Task-oriented recipes. Each section assumes you already have `splam` installed and running. See the [Tutorial](00-tutorial-getting-started.qmd) if not.

## Rebuild and preview the documentation site

```bash
great-docs build
great-docs preview --port 4173
```

`great-docs build` regenerates `great-docs/_site/` from `README.md`, `user_guide/*.qmd`, and the package's docstrings. `great-docs preview` serves that directory locally.

## Add a screenshot to the docs

Drop image files into `user_guide/images/` and reference them from a `.qmd` file with a path relative to `user_guide/` (no `../`), e.g.:

```markdown
![](images/ui-login.png){width='100%'}
```

`great-docs build` auto-copies any asset directory (one with no `.qmd` files, e.g. `images/`) that lives directly inside `user_guide/` into the built site. A project-root `images/` folder (a sibling of `user_guide/`) is **not** picked up. It must be inside `user_guide/`.