Swarm options and log destinations
Every Docker Swarm setting an application can have beyond its size and command, which ones need a cluster of your own, and how to send logs to Loki, Elastic, Graylog or Splunk.
An application's size, copies, environment, command, health check and rollout have their own panels. Swarm options is everything else Docker Swarm can set on the service. Each one left empty keeps Docker's default, and clearing one puts that default back at the next deploy.
Set them in the console (application page, Swarm options), with
isogrid apps options <app>, or in a stack file, where the usual compose keys
are read.
What you can set
| Area | Options | Compose keys |
|---|---|---|
| Scheduling | spread copies over a node label, most copies per machine, CPUs reserved per copy | deploy.placement.preferences, max_replicas_per_node, deploy.resources.reservations.cpus |
| Container | working directory, hostname, stop signal, init process, read-only filesystem, TTY, ulimits, process limit, extra groups, dropped capabilities | working_dir, hostname, stop_signal, init, read_only, tty, ulimits, group_add, cap_drop |
| Files | configuration files made from text (Swarm configs) | configs with an inline content: |
| Network | extra /etc/hosts lines, DNS servers, search domains and options, one address per copy (dnsrr) |
extra_hosts, dns, dns_search, dns_opt, deploy.endpoint_mode |
| Logs | where the output goes | logging.driver, logging.options |
Files are for configuration that is not secret. Secrets belong in a vault and reach the container as files or variables.
Options that need a cluster of your own
Some options reach the machine itself. They run only on a cluster that is yours: a private cluster granted to your organization alone, or one run by your Nomad agent. On a shared region they are refused, and the console says so.
- added capabilities (
cap_add) and kernel settings (sysctls) - the OOM score (
oom_score_adj) - the machine's network (
network_mode: host): the application then leaves its private network - directories of the machine (bind mounts)
- ports published on the machines (
portswithmode: hostormode: ingress) - node resources such as GPUs (
generic_resources) - one copy on every machine (
deploy.mode: global). Switching to or from it recreates the service, with a short interruption, and such an application cannot be scaled or autoscaled. - raising any limit but
nofile, or setting one to unlimited
On a cluster run by the Nomad agent, the ones that give root on the machine
(capabilities, sysctls, OOM score, host network, bind mounts) also need the
agent's consent: set ISOGRID_NOMAD_ALLOW_PRIVILEGED=true on the agent. Without
it, the agent refuses them, whatever ISOGrid asks.
Where logs go
By default the output stays on the machine, where the console and log search read it. You can send it somewhere else instead:
- Docker's own drivers:
syslog,gelf(Graylog, Logstash),fluentd,splunk, and, on your own cluster only,awslogs,gcplogsandjournald, which use the machine's identity or journal. - Log plugins: Grafana Loki and Elastic. A plugin is installed on every
machine of the region first:
isogrid regions log-plugins <region> --install loki, or the button in the Swarm options panel. A plugin runs as root on each machine, so you install one on a cluster of your own; on a shared region ISOGrid's operators decide.
isogrid apps options shop --log-driver loki \
--log-secret-opt loki-url=https://USER:TOKEN@logs.example.com/loki/api/v1/push --deploy
The console keeps showing the output whatever the destination: Docker keeps a local copy, sized like your retained logs. A destination that is slow or down never holds the application up: output is buffered (4 MB) and dropped past the buffer.
A token or password in a log option goes in the secret options: it is kept in
the secret store and never shown again. On a cluster run by your Nomad agent,
write it as {{secret:<reference>}} instead, so it stays in your Vault.
On a shared region a destination must be a public address: the machine sends the logs itself, and is not allowed to reach private networks on your behalf.
Searching JSON logs
When your application prints JSON lines, log search reads them as fields. The
level is the logger's own (level, severity, pino's numbers), the message is
its msg, and fields are filtered with @:
| Query | Matches |
|---|---|
@status>=500 |
a number compared (>, >=, <, <=) |
@http.method:GET |
equal, ignoring case; nested fields with dots |
@user.id:* |
the field is present |
@path:/api/* |
starts with |
@msg~timeout |
contains |
-@env:dev |
not that |
They combine with words, phrases, level: and replica: as before.