Swarm options and log destinations

Every Docker Swarm setting an application can have beyond its size and command, which ones need a cluster of your own, and how to send logs to Loki, Elastic, Graylog or Splunk.

An application's size, copies, environment, command, health check and rollout have their own panels. Swarm options is everything else Docker Swarm can set on the service. Each one left empty keeps Docker's default, and clearing one puts that default back at the next deploy.

Set them in the console (application page, Swarm options), with isogrid apps options <app>, or in a stack file, where the usual compose keys are read.

What you can set

Area Options Compose keys
Scheduling spread copies over a node label, most copies per machine, CPUs reserved per copy deploy.placement.preferences, max_replicas_per_node, deploy.resources.reservations.cpus
Container working directory, hostname, stop signal, init process, read-only filesystem, TTY, ulimits, process limit, extra groups, dropped capabilities working_dir, hostname, stop_signal, init, read_only, tty, ulimits, group_add, cap_drop
Files configuration files made from text (Swarm configs) configs with an inline content:
Network extra /etc/hosts lines, DNS servers, search domains and options, one address per copy (dnsrr) extra_hosts, dns, dns_search, dns_opt, deploy.endpoint_mode
Logs where the output goes logging.driver, logging.options

Files are for configuration that is not secret. Secrets belong in a vault and reach the container as files or variables.

Options that need a cluster of your own

Some options reach the machine itself. They run only on a cluster that is yours: a private cluster granted to your organization alone, or one run by your Nomad agent. On a shared region they are refused, and the console says so.

  • added capabilities (cap_add) and kernel settings (sysctls)
  • the OOM score (oom_score_adj)
  • the machine's network (network_mode: host): the application then leaves its private network
  • directories of the machine (bind mounts)
  • ports published on the machines (ports with mode: host or mode: ingress)
  • node resources such as GPUs (generic_resources)
  • one copy on every machine (deploy.mode: global). Switching to or from it recreates the service, with a short interruption, and such an application cannot be scaled or autoscaled.
  • raising any limit but nofile, or setting one to unlimited

On a cluster run by the Nomad agent, the ones that give root on the machine (capabilities, sysctls, OOM score, host network, bind mounts) also need the agent's consent: set ISOGRID_NOMAD_ALLOW_PRIVILEGED=true on the agent. Without it, the agent refuses them, whatever ISOGrid asks.

Where logs go

By default the output stays on the machine, where the console and log search read it. You can send it somewhere else instead:

  • Docker's own drivers: syslog, gelf (Graylog, Logstash), fluentd, splunk, and, on your own cluster only, awslogs, gcplogs and journald, which use the machine's identity or journal.
  • Log plugins: Grafana Loki and Elastic. A plugin is installed on every machine of the region first: isogrid regions log-plugins <region> --install loki, or the button in the Swarm options panel. A plugin runs as root on each machine, so you install one on a cluster of your own; on a shared region ISOGrid's operators decide.
isogrid apps options shop --log-driver loki \
  --log-secret-opt loki-url=https://USER:TOKEN@logs.example.com/loki/api/v1/push --deploy

The console keeps showing the output whatever the destination: Docker keeps a local copy, sized like your retained logs. A destination that is slow or down never holds the application up: output is buffered (4 MB) and dropped past the buffer.

A token or password in a log option goes in the secret options: it is kept in the secret store and never shown again. On a cluster run by your Nomad agent, write it as {{secret:<reference>}} instead, so it stays in your Vault.

On a shared region a destination must be a public address: the machine sends the logs itself, and is not allowed to reach private networks on your behalf.

Searching JSON logs

When your application prints JSON lines, log search reads them as fields. The level is the logger's own (level, severity, pino's numbers), the message is its msg, and fields are filtered with @:

Query Matches
@status>=500 a number compared (>, >=, <, <=)
@http.method:GET equal, ignoring case; nested fields with dots
@user.id:* the field is present
@path:/api/* starts with
@msg~timeout contains
-@env:dev not that

They combine with words, phrases, level: and replica: as before.