Moud

Running the catalog api#

The catalog api in api/ is the service the Moud client, the website and every fleet talk to. It runs as one Node process by default, and as several behind a load balancer when one is not enough. This page covers what changes between the two, the metrics it exposes, regions and serving big files. api/.env.example lists every setting.

One process#

With nothing extra set, the api keeps its rate limits, open event streams, sign in codes and matchmaking queues in its own memory. That needs no setup beyond the database, and a restart empties the queues and drops the streams, which clients and fleets open again on their own.

Several processes#

Set CATALOG_CLUSTER=on on every process that shares the database. Each process then:

  • creates the moud_cluster schema in the database on start if it is missing. Its tables are unlogged, so they are fast and empty after a database crash, which is what this state needs
  • counts rate limits and server quotas in postgres as well as in memory. A process sends its counts in a batch every 200 ms, so across processes a limit can be passed by what arrives in that time. Sign in limits are counted in postgres on every request
  • hands events to the other processes with LISTEN and NOTIFY on the channel moud_cluster. An event for a player whose client is connected to another process reaches it there, and so does a start request for a fleet listening on another process
  • keeps matchmaking queues in tables. Each process takes the queues no other process holds, with FOR UPDATE SKIP LOCKED, so the work spreads out and a queue is never matched twice
  • keeps sign in challenges and claim codes in a table, so a code made on one process is accepted on another

One process at a time leads, the one holding a postgres advisory lock. It runs retention, the blob sweep, the status checks, webhook deliveries, analytics sampling, the leaderboard, memory, ban, erasure and takedown sweeps, and the clean up of the cluster tables. When it stops, another process takes the lock within a few seconds.

Two processes asked for a server of the same place and channel at once send the fleet one start: the second waits on the first one's request, through a short lived row in the cluster tables.

Every process needs the same CATALOG_TICKET_KEY and CATALOG_CONTENT_SECRET. Without them each one makes its own, and a ticket signed by one is refused by another. The blob folder, CATALOG_BLOB_DIR, has to be one folder they all see.

The load balancer needs no sticky sessions. For nginx, list each process in the upstream moud_api block of api/deploy/nginx.conf.

Metrics#

/metrics answers in the Prometheus text format. It is off until you turn it on in one of two ways:

  • METRICS_PORT=9464 serves it on a port of its own, on 127.0.0.1 unless METRICS_HOST says otherwise, with no token. Use this with several processes, each on its own port, so a scrape always reaches the process it means to
  • METRICS_TOKEN=... serves it on the main port to requests with Authorization: Bearer <token>. The nginx config in api/deploy refuses /metrics from outside either way
Metric
moud_http_request_seconds time to answer, by method, route pattern and status. Its _count is the request count
moud_db_query_seconds time from sending a query to its answer
moud_db_queries_active queries in flight, at most DATABASE_POOL of them run at once
moud_sse_clients, moud_sse_fleets event streams open on this process
moud_matchmaking_waiting players waiting in matchmaking queues
moud_tickets_issued_total join and teleport tickets handed out, by kind
moud_tickets_redeemed_total tickets game servers used
moud_fleets_live, moud_servers_live fleets and servers in a recent heartbeat, by region
moud_rate_limit_refusals_total requests a rate limit or quota turned away, by limit
moud_retention_deleted_total rows retention and the blob sweep removed, by table

Counters start at zero when a process starts. The live and waiting numbers come from the database, so every process reports the same value for them.

Regions#

Fleets say where they run with region in fleet.toml, see Running a fleet. Players are matched to a region in one of two ways:

  • CATALOG_REGIONS lists the regions with an address the client can time, like eu=https://eu.example.net/v1/ping,na-east=https://na.example.net/v1/ping. Every api answers /v1/ping, so an api process in each region works. The client times each address once a session and sends the closest with its joins
  • CATALOG_REGION_HEADER names a header your edge proxy or cdn sets to the player's region. It is used when the client sent none, a play click on the website for example

Region names are short lowercase names, letters, digits and dashes.

Big downloads#

Bundles, game files and pictures are served with ETag and byte ranges. The Moud client asks for the rest of a file after a dropped connection instead of starting again, and a file it already has answers 304. Files are named by their hash, so they are cached as immutable.

To take the downloads off the api, set CATALOG_CDN_URL and CATALOG_CDN_SECRET. The api then answers those requests with a redirect to a signed link under that address, after it checked the request as usual. Public pictures get links that change once a week, so a cdn keeps one copy, and everything else gets links that run out after about fifteen minutes. The links use the nginx secure_link form, and api/deploy/cdn.conf serves the blob folder that way. A cdn in front of it works as long as it keeps the query string in its cache key.

Only requests that carry no token are redirected, and the ones from a Moud client that says it follows redirects itself. A fleet downloading a server bundle is always served by the api.