Health checks

In production nobody watches the gateway by hand. An orchestrator such as Kubernetes runs several copies of it and, every few seconds, asks each copy two questions. Are you alive? If not, the process is stuck and gets restarted. Are you ready? If not, it stays up but gets no traffic until it is. The gateway answers over HTTP, at routes the orchestrator is configured to call. A 200 means yes and a 503 means no. The body says why, for the people reading the dashboard.

The two questions have different answers on purpose. A gateway whose cats service is down should not be restarted, because a restart fixes nothing: it is alive and not ready. Put a dependency into the liveness check and a database outage becomes a restart loop across every copy.

Terminus

@nestjs/terminus builds these routes. Import TerminusModule and inject HealthCheckService. A check is a list of functions, each returning one indicator's result, and check() runs them all and combines them:

@Get('ready')
@HealthCheck()
ready() {
  return this.health.check([
    () => this.http.pingCheck('payments', 'http://payments.internal/ping'),
  ]);
}

The result has a status, plus info (indicators that are up or degraded), error (indicators that are down) and details (all of them). If any indicator is down, check() answers 503 with status error. If none is down but one is degraded, 200 with degraded. Otherwise 200 with ok. With no functions at all it answers ok, which is what a liveness check needs. @HealthCheck() marks the route and sets Cache-Control: no-cache, no-store, must-revalidate, because a cached health answer is a lie.

Built-in indicators

Terminus ships indicators for the usual dependencies. HttpHealthIndicator.pingCheck(key, url) is up when the URL answers with a success status. It needs the @nestjs/axios package, though not its HttpModule: Terminus 12.1 makes its own HttpService, whatever older docs say. MicroserviceHealthIndicator.pingCheck(key, options) connects to a microservice over the transport the options describe:

() => this.microservice.pingCheck<TcpClientOptions>('ledger', { transport: Transport.TCP, options: { port: 4200 } }),

A connection that fails gives { status: 'down', message: 'connect ECONNREFUSED 127.0.0.1:4200' }. There are also indicators for TypeORM, Mongoose, Sequelize, Prisma, memory and disk. The last two measure the process and its machine, which a browser does not have.

An indicator of your own

For anything the built-ins do not know, inject HealthIndicatorService. check(key) starts a result for that key. You then return up(), down() or degraded(), each taking an object whose fields are added to the result, or a message:

isHealthy(key: string) {
  const indicator = this.indicators.check(key);
  const waiting = this.queue.size();
  if (waiting > 1000) return indicator.down({ waiting });
  if (waiting > 100) return indicator.degraded({ waiting });
  return indicator.up({ waiting });
}

Degraded means "still serving, but look at me": it keeps the 200, so the orchestrator keeps sending traffic, while the dashboard shows the problem. An indicator must not throw. A thrown error is treated as a bug and aborts the whole check with a 500, not a 503. For an operation that may throw, check(key).attempt(fn) catches it and reports it as down.

Your task

The gateway has a KennelsModule: KennelsService.freeKennels() says how many kennels are free (three to start with), and POST /kennels/occupy fills one. The vet clinic's status page is at http://vet-clinic.internal/status.

1. Write KennelsHealthIndicator. It reports the kennels under the key it is given: up with { free } while three or more are free, degraded with { free } while one or two are, down with { free } when none is.
2. Write HealthController at /health. GET /health/live checks nothing. GET /health/ready checks the cats service over TCP on port 3001 as cats-service, the vet clinic's status page as vet-clinic, and the kennels as kennels. Both are health checks.
3. Write HealthModule (Terminus and the kennels, the controller, the indicator), and import it in GatewayModule.

Fill the kennels one by one, and read /health/ready and /health/live after each.

When it fails

• Nest can't resolve dependencies of the KennelsHealthIndicator (?, KennelsService): HealthIndicatorService comes from TerminusModule, which HealthModule does not import.
• The ready check is a 500 and the console shows your error: the indicator threw. Return down() instead.
• cats-service is down with connect ECONNREFUSED 127.0.0.1:3011: the ping names a port the service does not listen on.
• No Cache-Control on the answer: the route lost @HealthCheck().

Remember

• Liveness: should this process be restarted? Readiness: should it get traffic? Keep dependencies out of liveness.
• check() combines indicators: 503 when one is down, 200 with degraded or ok otherwise.
• Built-in indicators ping HTTP services and microservices; HealthIndicatorService builds your own.
• Degraded keeps serving and says why; an indicator reports its state, it never throws.