Health checks

Overview

Health checks are used to monitor the health of platform services.

Install

ASOL.Core.HealthChecks - Extends Microsoft's health checks functionality with additional configuration support platform health checks.

Platform NuGet packages in the table below already reference ASOL.Core.HealthChecks with minimum recommended version 0.0.1.42547-dev. Therefore, you don't need to reference it directly.

Package	Version
ASOL.Core.ApiController	0.0.1.42562-dev
ASOL.Core.Persistence.MongoDB	0.0.1.42561-dev
ASOL.Core.Persistence.EFCore	0.0.1.42561-dev
ASOL.Core.Messaging	0.0.1.42563-dev
ASOL.Core.Messaging.RabbitMq	0.0.1.42563-dev
ASOL.Core.ApiConnector	0.0.1.44962-dev

Changelog

• 0.0.1.44328-dev fix loading assemblies when discovering platform readiness health checks
• 0.0.1.43035-dev http communication improvements - http version
• 0.0.1.42547-dev significant performance improvements, add memory cache for platform readiness health checks.

Description

Platform uses three types of health checks - Startup, Liveness and Readiness.

	Endpoint	Description
Startup	/health/startup	Detects that a service has started successfully and processes that need to be completed before traffic can be routed have been completed.
Liveness	/health/liveness	Defacto ping to a service.
Readiness	/health/readiness	Checks whether a service has all dependencies available (RabbitMQ, MongoDB, PostgreSQL, other HTTP services) needed to successfully process incoming requests.

The Orchestrator periodically calls platform health checks to determine whether a service is healthy. If any health check fails, the service is considered unhealthy. The Orchestrator will not route traffic to the unhealthy service and will try to restart it. However, this does not happen after the first health check failure. The Orchestrator has set a failure threshold. If the number of consecutive health check failures exceeds the threshold, the service is considered unhealthy.

A platform readiness health check can be a quite expensive process, and therefore should not be called too often. It would be best to leave platform readiness health checks to the orchestrator. If some readiness health check fails, the orchestrator will stop traffic to that service and liveness health check will fail as well. So if you want to monitor your service, a liveness health check should be enough.

Example of a platform readiness health check process.

Registration

builder.Services.AddPlatformHealthChecks<Startup>(configuration) // namespace: ASOL.Core.ApiController.Extensions 
    .AddEFCoreHealthCheck()           // namespace: ASOL.Core.Persistence.EFCore.Extensions
    .AddMongoDbHealthCheck()          // namespace: ASOL.Core.Persistence.MongoDb.Extensions
    .AddRabbitMqHealthCheck()         // namespace: ASOL.Core.Messaging.RabbitMq.Extensions
    .AddRedisConnectionHealthCheck()  // namespace: ASOL.Core.Distributed.Redis.Extensions
    .AddPlatformApiHealthCheck<SsoAuthServiceHealthCheck>() // idp (namespace: ASOL.Core.HealthChecks.Extensions, namespace: ASOL.Core.Identity.HealthChecks)
    .AddPlatformApiHealthCheck<AuthorizationServiceHealthCheck>() // idm (namespace: ASOL.Core.HealthChecks.Extensions, namespace: ASOL.IdentityManager.AspNetCore.Authorization.HealthChecks)
    .AddPlatformApiHealthCheck<MyOwnProjectClientOptions>(); // connector (namespace: ASOL.Core.ApiConnector.Extensions)

app.UseEndpoints(endpoints =>
{
    endpoints.MapControllers();
    endpoints.MapPlatformHealthChecks(); // namespace: ASOL.Core.ApiController.Extensions
    endpoints.MapVersionHealthChecks();  // namespace: ASOL.Core.ApiController.Extensions (optional)
});

During the platform health checks registration, the following actions are performed:

• Registers a trivial “Startup” health check that immediately returns Healthy and is tagged with Startup
• Registers a “Liveness” health check that immediately returns Healthy
• Registers a “Version” health check that returns information about assembly versions.

Important Automatic discovery and registration of platform readiness health checks (RabbitMQ, EF Core, ApiConnectors) has been removed from the core library. To monitor these dependencies, you should now register each health check explicitly. This change was made to give developers better control over what is being monitored and to avoid unnecessary overhead when using different versions of packages.

Dependency Registration Recommendation

For native applications, it is recommended to explicitly register health checks for any used dependencies that you want to monitor. The AddPlatformHealthChecks<TStartup>(configuration) method returns an IHealthChecksBuilder, which allows you to chain further registrations.

1. Base Services: Call AddPlatformHealthChecks<TStartup>(configuration) to set up basic infrastructure, liveness/startup checks, and the version health check.
2. Persistence: If you want to monitor EF Core, add .AddEFCoreHealthCheck() to the chain. For MongoDB, add .AddMongoDbHealthCheck().
3. Remote cache and distributed Redis primitives: If the application uses DistributedCache:Storage = RemoteCache or Redis-backed distributed coordination, add .AddRedisConnectionHealthCheck() to the chain. Do not register it for MemoryCache-only cache usage.
4. Messaging: If you want to monitor RabbitMQ, add .AddRabbitMqHealthCheck() to the chain.
5. API Clients: To monitor an API client, add .AddPlatformApiHealthCheck<TOptions>(Configuration) for its options type or add .AddPlatformApiHealthCheck<THealthCheck>() for speficic implementation of api healthcheck (e.g. SsoAuthServiceHealthCheck or AuthorizationServiceHealthCheck).

Example of a registration with dependencies:

public void ConfigureServices(IServiceCollection services)
{
    // ... other services

    services.AddPlatformHealthChecks<Startup>(Configuration)
        .AddEFCoreHealthCheck()           // monitor EF Core
        .AddMongoDbHealthCheck()          // monitor MongoDB
        .AddRabbitMqHealthCheck()         // monitor RabbitMQ
        .AddRedisConnectionHealthCheck()  // monitor shared Redis connection
        .AddPlatformApiHealthCheck<SsoAuthServiceHealthCheck>() // monitor IDP
        .AddPlatformApiHealthCheck<AuthorizationServiceHealthCheck>() // monitor IDM
        .AddPlatformApiHealthCheck<MyServiceClientOptions>(); // monitor specific API client
}

Response

Platform health checks can respond with three states: Healthy, Degraded and Unhealthy.

	Status code	Content-Type	Body
Healthy	200	text/plain	Healthy
Degraded	200	text/plain	Degraded
Unhealthy	503	text/plain	Unhealthy

Extended readiness response

If you want to debug some problems with some particular health check, you can enable the extended health check readiness response.

Note: In a production environment, it is recommended to disable the extended health check response. It should be used only for development and testing purposes.

{
  "PlatformHealthCheckOptions": {
    "ExtendedHealthCheckResponse": true
  }
}

Or you can set via configuration API for a specific environment.

{
  "code": "PlatformHealthCheckOptions:ExtendedHealthCheckResponse",
  "value": "true"
}

With the extended response readiness health check on, the health check response will contain the results of all particular readiness health checks.

{
  "status": "Unhealthy",
  "results": {
    "EFCoreHealthCheck, ASOL.Core.Persistence.EFCore": {
      "status": "Unhealthy",
      "description": "Unable to connect to db.",
      "data": {}
    },
    "RabbitMqHealthCheck, ASOL.Core.Messaging.RabbitMq": {
      "status": "Healthy",
      "description": null,
      "data": {}
    },
    "AuthorizationServiceHealthCheck, ASOL.IdentityManager.AspNetCore.Authorization": {
      "status": "Healthy",
      "description": null,
      "data": {}
    },
    "ApiConnectorHealthCheck\u00601\u003CIdentityManagerClientOptions\u003E, ASOL.Core.ApiConnector": {
      "status": "Healthy",
      "description": null,
      "data": {}
    }
  }
}

Configuration

You can configure the health check mode for each platform readiness health check. The health check mode can be set in the appsettings.json file or via configuration API.

• Enabled: Default
• EnabledButOverwriteUnhealthyWithDegraded: Unhealthy is override to Degraded
• SkipAsHealthy: skp health check -> dependency is always healthy

Example:

{
  "ProductAppClientOptions": {
    "BaseUrl": "https://avaplace.com/api/asol/product",
    "HealthCheckMode": "SkipAsHealthy"
  }
}

Be aware of the circular dependency between services. When you have a circular dependency between services, you need to use the SkipAsHealthy mode for the health check. This mode skips platform readiness health check of another service and returns the health check status of the checked service as healthy.

Logging

In case of health check failure, a message is logged with information about the failure.

• Unhealthy: HealthCheck ASOL.Core.Persistence.MongoDb.HealthChecks.MongoDbHealthCheck is Unhealthy: MongoDB is in an unknown state
• Unhealthy with overwrite to Degraded according to the HealthCheckMode configuration: HealthCheck ASOL.Core.Persistence.MongoDb.HealthChecks.MongoDbHealthCheck is Unhealthy (overwrite with Degraded): MongoDB is in an unknown state

Platform integrated health checks

MongoDb

Platform readiness health check for RabbitMQ is implemented in the package ASOL.Core.Persistence.MongoDB. MongoDB platform readiness health check is based on the following steps:

• Ensuring a connection string is configured.
• Issuing a lightweight ping command against a database.
• Verifying the client’s cluster state is connected.

RabbitMQ

Platform readiness health check for RabbitMQ is implemented in the package ASOL.Core.Messaging.RabbitMQ. Package ASOL.Core.Messaging.RabbitMQ internally use MassTransit package. For platform readiness health check has been reused health check from MassTransit package.

PostgreSQ

Platform readiness health check for RabbitMQ is implemented in the package ASOL.Core.Persistence.EFCore. PostgreSQL platform readiness health check is based on the following steps:

• Ensuring that a connection to the database could be established.
• Connect to the database.
• Runs a lightweight SELECT 1 command and validates the scalar result equals 1.

ApiConnector

If you are implementing connector with ASOL.Core.ApiConnector package, you can use the IPlatformHealthCheckableOptions interface. The health check itself is already implemented in the ASOL.Core.ApiConnector package and can be registered using AddPlatformApiHealthCheck<TOptions>(Configuration).

ApiConnector platform readiness health check is based on the following steps:

• Calls health/liveness endpoint of the target API.
  • If the response status code is 200, return Healthy.
  • If the response status code is not 200 or 404, return Unhealthy.
• If health/liveness endpoint does not exist (404), try to call health/live endpoint. This is the previous version of platform health check. If it fails, return Unhealthy.

public class OtherServiceClientOptions : ApiClientOptions<IApiClient>, IPlatformHealthCheckableOptions<ApiConnectorHealthCheck<OtherServiceClientOptions>>
{  
    public const string DefaultSectionName = nameof(PlatformStoreOrderClientOptions);
    public static PlatformStoreOrderClientOptions Default => new PlatformStoreOrderClientOptions();
    public HealthCheckMode HealthCheckMode { get; set; } = HealthCheckMode.Enabled;
}

{
  "PlatformStoreOrderClientOptions": {
    "HealthCheckMode": "Enabled"
    // "HealthCheckMode": "EnabledButOverwriteUnhealthyWithDegraded"
    // "HealthCheckMode": "SkipAsHealthy"
  }
}

Implementing platform health checks

Platform readiness health check

You have two options of platform readiness health check implementation direct and indirect. You also need to add the ASOL.Core.HealthChecks NuGet package reference to your service.

Direct implementation

Direct implementation of a platform readiness health check inherits from the generic abstract class PlatformHealthCheck. This generic abstract class requires a generic typed parameter that implements the IPlatformHealthCheckOptions interface.

using System;
using System.Threading;
using System.Threading.Tasks;
using ASOL.Core.HealthChecks;
using ASOL.Core.HealthChecks.Options;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using Microsoft.Extensions.Options;

public class SampleHealthCheck(
    IOptions<SampleOptions> sampleOptions,
    ILogger<SampleHealthCheck> logger,
    IMemoryCache memoryCache,
    IOptions<PlatformHealthCheckOptions> platformHealthCheckOptions)
    : PlatformHealthCheck<SampleOptions>(sampleOptions.Value, logger, memoryCache, platformHealthCheckOptions)
{
    protected override Task<HealthCheckResult> PlatformHealthCheckAsync(HealthCheckContext context, CancellationToken cancellationToken)
    {
        return Task.FromResult(HealthCheckResult.Healthy());
        // or
        // return Task.FromResult(HealthCheckResult.Unhealthy("Resource is not available"));
        // or
        //return Task.FromResult(HealthCheckResult.Degraded("Resource is degraded"));                         
    }
}

public class SampleOptions : IPlatformHealthCheckOptions
{
    public HealthCheckMode HealthCheckMode { get; set; } = HealthCheckMode.Enabled;
    //public HealthCheckMode HealthCheckMode { get; set; } = HealthCheckMode.EnabledButOverwriteUnhealthyWithDegraded;
    //public HealthCheckMode HealthCheckMode { get; set; } = HealthCheckMode.SkipAsHealthy;    
}                      

Optional configuration in the target API

{
  "SampleOptions": {
    "HealthCheckMode": "Enabled"
    // "HealthCheckMode": "EnabledButOverwriteUnhealthyWithDegraded"
    // "HealthCheckMode": "SkipAsHealthy"
  }
}

Indirect implementation

The indirect implementation of platform readiness health check inherits from the non-generic abstract class PlatformHealthCheck and implements the IPlatformHealthCheckOptions interface. The class itself must be generic. The generic type parameter must implement IPlatformHealthCheckableOptions.

using System;
using System.Threading;
using System.Threading.Tasks;
using ASOL.Core.ApiConnector.Options;
using ASOL.Core.HealthChecks;
using ASOL.Core.HealthChecks.Options;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;

public class SampleHealthCheck<T>(
    IOptions<T> options,
    ILogger<SampleHealthCheck<T>> logger,
    IMemoryCache memoryCache,
    IOptions<PlatformHealthCheckOptions> platformHealthCheckOptions)
    : PlatformHealthCheck(options.Value.HealthCheckMode, logger, memoryCache, platformHealthCheckOptions)
    where T : class, IPlatformHealthCheckOptions, new()
{
    protected override Task<HealthCheckResult> PlatformHealthCheckAsync(HealthCheckContext context, CancellationToken cancellationToken)
    {
        return Task.FromResult(HealthCheckResult.Healthy());
        // or
        // return Task.FromResult(HealthCheckResult.Unhealthy("Resource is not available"));
        // or
        //return Task.FromResult(HealthCheckResult.Degraded("Resource is degraded"));  
    }
}

using ASOL.Core.ApiConnector.HealthChecks;
using ASOL.Core.ApiConnector.Options;
using ASOL.Core.HealthChecks;

public class SampleOptions : IPlatformHealthCheckableOptions<SampleHealthCheck<SampleOptions>> 
{   
    public HealthCheckMode HealthCheckMode { get; set; } = HealthCheckMode.Enabled;
}

{
  "SampleOptions": {
    "HealthCheckMode": "Enabled"
    // "HealthCheckMode": "EnabledButOverwriteUnhealthyWithDegraded"
    // "HealthCheckMode": "SkipAsHealthy"
  }
}

Platform startup health check

You can also implement and register a startup health check.

using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Diagnostics.HealthChecks;

public class StartupHealthCheck : IHealthCheck
{
    private volatile bool isReady;

    public bool StartupCompleted
    {
        get => isReady;
        set => isReady = value;
    }

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        if (StartupCompleted)
        {
            return Task.FromResult(HealthCheckResult.Healthy("The startup task has completed."));
        }

        return Task.FromResult(HealthCheckResult.Unhealthy("That startup task is still running."));
    }
}

using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;

public class StartupService(StartupHealthCheck startupService) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // Simulate the effect of a long-running task.
        await Task.Delay(TimeSpan.FromSeconds(15), stoppingToken);

        startupService.StartupCompleted = true;
    }
}

services.AddSingleton<StartupHealthCheck>();
services.AddHostedService<StartupService>();
services.AddHealthChecks()
    .AddCheck<StartupHealthCheck>(
        nameof(StartupHealthCheck),
        tags: [PlatformHealthCheckTags.Startup]); // namespace: ASOL.Core.HealthChecks

External links

• https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks
• https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/monitor-app-health
• https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/