• Table of contents
• Configuration storage
  • Storage architecture considerations
  • Client library
  • Configuration connection
    • Connection request
    • TLS encryption
    • RESP3 and authentication
    • Real-time updates
  • Resiliency
  • Configuration database endpoint
  • Benefits
  • Limitations

Configuration storage¶

Storage architecture considerations¶

The module can be loaded in master and replica redis-server processes, in a single-master or multi-master (distributed) database. Thus it can not rely on the locally available data to get the details about the underlying databases. The configuration is stored in a Redis database with standard native datastructures to have the benefits of RESP3 (for client-side caching), replication (HA), persistency, backups, ... and the module use an external endpoint (TCP or Unix socket) to connect to this database. The module use parameters to get the details to connect to the configuration database. This way, the configuration database can be remote or local, distributed or not, HA or not, Enterprise or Community. The module acts as a standard Redis client on a standard Redis database to fetch its configuration.

Client library¶

Given that the hiredis API is embedded into redis-server binaries, it can be used from the module without explicit extra linking. Unfortunately, hiredis does not support redis-sentinel and redis-cluster protocols.

Currently, the module does neither support redis_cluster protocol nor redis_sentinel protocol. Thus, it can use any database configuration in redis_enterprise, but only single master without replication database in Redis community.

Configuration connection¶

Connection request¶

As soon as the module loads, while is it initialized, it tries to open the connection. It the configuration database is the database with the module (itself) and if the module loads at redis-server startup, the module is initialized before the database reloads the persistency RDB file and before listening. The requested connection is non-blocking, asynchronous to avoid a deadlock, with a configurable timeout, and an auto-retry delay. In such a case, the first connection try is expected to fail, but the second will succeed. The auto-retry also triggers as long as the connection opening fails and everytime a disconnect occurs (connection loss).

It makes use of a timer, unloading the module while it auto-retries is almost impossible.

TLS encryption¶

If TLS encryption was requested with the TLS_USE parameter, the module starts it using the provided parameters as soon as the connection successfully opens. The module supports peer certificate verification or not with the TLS_CHECK and TLS_CACERT parameters, and can also use a client certificate/key pair with TLS_KEY and TLS_CERT to authenticate the connection against the server if requested by the configuration database.

RESP3 and authentication¶

The module relies on Redis' CLIENT TRACKING feature with PUSH notifications which are available from the RESP3 protocol version. I did not implement a PUBSUB based workaround and assume that the configuration database will support RESP3. The module starts by negociating a switch from RESP2 to RESP3. RESP3 is a pre-requisite, RESP2 is not supported.

If the optional USERNAME or PASSWORD parameter where provided, the module will use them to authenticate on the server. Password only authentication is currently still supported but ACLs' user/pass usage is recommended for security. The only recommended ACL for the module user account is +@all ~*

Real-time updates¶

Finally, CLIENT TRACKING is enabled to receive update PUSH notifications from the configuration database each time that an already fetched data is updated. Thus, whatever the architecture is, whereever a configuration is changed from, every module instance of the database will be notified to update its own configuration in real-time.

Resiliency¶

The configuration database is read at startup and then rarely used to persist configuration changes and to propagate them. Furthermore, these configuration access are not on the data access path, they have no impact on the performances. The strategy is to keep on reconnecting when the connection is lost, without any other special processing. This is compliant with both redis enterprise and redis community HA management. The module continues to use internally stored configuration and will not receive configuration update anymore.

When the connection is successfully reopenned, the whole configuration is read again and applied internally. As for any configuration update, if a change is detected, it will trigger a configChange event to let the underlying connection manage this change accordingly to their own policy.

It is up to each of the underlying cache connection to define a policy in case of configuration database connectivity loss, using onConnectionLoss, onReconnect, onDisconnectPre, onDisconnectPost, onConfigChange callbacks.

If the RETRYDELAY parameter is too low, it will quickly fill the logs in case of a config db connection issue. On the other hand, it is always triggered when the config db is the same as the Redis instance loading the module because the module is initialized (and first tries to open the connection) immediately before the database is ready. Thus, a long delay can impact the global database SLA when a redis process is restarted.

Configuration database endpoint¶

The Redis connection to the SmartCache configuration database has to be specified as module arguments at loading time. At least one endpoint must be specified, if several endpoints are specified, they will be tried with a round-robin algorithm until a valid connection or the end of the list. The module currently supports only one endpoint specification. The last one (from left to right) takes precedence.

As soon as a connection is established, the module will try to detect a redis-cluster database with CLUSTER NODES and potentially switch to the redis-cluster protocol. redis-cluster and redis-sentinel protocols are not yet supported by the module.

SmartCache config db endpoints (at least one specified):

• HOSTPORT Hostname or IP address, followed by port number (defaults to 127.0.0.1:6379). The host part has to be either a local unix socket absolute file path with a port explicitely set to 0 (ending with :0 or :), or it can be a remote host (IP or name) with (hostname:port) or without (hostname) a port number.

Global options:

• USERNAME [defaults: empty string]
• PASSWORD [defaults: empty string]
• TIMEOUT [defaults: 10] Connection timeout in ms
• RETRYDELAY [defaults: 500] Connection retry (if lost or not open) delay in ms
• TLS_USE [defaults: no]
• TLS_CHECK [defaults: yes] check or not the peer certificate against the CA chain
• TLS_CACERT [defaults: empty string] cacert absolute filename
• TLS_CERT [defaults: empty string] client certificate absolute filename (if mutual TLS auth is required server-side)
• TLS_KEY [defaults: empty string] client private key absolute filename (if mutual TLS auth is required server-side)
• PREFIX [defaults: scache:] Cachegroup name, aka tenant (eg accounting, orders, frontend, ...)

The credential are used for the authentication (unfortunately the module arguments used to configure the credentials are visible with the HELLO and the MODULE LIST commands and the module does not yet support credentials in the configuration file).

Benefits¶

Such a storage supports :

• both Redis Community and Redis Enterprise,
• both sharded (distributed) and single-process databases,
• a centralized configuration database hosting several distinct SmartCache databases.

This means that the connection configuration database can either be collocated with the smartcache cached data within the same Redis database or not.

This supports flexible architectures from the simplest one with everything in a single-master single-database with both the configuration and the data, to a complex architecture with several sets of caches in each smartcache database, and several smartcache databases using a single configuration database.

Limitations¶

A configuration database change implies a module unload and reload with different arguments.