• Table of contents
• Configuration storage
  • Storage architecture considerations
  • Client library
  • Client-side caching
    • Storage
    • Real-time updates (chapter to update and review)
  • Backends
    • Internal
    • Local
    • Remote
      • Parameters
      • Connection request
      • TLS encryption
      • Protocol version and authentication
      • Resiliency (chapter to update and review)

Configuration storage¶

Storage architecture considerations¶

The module can be loaded in master and replica redis-server processes, in a single-master or multi-master (distributed) database. Thus it can not rely on the locally available data only to get the details about the underlying databases. The configuration has to be storable in a Redis database with standard native datastructures to have the benefits of RESP3, replication (HA), persistency, backups, client side caching (via RESP2 PubSub or RESP3 Client Tracking )... and the module has to support a remote endpoint (TCP or Unix socket) to connect to this database. The module has to support at least module parameters and configuration file to get the details to connect to this configuration database. This way, the configuration database can be remote or local, distributed or not, HA or not, Enterprise or Community. The module can act as a standard Redis client on a standard Redis database to fetch its configuration.

Such a storage can support:

• both Redis Community and Redis Enterprise,
• both sharded (distributed) and single-process databases,
• a centralized configuration database hosting several distinct SmartCache database instances (multi-tenancy).

This means that the connection configuration database can either be collocated with the smartcache cached data within the same Redis database or not. This supports flexible architectures from the simplest one with everything in a single-master single-database with both the configuration and the data, to a complex architecture with several sets of caches in each smartcache database, and several smartcache databases using a single configuration database.

Currently, dynamic configuration database changes are not yet supported, a configuration database change implies a module unload and reload with different arguments.

Client library¶

Given that the hiredis API is embedded into redis-server binaries, it can be used from the module without explicit extra linking. Hiredis supports server TLS encryption and validation and Client TLS certificates and asynchronous non-blocking commands with callbacks. Unfortunately, hiredis does not support redis-sentinel and redis-cluster protocols, these have to be implemented by the module (not yet implemented). All the configuration database connections and operations leverage the asynchronous hiredis API to avoid blocking the Redis event loop or wasting CPU cycles.

Currently, the module supports any Redis Enterprise database configuration (with or without HA, with or without clustering) via Redis-Enterprise proxy and endpoints, and single instance Redis Community database (not yet Redis Community cluster or sentinel modes).

Client-side caching¶

Client-side cachine is available both for remote and local backends.

Storage¶

By default, the module queries the configuration database everytime it is needed. If client side caching is enabled, the module stores an internal copy of the configuration values fetched from the configuration database and use them for all read operations. The write operations are executed on the configuration database and updated internally, anticipating the notification for consistency in case of read-my-write pattern. If RESP3 protocol negociation was requested and succeeded, it relies on PUSH notifications, otherwise it relies on a dedicated PUBSUB channel.

If the authentication succeeded using RESP3, The module send a CLIENT TRACKING request to receive changes via PUSH notifications, registers a push callback to manage the notifications and automatically update the internally stored values. If the RESP2 authentication was used, the module opens a second connection to the configuration database, subscribes to a <PREFIX>connections:changes channel, registers a callback to manage the notifications and automatically update the internally stored values. RESP2 client-side-caching is not yet supported.

Real-time updates (chapter to update and review)¶

Finally, CLIENT TRACKING is enabled to receive update PUSH notifications from the configuration database each time that an already fetched data is updated. Thus, whatever the architecture is, whereever a configuration is changed from, every module instance of the database will be notified to update its own configuration in real-time.

Every write operation on the configuration data is directly made to the configuration database only, not to the local redis-server process' internal structures. Then, the database sends PUSH notifications to all the connected redis-server process so that they can apply the changes in their internal structure. Every read operation is made from the locap redis-server process' internal structure if the data is present. If the data is not present either because the key does not exist or because the process missed a PUSH notification, the process attempts a read from the database to fix the potential message miss situation and return an consistent reply.

The configuration databases uses a SET to store the list of defined connections and a dedicated HASH for each connection details (Connections_specifications).

The SET is read at connect/reconnect and used to preload all the existing connections at connect. The PUSH messages are used to be notified in case of a connection addition (to fetch the added connection and implicitely subscribe to changes) or removal (to remove the connection from the list and potentially remove the connection details if not already removed by another PUSH message). Connection list changes are applied incrementally if possible, only adding new connection to the internal structure and removing old ones without changing the current connections.

The individual and dedicated HASH are read and applied to the internal structures as soon as the module is not uptodate (initial load from the SET, addition/removal triggered by the SET, removal or changes triggered by a direct PUSH). A deletion removes both the details from the internal structure and the connection from the internal list.

Deletions are managed twice to limit inconsistencies in case of configuration database connectivity and notification loss.

Backends¶

The module configuration can be stored in different backends to match the needs and constraints from the simpliest to the more complex architectures.

Common parameters to all backends (default value is underlined)

• PREFIX [ scache:, any string] Cachegroup name, aka tenant (eg accounting, orders, frontend, ...)
• CFGDB [ local, internal, remote] Configuration database backend, see below

Internal¶

This backend stores the module configuration in modules internal structures, in RAM. The backend supports replication and persistency (not yet implemented). This means that the backup files can only be loaded in Redis databases with the module loaded, which can be an issue if the backups are for long-term storage (will the module still be available) and if the backups are for regulatory constraints, the module is loaded in the database and could trigger actions on the data immediately after restoration. The configuration can only be set and updated using the modules Redis commands, no access is possible otherwise. This storage does not need any change notification and event handling. It can not handle multi-instances databases, the module has to be loaded in master and replica instances, can synchronise its configuration between master and replica, but it can not scale to a multi-master deployment (redis-cluster or Redis-Enterprise cluster). It is probably the simpliest and fastest storage backend.

This storage does not need any specific parameter. It stores all the information in internal structure, in the module RAM.

Local¶

This backend stores the module configuration in Redis standard data structures in the local Redis instance. It relies on the instances replication and persistency. The configuration can be set and updated with standard Redis commands, despite not recommended, the module has to rely on keyspace notifications to be notified in case of configuration changes (not yet implemented). The persistency files can be loaded in a Redis database without the module, making this storage ideal for regulatory constraints. It can not scale to multi-master databases (redis-cluster or Redis-Enterprise cluster).

This storage supports the following parameters.

• CSCACHING [ enabled, disabled] Client side caching in internal module structures.

Remote¶

The remote storage uses a standard Redis connection to support the multi-master deployments with redis-sentinel support (not yet implemented), redis-cluster support (not yet implemented) and Redis-Enterprise proxy-endpoints support. The connection supports TLS encryption with or without mutual TLS, with or without peer certificate validation, it supports RESP2 and RESP3 protocol, it supports authentication (no authentication, password only authentication, ACLs user and password authentication), it supports client side caching to keep an up to date copy of the whole configuration in its internal structures using RESP3 client tracking or RESP2 PubSub (not yet implemented). It can use a prefix to all its keys, enabling multiple configurations in a single configuration database for multi-tenancy. It supports the database where the module is loaded (self) or any other external remote Redis database. The configuration details are stored in Redis standard data-structures to leverage clustering, persistency and replication for HA.

Parameters¶

Currently the Redis connection to the SmartCache configuration database has to be specified as module arguments at loading time. At least one endpoint must be specified, if several endpoints are specified, they will be tried with a round-robin algorithm until a valid connection or the end of the list. Several endpoints are not yet supported

As soon as a connection is established, the module will try to detect a redis-cluster database with CLUSTER NODES and potentially switch to the redis-cluster protocol. redis-cluster and redis-sentinel protocols are not yet supported

SmartCache config db endpoints:

• HOSTPORT Hostname or IP address, followed by port number (defaults to 127.0.0.1:6379). The host part has to be either a local unix socket absolute file path with a port explicitely set to 0 (ending with :0 or :), or it can be a remote host (IP or name) with (hostname:port) or without a port number (hostname).

Global options:

• PROTOCOL [defaults: auto] resp2, resp3 or auto
• CSCACHING [defaults: enabled] enabled, disabled
• USERNAME [defaults: empty string]
• PASSWORD [defaults: empty string]
• TIMEOUT [defaults: 10] Connection timeout in ms
• RETRYDELAY [defaults: 2000] Connection retry (if lost or not open) delay in ms
• TLS_USE [defaults: no]
• TLS_CHECK [defaults: yes] check or not the peer certificate against the CA chain
• TLS_CACERT [defaults: empty string] cacert absolute filename
• TLS_CERT [defaults: empty string] client certificate absolute filename (if mutual TLS auth is required server-side)
• TLS_KEY [defaults: empty string] client private key absolute filename (if mutual TLS auth is required server-side)

The credential are used for the authentication (unfortunately the module arguments used to configure the credentials are visible with the HELLO and the MODULE LIST commands and the module does not yet support parameters in the configuration file).

Connection request¶

As soon as the module loads, while is it initialized, it tries to open the connection. It the configuration database is the database with the module (itself) and if the module loads at redis-server startup, the module is initialized before the database reloads the persistency RDB file and before listening for incoming connections. The requested connection has to be non-blocking and asynchronous to avoid a deadlock, with a configurable timeout, and an auto-retry delay. In such a case, the first connection try is expected to fail, but the second will succeed. The auto-retry triggers when a connection request fails and each time a disconnect occurs (connection loss).

It makes use of a timer, unloading the module while auto-retries is almost impossible.

TLS encryption¶

If TLS encryption is requested, the module starts it using the provided parameters as soon as the connection successfully opens. The module supports peer certificate verification or not, and can also use a client certificate/key pair to authenticate the connection against the server if requested by the configuration database.

Protocol version and authentication¶

If the requested protocol is RESP3 or AUTO, the modules tries RESP3 negociation with authentication, in case of failure, if the requested protocol is AUTO, it tries a RESP2 authentication.
Otherwise, it the specified protocol is RESP2, it directly tries a RESP2 authentication.

If an optional username or password parameter where provided, the module will use them to authenticate on the server. Password only authentication is supported in RESP2 and also in RESP3 (with the default username automatically used by the module in this case), ACLs' username and password usage is recommended for security. The only recommended ACL for the module user account is +@all ~*

Resiliency (chapter to update and review)¶

The configuration database is read at startup and then rarely used to persist configuration changes and to propagate them. Furthermore, these configuration access are not on the data access path, they have no impact on the performances. The strategy is to keep on reconnecting when the connection is lost, without any other special processing. This is compliant with both redis enterprise and redis community HA management. The module continues to use internally stored configuration and will not receive configuration update anymore.

When the connection is successfully reopenned, the whole configuration is read again and applied internally. As for any configuration update, if a change is detected, it will trigger a configChange event to let the underlying connection manage this change accordingly to their own policy.

It is up to each of the underlying cache connection to define a policy in case of configuration database connectivity loss, using onConnectionLoss, onReconnect, onDisconnectPre, onDisconnectPost, onConfigChange callbacks.

If the RETRYDELAY parameter is too low, it will quickly fill the logs in case of a config db connection issue. On the other hand, it is always triggered when the config db is the same as the Redis instance loading the module because the module is initialized (and first tries to open the connection) immediately before the database is ready. Thus, a long delay can impact the global database SLA when a redis process is restarted.