The module can be loaded in master and replica redis-server processes, in a single-master or multi-master (distributed) database. Thus it can not rely on the locally available data to get the details about the underlying databases. The configuration is stored in a Redis database with standard native datastructures to have the benefits of RESP3 (for client-side caching), replication (HA), persistency, backups, ... and the module use an external endpoint (TCP or Unix socket) to connect to this database. The module use parameters to get the details to connect to the configuration database. This way, the configuration database can be remote or local, distributed or not, HA or not, Enterprise or Community. The module acts as a standard Redis client on a standard Redis database to fetch its configuration.
Given that the hiredis API is embedded into redis-server binaries, it can be used from the module without explicit extra linking. Unfortunately, hiredis does not support redis-sentinel and redis-cluster protocols. All the Configuration database connection and access leverage the ASynchronous hiredis API to avoid blocking and wasting CPU cycles.
Currently, the module does neither support redis_cluster protocol nor redis_sentinel protocol. Thus, it can use any database configuration in redis_enterprise, but only single master without replication database in Redis community.
As soon as the module loads, while is it initialized, it tries to open the connection. It the configuration database is the database with the module (itself) and if the module loads at redis-server startup, the module is initialized before the database reloads the persistency RDB file and before listening. The requested connection is non-blocking, asynchronous to avoid a deadlock, with a configurable timeout, and an auto-retry delay. In such a case, the first connection try is expected to fail, but the second will succeed. The auto-retry also triggers as long as the connection opening fails and everytime a disconnect occurs (connection loss).
It makes use of a timer, unloading the module while it auto-retries is almost impossible.
If TLS encryption was requested with the TLS_USE parameter, the module starts it using the provided parameters as soon as the connection successfully opens. The module supports peer certificate verification or not with the TLS_CHECK and TLS_CACERT parameters, and can also use a client certificate/key pair with TLS_KEY and TLS_CERT to authenticate the connection against the server if requested by the configuration database.
The module relies on Redis' CLIENT TRACKING feature with PUSH notifications which are available from the RESP3 protocol version. I did not implement a PUBSUB based workaround and assume that the configuration database will support RESP3. The module starts by negociating a switch from RESP2 to RESP3. RESP3 is a pre-requisite, RESP2 is not supported.
If the optional USERNAME or PASSWORD parameter where provided, the module will use them to authenticate on the server. Password only authentication is currently still supported but ACLs' user/pass usage is recommended for security. The only recommended ACL for the module user account is +@all ~*
Finally, CLIENT TRACKING is enabled to receive update PUSH notifications from the configuration database each time that an already fetched data is updated. Thus, whatever the architecture is, whereever a configuration is changed from, every module instance of the database will be notified to update its own configuration in real-time.
Every write operation on the configuration data is directly made to the configuration database only, not to the local redis-server process' internal structures. Then, the database sends PUSH notifications to all the connected redis-server process so that they can apply the changes in their internal structure. Every read operation is made from the locap redis-server process' internal structure if the data is present. If the data is not present either because the key does not exist or because the process missed a PUSH notification, the process attempts a read from the database to fix the potential message miss situation and return an consistent reply.
The configuration databases uses a SET to store the list of defined connections and a dedicated HASH for each connection details (Connections_specifications).
The SET is read at connect/reconnect and used to preload all the existing connections at connect. The PUSH messages are used to be notified in case of a connection addition (to fetch the added connection and implicitely subscribe to changes) or removal (to remove the connection from the list and potentially remove the connection details if not already removed by another PUSH message). Connection list changes are applied incrementally if possible, only adding new connection to the internal structure and removing old ones without changing the current connections.
The individual and dedicated HASH are read and applied to the internal structures as soon as the module is not uptodate (initial load from the SET, addition/removal triggered by the SET, removal or changes triggered by a direct PUSH). A deletion removes both the details from the internal structure and the connection from the internal list.
Deletions are managed twice to limit inconsistencies in case of configuration database connectivity and notification loss.
The configuration database is read at startup and then rarely used to persist configuration changes and to propagate them. Furthermore, these configuration access are not on the data access path, they have no impact on the performances. The strategy is to keep on reconnecting when the connection is lost, without any other special processing. This is compliant with both redis enterprise and redis community HA management. The module continues to use internally stored configuration and will not receive configuration update anymore.
When the connection is successfully reopenned, the whole configuration is read again and applied internally. As for any configuration update, if a change is detected, it will trigger a configChange event to let the underlying connection manage this change accordingly to their own policy.
It is up to each of the underlying cache connection to define a policy in case of configuration database connectivity loss, using onConnectionLoss, onReconnect, onDisconnectPre, onDisconnectPost, onConfigChange callbacks.
If the RETRYDELAY parameter is too low, it will quickly fill the logs in case of a config db connection issue. On the other hand, it is always triggered when the config db is the same as the Redis instance loading the module because the module is initialized (and first tries to open the connection) immediately before the database is ready. Thus, a long delay can impact the global database SLA when a redis process is restarted.
The Redis connection to the SmartCache configuration database has to be specified as module arguments at loading time. At least one endpoint must be specified, if several endpoints are specified, they will be tried with a round-robin algorithm until a valid connection or the end of the list. The module currently supports only one endpoint specification. The last one (from left to right) takes precedence.
As soon as a connection is established, the module will try to detect a redis-cluster database with CLUSTER NODES and potentially switch to the redis-cluster protocol. redis-cluster and redis-sentinel protocols are not yet supported by the module.
SmartCache config db endpoints (at least one specified):
127.0.0.1:6379). The host part has to be either a local unix socket absolute file path with a port explicitely set to 0 (ending with :0 or :), or it can be a remote host (IP or name) with (hostname:port) or without (hostname) a port number.Global options:
The credential are used for the authentication (unfortunately the module arguments used to configure the credentials are visible with the HELLO and the MODULE LIST commands and the module does not yet support credentials in the configuration file).
Such a storage supports :
This means that the connection configuration database can either be collocated with the smartcache cached data within the same Redis database or not.
This supports flexible architectures from the simplest one with everything in a single-master single-database with both the configuration and the data, to a complex architecture with several sets of caches in each smartcache database, and several smartcache databases using a single configuration database.
A configuration database change implies a module unload and reload with different arguments.