{{>toc}} h1. Configuration storage h2. Storage architecture considerations The module can be loaded in master and replica _redis-server_ processes, in a single-master or multi-master (distributed) database. Thus it can not rely on the locally available data to get the details about the underlying databases. The configuration is stored in a Redis database with standard native datastructures to have the benefits of RESP3 (for client-side caching), replication (HA), persistency, backups, ... and the module use an external endpoint (TCP or Unix socket) to connect to this database. The module use parameters to get the details to connect to the configuration database. This way, the configuration database can be remote or local, distributed or not, HA or not, Enterprise or Community. The module acts as a standard Redis client on a standard Redis database to fetch its configuration. h2. Client library Given that the _hiredis_ API is embedded into _redis-server_ binaries, it can be used from the module without explicit extra linking. Unfortunately, _hiredis_ does not support _redis-sentinel_ and _redis-cluster_ protocols. All the Configuration database connection and access leverage the ASynchronous _hiredis_ API to avoid blocking and wasting CPU cycles. *Currently, the module does neither support _redis_cluster_ protocol nor _redis_sentinel_ protocol. Thus, it can use any database configuration in _redis_enterprise_, but only single master without replication database in Redis community.* h2. Configuration connection h3. Connection request As soon as the module loads, while is it initialized, it tries to open the connection. It the configuration database is the database with the module (itself) and if the module loads at _redis-server_ startup, the module is initialized before the database reloads the persistency RDB file and before listening. The requested connection is non-blocking, asynchronous to avoid a deadlock, with a configurable _timeout_, and an auto-retry delay. In such a case, the first connection try is expected to fail, but the second will succeed. The auto-retry also triggers as long as the connection opening fails and everytime a disconnect occurs (connection loss). It makes use of a timer, *unloading the module while it auto-retries is almost impossible*. h3. TLS encryption If TLS encryption was requested with the @TLS_USE@ parameter, the module starts it using the provided parameters as soon as the connection successfully opens. The module supports peer certificate verification or not with the @TLS_CHECK@ and @TLS_CACERT@ parameters, and can also use a client certificate/key pair with @TLS_KEY@ and @TLS_CERT@ to authenticate the connection against the server if requested by the configuration database. h3. RESP3 and authentication The module relies on Redis' @CLIENT TRACKING@ feature with _PUSH_ notifications which are available from the _RESP3_ protocol version. I did not implement a @PUBSUB@ based workaround and assume that the configuration database will support _RESP3_. The module starts by negociating a switch from _RESP2_ to _RESP3_. *_RESP3_ is a pre-requisite, _RESP2_ is not supported.* If the optional @USERNAME@ or @PASSWORD@ parameter where provided, the module will use them to authenticate on the server. Password only authentication is currently still supported but ACLs' user/pass usage is recommended for security. *The only recommended ACL for the module user account is @+@all ~*@* h3. Real-time updates Finally, @CLIENT TRACKING@ is enabled to receive update _PUSH_ notifications from the configuration database each time that an already fetched data is updated. Thus, whatever the architecture is, whereever a configuration is changed from, every module instance of the database will be notified to update its own configuration in real-time. Every write operation on the configuration data is directly made to the configuration database only, not to the local _redis-server_ process' internal structures. Then, the database sends PUSH notifications to all the connected _redis-server_ process so that they can apply the changes in their internal structure. Every read operation is made from the locap _redis-server_ process' internal structure if the data is present. If the data is not present either because the key does not exist or because the process missed a PUSH notification, the process attempts a read from the database to fix the potential message miss situation and return an consistent reply. The configuration databases uses a _SET_ to store the list of defined connections and a dedicated _HASH_ for each connection details ([[Connections_specifications]]). The _SET_ is read at connect/reconnect and used to preload all the existing connections at connect. The _PUSH_ messages are used to be notified in case of a connection addition (to fetch the added connection and implicitely subscribe to changes) or removal (to remove the connection from the list and potentially remove the connection details if not already removed by another _PUSH_ message). Connection list changes are applied incrementally if possible, only adding new connection to the internal structure and removing old ones without changing the current connections. The individual and dedicated _HASH_ are read and applied to the internal structures as soon as the module is not uptodate (initial load from the _SET_, addition/removal triggered by the _SET_, removal or changes triggered by a direct _PUSH_). A deletion removes both the details from the internal structure and the connection from the internal list. Deletions are managed twice to limit inconsistencies in case of configuration database connectivity and notification loss. h2. Resiliency The configuration database is read at startup and then rarely used to persist configuration changes and to propagate them. Furthermore, these configuration access are not on the data access path, they have no impact on the performances. The strategy is to keep on reconnecting when the connection is lost, without any other special processing. This is compliant with both _redis enterprise_ and _redis community_ HA management. The module continues to use internally stored configuration and will not receive configuration update anymore. When the connection is successfully reopenned, the whole configuration is read again and applied internally. As for any configuration update, if a change is detected, it will trigger a configChange event to let the underlying connection manage this change accordingly to their own policy. It is up to each of the underlying cache connection to define a policy in case of configuration database connectivity loss, using @onConnectionLoss@, @onReconnect@, @onDisconnectPre@, @onDisconnectPost@, @onConfigChange@ callbacks. If the @RETRYDELAY@ parameter is too low, it will quickly fill the logs in case of a config db connection issue. On the other hand, it is always triggered when the config db is the same as the Redis instance loading the module because the module is initialized (and first tries to open the connection) immediately before the database is ready. Thus, a long delay can impact the global database SLA when a redis process is restarted. h2. Configuration database endpoint The Redis connection to the _SmartCache_ configuration database has to be specified as module arguments at loading time. At least one endpoint must be specified, if several endpoints are specified, they will be tried with a round-robin algorithm until a valid connection or the end of the list. *The module currently supports only one endpoint specification. The last one (from left to right) takes precedence*. As soon as a connection is established, the module will try to detect a _redis-cluster_ database with @CLUSTER NODES@ and potentially switch to the _redis-cluster_ protocol. *_redis-cluster_ and _redis-sentinel_ protocols are not yet supported by the module*. SmartCache config db endpoints (at least one specified): * *HOSTPORT* Hostname or IP address, followed by port number (defaults to @127.0.0.1:6379@). The host part has to be either a local unix socket absolute file path with a port explicitely set to 0 (ending with @:0@ or @:@), or it can be a remote host (IP or name) with (@hostname:port@) or without (@hostname@) a port number. Global options: * *USERNAME* [defaults: empty string] * *PASSWORD* [defaults: empty string] * *TIMEOUT* [defaults: 10] Connection timeout in ms * *RETRYDELAY* [defaults: 500] Connection retry (if lost or not open) delay in ms * *TLS_USE* [defaults: no] * *TLS_CHECK* [defaults: yes] check or not the peer certificate against the CA chain * *TLS_CACERT* [defaults: empty string] cacert absolute filename * *TLS_CERT* [defaults: empty string] client certificate absolute filename (if mutual TLS auth is required server-side) * *TLS_KEY* [defaults: empty string] client private key absolute filename (if mutual TLS auth is required server-side) * *PREFIX* [defaults: scache:] Cachegroup name, aka tenant (eg accounting, orders, frontend, ...) The credential are used for the authentication (unfortunately the module arguments used to configure *the credentials are visible with the @HELLO@ and the @MODULE LIST@ commands* and the module does *not yet support credentials in the configuration file*). h2. Benefits Such a storage supports : * both Redis Community and Redis Enterprise, * both sharded (distributed) and single-process databases, * a centralized configuration database hosting several distinct _SmartCache_ databases. This means that the connection configuration database can either be collocated with the _smartcache_ cached data within the same Redis database or not. This supports flexible architectures from the simplest one with everything in a single-master single-database with both the configuration and the data, to a complex architecture with several sets of caches in each _smartcache_ database, and several _smartcache_ databases using a single configuration database. h2. Limitations A configuration database change implies a module unload and reload with different arguments.