Imagine you've built a movie streaming app. You used PostgreSQL as your data store, and as you needed to scale you implemented caching using Redis. However, now you are experiencing slowness in reflecting of updated user profile or subscription.
For example, When a user purchases or modifies subscription, user expects the changes to be reflected immediately on his account so that the desired movie/ show of new subscription is enabled for watching. So you need a way of quickly providing strong consistency of user data. In such situation, What you need is called the "write-through pattern."
With the Write-through pattern, every time an application writes data to the cache, it also updates the records in the database, unlike Write behind the thread waits in this pattern until the write to the database is also completed.
Below is a diagram of the write-through pattern for the application:
The pattern works as follows:
Note : the Redis server is blocked until a response from the main database is received.
There are two related write patterns, and the main differences between them are as follows
Learn more about Write behind pattern
Write-through caching with Redis ensures that the (critical data) cache is always up-to-date with the database, providing strong consistency and improving application performance.
consider below scenarios of different applications :
RedisGears is a programmable serverless engine for transaction, batch, and event-driven data processing allowing users to write and run their own functions on data stored in Redis.
Functions can be implemented in different languages, including Python and C, and can be executed by the RedisGears engine in one of two ways:
Some batch type operations RedisGears can do:
person:person: to a setperson: (assume all of them are of type hash)Some event type operations RedisGears can do:
I-AM-IMPORTANT: and asynchronously dump them in a "deleted keys" log fileplayer: and synchronously update a user's level when the score reaches 1000Run the Docker container:
For a very simple example that lists all keys in your Redis database with a prefix of person: create the following python script and name it hello_gears.py :
Execute your function:
The gears-cli tool provides an easier way to execute RedisGears functions, specially if you need to pass some parameters too.
It's written in Python and can be installed with pip:
Usage:
For our sample code, we will demonstrate writing users to Redis and then writing through to PostgreSQL. Use the docker-compose.yml file below to setup required environment:
To run the docker-compose file, run the following command:
This will create a Redis server, a PostgreSQL server, and an Adminer server. Adminer is a web-based database management tool that allows you to view and edit data in your database.
Next, open your browser to http://localhost:8080/?pgsql=postgres&username=root&db=example&ns=public&sql=. You will have to input the password (which is password in the example above),
then you will be taken to a SQL command page. Run the following SQL command to create a table:
Developers need to load some code (say python in our example) to the Redis server before using the write-through pattern (which syncs data from Redis to the system of record). Redis server has a RedisGears module that interprets the python code and syncs the data from Redis to the system of record.
Now, we need to create a RedisGears recipe that will write through to the PostgreSQL database. The following Python code will write through to the PostgreSQL database:
Make sure you create the file "write-through.py" because the next instructions will use it. For the purpose of this example we are showing how to map Redis hash fields to PostgreSQL table columns. The RGWriteThrough function takes in the usersMapping, where the keys are the Redis hash keys and the values are the PostgreSQL table columns.
The python file has a few dependencies in order to work. Below is the requirements.txt file that contains the dependencies, create it alongside the "write-through.py" file:
To run our write-through recipe using gears-cli, we need to run the following command:
You should get a response that says "OK". That is how you know you have successfully loaded the Python file into the Redis server.
2. Using the RG.PYEXECUTE from the Redis command line.
The next step is to verify that RedisGears is syncing data between Redis and PostgreSQL. Note that in our Python file we specified a prefix for the keys. In this case, we specified __ as the prefix, users as the table, and id as the unique identifier. This instructs RedisGears to look for the following key format: __{users:<id>}. Try running the following command in the Redis command line:
Check RedisInsight to verify that the hash value made it into Redis. After RedisGears is done processing the __{users:1} key, it will be deleted from Redis and replaced by the users:1 key. Check RedisInsight to verify that the users:1 key is in Redis.
Next, confirm that the user is inserted in PostgreSQL too by opening up the select page in Adminer. You should see the user inserted in the table.
This is how you can use RedisGears to write through to PostgreSQL, and so far we have only added a hash key. You can also update specific hash fields and it will be reflected in your PostgreSQL database. Run the following command to update the username field:
In RedisInsight, verify that the username field is updated
Now go into Adminer and check the username field. You should see that it has been updated to bar.
You now know how to use Redis for write-through caching. It's possible to incrementally adopt Redis wherever needed with different strategies/patterns. For more resources on the topic of caching, check out the links below:
Write Behind | Write through |
|---|---|
Syncs data asynchronously | Syncs data synchronously/ immediately |
Data between the cache and the system of record (database) is inconsistent for a short time | Data between the cache and the system of record (database) is always consistent |
You can skip reading this section if you are already familiar with RedisGears)
A collection of RedisGears functions and any dependencies they may have that implement a high-level functional purpose is called a recipe. Example : "RGJSONWriteThrough" function in above python code
If you are on Windows, we recommend you use WSL to install and use gears-cli.
The RG.PYEXECUTE command can also be executed from the Node.js code (Consult the sample Node file for more details)
Find more examples in the Redis Gears GitHub repository.
RedisInsight is the free redis GUI for viewing data in redis. Click here to download.
docker run -p 6379:6379 redislabs/redisgears:latestgb = GearsBuilder() gb.run('person:*')docker exec -i redisgears redis-cli RG.PYEXECUTE "`cat hello_gears.py`"pip install gears-cligears-cli hello_gears.py REQUIREMENTS rgsyncgears-cli --help
usage: gears-cli [-h] [--host HOST] [--port PORT]
[--requirements REQUIREMENTS] [--password PASSWORD] path [extra_args [extra_args ...]]version: '3.9'
services:
redis:
container_name: redis
image: 'redislabs/redismod:latest'
ports:
- 6379:6379
deploy:
replicas: 1
restart_policy:
condition: on-failure
postgres:
image: postgres
restart: always
environment:
POSTGRES_USER: root
POSTGRES_PASSWORD: password
POSTGRES_DB: example
adminer:
image: adminer
restart: always
ports:
- 8080:8080$ docker compose up -dCREATE TABLE users (
id SERIAL PRIMARY KEY,
username VARCHAR(255) UNIQUE NOT NULL,
email VARCHAR(255) UNIQUE NOT NULL,
password_hash VARCHAR(255) NOT NULL,
first_name VARCHAR(255),
last_name VARCHAR(255),
date_of_birth DATE,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);from rgsync import RGWriteThrough
from rgsync.Connectors import PostgresConnector, PostgresConnection
'''
Create Postgres connection object
'''
connection = PostgresConnection('root', 'password', 'postgres:5432/example')
'''
Create Postgres users connector
'''
usersConnector = PostgresConnector(connection, 'users', 'id')
usersMappings = {
'username': 'username',
'email': 'email',
'pwhash': 'password_hash',
'first': 'first_name',
'last': 'last_name',
'dob': 'date_of_birth',
'created_at': 'created_at',
'updated_at': 'updated_at',
}
RGWriteThrough(GB, keysPrefix='__', mappings=usersMappings,
connector=usersConnector, name='UsersWriteThrough', version='99.99.99')rgsync
psycopg2-binary
cryptography# install
pip install gears-cli$ gears-cli run --host localhost --port 6379 write-through.py --requirements requirements.txt# Via redis cli
RG.PYEXECUTE 'pythonCode' REQUIREMENTS rgsync psycopg2-binary cryptographyhset __{users:1} username john email john@gmail.com pwhash d1e8a70b5ccab1dc2f56bbf7e99f064a660c08e361a35751b9c483c88943d082 first John last Doe dob 1990-01-01 created_at 2023-04-20 updated_at 2023-04-20> hset __{users:1} username bar