How to use Redis for Write-behind Caching

Imagine you've built a movie streaming app. You used MongoDB as your data store, and as you needed to scale you implemented caching using Redis. This allows you to drastically speed up reads. However, now you are experiencing slowness when writing to MongoDB.

For example, maybe you want to allow users to continue watching movies where they last left off. This requires you to store the timestamp of where a user is when they decide to pause the movie. With millions of users, this is starting to cause MongoDB to slow down when you have peaks in demand.

You need a way of flattening the peaks in demand, allowing you to write data quickly and then persist it to MongoDB when the demand dies down. What you need is called the "write-behind pattern."

The pattern is simple, your application writes data to Redis and then asynchronously data gets written to MongoDB. Write operations are queued up so that the application can move on quickly and the cache can catch up over time. However, this does mean there is a short time when the data between the cache and the system of record is inconsistent.

Below is a diagram of the write-behind pattern for the application:

The pattern works as follows:

1. 1.The application reads and writes data to Redis.
2. 2.Redis syncs any changed data to the MongoDB database asynchronously.

There are two related write patterns, and the main differences between them are as follows

Learn more about Write through pattern

Consider Redis with this pattern when you need to

1. 1.Flatten peaks in demand: Under stress, an application may need to write data quickly. If your application needs to perform a large number of write operations at high speed, consider Redis. The programmability capabilities of Redis make sure the data stored in the cache is synced with the database.
2. 2.Batch multiple writes: Sometimes it's expensive to write to a database frequently (for example, logging). In those cases, it can be cost-effective to batch the database writes with Redis so that data syncs at intervals.
3. 3.Offload the primary database: Database load is reduced when heavy writes operate on Redis, So we can spread writes to improve performance during the peak time of application usage.

RedisGears is a programmable serverless engine for transaction, batch, and event-driven data processing allowing users to write and run their own functions on data stored in Redis.

Functions can be implemented in different languages, including Python and C, and can be executed by the RedisGears engine in one of two ways:

1. 1.Batch: triggered by the Run action, execution is immediate and on existing data
2. 2.Event: triggered by the Register action, execution is triggered by new events and on their data

Some batch type operations RedisGears can do:

• •Run an operation on all keys in the KeySpace or keys matching a certain pattern like :
  • •Prefix all KeyNames with person:
  • •Delete all keys whose value is smaller than zero
  • •Write all the KeyNames starting with person: to a set
• •Run a set of operations on all(or matched) keys where the output of one operation is the input of another like
  • •Find all keys with a prefix person: (assume all of them are of type hash)
  • •Increase user's days_old by 1, then sum them by age group (10-20, 20-30 etc.)
  • •Add today's stats to the sorted set of every client, calculate last 7 days average and save the computed result in a string

Some event type operations RedisGears can do:

• •RedisGears can also register event listeners that will trigger a function execution every time a watched key is changed like
  • •Listen for all operations on all keys and keep a list of all KeyNames in the KeySpace
  • •Listen for DEL operations on keys with a prefix I-AM-IMPORTANT: and asynchronously dump them in a "deleted keys" log file
  • •Listen for all HINCRBY operations on the element score of keys with a prefix player: and synchronously update a user's level when the score reaches 1000

Run the Docker container:

For a very simple example that lists all keys in your Redis database with a prefix of person: create the following python script and name it hello_gears.py :

Execute your function:

The gears-cli tool provides an easier way to execute RedisGears functions, specially if you need to pass some parameters too.

It's written in Python and can be installed with pip:

Usage:

• •RedisGears docs
• •rgsync docs
• •Installing RedisGears
• •Introduction to RedisGears blog
• •RedisGears GA - RedisConf 2020 video
• •Conference talk video by creator of RedisGears
• •Redis Gears sync with MongoDB

The demo application used in the rest of this tutorial showcases a movie application with basic create, read, update, and delete (CRUD) operations.

The movie application dashboard contains a search section at the top and a list of movie cards in the middle. The floating plus icon displays a pop-up when the user selects it, permitting the user to enter new movie details. The search section has a text search bar and a toggle link between text search and basic (that is, form-based) search. Each movie card has edit and delete icons, which are displayed when a mouse hovers over the card.

To demonstrate this pattern using the movie application, imagine that the user opens the pop-up to add a new movie.

Instead of the application immediately storing the data in MongoDB, the application writes the changes to Redis. In the background, RedisGears automatically synchronizes the data with the MongoDB database.

Developers need to load some code (say python in our example) to the Redis server before using the write-behind pattern (which syncs data from Redis to MongoDB). The Redis server has a RedisGears module that interprets the python code and syncs the data from Redis to MongoDB.

Loading the Python code is easier than it sounds. Simply replace database details in the Python file and then load the file to the Redis server.

Create the Python file (shown below, and available online). Then update the MongoDB connection details, database, collection, and primary key name to sync.

There are two ways to load that Python file into the Redis server:

1. 1.Using the gears command-line interface (CLI)

Find more information about the Gears CLI at gears-cli and rgsync.

2. Using the RG.PYEXECUTE from the Redis command line.

Find more information at RG.PYEXECUTE.

The RG.PYEXECUTE command can also be executed from the Node.js code (Consult the sample Node file for more details)

Find more examples at Redis Gears sync with MongoDB.

The next step is to verify that RedisGears is syncing data between Redis and MongoDB.

Insert a key starting with the prefix (that's specified in the Python file) using the Redis CLI

Next, confirm that the JSON is inserted in MongoDB too.

You can also check RedisInsight to verify that the data is piped in via Streams for its consumers (like RedisGears).

How does all that work with the demo application? Below is a code snipped to insert a movie. Once data is written to Redis, RedisGears automatically synchronizes it to MongoDB.

You now know how to use Redis for write-behind caching. It's possible to incrementally adopt Redis wherever needed with different strategies/patterns. For more resources on the topic of caching, check out the links below:

• •Caching with Redis
• •Write through caching
• •Cache prefetching
• •Query caching
• •Redis YouTube channel
• •Clients like Node Redis and Redis om Node help you to use Redis in Node.js applications.
• •RedisInsight : To view your Redis data or to play with raw Redis commands in the workbench
• •Try Redis Cloud for free