Debugging Redis Module

Debugging Redis Module

Admin aaTuesday, February 20, 2018

Lately, I'm working with custom redis-modules and I need to say: redis-modules kick ass. I started working with Cthulu and coding modules with javascript which was nice and very productive but given some of my current use cases I had to go to C.

I recommend you take a look at my previous posts here:
  - Building Redis-Modules with JavaScript using Cthulhu
  - Dispatching Custom Commands with Lettuce and Redis-Modules

Today I want to you how to do proper debug with Redis modules written in C language. I will be using eclipse Oxygen CPP in order to debug redis module and redis 4.

Here there is a video which I recorded with a simple live demo about how to debug a redis-module project and redis. The source code is available on my GitHub. I hope this is useful for you.

How to Debug a Redis Module?


Slides

Redis Module Debugging from Diego Pacheco

Cheers,
Diego Pacheco

Dispatching Custom Commands with Lettuce and Redis-Modules

Dispatching Custom Commands with Lettuce and Redis-Modules

Admin aaTuesday, December 19, 2017

Lettuce is a very efficient and extensible Redis client for Java.  Lettuce is interesting for many reasons but one of the big ones is the fact that provides an Observable / Flux API. So you can very efficient reactive programming with RxJava or Reactive-Streams power by Reactor.

I'm using lettuce in production in the last 2 years and have been really great. For this blog post, I will how to dispatch custom commands to redis-modules via Lettuce.


Lettuce Features - Why Lettuce?

Lettuce is maintained by Pivotal and has many great features like:

• Non-Blocking IO
• Java 8 types: Flux | Mono
• Dynamic Commands
• Redis-Modules Support
• Cloud Ready

Building a Custom Redis Module

Some time ago I show how to build a simple redis-module using C. If you did not read the previous post please read it here. We will need that custom redis module in order to send our commands in Lettuce. There are 2 options EVAL and DISPATCH.

Sending Custom Commands via Lettuce Dispatch

EVAL is a simple way, we basically sensing some simple Lua Script returning the command and this will be run inside Redis. Let's take a look at the code:

So what do we have here? We basically create a simple connection to Redis. We are creating RedisAsyncCommands in order to leverage async and non-blocking IO. Them we do the EVAL with this: return redis.call('dp.DATE') .This means "dp.DATE" is the name of the command and we are just calling the command and get whatever it will return - for this case it will be the current date. This is fine considering that the Lua Script just call the command so the overhead is pretty minimal. The main advantage of this approach is that EVAL support is old in Redis - since 2.6 version.  There is a better approach but not all clients / eco-system is ready for it. So let's look 2 and best option.

Second Option is to use DISPATCH so we can send custom protocol(Which means raw encoded command) and a list of arguments. This will be sent straight to Redis without Lua Script. Let's take a look at the code:



So basically here we need to define a ProtocolKeyword with is pretty much the encoding of the command. Them we can get the Async command and just DISPATCH. Keep in mind this is done in Async fashion. I'm also not passing any parameters but we could if we need. 

Redis-Modules are getting more popular and this is the right way to customize Redis. Lettuce proved to be very efficient and flexible client in production for Java. Now you can wrap this EVALs or DISPATCH under some java Interfaces and have a very nice extended driver for your use cases.

You can get the full code on my GitHub here.

Cheers,
Diego Pacheco

Building Redis-Modules with JavaScript using Cthulhu

Building Redis-Modules with JavaScript using Cthulhu

Admin aa

Cthulhu is an interesting project. It allows you to code custom redis-modules in JavaScript. This is very interesting because JS is very productive and easy to write code.

You might be wondering how they made this efficient? Well, the secret is a project called Duktape which is an efficient JS engine written in C. So Cthulhu + Duktape will pre-compile your javascript function and run it on C which will be efficient and will be able to talk with redis core.

Unfortunately, not all functions are available -- you can check the ones that are available here: List of available APIS.

You can so a simple comparison with a module In C I wrote some time ago in this blog post. C is way more powerful but also way more error-prone and might take more time to write if you are not used to it. So for this Blog Post, I will how to download and compile Cthulhu and they install in Redis 4 and code some custom function in JavaScript. Let's get started.




Building Redis-Module in JavaScript



So here we are downloading and building Redis 4.0.6(Redis modules was introduced on Redis 4) and then we are making Cthulhu so Lib. As you can see we just need to build Cthulhu one time and then we link Cthulhu with our custom js function.

There are 3 JS functions: my_incr, my_append and echo. The function my_incr gets a key turns the content into a number and the increment 1 to the number and set back to the key and return to the user.

Second function my_append does text appending on existing key. Getting the current value and appending the parameter. Both functions already exist so they are here just to show how easy is to work with JS and Redis.

The last function is just an echo -- Whatever string you send you will receive back. This functions can be called on the Redis-CLI but also by any client like Jedis(>=3.0.0-SNAPSHOT) or Lettuce(4.x,5.x) versions. So its possible to call this commands by any language.

Cheers,
Diego Pacheco

Extending Redis with Redis-Modules

Extending Redis with Redis-Modules

Admin aaTuesday, November 7, 2017

Redis is a great K/V store written in C. Redis can accomplish a lot for a single thread process. There are many applications for redis like for instance: 

- Frontend Database
- Real-time Counters
- Ad Serving
- Message Queue
- Geo and TimeSeries DB
- Session State
- Cache
That's all great but let's say I want more, how can we customize redis? There are some options like a Lua Script, Fork Redis, Tak to @Antirez or Create your own NoSQL database based on redis or not. Redis provided another solution - Redis added support for external modules in 2016.

Redis Modules

Redis Modules are just dynamic libraries(.so files) loaded on redis - Often these libraries are written in C or C++ but there are some bindings where you write modules in Go or JavaScript for instance.

The best news is if you write one library you have ZERO LATENCY to access data in Redis - So this is a very big win and strong reason to write your own library.

Redis Modules enable a whole new set of extensions for you. For instance is possible to create new Datatypes. Its possible to create new Commands. It's possible also to combine existing commands and add custom new functionality using an existent command, datatypes in order to archive your goals.

There is just 1 bad news. Remember is C. C often can be fragile so you need to have great discipline and test very well your code. One wrong thing can tear down the whole redis process and I don't need to say how bad this is.

Redis Modules are supported by version 4.0. There are some interesting available models already like this ones here:
- RediSearch - Search support for Redis
- ReJSON - JSON data type and operations support for Redis
- Redis-ML - Machine Learning with Redis
- rebloom - Scalable Bloom Filters for Redis.

You can also get more information on additional modules here.

Creating a Simple Redis Module

There are some dependencies in order to build and run the redis module. Make sure you have installed:
  - docker
  - redis-cli
  - build-essential

So let's start with the Dockerfile. We will use docker to run redis 4.0 with the custom module we will build. Let's define the Dockerfile. 

Dockerfile

This Dockerfile is pretty simple we are extending the Latest Redis docker image and updating the OS for the latest packages and installing gcc and we are building the module(.so file) and loading on Redis. So when Redis process boots up on this docker container it will contain the module.

date.c

In this file(date.c) we have the module code. We basically need:
RedisModule_OnLoad: We use this function to load the module and all commands.
diegoDate_RedisCommand: The command implementation code.
getDate: A helper function which returns the current date as string.

As you might realize there is an include for redismodule.h which is the redis module API we need to interact with Redis.

redismodule.h

Here we have redis functions prototyping and also some includes of libraries like stdio, stdlib, time, stdint and so on and on. You don't need to worry much about this header file.

Makefile

We use Makefile to build the module file, Bake the docker image and also to perform some tests on the SO using docker and redis-cli.

Running

Now in bash, we can do:
$ make clean
$ make
$ make docker
$ make run 

This will build the module and bake the docker image and run the docker container. After doing that we can open redis-cli by doing: $ redis-cli

So finally we can test our command by doing $ dp.DATE and you should see the current date. That's it, folks, we have a very simple redis module working.

If you want have the full code you can get on my github here.

Cheers,
Diego Pacheco