Showing posts with label Redis. Show all posts
Showing posts with label Redis. Show all posts

Sunday, May 10, 2020

Feature Enrichment with Spring Data Redis and Lettuce

Lettuce by far is the best Redis client out there for Java. In the java world, Annotations and Mappings are pretty popular. I build services using these concepts before however they were always under the service boundaries, never as shared libraries in order to avoid coupling. So today I record a video showing how we can use this feature and also extend the functionality to add custom enriched behavior.  We will be using Spring Boot 2, Spring Data Redis, and Lettuce as redis Client. We also will need to have redis running either as standalone / cluster or running on docker. We will create custom annotations in order to describe the behavior we want. I need to say that some people love and other people hate annotations, I have mixed feelings. Often for those that don't like it - check this out. Besides that, let's take a look. So Let's get started.


Video



Code

Model

Our model is pretty simple. Simple Person POJO with Getters/Setters, toString, equals/hashCode all generated by Idea. You can see there is a custom annotation called UpperCase. The idea is to add custom features enrichment and make sure some fields are stored in the Upper case.



Annotation

Now let's look at the annotation.



Custom Repository

Here is where the magic happens. I wrapped all CurdRepository operations and I'm delegating to the proper CrudRepository implementation at the end of the day. However I', also I have a reflection code that adds the feature enrichment I want based on the existence of the UpperCase annotation at the Entity fields.



The complete code is here.

Cheers,
Diego Pacheco

Tuesday, March 12, 2019

Running Multithreaded Redis using KeyDB and Dynomite

Keys is the default oss standard for K/V store for many use cases and solutions. When you need run low latency, high throughput IN Memory Persistence - Dynomite is your solution. I always wanted to have multi-thread support in redis like Memcached has. Now someone made it and call it KeyDB. The best thing is the fact that this is done supporting the redis RESP protocol, so all your tools, scripts and code using redis protocol directly still works. Another killer feature in KeyDB is the support for FLASH storage, only available on Redis Enterprise. KeyDB also supports backup to S3. For this post, I will show how to build and run KeyDB which is dead simple and also how to use it with Dynomite like it was Redis.





KeyDB Benchmarks

As you can see on the KeyDB benchmarks, KeyDB does 1.5x less latency and 50% more throughput. These numbers are pretty impressive. The best part is the fact that is all transparent.

Running KeyDB


Running KeyDB with Dynomite



See the Results

KeyDB running


Dynomite Running


redis-cli(via dynomite port 8102)


Other Dynomite related Posts

Cheers,
Diego Pacheco

Tuesday, February 20, 2018

Debugging Redis Module

Lately, I'm working with custom redis-modules and I need to say: redis-modules kick ass. I started working with Cthulu and coding modules with javascript which was nice and very productive but given some of my current use cases I had to go to C.

I recommend you take a look at my previous posts here:
  - Building Redis-Modules with JavaScript using Cthulhu
  - Dispatching Custom Commands with Lettuce and Redis-Modules

Today I want to you how to do proper debug with Redis modules written in C language. I will be using eclipse Oxygen CPP in order to debug redis module and redis 4.

Here there is a video which I recorded with a simple live demo about how to debug a redis-module project and redis. The source code is available on my GitHub. I hope this is useful for you.

How to Debug a Redis Module?


Slides



Cheers,
Diego Pacheco

Tuesday, December 19, 2017

Dispatching Custom Commands with Lettuce and Redis-Modules

Lettuce is a very efficient and extensible Redis client for Java.  Lettuce is interesting for many reasons but one of the big ones is the fact that provides an Observable / Flux API. So you can very efficient reactive programming with RxJava or Reactive-Streams power by Reactor.

I'm using lettuce in production in the last 2 years and have been really great. For this blog post, I will how to dispatch custom commands to redis-modules via Lettuce.


Lettuce Features - Why Lettuce?

Lettuce is maintained by Pivotal and has many great features like:
  • Non-Blocking IO
  • Java 8 types: Flux | Mono
  • Dynamic Commands
  • Redis-Modules Support
  • Cloud Ready
Building a Custom Redis Module

Some time ago I show how to build a simple redis-module using C. If you did not read the previous post please read it here. We will need that custom redis module in order to send our commands in Lettuce. There are 2 options EVAL and DISPATCH.

Sending Custom Commands via Lettuce Dispatch

EVAL is a simple way, we basically sensing some simple Lua Script returning the command and this will be run inside Redis. Let's take a look at the code:


So what do we have here? We basically create a simple connection to Redis. We are creating RedisAsyncCommands in order to leverage async and non-blocking IO. Them we do the EVAL with this: return redis.call('dp.DATE') .This means "dp.DATE" is the name of the command and we are just calling the command and get whatever it will return - for this case it will be the current date. This is fine considering that the Lua Script just call the command so the overhead is pretty minimal. The main advantage of this approach is that EVAL support is old in Redis - since 2.6 version.  There is a better approach but not all clients / eco-system is ready for it. So let's look 2 and best option.

Second Option is to use DISPATCH so we can send custom protocol(Which means raw encoded command) and a list of arguments. This will be sent straight to Redis without Lua Script. Let's take a look at the code:



So basically here we need to define a ProtocolKeyword with is pretty much the encoding of the command. Them we can get the Async command and just DISPATCH. Keep in mind this is done in Async fashion. I'm also not passing any parameters but we could if we need. 

Redis-Modules are getting more popular and this is the right way to customize Redis. Lettuce proved to be very efficient and flexible client in production for Java. Now you can wrap this EVALs or DISPATCH under some java Interfaces and have a very nice extended driver for your use cases.

You can get the full code on my GitHub here.

Cheers,
Diego Pacheco


Building Redis-Modules with JavaScript using Cthulhu

Cthulhu is an interesting project. It allows you to code custom redis-modules in JavaScript. This is very interesting because JS is very productive and easy to write code.

You might be wondering how they made this efficient? Well, the secret is a project called Duktape which is an efficient JS engine written in C. So Cthulhu + Duktape will pre-compile your javascript function and run it on C which will be efficient and will be able to talk with redis core.

Unfortunately, not all functions are available -- you can check the ones that are available here: List of available APIS.

You can so a simple comparison with a module In C I wrote some time ago in this blog post. C is way more powerful but also way more error-prone and might take more time to write if you are not used to it. So for this Blog Post, I will how to download and compile Cthulhu and they install in Redis 4 and code some custom function in JavaScript. Let's get started.




Building Redis-Module in JavaScript



So here we are downloading and building Redis 4.0.6(Redis modules was introduced on Redis 4) and then we are making Cthulhu so Lib. As you can see we just need to build Cthulhu one time and then we link Cthulhu with our custom js function.

There are 3 JS functions: my_incr, my_append and echo. The function my_incr gets a key turns the content into a number and the increment 1 to the number and set back to the key and return to the user.

Second function my_append does text appending on existing key. Getting the current value and appending the parameter. Both functions already exist so they are here just to show how easy is to work with JS and Redis.

The last function is just an echo -- Whatever string you send you will receive back. This functions can be called on the Redis-CLI but also by any client like Jedis(>=3.0.0-SNAPSHOT) or Lettuce(4.x,5.x) versions. So its possible to call this commands by any language.

Cheers,
Diego Pacheco

Tuesday, November 7, 2017

Extending Redis with Redis-Modules




Redis is a great K/V store written in C. Redis can accomplish a lot for a single thread process. There are many applications for redis like for instance: 
- Frontend Database
- Real-time Counters
- Ad Serving
- Message Queue
- Geo and TimeSeries DB
- Session State
- Cache
That's all great but let's say I want more, how can we customize redis? There are some options like a Lua Script, Fork Redis, Tak to @Antirez or Create your own NoSQL database based on redis or not. Redis provided another solution - Redis added support for external modules in 2016.

Redis Modules

Redis Modules are just dynamic libraries(.so files) loaded on redis - Often these libraries are written in C or C++ but there are some bindings where you write modules in Go or JavaScript for instance.

The best news is if you write one library you have ZERO LATENCY to access data in Redis - So this is a very big win and strong reason to write your own library.

Redis Modules enable a whole new set of extensions for you. For instance is possible to create new Datatypes. Its possible to create new Commands. It's possible also to combine existing commands and add custom new functionality using an existent command, datatypes in order to archive your goals.

There is just 1 bad news. Remember is C. C often can be fragile so you need to have great discipline and test very well your code. One wrong thing can tear down the whole redis process and I don't need to say how bad this is.

Redis Modules are supported by version 4.0. There are some interesting available models already like this ones here:
- RediSearch - Search support for Redis
- ReJSON - JSON data type and operations support for Redis
- Redis-ML - Machine Learning with Redis
- rebloom - Scalable Bloom Filters for Redis.

You can also get more information on additional modules here.

Creating a Simple Redis Module

There are some dependencies in order to build and run the redis module. Make sure you have installed:
  - docker
  - redis-cli
  - build-essential

So let's start with the Dockerfile. We will use docker to run redis 4.0 with the custom module we will build. Let's define the Dockerfile. 

Dockerfile

This Dockerfile is pretty simple we are extending the Latest Redis docker image and updating the OS for the latest packages and installing gcc and we are building the module(.so file) and loading on Redis. So when Redis process boots up on this docker container it will contain the module.

date.c

In this file(date.c) we have the module code. We basically need:
RedisModule_OnLoad: We use this function to load the module and all commands.
diegoDate_RedisCommand: The command implementation code.
getDate: A helper function which returns the current date as string.

As you might realize there is an include for redismodule.h which is the redis module API we need to interact with Redis.

redismodule.h

Here we have redis functions prototyping and also some includes of libraries like stdio, stdlib, time, stdint and so on and on. You don't need to worry much about this header file.

Makefile

We use Makefile to build the module file, Bake the docker image and also to perform some tests on the SO using docker and redis-cli.

Running

Now in bash, we can do:
$ make clean
$ make
$ make docker
$ make run 

This will build the module and bake the docker image and run the docker container. After doing that we can open redis-cli by doing: $ redis-cli

So finally we can test our command by doing $ dp.DATE and you should see the current date. That's it, folks, we have a very simple redis module working.

If you want have the full code you can get on my github here.

Cheers,
Diego Pacheco



Friday, September 29, 2017

Dynomite Eureka Registry with Prana

Dynomite it's a great solution for clustering with NoSQL Databases like Redis. Eureka is a nice Registry & Discoverability Solution.

We can get best of both worlds using Prana. Prana is a sidecar that enables non-JVM applications to register in Eureka.

Sometimes we could easily use multiple discoverability solutions like DNS, ETCD, Eureka etc... However not all discoverability provide the same benefits some tools are better suited for some jobs them other. I like ETCD and make sense onKubernetes world but if you are doing Java Microservices eureka makes more sense.

The triad(Eureka, Prana, Dynomite) is great, This is great because then you can do discoverability on your database nodes, this is not great for several reasons like:
  • Use the same tool for Registry / Discoverability
  • Enable all sorts of dynamic programming which is great for DevOps Engineering
  • Avoid AWS Throttling issues
  • Make dyno clients more dynamic and this is a better solution them DNS like route53
I made a simple video with a simple presentation and live demo how to do this work on the server side with Dynomite 0.5.9. I hope you enjoy, have fun.


Slides from the presentation



Update: After talking with a Netflix Engineer, I got some new information about the state of Prana which can be found here. If you are using Dynomite-manager you should let DM do the Registry for you so there is no need to use Prana. For DM case you check this out. IF you are using Dynomite without DM you can still use Prana however you would consider DM in case you are running on AWS.

Cheers,
Diego Pacheco

Monday, August 7, 2017

Go and Redis running on Kubernetes with Minukube


Go is a simple, fast and powerful programing language. Go is growing a lot into the DevOps Engineering scene like Hashicorp Stack or even Kubernetes.  One of the main advantages of GO is the fact that you can generate a single binary with all you needed, bundled in a single file. This makes distribution so much easier.

Go is also very compact for some use cases and you can write so less code and still very very efficient and get best of performance.  Today we will see how to create a very, very, very simple service in go. This service will access redis to increment how many times it was called.  We will use Minikube in order to run kubernetes locally and we will store our data in Redis.




The Go Service

Let's get down to the code.

Here we are using an external library called go-redis. We need this library to communicate with Redis using Go. You can run this code on your machine right now but first, you need to install the Redis driver you can do it so by running $ go get -u github.com/go-redis/redis .Now you can simply run the app with $  go run main.go .Keep in mind this won't work right now because we don't have Redis running. You can download, install and run Redis or you can use docker. We will run redis with docker but on Kubernetes using Minikube.

As you can see we are exposing a service on the port :9090 and he have a handler function which takes care of the HTTP request to this very server. We want to avoid double counting for this service and some user might call it from the browser so we need to ignore the /favicon.ico request that's why we use the if on the very first lines of the code.

We are connecting on Redis but we are reading Redis URL from an OS env var. I'm doing this so we can run this code in many ways like locally, docker, docker-compose, and kubernetes. IF you have Redis locally now you can do $ REDIS_URL=localhost:6379 go run main.go .For this simple service, we are using Redis commands INCR and GET in order to increment keys(+1) and get the current value of the key.

Go Dockerfile

Right now we need to create a Dockerfile to run our go simple service. In order to do that we will create a Dockerfile file and add the following content:

Dockerfile is quite straightforward we are using GO lang 1.8 and we are installing the Redis driver and we copy the go source code to the container.

Kubernetes deployment

Now we need create 2 yaml files in order to describe the kuerbenetes deployment and Service for the go web app. However before doing that we will create 2 other yaml files in order to deploy Redis so our redis docker container runs on kubernetes as well. We also link connect this 2 containers using kubernetes internal DNS service.

Redis Deployment on Kubernetes

Let's create a file called: redis-deployment.yaml.

We also need to create a service. So create a file called redis-service.yaml.

The most important information here is that we are using the image called redis. We are also setting the spec on the service for LoadBalancing on the port 6379 and this means we will be able to access outside of the cluster. This is not required but is quite useful so you can use redis-cli and access the redis instance on Kubernetes.

Right now we can create the deployment and service yaml files for the web go simple service application.  So first create a file called: webappgo-deployment.yaml


Then, create another file called webappgo-service.yaml.

Here we have some important things that need to be noted such as the image which is: webappgo:v1. This image does not exist on public docker repo so we need to build on our local machine and send the docker image to minikube dockers registry.

When we write the go code we were expecting to receive the redis URL via OS ENV var. We can set this vars via Kubernetes and kubernetes will forward this vars to Docker we did this on the property called env. You might notice some specific name for the redis url which is: redis.default.svc.cluster.local:6379 .This is the kubernetes internal DNS. Where redis is the name of the service, the default is the default namespace and cluster.local is the domain.

Another important thing to notice is that the spec type on the service in LoadBalancer so we are exposing the port 9090.

kubectl and Minikube

okay, now we can go on and deploy theses files in kubernetes on our minikube local cluster.

There are some important steps here.  We are doing eval on minikube docker-env so we can use docker commands on out machine but send docker images to minikube docker registry. We are also baking a docker images for the go service application.

You might notify we are using kubectl create -f and passing a directory. This is created because you don't need to specify a file by files and long as you have redis and go deployment and services files properly in each respective directory.

Now we can run the GO simple service in our browser. You might need to run  $ kubectl describe services webappgo to get the proper port mapping.



IF you go to the minikube dashboard(http://192.168.99.100:30000/#!/pod?namespace=default) you can the see the GO service LOGs, like this:



You can get all the files in my github.

Cheers,
Diego Pacheco

Tuesday, November 10, 2015

Netflix Dynomite/Dyno: The Cluster for Redis


Dynomite is brilliant. Kudos for NetflixOSS team because it kicks ass. First all they mixed several interesting, battle tested and sexy architectural ideas and deliver into a single solution. What would be Dynomite? You can think as a kick Ass Cluster for Memcached and Redis. But its way more than that. Dynomite is integrated with the Netflix Stack so you can use with Eureka and the rest of the stack. You dont need use Redis or Memcached if you dont want because Dynomite is modular so you can use the NoSQL or thing behind it.

Dynomite is based on the Amazon Dynamo paper, so it implements the Consistence Hashing Ring, with quorum-like mechanisms, so you can have strong consistency and dont loss data(similar to Cassandra and Riak) and also have some low latency and high throughput using Redis or Memcached Behind. Dynomite is written in C and its a proxy, it uses the twitter twemproxy as base solution. Replication is a aymetric, dynomite has a java client called Dyno with has Token Aware load balancing. On the consistency side you can do: DC_ONE: Sync same AZ, Async other Region or DC_QUORUN: Sync to the mun of the quorum.
Performance is amazing, check this benchmarks by Netflix folks. They used a R3.Xlarge instance with replication factor set to 3 in 3 amazon zones, they used in front of Redis and did some set of GET and SET operations. The ratio between reads and writes was 80% reads and 20% writes(pretty much Netflix scenario)



Dyno Client Features

One of the great things about the client(Dyno) is that you can choose the client you want use, so for redis you can use Jedis or Redisson for instance but since dyno is modular you can code to integrated other clients if you like it more. Some key features in dyno are:

    * Connection pooling of persistent connections - this helps reduce connection churn on the Dynomite server with client connection reuse.
    * Topology aware load balancing (Token Aware) for avoiding any intermediate hops to a Dynomite coordinator node that is not the owner of the specified data.
    * Application specific local rack affinity based request routing to Dynomite nodes.
    * Application resilience by intelligently failing over to remote racks when local Dynomite rack nodes fail.
    * Application resilience against network glitches by constantly monitoring connection health and recycling unhealthy connections.
    * Capability of surgically routing traffic away from any nodes that need to be taken offline for maintenance.
    * Flexible retry policies such as exponential backoff etc
    * Insight into connection pool metrics
    * Highly configurable and pluggable connection pool components for implementing your advanced features.

Installing, Configuring and Running


Cheers,
Diego Pacheco

Redis Cluster: Should I go or should i stay?

First all i just want make clear a respect Redis and Salvatore a lot. This is just my view of the topic, no intention to create any kind of FUD or flame war here, just sharing my toughs and experiences. I use redis in production since 2011, its a great piece of technology. Redis has amazing throughput for single instance solution given the fact redis does not have threads(compared with Memcached). Do i recommend use Redis in production? Yes i do :-)

Redis: The K/V Solution

Whats is Redis? What is good for? Redis is a K/V store and its great for caching. Memcached have few key/set operations in comparison with Redis, there are lots of commands todo very precision operations. Redis cluster was added recently but originally redis was used more like a farm of servers like Memcached case, so replication and other tasks was carry out by the clients apps. Redis has a cluster now. Redis is written in C and has very fast and direct access do data structures like lists, sets and maps. Single instance(Also single thread) redis has support for some small level of transactions so you got atomic and isolated.
Redis Cluster: Enter the Room

Redis clusters uses the master/slave model and does async data replication, any system design like that will have data loss, i was checking last redis presentations and they are being very clear about that witch is nice. Redis has another process called Sentinel but during a network partition is easy to have the split-brain problem.

Operation and Awareness

What about operation: Operation is not mature yet, has some lake of tooling, you need use some bash and ruby scripts bundled with redis to operated, so in other words right now you will need write down your tools to operate redis if you want use its default built in cluster mechanism. They key question is can i afford Data Loss? If you are building a cache yes, even if you are not using as a cache, like some people use redis for different purposes like session store, message queue or even distributed lock it might be okay but given the fact you can lost data it might not be the best tool for the job. 

Installing, Configuring and running the cluster

How to Install Redis 3 Amazon Linux OS / CentOS 

redis.conf
This is the Redis single instance config file, the most important thing is the bind 0.0.0.0 to be able to accept connections from other boxes(in the case you upgrade to use the cluster). You also will need change the default create-cluster script, here is the one i changed a bit.
Redis Create Cluster

As i said before you will need replace the default script with this one. 

Create-Cluster Script

Cheers,
Diego Pacheco

Chuyên mục văn hoá giải trí của VnExpress

.

© 2017 www.blogthuthuatwin10.com

Tầng 5, Tòa nhà FPT Cầu Giấy, phố Duy Tân, Phường Dịch Vọng Hậu, Quận Cầu Giấy, Hà Nội
Email: nguyenanhtuan2401@gmail.com
Điện thoại: 0908 562 750 ext 4548; Liên hệ quảng cáo: 4567.