Showing posts with label NOSQL. Show all posts
Showing posts with label NOSQL. Show all posts

Tuesday, March 12, 2019

Running Multithreaded Redis using KeyDB and Dynomite

Keys is the default oss standard for K/V store for many use cases and solutions. When you need run low latency, high throughput IN Memory Persistence - Dynomite is your solution. I always wanted to have multi-thread support in redis like Memcached has. Now someone made it and call it KeyDB. The best thing is the fact that this is done supporting the redis RESP protocol, so all your tools, scripts and code using redis protocol directly still works. Another killer feature in KeyDB is the support for FLASH storage, only available on Redis Enterprise. KeyDB also supports backup to S3. For this post, I will show how to build and run KeyDB which is dead simple and also how to use it with Dynomite like it was Redis.





KeyDB Benchmarks

As you can see on the KeyDB benchmarks, KeyDB does 1.5x less latency and 50% more throughput. These numbers are pretty impressive. The best part is the fact that is all transparent.

Running KeyDB


Running KeyDB with Dynomite



See the Results

KeyDB running


Dynomite Running


redis-cli(via dynomite port 8102)


Other Dynomite related Posts

Cheers,
Diego Pacheco

Friday, April 13, 2018

Dynomite Remediation

Working with the cloud is great but it's not easy. The Cloud allows us spins machines pretty easily, however, update these machines are not that an easy task, especially if we talking about databases. Today I want to share some experience with remediation process I've built for Dynomite in my current project.  I'm using NetflixOSS Dynomite in production(AWS) for more than 2 years now both as cache but also as Source of Truth. I need to say that is rock solid and just works. However often we need to increase some machine capacity, i.e: Add more memory or increase the disk size. There are times where we need to apply patches on the OS - I use Amazon Linux on Production on EC2 and recently there was the Meltdown and Spectre situation. Other times there are telemetry or even provisioning and configuration bugs that need to be fixed.  No matter the reason you will always need, time to time, change something on the underlying AMI or in the Java or C application which is running. For this times there is a need to apply changes in a very smooth way, this is hard, It's not only hard in the sense that bad things will happen if you do it wrong but also because it's a long process and it's easy to do something wrong.


Why Manual Remediation is a bad idea?

First of all, manual operations are complicated at Cloud / Microservices era, since we have more software and this software is often way more distributed it means we have more machines to deal with. More machines mean more steps because sometimes the changes are not only on the machine but on the underlying infrastructure like Launch configurations or auto-scaling groups. Apply this changes to Database infrastructure requires very specific ordering. Ordering is easy to determine but proper execution is hard. As humans we are great but we always make mistakes, also like Google SRE folks said in the past, everytime you add a human you will introduce latency. Latency might be okay for 1 cluster, but for several clusters, it's a bigger problem.  Wrapping up: Cloud Operation Manual work has these potential problems:

  • Error-Prone: Mistakes or lack of attention to detail
  • Latency: It's will take more time to get the job done
  • Scalability Issues: It will be a bottleneck if a team need to do for all customers
  • Data Loss: If some task is executed out of order you might lose data.
  • Downtime: Depending on the mistake or error it could affect availability.

Dynomite Use Cases and Topology

I use dynomite as cache but also as a source of truth. When you think as a cache only remediation is great but less critical since something went wrong or takes more time it's fine since is a cache and you are not losing any data. Worst case you are adding more latency to the application, depending on your volume that might or might not be acceptable at all.

When we are talking about Source of Truth use case is a different deal since we can't afford losing Data. Thanks to Netflix awesome engineers there is a killer feature in Dynomite Manager called Cold Bootstrapping. This feature is very important for 2 reasons, first because allow dynomite / redis nodes to a recovery process when a node dies or got terminated - so the node can peer up with other nodes and get nearby data and they join the cluster without losing data. The same feature allows to perform remediation without downtime if we are careful and execute steps in very narrow and specific order.

Typical a dynomite cluster has 3 nodes as minimum topology. There is 1 node per AZ so this helps us to achieve high availability. You can have bitter clusters but will be always multiple of 3, like 3,6,9,12,15,18,21 nodes and so on and on.



Dynomite Requirements

In order to do a proper remediation, there are 3 main requirements we need to meet. First of all, we need to be fully automated them we also need to make sure we don't lose data and we don't create downtime for our consumers.

It's possible to avoid downtime because dynomite has dyno. Dyno is cluster and topology aware so it knows other nodes in other AZ so when a node is down it will failover to the other node and that's the reason why we can do the process without downtime. Dyno has some other interesting concept called LOCAL_RACK or LOCAL_AZ which allow you to have a preferable AZ to connect this allow you reduce latency and get nearby AZ.

In order to the remediation process work, dyno needs to be fully and properly configured and you need to have a proper cluster, in sense of topology. Othwerise you might not meet the requirement of no downtime.  Automation can be archived with AWS SDK which is available to most of the popular languages. The remediation solution I built was coded with Java 8. Overall JVM ecosystem is really great for troubleshooting, remote debug and profiling so this was one of the reasons behind the rationale plus my company I work for is a Java shop so :D.


Dynomite Remediation Process Flow

Right now we can talk about the remediation process. This process had to be coded very carefully in sense of timeouts, retry and make sure the steps and execute at the right moment. Forst instance first thinks the process does is to discover the cluster based on the ASG Name. Here I use the AWS API in order to get all dynomite nodes given cluster name which is unique.

Once I have the IPs that I can check is the nodes are healthy and if is a good time to start a remediation process. The first question makes sense and is easy to grasp because we want to know if the node is UP and RUNNING howe the er second question makes more sense if we think about Dynomite-Manager(DM). DM might be running some Backup yo S3 or might be performing some cold bootstrapping with redis since a node might just die or could be restoring data from S3 no matter the case this is pretty good examples of things that could be happing and you supposed to wait for all these events finish before remediation starts or proceed. That's why calling the Dynomite-Manager Health Checker is so important and thanks to Netflix great engineer it exposes all information we need to know in order to perform this process in a safe and reliable manner.

In order to apply patches or increase instance family, we need to create new Launch configurations pointing to new AMI IDS and we need to update Auto-Scaling Groups to point to these new LCs. Them we can kill node by node in order to remediate the cluster. After all, nodes are remediated we can delete the 3 old LCs(1 per AZ) and we are done.

When we kill a node there are operations that will happen and we will need to wait for this operations to be finished in order to move to the next node. First we need wait AWS ASG bootup a new node, then we need wait dynomite and DM bootup and DM will realize it needs to run some cold bootstrap in order to copy data from other Redis nodes, once that process is done DM health Checker will let us know and then we can proceed to next node.

Here is the visual picture of the whole remediation process flow.






















Remediation process is great and saves lots of times. The whole remediation process is a bit slow since we need wait AWS to kill and boot up new instances. On average a 3 node cluster takes about 15 minutes to be remediated. This time may vary depending on how much data do you have in Redis.

Other Dynomite related Posts
Cheers,
Diego Pacheco

Tuesday, February 20, 2018

Debugging Redis Module

Lately, I'm working with custom redis-modules and I need to say: redis-modules kick ass. I started working with Cthulu and coding modules with javascript which was nice and very productive but given some of my current use cases I had to go to C.

I recommend you take a look at my previous posts here:
  - Building Redis-Modules with JavaScript using Cthulhu
  - Dispatching Custom Commands with Lettuce and Redis-Modules

Today I want to you how to do proper debug with Redis modules written in C language. I will be using eclipse Oxygen CPP in order to debug redis module and redis 4.

Here there is a video which I recorded with a simple live demo about how to debug a redis-module project and redis. The source code is available on my GitHub. I hope this is useful for you.

How to Debug a Redis Module?


Slides



Cheers,
Diego Pacheco

Tuesday, December 19, 2017

Dispatching Custom Commands with Lettuce and Redis-Modules

Lettuce is a very efficient and extensible Redis client for Java.  Lettuce is interesting for many reasons but one of the big ones is the fact that provides an Observable / Flux API. So you can very efficient reactive programming with RxJava or Reactive-Streams power by Reactor.

I'm using lettuce in production in the last 2 years and have been really great. For this blog post, I will how to dispatch custom commands to redis-modules via Lettuce.


Lettuce Features - Why Lettuce?

Lettuce is maintained by Pivotal and has many great features like:
  • Non-Blocking IO
  • Java 8 types: Flux | Mono
  • Dynamic Commands
  • Redis-Modules Support
  • Cloud Ready
Building a Custom Redis Module

Some time ago I show how to build a simple redis-module using C. If you did not read the previous post please read it here. We will need that custom redis module in order to send our commands in Lettuce. There are 2 options EVAL and DISPATCH.

Sending Custom Commands via Lettuce Dispatch

EVAL is a simple way, we basically sensing some simple Lua Script returning the command and this will be run inside Redis. Let's take a look at the code:


So what do we have here? We basically create a simple connection to Redis. We are creating RedisAsyncCommands in order to leverage async and non-blocking IO. Them we do the EVAL with this: return redis.call('dp.DATE') .This means "dp.DATE" is the name of the command and we are just calling the command and get whatever it will return - for this case it will be the current date. This is fine considering that the Lua Script just call the command so the overhead is pretty minimal. The main advantage of this approach is that EVAL support is old in Redis - since 2.6 version.  There is a better approach but not all clients / eco-system is ready for it. So let's look 2 and best option.

Second Option is to use DISPATCH so we can send custom protocol(Which means raw encoded command) and a list of arguments. This will be sent straight to Redis without Lua Script. Let's take a look at the code:



So basically here we need to define a ProtocolKeyword with is pretty much the encoding of the command. Them we can get the Async command and just DISPATCH. Keep in mind this is done in Async fashion. I'm also not passing any parameters but we could if we need. 

Redis-Modules are getting more popular and this is the right way to customize Redis. Lettuce proved to be very efficient and flexible client in production for Java. Now you can wrap this EVALs or DISPATCH under some java Interfaces and have a very nice extended driver for your use cases.

You can get the full code on my GitHub here.

Cheers,
Diego Pacheco


Building Redis-Modules with JavaScript using Cthulhu

Cthulhu is an interesting project. It allows you to code custom redis-modules in JavaScript. This is very interesting because JS is very productive and easy to write code.

You might be wondering how they made this efficient? Well, the secret is a project called Duktape which is an efficient JS engine written in C. So Cthulhu + Duktape will pre-compile your javascript function and run it on C which will be efficient and will be able to talk with redis core.

Unfortunately, not all functions are available -- you can check the ones that are available here: List of available APIS.

You can so a simple comparison with a module In C I wrote some time ago in this blog post. C is way more powerful but also way more error-prone and might take more time to write if you are not used to it. So for this Blog Post, I will how to download and compile Cthulhu and they install in Redis 4 and code some custom function in JavaScript. Let's get started.




Building Redis-Module in JavaScript



So here we are downloading and building Redis 4.0.6(Redis modules was introduced on Redis 4) and then we are making Cthulhu so Lib. As you can see we just need to build Cthulhu one time and then we link Cthulhu with our custom js function.

There are 3 JS functions: my_incr, my_append and echo. The function my_incr gets a key turns the content into a number and the increment 1 to the number and set back to the key and return to the user.

Second function my_append does text appending on existing key. Getting the current value and appending the parameter. Both functions already exist so they are here just to show how easy is to work with JS and Redis.

The last function is just an echo -- Whatever string you send you will receive back. This functions can be called on the Redis-CLI but also by any client like Jedis(>=3.0.0-SNAPSHOT) or Lettuce(4.x,5.x) versions. So its possible to call this commands by any language.

Cheers,
Diego Pacheco

Friday, December 1, 2017

Deploy & Setup a Cassandra 3.x Cluster on EC2

Cassandra is a Rock solid AP NoSQL database. For this blog post, I will share a simple recipe to deploy a Cassandra cluster on AWS/EC2.

There is the DataStax AMI or DSE you should consider for production workloads.  This recipe I'm sharing is for Amazon Linux(CentOS based) but you can do for Ubuntu or even in Docker if you want to.

Keep in mind this is for Development / Experimentation purpose. I'm not covering proper tunning for your workload, Compaction Strategy and Keyspace design here and you also should be doing this under multiples ASG 1 per AZ ideally. So Let's get started.



Cheers,
Diego Pacheco

Tuesday, November 7, 2017

Extending Redis with Redis-Modules




Redis is a great K/V store written in C. Redis can accomplish a lot for a single thread process. There are many applications for redis like for instance: 
- Frontend Database
- Real-time Counters
- Ad Serving
- Message Queue
- Geo and TimeSeries DB
- Session State
- Cache
That's all great but let's say I want more, how can we customize redis? There are some options like a Lua Script, Fork Redis, Tak to @Antirez or Create your own NoSQL database based on redis or not. Redis provided another solution - Redis added support for external modules in 2016.

Redis Modules

Redis Modules are just dynamic libraries(.so files) loaded on redis - Often these libraries are written in C or C++ but there are some bindings where you write modules in Go or JavaScript for instance.

The best news is if you write one library you have ZERO LATENCY to access data in Redis - So this is a very big win and strong reason to write your own library.

Redis Modules enable a whole new set of extensions for you. For instance is possible to create new Datatypes. Its possible to create new Commands. It's possible also to combine existing commands and add custom new functionality using an existent command, datatypes in order to archive your goals.

There is just 1 bad news. Remember is C. C often can be fragile so you need to have great discipline and test very well your code. One wrong thing can tear down the whole redis process and I don't need to say how bad this is.

Redis Modules are supported by version 4.0. There are some interesting available models already like this ones here:
- RediSearch - Search support for Redis
- ReJSON - JSON data type and operations support for Redis
- Redis-ML - Machine Learning with Redis
- rebloom - Scalable Bloom Filters for Redis.

You can also get more information on additional modules here.

Creating a Simple Redis Module

There are some dependencies in order to build and run the redis module. Make sure you have installed:
  - docker
  - redis-cli
  - build-essential

So let's start with the Dockerfile. We will use docker to run redis 4.0 with the custom module we will build. Let's define the Dockerfile. 

Dockerfile

This Dockerfile is pretty simple we are extending the Latest Redis docker image and updating the OS for the latest packages and installing gcc and we are building the module(.so file) and loading on Redis. So when Redis process boots up on this docker container it will contain the module.

date.c

In this file(date.c) we have the module code. We basically need:
RedisModule_OnLoad: We use this function to load the module and all commands.
diegoDate_RedisCommand: The command implementation code.
getDate: A helper function which returns the current date as string.

As you might realize there is an include for redismodule.h which is the redis module API we need to interact with Redis.

redismodule.h

Here we have redis functions prototyping and also some includes of libraries like stdio, stdlib, time, stdint and so on and on. You don't need to worry much about this header file.

Makefile

We use Makefile to build the module file, Bake the docker image and also to perform some tests on the SO using docker and redis-cli.

Running

Now in bash, we can do:
$ make clean
$ make
$ make docker
$ make run 

This will build the module and bake the docker image and run the docker container. After doing that we can open redis-cli by doing: $ redis-cli

So finally we can test our command by doing $ dp.DATE and you should see the current date. That's it, folks, we have a very simple redis module working.

If you want have the full code you can get on my github here.

Cheers,
Diego Pacheco



Friday, September 29, 2017

Dynomite Eureka Registry with Prana

Dynomite it's a great solution for clustering with NoSQL Databases like Redis. Eureka is a nice Registry & Discoverability Solution.

We can get best of both worlds using Prana. Prana is a sidecar that enables non-JVM applications to register in Eureka.

Sometimes we could easily use multiple discoverability solutions like DNS, ETCD, Eureka etc... However not all discoverability provide the same benefits some tools are better suited for some jobs them other. I like ETCD and make sense onKubernetes world but if you are doing Java Microservices eureka makes more sense.

The triad(Eureka, Prana, Dynomite) is great, This is great because then you can do discoverability on your database nodes, this is not great for several reasons like:
  • Use the same tool for Registry / Discoverability
  • Enable all sorts of dynamic programming which is great for DevOps Engineering
  • Avoid AWS Throttling issues
  • Make dyno clients more dynamic and this is a better solution them DNS like route53
I made a simple video with a simple presentation and live demo how to do this work on the server side with Dynomite 0.5.9. I hope you enjoy, have fun.


Slides from the presentation



Update: After talking with a Netflix Engineer, I got some new information about the state of Prana which can be found here. If you are using Dynomite-manager you should let DM do the Registry for you so there is no need to use Prana. For DM case you check this out. IF you are using Dynomite without DM you can still use Prana however you would consider DM in case you are running on AWS.

Cheers,
Diego Pacheco

Wednesday, July 5, 2017

Dynomite and RocksDB running on Docker

Dynomite is a kick ass cluster/proxy solution that provides high availability and strong consistent to databases. Dynomite was created and battle tested by Netflix using Memcached and Redis as primary store backend.

RocksDB it's a Facebook Embedded Key/Value store which is growing up a lot because of his incredible high performance and low latency.

For this blog post, I will show some simple project I create for the developers to get started with Dynomite and RocksDB together. This project uses docker and creates a Dynomite cluster using RocksDB as backend.

Let's get started...




Dynomite Super Powers

Dynomite is interesting for many reasons but one very simple and limited way to see Dynomite as a supercharged Redis with batteries included. Why not use ElastiCache(Amazon Redis managed service) well because first of all you should think twice before get fully coupled to Amazon or any public cloud vendor.  In the beginning, it might look like a good idea since you have all sorts of services that are integrated but there are many limitations and as you get a vendor LOCK IN you won't be able to get better and cheaper competitors offers and also it can be less available. I guess everybody remembers last S3 outage almost tear down the whole with it. If you have a portable solution you can run your solution in other cloud vendors like Google or Azure. Solutions like Spinnaker can help you a lot with that but they won't help you if your persistence system is fully coupled to one vendor.

IMHO because Dynomite is based on Amazon Dynamo paper still has superior clustering solution them original Redis cluster(Master/Slave) and elastiCacheit's pretty much the same cluster as OSS Redis Cluster but managed by AWS.

Another more comprehensive and correct way to see Dynomite is looking as a Generic Supercharger because as long as you have something that talks Redis protocol you can benefit from all Redis ecosystem and Dynomite as well.  There is a very nice project which helps on that front called ARDB.

Dynomite-Docker-RocksDB project

Basically this a very simple and tiny project. Where I have some bash scripts to create Dynomite clusters using ARDB and RocksDB as persistent backend. Let's see how to get started. First of all, you need to make sure you have Docker installed on your Linux.

Cheers,
Diego Pacheco


Sunday, July 2, 2017

Configure a Cassandra 2.x Cluster on EC2

Cassandra is a kick ass NoSQL Database that guarantees strong consistency.  Inspired by Amazon Dynamo paper Cassandra can deliver high availability and strong consistency.  Battle tested by companies like Apple, eBay, Netflix, Github, Nasa and several others. For this blog post, I will show how to setup a simple Cassandra cluster. This setup is for development purpose and it's not ready for production.


Cheers,
Diego Pacheco

Saturday, December 12, 2015

Having fun with ArangoDB

Besides performance, strong consistency and scalability we also pick NoSQL DBs based on the Design of your Data this is one of the cool things about ArangoDB because it has multi-models.

ArangoDB is a distributed NoSQL database. It has a flexible data model for documents, graphs, and key-values this is also called multi-model db. ArangoDB has convenient SQL-like query language or JavaScript extensions.

In this post i will cover the Document Model and show some have code for such task i will use gradle and Java 8.  ArangoDB have drivers for lots of languages you check it out here.

I'm using ArangoDB in Ubuntu Linux, so to install you just do $ sudo apt-get install arangodb. If you are using another OS or Docker for instance is fine you check it out here as well.



Build.gradle

So first of all lets see the build.gradle file, here we are just adding ArangoDB Java Driver dependency. So it's very straightforward. Once you do this you can run $ gradle build and $ gradle eclipse to generate the eclipse project. Now let's see some Java code.

Java Driver

Now we will create a very simple DB and query a document using the Java Driver.

First we connect on the server running on localhost on port  8529. Next step is create a Database called db1. Moving forward there is a collection being created called myCollection in this collection there are several documents being add, as you can see us the Simpsons Family. To finish there is  query and a for loop to show all results.


You can then run $ gradle run . Full project is available in my GitHub.

That`s it :-)

Thursday, December 10, 2015

Having fun with Etcd

Etcd is a distributed, consistent key-value store for shared configuration and service discovery, with a focus on being. Etcd is very useful in DevOps context because today Immutable Infrastructure is one of the core principles. Often what people do is store the configs into a K/V store like Etcd and them just export OS_VARS into the OS level that could be an Amazon AMI, Linux or Bare metal, Mesos or even a Docker container.

One of the cool things about Etcd is that you have a pretty simple REST interface to operate over the K/V store.  When you are doing provisioning sometimes you need store IPs, IDS, Tags, etc... and this is very viable solution.

Having fun with Etcd

Cheers,
Diego Pacheco

Tuesday, November 10, 2015

Redis Cluster: Should I go or should i stay?

First all i just want make clear a respect Redis and Salvatore a lot. This is just my view of the topic, no intention to create any kind of FUD or flame war here, just sharing my toughs and experiences. I use redis in production since 2011, its a great piece of technology. Redis has amazing throughput for single instance solution given the fact redis does not have threads(compared with Memcached). Do i recommend use Redis in production? Yes i do :-)

Redis: The K/V Solution

Whats is Redis? What is good for? Redis is a K/V store and its great for caching. Memcached have few key/set operations in comparison with Redis, there are lots of commands todo very precision operations. Redis cluster was added recently but originally redis was used more like a farm of servers like Memcached case, so replication and other tasks was carry out by the clients apps. Redis has a cluster now. Redis is written in C and has very fast and direct access do data structures like lists, sets and maps. Single instance(Also single thread) redis has support for some small level of transactions so you got atomic and isolated.
Redis Cluster: Enter the Room

Redis clusters uses the master/slave model and does async data replication, any system design like that will have data loss, i was checking last redis presentations and they are being very clear about that witch is nice. Redis has another process called Sentinel but during a network partition is easy to have the split-brain problem.

Operation and Awareness

What about operation: Operation is not mature yet, has some lake of tooling, you need use some bash and ruby scripts bundled with redis to operated, so in other words right now you will need write down your tools to operate redis if you want use its default built in cluster mechanism. They key question is can i afford Data Loss? If you are building a cache yes, even if you are not using as a cache, like some people use redis for different purposes like session store, message queue or even distributed lock it might be okay but given the fact you can lost data it might not be the best tool for the job. 

Installing, Configuring and running the cluster

How to Install Redis 3 Amazon Linux OS / CentOS 

redis.conf
This is the Redis single instance config file, the most important thing is the bind 0.0.0.0 to be able to accept connections from other boxes(in the case you upgrade to use the cluster). You also will need change the default create-cluster script, here is the one i changed a bit.
Redis Create Cluster

As i said before you will need replace the default script with this one. 

Create-Cluster Script

Cheers,
Diego Pacheco

Some fun with Memcached

Memcahed is a K/V store. Its fast pretty much all operations are O(1). Memcached use Threads(Async and non-blocking IO) different from Redis with is single-thread. Memcached does not have a cluster, its a farm, some concerns like Replication need be handled by the clients or 3rd party application, nodes on the farm don't talk o each other so there is no GOSSIP. Memcached has few key/set operations in comparison with Redis. Redis have several. Some clients have the concept of the master-key / sub-key based on this it can locate what server has that specific key. Often keys are smaller than 250 chars and objects smaller than 1MB. Memcached if used for a lot lot of scenarios like: Profile Caching(Social Networking), Page Caching(HTML), Ad Targeting with cookie / profile tracking, Session caching for games and entertainment its pretty much for caching.

Installing, Configuring and Running

cheers,
Diego Pacheco

Chuyên mục văn hoá giải trí của VnExpress

.

© 2017 www.blogthuthuatwin10.com

Tầng 5, Tòa nhà FPT Cầu Giấy, phố Duy Tân, Phường Dịch Vọng Hậu, Quận Cầu Giấy, Hà Nội
Email: nguyenanhtuan2401@gmail.com
Điện thoại: 0908 562 750 ext 4548; Liên hệ quảng cáo: 4567.