Amazon Builder Library: Review Notes
Review Notes
My Amazon Builder's Library Notes
Companion Deck
I hope you enjoy!
Cheers,
Diego Pacheco
Showing posts with label aws. Show all posts
Showing posts with label aws. Show all posts
Thursday, January 2, 2020
Review Notes
My Amazon Builder's Library Notes
Companion Deck
I hope you enjoy!
Cheers,
Diego Pacheco
Wednesday, January 1, 2020
Amazon Aurora: Papers Review
Papers
Engineers often dont read papers, which is a big mistake because they loose often Deep Dive tech content related to Algorithms, Datastructures, Techniques, and lessons learned. So If you are reading this, start reading papers. The 2 amazon papers I'm talking about are:
Amazon Aurora: On Avoiding Distributed Consensus for I/Os, Commits, and Membership Changes
https://dl.acm.org/doi/10.1145/3183713.3196937
Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases
https://www.allthingsdistributed.com/files/p1041-verbitski.pdf
I highly recommend you read the papers if you want to learn more about cloud-native databases.
Reviews
Amazon Aurora: On Avoiding Distributed Consensus for I/Os, Commits, and Membership Changes
https://github.com/diegopacheco/notes/blob/master/paper_reviews/aws_aurora/diegopacheco_paper_review_Amazon_Aurora_On_Avoiding_Distributed_Consensus_for_IOs_Commits_and_Membership_Changes.pdf
Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases
https://github.com/diegopacheco/notes/blob/master/paper_reviews/aws_aurora/diegopacheco_paper_review_Amazon_Aurora_Design_Considerations_for_High_Throughput_CloudNative_RelationalDatabases.pdf.pdf
I hope you enjoy it. More to come.
Cheers,
Diego Pacheco
Tuesday, March 12, 2019
Running AWS ES Open Distro locally
The Script
The script uses docker and link 2 containers(es and kibana). The script also creates an index in ES called Twitter having 3 documents getting indexed with the following props: user, post_data and message. The script copy a custom kibana.yml file into kibana container in order to pass proper docker DNS / Container link to ES container - by default config looks up to localhost and I have to change to es DNS(same name used on docker link).
Checking ES and Kibana
ES: https://localhost:9200/_all
Kibana: http://localhost:5601/app/kibana
ES (_all)
Kibana
Perf Tool
Node Analysis
Cluster Thread Analysis
Cluster Network Analysis
Cluster Overview
When you are done you can simply do:
docker kill es kibana
Cheers,
Diego Pacheco
Monday, February 11, 2019
Running Istio on EKS
Running Istio in AWS with EKS and eksctl
Cheers,
Diego Pacheco
Wednesday, February 6, 2019
Running k8s on EKS
EKS Benefits
- Managed Service without lock-in (Kubernetes API, specs, kubectl)
- There is no Control Plane management (Multi-AZ, Automatic patches, and updates)
- Secure by Default (Secure and encrypted communications between worker nodes and master)
- Conformant and Compatible (EKS runs certified kubernetes. Compatible with standard k8s envs)
IMHO this is what most of the companies are looking for, in other words, *Serverless*.
The Pain Points
I don't want to paint a rosy picture of EKS like any new technology there are problems such as:
- Poor Documentation - very few docs
- Lack of Observability - It's a kind of black blocks compared with Kops.
- When you do something wrong and har to figure it what you did it wrong.
- Error-Prone - Is very easy to make mistakes(eksctl fix that)
- In a Long Run is more expensive than running on EC2 with Kops
- Alpha APIS is not supported.
It's important to keep in mind that EKS is a very new service and it should get better and the time passes.
Running Kubernetes in EKS using EKSCTL
EKSCTL is a great tool. Written in go, makes eks cluster creatin way less error-prone and many simples to get a cluster up and running. So let's get started! Basically, we will install the AWS Authenticator and EKSCTL them we can create the cluster and deploy nginx in kubernetes after that we can access the nginx application in the browser(before that you will need to enable the SG access for your IP). Them we destroy the cluster.
Creating a cluster
Deploying nginx in Kubernetes
Nginx up and running on AWS(Need to enable 80 port SG access)
Nginx AWS ELB Created by EKS and K8s with EKSCTL
Destroying the cluster
Cheers,
Diego Pacheco
Tuesday, February 5, 2019
Running Istio on AWS with Kops
This installation is Linux based(Ubuntu), I'm running all commands from my local-desktop, if you don't use Linux(shame on you) you can create a virtual-machine on AWS with ubuntu and run this commands there, also is possible to run Vagrant with Linux and them run this commands on Vagrant box as well. Istio runs smoothly in AWS with Kops. You don't need much, pretty much 3 machines(1 master node, 2 minions). Keep in mind this is not a production-grade setup, for production, you should be running with 3 masters at least for High Availability.
Installing and Running Istio with Kops
Master and Worker nodes on AWS EC2 Console
Istio Metrics in Grafana
Jaeger - Distributed Tracing
Kiali - Observability
BookInfo ServiceMesh (4 microservices) running on Istio / Kubernetes in AWS
Prometheus(Cloud-Native Observability) - Metrics, Dashboards, and Alerts
ServiceGraph
That's it - I hope you enjoyed.
Cheers,
Diego Pacheco
Running Kubernetes on AWS with KOPS
Installing and running Kubernetes in AWS with KOPS
Cheers,
Diego Pacheco
Monday, July 30, 2018
Experiences Building a Cassandra Orchestrator(CM)
Why Build CM?
Basically, we won't be able to Scale AWS Single and Multi-Regional deploys and Operations so we don't need to code any Jenkins or bash script in order to serve more clusters. Before CM we were doing backups with Jenkins jobs and we did not have ASG on top of Cassandra nodes so cloud ops team always had issues when a node goes down and had to run bash scripts to put the node back online with CM all that changes. The single region is not that complex but when we talk about multi-region things gets more complicated. So automation is really the solution.
Who we build complex system on the fly?
In order to Build CM we had to some POCs and also drawn some Color UML diagrams based on Responsibility-Driven Design(RDD). This might sound a bit old school and design up to the front, however, there is no way to build such a system without some up to front thinking and strong design and architecture leadership. My team works with Kanban and I do multiple roles in my team playing as Tech Manager / Agile Coach, Engineering, Architect, and DevOps Engineer Testing and Developer Support. My team does multiple roles as well so an engineer in my team does Software Engineering, DevOps Engineering, Testing and Developer Support. I and my team work with the idea of Self Service systems where a developer could easily consume what we do, so there are generic Jenkins jobs and good documentation is the place(Internal Wiki).
We use Kanban and track stories per week with simple math items/weeks we easily could keep develop and deliver CM in time. CM took 3 months to get it done. Thanks to our Kanban management, and internal system design with RDD and Color UML we were able to work normal hours. However, I underestimate the effort to get the system STABLE and the stabilization period that should be 2 weeks become 2 months. During CM tests me and my team found ~30 bugs in CM. I work with a small team right now(me and 2 people) when we start coding CM was 4 people and 2 guys left so after stabilization all gets a bit complicated and we had to work several hours in order to keep with new team and lots of CM bugs this was a bit stressful but we overcome that and deliver to production :D
It's all about Availability
When I was thinking about CM I knew we will need to make CM highly available. In order todo, we made CM optinional so Cassandra can operate without if CM dies or crashes after deploy. In case CM dies or crashes Cassandra keep operating normally and since CM is an orchestrator developer and application who w/r in cass don't see cm. So if CM goes down we lose some operations like:
- Backups
- Repairs
- Node_Replaces
CM has an Autoscaling group for it and persists its internal state in S3 so we build a recovery process for CM in case that happens ASG will spin a new ec2 instance and CM will keep running again. CM was built with multiple layouts in mind so you can have 1 CM for several Cassandra clusters since CM has it own thread Pool or you can have 1 CM per Cassandra cluster, this is great because it provides more availability and allows us to save costs in low production environments like DEV and TEST where we can use shared CM or in production for small clusters we can share CM as well. For clusters that are sensible, they can have they own CM. This is so true because we had an outage in production with CM, however, Cassandra was not affected and was running fine. All this can be seen as Reliable or Anti-Fragility system and as Failure Degradation mode(meaning CM fails don't make cass fails) all great Netflix ideas we stole :D
CM was a set of philosophies like Self Healing and Self-Operating system. Ops teams are used to Operate systems in production due to the lack of built-in philosophy. CM has a different philosophy so basically, you don't operate CM - CM does everything by itself(Unless there is a BUG in CM - them you need RECYCLE / OPERATE CM). CM was highly inspired by Dynomite, Dynomite-Manager, and Priam from Netflix, Although CM has a different architecture. I worked a lot with Dynomite and Dynomite manager on the last 3 years and this experience was very important and allow me to get it here.
Why not use Priam? Priam is great don't get me wrong. However, for my use case I need to support Cassandra 2, Cassandra 3 and soon Cassandra 4(when gets released). Priam has support for Cass 2 and cass 3x, however, Priam has branches and Netflix is migrating the whole fleet to latest Cassandra and we want to have a more long-term support for cass 2 and cass 3 and have our time windows. Priam is a co-processor also know as sidecar and Cassandra Manager is an orchestrator meaning it doesn't have in the same node as Cassandra so we can precisely control what runs in each not and when.
We don't have run multiple things on same time for same clusters like backups and repairs so we want to make TAKS run SERIALIZED like 1 at the time per cluster and parallel within multiple clusters in CM - this will be better covered later on this post.
Jenkins Builds
There are any Jenkins jobs - Basically, there are 2 main jobs. 1 to bake AMIS we can bake amis for Cassandra 2x, Cassandra 3x and CM. Them we have a job for lunch CM. For Devops Engineering we use Ansible in order to provision things in Amazon-Linux OS(Our Cassandras are running in Amazon-Linux latest version - we don't use ubuntu anymore). We use Packer in order to build the AMI for AWS ec2 - We do build in 2 regions. We create LC, ASG and EC2 instances for CM. Once CM is deploying we use CM to deploy Cassandra clusters. CM has java code in order to provision all Cassandra clusters this way we do almost all work in Java, so we have less code in bash as possible and this is great because we take advantage of great troubleshooting infrastructure and tools for Java and also the JVM compiler and good IDEs for refactoring like Eclipse.
Flacky Integration Tests
Since almost all the system is built in java it was very easy to build integration tests. We had to have 2 Tests Suites. One suite for Cassandra 2.x and other for Cassandra 3.x We were able to have the same code and same integration(end-2-end-tests) for cass 2.x and 3.x. However, we had to create a configuration that in the beginning of the sets the AMI and configs for cass 2x and later one switch to cass 3.x. My goal was to use Junit 5, however, looks like more and more Junit is moving away from integration tests and remove suite features and runners(we don't want to use legacy support thanks) so we end up using JUnit 4 which was better for integration tests.
Integration tests stated great and they easily were taking 20min or more to run since we deploy real Cassandra clusters(2x and 3x) during tests and we need to wait for aws to create objects and later on in the same suites we shut down this cass clusters. We test all major capability like backups, restore, seeds manager, restore. So as expected out tests get Flacky. Flacky tests suck. The main issues with our tests(discussed in an Agile Retrospective) were:
1. Timeouts are evil - We can't keep tracking timeouts - So we decide to switch to progressive timeouts.
2. Lack of 9 Rules: In math we can veryfied everything using math so we can check if is correct. we was not doing that in the tests and also in several tasks on the code. So if you dont 1 like that create a file - next line should check if the file as proper created.
We still have lots of work to improve our automated integration tests. What really saved us for the release was combination of Chaos Testings with Exporatory tests and checklists. I don't want to paint a rosy picture about CM so I still have mixed feeling with integration tests and we lost so much time with this.
Generic Remediation for Cassandra
Since dynomite, I had build remediation systems, So which Dynomite I was able to replace AWs AMI or scale UP dynomite without data loss or downtime. This was possible because 2 things: First because dynomie-manager had cold bootstrap feature and second because I coded a remediation process which would call DM(Dynomite-Manager) health checker and check if the node is ok and not doing anything and them kill the node wait for the node finish bootstrap and move on and so on and son. I basically refactord this code(which was and is written in java) and extend it to be a generic Gemeriation so same code wotks for Dynomite and Cassandra. We have some properties we can remediatie any cassandra node without data loss or downtime, who? Same idea we have the generic generiation process going node to node but calling CM(Cassandra-Manager) health-checker and asking if CM is not doing anything with thay node them we run a node_repair in cass so data if copied from other node this is very similar to a node_replace operation when a node dies. If you want leran more about this I made a separated post in the past its here and here.
CM Architecture
CM is built primarily with Java 8 and Inside of Cassandra node we have a Heart-Beat daemon written in Python which calls CM and decides with the node need to check in in CM(new node) or recovery(meaning node_replace) in cass.
We deploy high available Cassandra clusters where we have at least 1 node per AZ begin at least 3 nodes cluster. For multi-region, we do at least 6 nodes cluster. CM replicate its internal state to other CMs meanly in other regions in order to sync multi-region check-in for seeds management. CM was build using PEM files(Cassandra-Nodes) and we are currently looking to change this code in order to use AWS Multi-Region VPC Peering feature so we can switch from EIP and public IPs to private IPs.
Let's take a closer look at CM architecture now.
CM Architecture - More Internal Details
Basically, we have REST interfaces exposing all CM core functionality - This interfaces can be accessed with any REST tool - we build ours on a tool called CMSH which was built in Go and will be cover later on in this very post.
CM has a core class called Internal State Manager which managed all Cassandra clusters and state transitions. There is a bunch of
CM has an internal Tracker class which knows what every class is doing we have an internal Framework called Step Framework which keeps tracks of all sub-task level execution state and stores metadata internal. Later one this internal state in CM is persisted in disk and time to and also sent to S3. In the future, we will backup CM state in Cassandra nodes.
When CM talks to Cassandra it uses SSH in order to manipulate Cassandra filesystem and run local nodetool for backup and repairs. All heavy work happens in cass node, not in CM - Cm just does light state and coordination work. Even s3 Upload is done in cass node, not in CM.
CM Thread Model and Issues
First Thread Scheduling Implementation was using Quartz and Reflections and we realize this was open too much treads. Basically, you want to open your threads up to the front and they just re-use them like a Thread-Pool. However, with CM we have an interesting problem since we don't know how many cass clusters we would need to deploy. For the first implementation was creating a group of threads to run each cluster operation so was basically 4 threads per group. However we were running this on Quartz Threads without the need so first refactor was move some of the work that was in quartz threads like Heart-Beats, State Request, HealthChecker to REST operation only since they only interacted with cm internal state and was a very short living process. Second refactoring was changed quartz and open just one time the pool, however, this created a problem because if we have a big pool we would run things(TASKS) in parallel that we did not want like(backups and repairs) and if we put 1 thread it would be inefficient since the machine could handle more load and we would have 2 different Cassandra clusters competing with same resources. So it was clear that we need a new thread model.
CM Old Thread Model - Thread Issues
So basically the new Thread model would need to provide the following properties/requirements:
- Within the same cluster be serial - run tasks in sequential arrival order
- Within different clusters - run all in parallel
- Run CM tasks in parallel - SignalFX metrics, Internode communication, Recovery.
- Don't break current TASKS - we don't want re-write all tasks code(quartz tasks)
- Replace Threads from Tasks and reuse-tasks
Whats the solution? Simple we use QUEUEs and Workers. So we create and destroy Queues and workers all the time(As a Cassandra clusters do check-in(created) and checkout(destroyed). In order to make this work I had to create a QueueManager and a WorkerManager, So the QueueManager is responsible to assign Tasks to Queues and the WorkerManager to allocate workers in Threads from the pool. This solutions turn out to be more efficient and provided all design requirements we want.
CM New Thread Model - Queues and Workers
Now Quartz is gone and all threads are managed by java concurrent Executors. There are 4 main thread pools. 2 Pools for Cassandra Clusters and 2 Pools for CM. 1 Pool is for Queue and 2 Pool is to schedule recurrent tasks(like backups and repairs) on the queue. Since is a queue one you consume it there is nothing more to do there.
Footnote: Back in 2010 I built a Scala Middleware using this concepts and ActiveMQ and JBoss 5 so I basically apply similar ideas on a smaller scale.
CMSH - Build a REPL in Go
If you work with Cassandra you might be familiar with cqlsh. Cmsh is like Cqlsh but for CM and for Cassandra clusters. Cmsh is written in go, it allows us to stress tests Cassandra and create a schema, insert data and remove data and also plays a role and centralized control panel for CM so we can trigger backups and repairs from cmsh and we can see logs can all cm rest operations pretty easily. CMsh is a REPL.So its much more productive them ssh each box and run alias scripts. This is also created because we can add functionality without having to rebake Cassandra or cm amis. Cmsh is very productive because you can connect in multiple cms with cmsh and also there is tab autocomplete, reverse back search, persistent history and much more. First was coding my own REPL later one moved to iShell which was great and save me lots of time.
Whats Next
There are many cool and challenge aspects of CM that was not covered in this post like Telemetry and Self-Healing, Multi-Region deploys and Checking, TTL Eviction for Nodes, Stress Testing CM, Unit Testing, Tuning and optimizations and much more this might be covered in future posts. Now the main focus of my team is to keep improving CM and support VPC Peering for multi-region deployments maybe later on this year we might do experiences with Containers(Docker). although I'm the tech leader and lead engineer this is a team effort and I want to thank Jackson and Tarzan for the hard working hours and fun for working together as an awesome team and tech challenged that is CM.
I hope you guys like it, take care.
cheers,
Diego Pacheco
Monday, June 18, 2018
Running Ansible with Docker
Getting Started
In order to get started, we need have docker and git installed. Next, we need to git clone ansible-docker and then bake it. Bake just need to happen 1 time. Baked might take some time depending on your internet connection. After baking you can run "run" command which will run ansible linter and then ansible on docker image.
The Ansible Project
For this ansible sandbox, we have simple and default ansible project structure. Which you can see here on the src folder. There is main.yml which is the file it will be run by ansible. You can see this file delegates to a git role in ansible which is located in roles/git/tasks/main.yml
The Dockerfile
Now let's take a look at the Dockerfile. So here are installing Ansible and we are using Amazon Linux Latest version as our base Docker image. As you can see on the Dockerfile we call run.sh which will run ansible as soon as the container get up. You might see a different path that happens because I'm doing some volume mapping - You can check it out here.
That's it. Now we can run ansible locally with ansible-docker. Using this ideas and scripts you can speed up your development time.
Cheers,
Diego Pacheco
Sunday, April 15, 2018
Mocking and Testing AWS APIs with TestContainers and LocalStack
Running LocalStack locally
In order to run LocalStack locally we need to have Python and Docker installed. After you install them we can get and run LocalStack.
After you run LocalStack docker container make sure you shut down because when we run with TestContaoiners we will be able to do it over JUnit.
Setting Up Gradle Project
Now we need to set up a gradle project. Let's take a look at the build.gradle file.
Great. Now we can proceed and work with localstack and testcontainers together.
Hacking to use Latest LocalStack Image
There is a java project on TestContainers that does the Integration between LocalStack and TestContainers we will hack that project because we want to use the latest version of LocalStack. Right now the code is using an old version. So let's take a look at the code.
The changes I did above are quite simple - I just change the docker tag from 0.6 to latest and rename the name of class + enum and that's it. Now we can move to testcontainers test code.
Testing S3 API and Running with TestConainers
Now we can focus on the unit test and mock S3. So let's go for it.
There are some important things here we need to cover. Let's start with the Annotation @Rule. This is important because the runner will use it when JUnit boot up and will boot up the Docker container needed. In regards to LocalStack as you can see I need to pass a Specific Service/Endpoint from AWS that we want to mock.
After the Rule we initialize AWS API - It's important to note here we are passing a custom endpoint otherwise we will reach the real AWS API. Then we can use S3 API normally and all commands will go to your Docker image. So we have mocked S3 Succesful. LocalStack support most of AWS endpoints and the combination with TestContainers is killer now is very easy to test AWS specific code.
The complete code is available on my GitHub here.
Cheers,
Diego Pacheco
