Java Mutation Testing with Pitest
Video
Code
Cheers,
Diego Pacheco
Showing posts with label testing. Show all posts
Showing posts with label testing. Show all posts
Monday, August 3, 2020
Friday, July 24, 2020
Property Based Testing with Jqwik
Video
Code
Cheers,
Diego Pacheco
Monday, July 6, 2020
Code Forensics
Thursday, February 27, 2020
Going Faster with Testing
Agile View on Engineering Speed
The heart of agile is about Feedback. Why feedback? Because we need to Learn. Going Faster the right way means Learning Faster via more deploys in production, MVPs and Mindset shift. Just following a plan does not make us go faster, no matter how many user stories we deliver per week. If we end up delivering +300 user stories per week - we need to think if:
1) Are we making an impact on users? Are we getting the right outcomes?
2) Are we pilling up debt that will kill our productivity soon?
3) Are we getting more organic growth or we are just following plans?
So in order to go faster, we need to learn faster, how do we learn faster?
1) We make experiments, We fail fast and we discuss ideas and reduce the backlog.
2) We apply user experiments via UX usability Testing, Landing Pages, Surveys, A/B Testing, MVTs(Minimum viable testing).
3) We adapt to changes, new variables, variation and respond to change via learning in a blameless(Devops culture) and Psychological Safety Envimento(Google principle).
In order words, we need more testing. Lean Startup is about Testing. Testing is not only for engineering but for Discovery and for Goals(OKRs). We need to Test more and learn from our tests(experiments). Testing is not only a Discovery matter but also a delivery one, currently, there are a lot of techniques that need to be considered in order to make sure also engineering is building the product the right way.
Testing Pyramid and Sustainability
Testing is old practice in engineering. However, it's often we see products with poor coverage and lack of proper test balance(how many units vs integration test should I do?).
The classical Test Pyramid cover from 3 levels(Unit, Integration, UI) to 10 levels(considering E2E, Acceptance, Regression, Stress, etc..). However, is this enough for 2020? I dont think it is. There are other ways to do testing. I'm not saying that the classical Testing pyramid is wrong. I'm saying: We need more.
Why do we need more? There are several reasons, like:
1) Software become more distributed. i.g: Cloud, Edge, IoT.
2) There is more specialization in Engineering: Frontend, Backend, IoT, DevOps Engineers...
3) Testing can be expensive(Unit testing it's not a silver bullet) due explosion of options.
4) Difficulty to replicate REAL prod env.
5) Multiple teams, cultures, countries, and timezones.
6) Constant software change and Technology evolution
Considering this scenario it's clear that Unit Testing is not enough. Because it happens BEFORE PRODUCTION and we need start doing whats SV is doing, Testing in production. Even on the classical Testing pyramid, there are other testing approaches that can be used like:
* Property Testing
* Mutation Testing
* Snapshot Testing
* Statical Analysis Testing (Covertura / Security Checks)
Let's talk about production now. Testing in Production: Oh Boy!
Why It's not enough? Brand New Testing Approaches
I know, you think I'm crazy. Often when the words Production and Testing come together they are associated with the following issues:
1) Lack of Maturity
2) Risk
3) Damage to the brand and final user
4) Risk of creating outages
5) Lack of Professional Behavior
6) You are nuts, we are not SV feelings
True. Poorly executed Testing in production will create these issues. However not Testing in production also create issues. Do you trust your: Dev, QA, Staging Environments? Would the QA/Staging Environment the new "works on my developer machine mantra?". Often lower environments are not like production. Lower envs often lack:
A) Real Traffic
B) Real cluster sizes and machines
C) Real Configurations
D) Real Users
E) Real Observability
So if QA/Staging is lacking all that, what's the point of testing in QA/Staging? I would say that the main reason is because is how we always did it. Secondly Fear of creating damage and last but not least lack of proper automation, observability, isolation in Production. Great so dont test in Production. Not testing in production does not give you feedback, so are we going faster or slower?
The KEY thing is the difference between deployment and Releases. For most companies, they are something. For smart SV companies DEPLOY is add things in production but do not expose to users. This difference is the key to safety and reducing the blast radius. Netflix runs chaos monkeys in production for many years and no customer impact is noticed. So they do Test and they do learn. If you do not Test you will learn when outages happen which are much more expensive and much more damaging than testing in production.
The previous image show testing approaches classified in Pre-production and In-Production. There are many well know ideas like Chaos Engineering, Stress Testing, Observability(Logs, Traces, Profiling, Auditing) but there is new stuff like Shadowing which means recording prod traffic and replay to a service or via Service Mesh(i.g Envoy) isolate the call in a canary.
Testing in production requires:
1) Automation
2) Canary
3) Observability
4) Isolation
5) Proper Design (Rollback, metadata isolation, different DB cluster, etc..)
6) Ability to perform Rollback
7) Caution
However, is a very powerful tool in order to make sure our product works and we deliver the best user experience we can do at the cloud. Doing that we will be having super important feedback, with the feedback we will be learning and then going faster.
The Way Forward
A great starting point is to explore other ways of pre-production testing like Mutation, Property, Snapshot and Statical Analysis. As you increase your Observability and Automation you really should consider testing in production because you will get the right feedback and also you can save money on lower environments and from avoiding outages.
Proper execution requires maturity and breaking tabus on the sense of Production being something holy and unsuitable. Production is the most important environment however with the right set of mechanisms we can take it much more advantage for the business.
Cheers,
Diego Pacheco
Monday, September 11, 2017
Chaos is the New Normal
Tests are at same time a simple and a complex subject. Testing is something basic. Are you don't do testing how do you know it works? How do you know your software won't stop working after some refactorings?.
Unit Tests are the basic level of testing you could do. In the past unit test was normal. When we deliver software we need to deliver unit tests whiten so level of coverage. When we talk about coverage things can get tricky since you might end up coding tests for parts of your application where it is little or no value what so ever.
Coverage can be something questionable as you language might force you to do less or more tests. For instance is you have a strongly typed language where you enforce as much thing as you can via compiler you might need to do less testing since the compiler works in your favor. As you work with more dynamic and weak-typed languages you might need to do more tests since the compiler might catch fewer things.
As much the language ends up affecting how much tests you do I can say the same or even more about software architecture. How we run our software today affects a lot on how and what kind of strategies we should be considering to test out software.
From Monolith to Microservices
Today microservices are the standard de facto architecture style. Microservices bring effects into your software. There are lots of benefits when we do microservices like:
- Independence: Different things can happen at same time.
- Different teams can use different technologies and versions
- Easier to scale
- Easier to maintain and evolve since there are different and independent code bases
- Isolation: Each microservice has they own:
- Database
- Operation System / VM
- Configuration
- Release Process
However, microservices are not a free lunch. There are several drawbacks or issues you need to address with microservices which were not present before, like:
- How do we do joins? There is no central DB. The need for Streaming.
- Infrastructure Complexity: Microservices require DevOps Engineering
- The need for Observability: Health Checkers, Centralized Logs, and Distributed Tracking.
There is one big effect on microservice that is the fact that we moved from a CENTRALIZED solution to a DISTRIBUTED one. As we have distribution we will have way more FAILURE. Failure will happen and we need to design for failure. That's something it's very hard to do it later. As we need to design for failure, we also need to test for failure, right? Yes. We need to test our microservices in a different way. That's why we need to have chaos engineering.
Chaos: Different Strategy
When we are doing unit testing, integration testing, and contract testing we are testing our application(microservice) but we are not testing the infrastructure. Cloud-Native microservices implies that we run code on the cloud. How do we know that our infrastructure can survive failures? There are many kinds of failures for instance:
- A machine can have no more CPU available
- A machine can have no more MEMORY available
- A machine can have no more DISK available
- A machine can have no more NETWORK available
- An Instance can be terminated any time
- An AZ can go down
- A Region can go down
Is out software ready to deal with all these kinds of failures? This failure eventually will happen and you might learn on the worst time and in the worst way possible. So it's better to test before it happens. This makes the chaos as the new normal. Is chaos is the new normal this could be the new Definition of Done? So now when we finish a story we can say we need to complement our testing strategy with chaos testing. If you are not doing this on the story level you should be doing at least at the delivery level. So you need to have some kind of Production Ready Checklist where applying chaos is one critical item.
Netflix's Simian Army(https://github.com/Netflix/SimianArmy) and Chaos Monkey(https://github.com/Netflix/chaosmonkey) which are chaos tools that can help you to simulate some of this scenarios. However, you just need very few things to create this calls. You can use most of Linux APIS and AWS APIs in order to generate this chaos.
Don't forget Network testing
Chaos testing it's not everything. You need to do more. When you have several microservices calling each other via a network. Several things can happen. For instance:
- The network might completely fail
- You may get 20% extra lag/latency
- What happens is your call never return? The code will be hanging?
- What happens is the return is completely mess up(corruption)
- What IF the return is too big(10MB string for instance)?
Would your code be ready to deal with this scenarios? Well, there is the only way to know it. Doing some kind of network failure testing. Some of this network failures you can use tools like Toxy Proxy(https://github.com/h2non/toxy).
What about Databases?
Do you database infrastructure ready for failures? I'm really into open source. However, I know when I use NoSQL databases like Cassandra, Redis, ElasticSearch, for instance, I need to have a great automation in place, not only for deployments but for operations as well. If a Cassandra node die? Would the cluster recover? Who would spin out new instances? Do you have all under an ASG? Well, we can only know if we test it. The same kind of chaos testing we do for microservices we need to apply for all databases that are managed by us. When I say managed by us I mean any hosted services that is not cloud-vendor managed.
You might go even further and test the database itself. Do you know is your database deliver what the database promise to you? It's your database strong consistent? Are the DB losing data or not? Well, there is chaos testing for databases. Aphyr is doing a while with Jepsen(https://aphyr.com/tags/jepsen) and it's open source so is your DB is not there you can add it https://github.com/jepsen-io/jepsen.
Assertions on Chaos
Junit has this assertion class with several methods to do help you to check certain properties are correct as you expect. How we can do that with Chaos, Network and provisioning testing? For provisioning, you can use ServerSpec(http://serverspec.org/) and a check is your infrastructure is in place. For networking failure. As you use ToxyProxy you can code with JUnit or any other testing framework since you will do a remote call and you expect your code to survive. In order to survive its importance to code basic concepts like:
- Timeouts - Hystrix or any good request lib have it.
- Retries - at least 3 times? Or use Retry Budget is latency sensible.
- Error Observability: Are you fail - log it, send some metric somewhere.
For Simian Army cases, you actually should not care is a machine die or a microservice stop working. You need to worry is has you disrupted the service for the final users or is the failure is perceived by the final user. A great way to test this is using a stress test tool like Gatling(http://gatling.io/) because then you can simulate a number of users let's say 10k concurrent users per minute and then you can run the Simian army and see if a machine, az or region dies will after the users or not.
The software has changed a lot in the last 5 years. Software architecture changed, runtime changed. So your tests need to change as well otherwise you will not be ready for what you are building. Chaos testing requires discipline but keeping the user experience great and not increasing much latency pays off in the end of the day.
Cheers,
Diego Pacheco
Sunday, August 30, 2015
Stress Test with Gatling
But thats not enought you are into SOA/microservice land. As much as you distribute your software you have way more wants to fail. Tests need to catch up, Stress test are something that unfortunately people dont do - sometimes they do but when they know that there is a scalability problem witch is sad.
This practice of stress tests could be automated(it really should) or at least make sure you run periodically as far as i know Netflix does 2-3 times per week - because this is not just about scalability. Stress Tests are bigger than just how many users can i handle.
What Stress Tests are about?
First of all i will not try to make difference of terms, like is load tests of stress tests, call it however you want i dont think the name matters much. So for sure stress tests let you know how much you can scala and thats not only for Web Scalae business but for your company too. Stress Tests can review a serious of problems that you might not them at all like:
* Services can hide Bottlenecks - Could be ground for optimization
* Services can have race conditions - Could be bad coding practices
* Services can not recover on failure - Could be just hanging forever
* Service maybe get affected by dependencies - a.k.a other service not functining well.
* Services could be hiding other botlenecks like databases, messaing systems, datagrids or even infrastructure issues.
The list goes on and on and thats i just Stress Testing not to mention Chaos testing. Like introducing network faliures, drop packages, issues with ack, hanging, os issues like permissions, sicurity groups or firewall issues, lack of oss problem tunning to handle more threads and sockets, etc...
Why Gatling?
Gatling(http://gatling.io/#/) is a great tool. I worked a lot with Jmeter and the problem with jmeter to me is that besides the fact is old is that JMeter is very hard to use it - They UI is a mess and confusing. The XML is not the best structure to code your tests and is not clear what are the things you are doing. Gatling however get all this things right IMHO. So no XML at all you have Scala DSL witch is simple and easy to read and understand what are you doing. Gatling is REST native so is so, so, so easy to test REST services. Gatling have good documentation and in the end of the day is scala code so you can add whatever do you need.
What i really like in Gatling besides the simplicity is the LATENCY orientation, reports are pretty much around latency so you know how fast your service is responding or not and i really appreciate that vision.
Gatling is not so great(at least today) when we talk about high scales, they clustering is not there yet and you need do things by yourself and no hel with the aggregation but for lots of scenarios is a great tool and can help you alot.
Having some Fun
So you need download Gatling from here() and them you just need code your scripts at Gatling: $GATLING_HOME/user-files/simulations/ and them you can place your scala code there. Here is a sample.
For this script we are calling the open weather REST service http://api.openweathermap.org/data/2.5/weather?q=London,uk querying for London. We are checking the http status code so see if is 200 and this call should not take more than 1 second otherwise the test will fail. For this test we have 30 active users, that means 30 threads hammering for 60 seconds.
ÍF you run $GATLING_HOME/bin/gatling.bat
Gatling will list all stress tests scripts you have - them you just pick the one you want run and wait for it. The results come in 2 flavors a summary and a HTML report like this.
There is a Jenkins plugins https://wiki.jenkins-ci.org/display/JENKINS/Gatling+Plugin I recommend you create a separate job so this dont delay your build - whats even better is have a dedicated slave with a dedicated environment just todo stress tests.
Have Fun :-)
Cheers,
Diego Pacheco
