SOA for Internal Shared Libraries

SOA for Internal Shared Libraries

Admin aaSaturday, September 5, 2020

SOA has the right principles for scalability with cohesion and consistency. Microservices are dying nowadays for several reasons like Lack of Isolation, Distributed Monolith & Shared Libs Abuse, and many other reasons such as the wrong boundaries. I deeply believe we need to double down in Services and SOA. Sometimes the Sidecar Pattern also makes sense and is an interesting approach.  However, there are times where we really need to go to Internal Shared Libraries.  When that's the case we can still apply SOA for Internal Shared Libraries and have a better a clean design such as having a clean CORE. Keep in mind there is a lot of shared libs abuse in companies but in the cases where internal shared libs are the right choice, there are some important considerations. SOA can philosophy can help internal shared libs to do better and by doing so we can have better solutions with much fewer headaches. I need to repeat Internal Shared Libs should not be your goto approach. Services should be your number 1 choice to deliver software.  

It's not about being a purist

So should we never ever use any lib and code in Java 1.2? NO. We should use libraries. We should use Frameworks. We should not re-invent the wheel. However, do we mind the dependencies we are imposing in people's classpath? Do we even realize how many jars we are adding there? That's the issue. There is a binary coupling, where upgrading libs is very hard. So the solution is not to go to the other extreme and say no libs any more, No that's not what I'm saying. We need to reach some middle ground where we need to Judge and analyze case by case. However, I need to say that we need to be much more RIGID and DISCIPLINED when come down to using shared libraries.  

Discipline is the key

The discipline is the key. We can copy and paste code from the internet on your CODE, that's 100% fine. We can also use plugging like SHADE and have a single jar with no dependencies. We can introduce dependencies as well as long as we are conscious and paying lots of attention to that. The whole java ecosystem of OSS libs often depend on Guava, People use guava a lot, guava is very hard to upgrade when you have different versions. NetflixOSS stack was very binary, Spring cloud used NetflixOSS and is even more binary. There is this effort to push reactive programming everywhere, meaning to call services you will need to use clients it would be impossible to do REST calls very soon. This is coming.  We should not trade performance for flexibility(never). 

So how can we fix this? Discipline is the answer. We need to Think, case by case, and analyze the options and make sound decisions. Changing one extreme for others is not the answer. We dont need to ban shared libs we need to be much much more disciplined about them. 

Functional Programing is about discipline as well(Discipline state management). DevOps, CI/CD are about discipline too, I should say there is no way you can get internal shared-libs right without discipline.

There is no Silver Bullet

There is no way we could create a single rule and have simple decisions we could just apply to all cases. Things are not that black/white we can't have a clear cut when we talk bout design and this particular libraries issue. In other words, there are no silver bullets. We need to fight complexity and all its forms and hidden costs. We need to stop creating Debt that kills us in the long term. So it's important to Reflect on old dogmas and sometimes the answer is an unpopular opinion. 

This is hard because it leaves us with no good default choice. If you are looking for a default choice should be serviced, not libraries. We have an obsession with Reuse. We should have an obsession with your consumers and freedom and isolation instead. The IT industry often takes a long time to see it's mistaken so just because few people are talking about(or you think is few people) it does not mean you should not pay attention.  Hopefully, we are just judging and making decisions by popularity.

Design Libs using SOA

SOA has principles. There is the SOA Manifesto. Jeff Bezos wrote in 2002 an SOA Mandate for Amazon. One way to see it is "Talk via Services not via Internal Shared Libs". 

SOA has Services. Services have Contracts and Implementations. The API in your Internal Shared Library should be your Contract. You should decouple you contract from your implementation and hide as many things as possibles. Doing so It's possible to apply SOA to Internal Shared Libraries. Which will give you more flexibility and less Intrusion to your lib consumers.  Proper versioning is also very important you should not break your consumers with MINOR and SECURITY/PATCH versions changes. IF you pay attention to your 3rd party libs you can make this possible. The benefits are countless such as:

  * Reduce the number of Lib Migration Efforts (Time and Money).

  * Increase the TRUST and IMAGE of your Platform team among your consumers.

  * Reduce the blast radius impact in your consumers via(3rd party deps and better contracts).

  * More time to Consumers focus on innovation + Increase the value of your solutions

  * More Freedon to change the Internal Shared Lib under the hood

  * Less coupling and more innovation for both sides (Provider and Consumers).

SOA is key for Better Designs and reducing complexity. SOA can be applied to Internal Shared libs and is not only a Service Thing. 

Cheers,

Diego Pacheco

Shared Libraries Trap

Shared Libraries Trap

Admin aaMonday, August 31, 2020

SOA it's all about Services. Services are the first-class citizen and there is no mistake in that decision. Some problems requires shared libraries and thats fine, however, I would say most of the internal shared libraries are not needed and they only cause trouble and little benefit. There is plenty of abuse of shared libraries and we need to start to recognize that we need to reduce the number of shared libraries and do fewer libraries and accept more code duplication and others from os package solutions. Shared libraries tend to spread and as a side effect, they create lots of binary coupling, complexity, and maintenance issues that kill any real Service/Microservice benefits. Services require flexibility and freedom to upgrade and release in different schedules. You might not realize but Shared Libraries are killing services and making them just a big complex distributed monolith. That's not the first time I blog about, but I still think the industry still did not get this right, and still super important to understand this big issue. There are plenty of mistakes that are killing microservices, shared libraries is a big offender.

Don't Repeat Your Self (D.R.Y)

It always starts with that. I don't want to duplicate code. We dont want to fix the same bug in 2 places. That often is the argument to start a shared library. However, what really boils down to is developer experience and commodity. It's funny but as engineers, we are paid to solve problems, often by writing code so we try to write less code. There are good and SOLID reasons to have internal libraries but I would argue is 10% of the cases and 90% of the cases are just commodities and some particular programming model that is desired such as Annotations, Aspects, Spring Integrations, and other extensions/integration that are mostly style based. It all starts as "good" ideas but ends with coupling, technical debt, slowing down migrations, complexity pilling up, and deprecated old code that people don't like. It does not take long to be stuck into some old frameworks that you cannot upgrade unless you migrate all Services. 

We need to leverage Services, SideCars, Tooling over libraries. Because there is so so so much abuse and miss use of libraries and the end result is always the same: Tech debt, Bad software written, people left and go make the same mistakes in other companies and we need end this cycle. How many times you saw someone saying: "Dont write a shared library"? Never! People know "OH we need to reuse code", "OH we need to be D.R.Y which is a principle". Maybe do we need different and new principles?

Scalability is about duplication 

NoSQL scales. The key to NoSQL Scalability is DUPLICATION. We duplicate Data. The KEY for CQRS/ES(A practical way to scale applications) is to DUPLICATE data again. SOA is about duplication. Microservices is about duplication. Duplicating CODE is 100% fine. Data Lakes and Data Meshes are about DUPLICATION. Everything that is BIG and SCALE is about DUPLICATION is time for us to accept we need to let it go. It's 100% fine to duplicate code between services. 

When you duplicate your CODE you are DECOUPLED and that allows you do have ISOLATION. Meaning your service can be independent of another service and you can have different stacks, versions and work things much faster. 

Sharing is a BOTTLENECK! The best Multi-threading solutions happen in architectures called "SHARE NOTHING". The worst code to refactor is the ones with a global shared state, state isolation is the key for performance. Isolation and duplication walk side by side. What you think Multi-Region means? Duplication. What you think High Availability mean? Duplication. So you don't believe me, well if you look into 70-90s database architecture we have OLTP and OLAP 2 different words, never run an expensive report on a transaction DB. So how you fix this? DUPLICATION. 

You don't need shared libraries

So just you don't think I'm a crazy Brazillian, that might a very few use cases where doing an internal shared library is 100% the right solution for that case you need to be very careful and try to add any external library as transitive dependencies and there are techniques for that. 

It's 100% ok use shared libraries from the internet but is not fine to use your internal libraries because you are abusing them. Libraries from the internet are FREE and you did not pay to use FOSS. When you have a team building a shared library you have an investment issue because often the team who are internal consumers of that library cannot afford to build something better by themselves. Shared Libraries have HIGHLY coupled with services since they are binary dependencies and any update on them requires migration if you break the API or not. We can reduce some of the pain by adding automation so for minor and security/patch versions changes you dont require people to do nothing.  

We need to understand internal shared libraries are not making any good us, actually it's the other way around there is so much abuse that we really need to consider stop or drastically reduce the number of shared libs we do.  Look Istio and Service Meshes(Kubernetes space) they push things to the PLATFORM, there is ZERO couplings, ZERO binary dependencies, ZERO shared libraries. 

Cheers,

Diego Pacheco

Double Down on Service Orientation

Double Down on Service Orientation

Admin aaFriday, July 3, 2020

Service-Oriented Architecture(SOA) is not new. Microservices which are a specific flavor of SOA are the default architecture you will find out in digital products. I'm not sure if Microservices will be here for much longer. However, thanks to the silicon valley, companies can be structured in 2 organizations(Which is a good thing IMHO). Product and Engineering organization. An engineering organization means that you create software for your product engineers to be more productive and focus on the product and improving the user experience. Engineering organizations have different ways they could be distributing software. Software distribution is a fascinating theme for me. There are several ways we can distribute software however we often dont consider other distribution models. As your company grows and depending on your business segment you might have several specializations insider engineering like Data Science, Security, Big Data, Architecture, Storage, Mobile Devices, and so on and on. IMHO it does not matter the kind of software specialization we are not limited to binaries.  

SOA It's all about Services

SOA only talks about Services. Services are 1st class citizens and if you want to reuse you do it via services. Product organizations leverage Services however it's very common to find engineering organizations being too much binary, I saw it so many times. Binaries are not wrong if you are very careful about dependencies(most of the time people are not). However, there are other distribution models beyond binaries. For instance, a model that is growing a lot nowadays is the Side Car pattern. Besides sidecars, you could rely on Services. Services dont need to be a consumer only via APIs or JARs or binaries, services can be consumed via generic UI or even a generic Jenkins Job. 

One important thing that Services provide is abstractions. Meaning you dont need to CODE or you dont need to configure it a good abstraction if done the right way. Within a remote service interface like API/REST, UI or Jenkins job is possible, with binaries is not possible and often results in lots of coupling.

Issues with Binaries

By definition binaries are "EMBEDDED" meaning you need to have them on your side(classpath). Having binaries on your side means you get all side effects like Configuration, Schemas, 3rd party dependencies. By nature, binaries are much more coupled with the remote services.  +10 years ago I remember when I was in a big international SOA Project. Then we had an idea to have EMBEDDED Services. Since all our SOA Services was written in Java we just important the jars in the target/consumer project and thats it you have an embedded service and it works. There was a great performance benefit for killing the remote call however it resulted in several other issues like:

 * Much more difficult to configure since the jars had to have the configs our you would duplicate it.

 * Dependency hell - since 3rd party dependencies were brought put there were jar conflicts. 

 * Potential Coupling - Now there is noting blocking developer to use non-contract classes(contract by accident will happen eventually). 

 * Difficult to upgrade individually:  If you change the service need to change the embedded deployed service as well - which was always hard.  

Services Benefits

Services provide abstractions. Service Abstractions are potentialized by remote interfaces like REST/APIs, Generic UI/Jenkins because:

 * Natural Hiding: Since you are "remote" or at least running the code someplace else. 

 * No contract By Accident:  It's basically impossible for the consumer to use anything beyond the public contract. When we think about the binary model, it's every use to couple beyond the public contract. 

 * Easier to Scale: as you have control over how you operate your solution it's easier to scale and provide a better experience and abstract that from consumers.

 * Easier to Maintain: Since we are exposing less it's much easier to upgrade the underlying operating system, data stores, code, languages, libraries, or whatever aspect of the solution besides the contract.

 * Easier to Reason about it:

Service provides many benefits, they will always be less optimized as a specific solution however they often make more sense and provide a stable way to the company to build products and other services on top of each other which is how the internet works and how companies operate in the real world. 

Operationalizing Services beyond Service Interfaces

SOA is not restricted to APIs(REST). You can do SOA with other interfaces / API, for instance, it does not matter the language or serialization format(XML, Json, Yaml, Protobuf, Flatbuffer, Avro, Parquet, MessagePack, whatever). Having said so, it's also possible to provide generic UI's and Jenkins jobs, which can be useful to consumer stateful services like Databases, Caches, Object Storages, DNS, Pipeline Platforms, Observability platforms, Stress Test / Chaos platform and much more. 

If you take a look at AWS there are managed services like S3, RDS, Kinesis, and many others. These services abstract the operation, scalability, upgrade, patching, and security aspects for you. It's possible to use some ideas and deliver internal services to your company. Netflix pig bank a lot in that idea and has several of internal services(Similar to AWS managed Services). 

You can do the same, therefore you can consider Operacionalizing services as an interesting distribution channel/model for your engineering solutions. Service approach means you have a centralized operation for those services, you can have different teams operationalizing different services. 

Consuming a service does not need to be done always via an API. API(REST) has one big drawback when we compare it with Generic UI / Generic Jenkins job. They require you to CODE. In order to use it, you need to code. Sometimes coding is fine, as much as possible you should not require your consumers to code. For instance, think about a Database Backup Service. It could be a generic UI or Generic Jenkins Job you go there and just click in a button and thats it, you enable automatic backup for your application rather than have a DSL or Java API someone would need to use and run the code against it. 

SDKs are fine, DSLs are fine. API's are great. However, you need to consider whenever is possible to not require your consumers to code will be great and will increase productivity, abstraction and you will be leveraging more service orientation than binaries. 

Double Down on Services

Remote Services with standard interfaces often could have performance drawbacks. For instance, REST Interface + JSON will be always slower than the LOCAL IPC call with a binary format for instance. However, trading performance for flexibility and abstraction is often a bad deal. Being remote often means you can scale beyond a single machine.  Services are great as distribution channels because of the abstraction, easy to upgrade, isolation and it's much easier to organized teams around it. 

Services can be used for non-end-user products. Internal Services are an interesting and yet powerful solution to avoid inertia in software architecture. When you have a binary, if you improve your code, you often need to do a migration, which can be hard, tedious, risky, and expensive. Services provide a great abstraction and reduce lots of migration needs. Migration is a big concern because refactoring / re-testing all your consumers could be expensive and risky as you grow and have a big organization. 

Internal Services make consumer life easier, compare DVD with Stream model for instance. DVD was like a binary JAR and Streaming is like an API(REST) or Generic UI or Generic Jenkins Job. There are several concerns that get abstracted by the service model. All models have tradeoffs, there will be cases were binaries are the right answer. IMHO we should be open to considering using more services whenever is possible. 

Cheers,

Diego Pacheco

The Death of microservices - Distributed Monolith 101

The Death of microservices - Distributed Monolith 101

Admin aaFriday, May 15, 2020

It does not matter the question; the answer is Microservices Architecture(MSA). Feel that way too? Are companies doing proper microservices at all? Would it be just everybody just calling "microservices" and doing all sins against microservices? Let's adopt microservice now, okay. What problems are we trying to fix? I know I want to do it because it is everybody doing, are you sure? I think I know what I'm doing, are you? Well, "the road to hell is paved with good intentions." The problem was happening before the SOA era; we had this problem before, microservices still have it, Serverless the same. OH, this is backend thing, buddy we are good at Frontend or in BigData, DevOps Wrong! It's everywhere! If there are software and distribution, potentially, you will have this issue, which is not minor. 

The Reality Mismatch 

Very few companies are doing proper microservices. What companies are doing are not microservices. Microservices will likely stop to be a thing(to be honest if most of the company never did it - was they real anyway or just an illusion?). Do I believe in Microservices(MSA) for sure, do I think we should do it? Yes, but that will require discipline and effective learning, leadership and year, are we ready for it? Most of the cases no. It hurt a lot to admit, but there is one little thing called "Reality," and we are not doing an excellent job with her. Should we give up? Hell no! Like everything in this life, it's about education and discipline. Software is fragile, people come and go, so it's easy to lose control, and things fall apart. I wish reality were different, but, unfortunately, it is not. 

What problem are you trying to Fix? 

If you are picking microservices, you SHOULD have the following problems:

Difficulties to Scale: It's hard to scale up a monolith because of the coupling. It's hard to work appropriately with specialized teams. It's expensive to scale, and there is lots of waste. 

Specialized Solutions: You want to work with different languages, different technologies, separate databases. Why? Because it is cute? No. There is the right tool for the job. Also, as you have more freedom, you can use people's skills more productively and intelligently.   

Freedom to Upgrade: Can you upgrade any lib, language version, server quickly in your solutions? If you have coupled, the answer will be no. Upgrading a lib could mean:

• Fixing a security potential breach
• Improving performance
• Reducing costs - Better code - using fewer resources
• Pure developer joy and productivity with a modern, up-to-date solution - a.k.a productivity. 
• A simple bug fix

However, can you upgrade your libs(independent), wich out a mass migration? Often the answer is not. 

However, microservices are not a free lunch. There are prices you need to pay. Here are the following prices you MUST pay:

1. Infrastructure Complexity: Separate deploys, Dbs, Code bases.
2. Complexity: in form, but not limited by Eventual Consistency, Distributed programming(failures). 
3. Duplication: Data duplication, Code duplication, DevOps code duplication. 

However, naive managers have problems with code duplication. Even engineers think repetition is terrible. Well, it is awful, it's a source of bugs, right? Yeah. However, simple is not that simple. You cannot eat the cake and have the cake. 

You don't want code duplication: Why did you pick microservices in the first place? 

That's it. It would help if you let it go. Code duplication is not bad as in comparison with coupling. Coupling is the trait of a monolith. Hold on; we have several services we don't have a monolith, correct? Yes, you do not have a monolith. You have something far more worst them a monolith. You have a distributed monolith. Why is a distributed monolith worst them a monolith? 

1. It's more complicated - you made it distributed buddy. 
2. It has more failure points - you made it distributed buddy.

Not only is worst but you still dont get:

Freedom to Upgrade - Because it is all coupled, thanks to you smart code reuse. Saved some lines of code but kill the main microservice benefit (ISOLATION). IF you are doing a poor microservice, stick with a monolith. That's why I imagine Sam Newman often says, do not start with a microservice. 

Botton line here: Code Duplication is fine. Go Type. Seriously. OH: but I have the same problems everywhere. It's expensive, it sucks. Well, do you think a distributed monolith is cheap? I felt you moved to microservices because you want ISOLATION meaning code duplication. 

Ways you can shoot-your self: Traps in Details.

There are so many ways you can kill yourself trying to implement microservices properly. It looks like this time; we did not make the same errors in SOA in the sense of tooling. We still have a long road ahead. What are the ways we can do microservices wrong, or how things can go wrong? 

1. You have too many microservices with the wrong boundaries - Go luck refactoring your distributed monolith. OH, Maybe a monorepo can help us - yeah hold that thought. 
2.  Shared-Libs and Drivers: Yeah. It's is how you create a distributed monolith. More to come. 
3. Lack of Encapsulation: Tables. People still share tables; you know the worst? It's not only "tables," you need to worry. 

Is it a monorepo the solution or a part of the problem? 

First of all, you are not Google. You do not have 2B lines of code. Second, if you already have a distributed monolith, yeah, a monorepo will make it easy to upgrade the best, It's all or nothing. However, if you have proper isolation, proper design, no abuse of shared-libs, why do you need all code in one freaking place? You will likely start coupling things. Cripling you microservice! So dreamed isolation, and you might create a distributed monolith. Think twice. What problem are you trying to fix, and what difficulty you might create? Do no think decisions are free of side effects and consequences. 

Shared-libs are evil

Don't believe me? Look this video from a former Netflix Engineer who is on Facebook right now. Why are shared libs evil? Because of the 3rd-party dependencies, they introduce and coupling. Shared-libs start small, and like any addiction, they increase; you have hundreds of dependencies to make your software work. It won't take time. When you have 1,10,100, 1k, or 10k services, you won't be able to upgrade anything. Sounds familiar, which the monolith? Yeah, but worst. The issue is if you are a big product company, you know what I'm talking about, excellent. If you are in a service company might never see this issue as you leave the project after "deliver"(Good old waterfall), and it might take some time to see this problem to manifest. 

Drivers are evil

There are only one thing wors the shared-libs, which is drivers. If to access a server, you need a driver, and this driver introduces lots of dependencies you are in deep trouble. Let me talk about an unpopular thing today called SOA. In SOA, we use to like to have: Services. Nothing else, nothing more. That's why it is called Service Oriented Architecture. Services need to have a Contract and an Implementation. The contract should abstract a lot and be flexible and provide intrinsic interoperability. Which your custom driver just killed. Now no one that nows REST can talk yo you service you need that Jars of yours. What if you want to call the service with Python, Rust, V, GoLang? You need to have a driver per language.  

It's possible to have drivers and shared-libs that don't suck, yes it is possible. But the implementation needs to be chirurgical. Meaning you do things you are not used to doing like:

1. Mind your dependencies - Add as little as possible - ideally ZERO 3rd party dependencies. Be Lean on Dependencies.
2. Copy Code: Why dont you copy some code instead of added 30 dependencies for five lines of code? Sound bad but isn't.  
3. Use package explosion: Plenty of tools that can help you with that. More to come - will comment more soon. 
4. Encapsulation: Hide your code, have public interfaces(few), and do not leak your implementation; neither make it public. 

In my previous post, I was talking more about coupling in the context of backward and forward compatibility, which is related to this post. There is only one thing worst than drivers, which is(often know as client-drivers or network-drivers)— corporate frameworks. 

Corporate Frameworks

In the past, every company in brazil (10-20 years ago) had a corporate framework; these frameworks never worked well and often presented all these issues:

• Expensive: Often, they spend more time doing a framework than fixing real problems that the company needs to make money.  
• Complex to use: Often based on a popular framework like Spring, Hibernate, or JEE but much harder to use. 
• Lack of documentation: No Javadoc, no samples, no wiki, nada!
• Lack of abstractions: Often, make your life worst. Often do not provide the classical two levels of concepts to fix problems—level 1: Abstraction, Level 2: Low-level access. 
• People hate it: They can't use the web to search, they can't tell their friends, they cant use it in other jobs. 

Today corporate frameworks are less popular on their original "form"; however, they still exist—often called "shared-libs." Just because it is the package like lib and you "call" a lib, it does not mean it is a lib. Today libs are the new corporate frameworks. 

A Lib == framework is terrible when it falls in the following issues:

1. Impose a specific programming model: Teams should have freedom of programming mode. Some people like Annotations, Mappings, ORM, Reactive others don't. 
2. It has coupled with other libs: It doesn't take time for your libs to depend on all your other libs. Coupling is the root of all evil. Them you can't upgrade anything, can't add anything in your service until all other services do the same. 
3. Hides to many things from you: This is one of the main traits of a framework, which means: "To frame the work." 

It does not matter how you can; it matters what it is. 

It's not only about Tables.

People still fail in isolating databases. Often companies have the same database cluster for multiple microservices(cost reasons, lack of automation, or simple corrupt practices); however, databases(when is your software of truth) is not the only thing you need to abstract and encapsulate. These are the other "things" you need to hide:

1. All Datastores (SQL, NoSQL, NewSQL, In-memory) all of them.
2. Caches (IMOC): Redis, Memcached, RocksDB, you name it.
3. Configurations: XML, Json, Yaml, whatever.  

Everything that is "public" is a contract. Kafka/Kineses/RabbitMQ queues are part of your agreement, You might not be able to hide them, but you should and need to treat them as contracts. 

How can we fix this? 

Not easy. However, there are things we can do to avoid or minimize these issues. To fix these problems, we need discipline, education, and other approaches. We also need to start accepting the penalty that every choice has, for this case, stop worrying too much about code duplication. 

Do not accept jars from strangers.

We need to stop creating shared-libs for everything. We need to stop coupling shared libs together. We need to start leveraging: Protocols, Good Practices, Templates, and even looking at our work-mate microservice and copying the code from there. We need to stop pushing opinionated programming models on everybody like RxJava or Spring Boot. Do you want to use spring boot in your service, fine, do it, be happy! But don imposes on everybody else. Even if everybody wants the same capability or shared-lib right now, I bet you a beer that people will not want to upgrade at the same time as you will wish to in the future. 

Currently, shared-libs are the number 1 cause of death to a microservices worldwide, right followed by shared databases. Protect your classpath. 

Packages are your friends.

Packages provide isolation and abstraction. We need to use more of them. There are several libs and plugins for building tools that can get your 3rd-party libs and create a FAT jar or even explode the 3rd party dependencies in your classpath. OH, but Diego, it will use much memory, and that's fine. Again accept the tradeoff. 

Isolate Everything

Isolate your datastores, caches, and config-files. Do not let. They see beyond the service contract. Keep them internal to your service. Having proper isolation means you only need to apply backward/forward compatibility at your contract level, you are free on the database now if you need to provide back and forward compatibility. 

Time passes fast. Companies grow, global refractories are not top priorities for most executives. You are protecting your data now will pay off in the long run.

It's about Plataforms

Shared-libs, Frameworks, Drivers are not the only place for you to put your code. You have other better options like Services, for instance. Instead of creating a lib, you can create a service. However, not all problems can be resolved with services, especially if we go remote calls and pay the right price, for that can have at least four other options. 

1. Tooling
2. Self-Service Automated Service
3. Sidecars
4. Runtime Platforms

Tooling: It could be a build tool, simple script, UI. Any self-service tool your engineers can use in build time to fix their problem. For some use, cases will work, for others don't. 

Self-Service Automated Service: Instead of making a lib, you can make an internal self-service solution. i.g.: You could create a shared lib to do Cassandra Backups or create a transparent service that does not improve jars in people's code. 

Sidecars: Sidecars have their issues like hard to debug and not idiomatic. However, they isolate and abstract solutions without coupling with your classpath—another option instead of a lib. i.g: Lyft Envoy. 

Runtime Platforms: Like JEE, Erlang, Spark, Kubernetes, where the abstractions are at the platform level and not at the code level. 

There are options. We need to make our code pure and clean. Otherwise, we dont get the benefits we are advertising, and it will be a never-ending cycle of issues. 

Education

People need to get a deeper understanding of architecture and understand that choices(tradeoffs) have consequences which we must learn to accept(talking about duplicated code). We need to train people, and like in Lean thinking: "Learn how to see waste." Education does not mean a simple training and game over. It is a continuous effort. Ideas need to be alive in people's heads; otherwise, we won't be able to him this war. Education is a must.