Evaluating a new Software Solution
Choosing a software solution is very critical to successfully addressing the problem we are trying to solve. So we need to follow a proper decision-making framework. In this blog, I will list down the things that I consider before choosing a software solution.
Identifying the pain point you are trying to solve.
We need to have a clear goal, what is that we are trying to address. Let's say we are seeing a heavy load on your DB instances. Then we need to find what is causing that, too many writes vs too many reads or skewed data partitioning. So first we figure out the problem and then write a goal according to that.
As a DB user I want my reads to be faster and response within 10 milliseconds.
Involving Future users
We need to try to find out future users and how they might use the solution in the future. This helps in designing our solution in terms of consumption patterns. Let's say in the above use case that the team who complained about the issue might have plans to expose some read-only APIs to downstream apps. So we might look at a Read replica of the DB.
Defining need vs want
We list down the features we need into categories like Must Have(need), Should Have(want), and Could Have(wish list). This helps in prioritizing the work.
Tool Purpose
Once we have a clear goal in our minds and have found a tool/software which seems a viable fit, we list down the purpose and value of that.
Comparative tools
I always compare tools with their alternatives. It is always a good practice to evaluate similar tools. So we check out what other options are available out there in the market. Let's say for example if we are evaluating Tableau then you should consider comparing it with Superset & Metabase etc.
Tech Stack
I always check if the tool is Open sourced or proprietary. Depending on our team’s capability and ecosystem variables, we choose open-source or proprietary solutions. With open-source tools, we get the speed of innovation, but at the same time, we have to be able to keep up with that pace. In the case of proprietary software, the job of keeping a stable version of the product is outsourced to the company and we can focus on consuming it rather than fighting with it. But that comes with a cost.
Product Roadmap
If we have opted for an open-source solution then we see the product’s upcoming features and Long Term Evolution support. In the case of proprietary software, we see how much the latest version is ahead of the one that we decide to use. Also, we see if that company has a future road map for the software.
Viability
Open source (maturity of tech, developer community) vs Proprietary (financial viability and vendor lock-in cost).
Pricing
In both open-sourced and proprietary software always take into account Capital Expenditure and Operation Expenditure and then decide on TCO for at least 3 years.
Support model
I look at the support model of the product. Free vs paid. I also look at SLA, MTTR, MTTD.
Functional
Scope of customization and gaps between tech and business needs. Generally, there are scopes of modification in the core code base in open-sourced software, while in the case of proprietary software we might need to check with the vendor to see if it can be customized or not.
Nonfunctional
Always keep in mind the Scale at which we will operate our solution. Some solutions will be excellent in short term but may be a pain point in case it needs to scale. Also, we see benchmarks of how the software will work under different stress levels.
Architecture
This is very important. I ensure to verify if the solution is Cloud Native vs Cloud Integrated vs Cloud Compatible.
Tenancy
I check if the solution will support multitenancy. If yes then at what level, user space level vs code space level? For example, elasticsearch+Kibana supports document-level separation in roles but when it comes to dashboards it is slightly ambiguous.
Maintainability
I verify how easy it is to maintain the solution. Can we perform Zero downtime upgrades? I verify the frequency of updates. What is the effort needed to perform an upgrade? I check for the availability of a knowledge base, and playbooks.
Observability
I check how verbose the solution is when it comes to debugging it. Is there Native support for metrics, dashboards, and alerts? These help in being agile when we see some bug.
Implementation Complexity
I check how complex it is to integrate it with the existing ecosystem. Also what skills will be needed as extra to implement the solution?
Security
I check to see if the solution has the scope of integration with LDAP, and Kerberos. Is there a CVSS evaluation or framework defined for it?
Other customers
I explore who else is using the product and find out their experiences.
List down Capabilities, Limitations, Efforts, Risks
Once I have done the above checklist, I list down all capabilities, limitations, efforts, and risks. Then only I make a decision based on the noted features.
