Our assessment employs a lightweight version of the Architectural and Tradeoff Analysis Method (ATAM) developed by the Software Engineering Institute at Carnegie Mellon University.

The five areas assessed include:

• Scalability & Performance: Can the application scale to accommodate a growing user base or increased system load?
• Maintainability & Extendibility: Is the application structured to facilitate growth and support additional integrations and functionality?
• Usability: Is the code clean and navigable, and does it utilize the common semantics and paradigms of the languages and frameworks used?
• Testability: Does the codebase incorporate testing, use common test paradigms, or is it inherently testable?
• Security: Does the codebase adhere to best practices and make considerations around common security pitfalls?

Our assessment includes the manual inspection of the system’s source code and any technical design or architecture documentation that may exist. We also leverage a static code analysis tool to grade the code on Reliability, Code Security, Maintainability, and Duplications. We dig into each of these scores to then provide our own “TechCXO Score” for each of these areas that takes into account the issues identified and the practical relevance of each – static code analysis doesn’t always provide an accurate depiction of the health of the code and requires expert analysis to determine what the important issues are that require focus.

For any issues identified, we will provide both suggested remediation for that issue and a priority. With this approach, the stakeholders have a clear picture of the critical issues that must be addressed immediately, those that can be addressed in the near term, and those that can wait a while. It is common for us to review these findings not just with executive stakeholders and investors but also with the engineering team (if desired). This approach yields the most actionable results since it promotes dialog about the issues identified with the people who can do something about it, which leads to better solutions and more buy-in than just handing them a report.

A review of the use of open-source libraries in the development of an application is important to

1. Ensure proper ownership and rights to sell and distribute the code and
2. Mitigate the exposure to known vulnerabilities in certain open-source libraries

The need to review how open-source is leveraged is often overlooked if the application is not built on top of an open-source version of the application’s core functionality – for example, a Learning Management System (LMS) built by extending the Moodle open-source LMS. While this is an obvious case where there is the need to ensure rights to sell and distribute, there are other ways open-source can be leveraged. Many controls and toolkits are available to developers to help speed up development that leverage open-source components. Furthermore, modern application development frameworks all leverage open-source components. By the time you factor in the open-source in direct use plus all the dependent libraries in use by the development components (and the components they use), there can be thousands of open-source libraries in the mix that need to be validated. In these latter cases, ownership of the code and distribution rights are not the primary concern since these libraries and frameworks were built to be included in applications that may be sold one day. The primary concern is whether the company is using older versions of libraries that use older versions of open-source libraries, which may contain known security vulnerabilities.

To determine if there is exposure for the company, all the open-source libraries must be identified and compared against known lists of open-source versions with security vulnerabilities. In addition, the license type of each open-source component must be reviewed to determine whether any core application code is built using an open-source library where updates must be contributed back to the open-source community – essentially making the proprietary code available to the world.

When the scope of an assessment includes an application that is being developed (either customer-facing in a Product company or internal in a tech-enabled Services business), how that application is deployed, hosted, monitored, and supported are all critical factors in the overall health of a technology organization and the experience of the customer.

In most cases, contemporary applications are hosted either in AWS, Azure, or Google Cloud, and each comes with myriad available services and configuration options for monitoring, security, and scaling. How these environments are set up and managed can have just as much impact on customer experience as the quality of the application itself. Because of that, it is important to have resources who are experts in these platforms to perform this assessment – not just an Architect or Developer who is generally familiar with the applicable cloud environment, as is frequently the case.

Similar to the approach of how we look at the application architecture, we assess the hosting environment across five key factors:

1. Scalability – how well can the environment scale up from a system resources perspective to handle growing demand?
2. Maintainability – how easily can existing personnel maintain the systems? Do the necessary skill sets exist within the current team, or will external help be required?
3. Flexibility – how easily can the hosting environment be changed to adapt to changing business and technical requirements?
4. Security – are best practices followed to ensure user access is set up to limit user access to only the minimal capabilities they need. Is the environment configured in a way that best prevents any kind of breach?
5. Reliability – how well is the environment configured to ensure uptime in the event of a failure in any given architecture component? Is the system highly available?

From a DevOps perspective, we also look at the processes, systems, and tools used to deploy code, maintain the production environment, and monitor and troubleshoot production issues. Again, these practices have a major impact on the overall customer experience.