You have completed your AO or RFI/RFP, and you have selected your shortlist of two publishers/integrators. They offer you the solution for one month, with a pre-sales/technical account manager team at your service to set it up and test it. Some publishers even talk more about POV (proof of value) than POC (proof of concept).
How can you get the most out of it and make the best choice (or at least one you won't regret too much)?
Onboarding
- The publishers will provide you with prerequisite documents. Read them, and not five minutes before the first workshop!
- Involve your cloud administrators, especially those with tenant-level (Azure) or organization-level (AWS, GCP) rights. I have encountered several cases of workshops that were unproductive because no one from the organization with the required rights was present to integrate the tool (even though this is indicated in the documentation).
- Terraform is often the method preferred by providers, but administrators at the tenant/org level often do not have access to it or are not comfortable using it. Manual onboarding is preferable, particularly on Azure, where it often amounts to creating an app and granting it rights.
- CNAPP solutions do not interfere with each other on most features (the CWPP component with its agent to be installed is the exception). Integrate them on the same perimeters so that you can compare the detection capabilities of each provider on an equivalent perimeter.
CWPP case
If you are also testing CWPP (Cloud Workload Protection Platform: what runs on VMs/Containers) and therefore agent installation:
- Compare bulk installation methods (not just single-unit deployment) and how to ensure you cover all VMs/containers for a tenant/organization.
- Do not do this only in test environments; the lack of impact on performance must be validated with machines in real-world conditions.
- Test the deployment of a cryptominer (xmrig, for example) to verify its detection and reporting in the solution interface, as well as possible responses.
- Check behavior when deleting machines or agents (offline agents must be deleted manually and affect statistics, etc.)
What to compare
- Often, the inventory section is scrutinized first, but this is not very discriminating, as all serious solutions easily pass this criterion.
- Even though you will often be assisted by a TAM/SE from the publisher, it is important to familiarize yourself with the interface and note how easy/intuitive the solution is to use. It will potentially be used by many different people, and you will not have time to train them all.
- Compare the risks found between the solutions. Export them to Excel, rank them by criticality, and do your own analysis to confirm the level (always remember a solution that flags a scenario as critical that shouldn't even be in the information...).
Integrations
Unfortunately, it is often too complex to fully test integration between the solution and other components of your environment during the POC (time-consuming, requires involvement of teams from other departments, etc.). You will often have to settle for the list of available partners.
- Try to validate the link with your SIEM/SOC anyway. I haven't seen any unpleasant surprises with standard market solutions, but validate the quality of the following points:
- Control over what is sent (no need to send all alerts to the SOC)
- The context that the SOC does not just receive an alert but a little more information to help it process
- Implementing a CNAPP is far from being solely a technical matter. The process aspect and RACI of the various teams are key. Think about your integrations so that they are relevant right from the POC stage.
CDR
The Cloud Detection and Response section is relatively new. The key points to note are:
- Understand the integration mechanics and the additional cloud resources required on your side (particularly for log flows).
- The nature of the integrated logs and their relevance to your organization; focus on audit logs as a priority.
- Have your SOC/CERT try out the interface, as they are rarely familiar with cloud environments. Make sure they adopt the product and can easily trace a chain of events with the relevant context.
Cost
In addition to the quote, it is critical to understand the billing criteria: workload (what is a workload?), credit based on functionality, active user, or functionality level, etc.
Each provider has its own method, which is always relatively complex, but you need to be able to measure the cost of adding a new project to your cloud. Most solutions offer a screen where you can view your usage. Check before and after account integration to note any differences.
Conclusion
The cloud security market is booming, with annual growth of 20-30% expected over the next five years. The CNAPP world, with around 20 publishers, will undergo rationalization (acquisitions/mergers) and combine with other products (DDR, CADR). Beyond the product itself, it is important to choose a partner that can evolve as quickly as the cloud and keep your infrastructure secure.
