Home / News / Introducing Cloudflare Application Confidence Score For AI Applications

Introducing Cloudflare Application Confidence Score For AI Applications

Introduction

The availability of SaaS and Gen AI applications is transforming how businesses operate, boosting collaboration and productivity across teams. However, with increased productivity comes increased risk, as employees turn to unapproved SaaS and Gen AI applications, often dumping sensitive data into them for quick productivity wins. 

The prevalence of “Shadow IT” and “Shadow AI” creates multiple problems for security, IT, GRC and legal teams. For example:

In spite of these problems, blanket bans of Gen AI don’t work. They stifle innovation and push employee usage underground. Instead, organizations need smarter controls.

Security, IT, legal and GRC teams therefore face a difficult challenge: how can you appropriately assess each third-party application, without auditing and crafting individual policies for every single one of them that your employees might decide to interact with? And with the rate at which they’re proliferating — how could you possibly hope to keep abreast of them all?

Today, we’re excited to announce that we’re helping these teams automate assessment of SaaS and Gen AI applications at scale with the introduction of our new Cloudflare Application Confidence Scores. Scores will soon be available as part of our new suite of AI Security Posture Management (AI-SPM) features in the Cloudflare One SASE platform, enabling IT and Security administrators to identify confidence levels associated with third-party SaaS and AI applications, and ultimately write policies informed by those confidence scores. We’re starting by scoring AI applications, because that’s where the need is most urgent.

In this blog, we’ll be covering the design of our Cloudflare Application Confidence Score, focusing specifically about the features of the score and our scoring rubric.  Our current goal is to reveal the details of our scoring rubric, which is designed to be as transparent and objective as possible — while simultaneously helping organizations of all sizes safely adopt AI, and encouraging the industry and AI providers to adopt best practices for AI safety and security.  

In the future, as part of our mission to help build a better Internet, we also plan to make Cloudflare Application Confidence Scores available for free to all our customer tiers. And even if you aren’t a Cloudflare customer, you will easily be able to browse through these Scores by creating a free account on the Cloudflare dashboard and navigating to our new Application Library.  

Transparency, not vibes

Cloudflare Application Confidence Scores is a transparent, understandable, and accountable metric that measures app safety, security, and data protection. It’s designed to give Security, IT, legal and GRC teams a rapid way of assessing the rapidly burgeoning space of AI applications.

Scores are not based on vibes or black-box “learning algorithms” or “artificial intelligence engines”.  We avoid subjective judgments or large-scale red-teaming as those can be tough to execute reliably and consistently over time. Instead, scores will be computed against an objective rubric that we describe in detail in this blog. Our rubric will be publicly maintained and kept up to date in the Cloudflare developer docs. 

Many providers of the applications that we score are also our customers and partners, so our overarching goal is to be as fair and accountable as possible. We believe that transparency will build trust in our scoring rubric and guide the industry to adopt the best practices that our scoring rubric encourages. 

Principles behind our rubric

Each component of our rubric requires a simple answer based on publicly available data like privacy policies, security documentation, compliance certifications, model cards and incident reports. If something isn’t publicly disclosed, we assign zero points to that component of the rubric, with no further assumptions or guesswork.  Scores are computed according to our rubric via an automated system that incorporates human oversight for accuracy.  We use crawlers to collect public information (e.g. privacy policies, compliance documents), process it using AI for extraction and to compute the resulting scores, and then send them to human analysts for a final review.   

Scores are reviewed on a periodic basis. If a vendor believes that we have mis-scored their application, they can submit supporting documentation via app-confidence-scores@cloudflare.com, and we will update their score if appropriate.

Scores are on a scale from 1 to 5, with 5 being the highest confidence and 1 being the most risky. We decided to use a “confidence score” instead of a “risk score” because we can express confidence in an application when it provides clear positive evidence of good security, compliance and safety practices. An application may have good practices internally, but we cannot express confidence in these practices if they are not publicly documented. Moreover, a confidence score allows us to give customers transparent information, so they can make their own informed decisions. For example, an application might get a low confidence score because it lacks a documented data retention policy. While that might be a concern for some, your organization might find it acceptable and decide to allow the application anyway.

We separately evaluate different account tiers for the same application provider, because different account tiers can provide very different levels of enterprise risk. For instance, consumer plans (e.g. ChatGPT Free) may involve training on user prompts and score lower, whereas enterprise plans (e.g. ChatGPT Enterprise) do not train on user prompts and thus score higher. 

That said, we are quite opinionated about components we selected in our rubric, drawing from deep experience of our own internal product, engineering, legal, GRC, and security teams. We prioritize factors like data retention policies and encryption standards because we believe they are foundational to protecting sensitive information in an AI-driven world. We included certifications, security frameworks and model cards because they provide evidence of maturity, stability, safety and adherence with industry best practices.

Actually, it’s really two Scores

As AI applications emerge at an unprecedented pace, the problem of “Shadow AI” intensifies traditional risks associated with Shadow IT. Shadow IT applications create risk when they retain user data for long periods, have lax security practices, are financially unstable, or widely share data with third parties.  Meanwhile, AI tools create new risks when they retain and train on user prompts, or generate responses that are biased, toxic, inaccurate or unsafe. 

To separate out these different risks, we provide two different Scores: 

  • Application Confidence Score (5 points) covers general SaaS maturity, and

  • Gen-AI Confidence Score (5 points) focused on Gen AI-specific risks.

We chose to focus on two separate areas to make our metric extensible (so that, in the future, we can apply it to applications that are not focused on Gen AI) and to make the Scores easier to understand and reason about.   

Each Score is applied to each account tier of a given Gen AI provider. For example, here’s how we scored OpenAI’s ChatGPT:

  • ChatGPT Free (App Confidence 3.3, GenAI Confidence 1) received a low score due to limited enterprise controls and higher data exposure risk since by default, input data is used for model training.

  • ChatGPT Plus (App Confidence 3.3, GenAI Confidence 3) scored slightly higher as it allows users to opt out of training on their input data.

  • ChatGPT Team (App Confidence 4.3, GenAI Confidence 3) improved further with added collaboration safeguards and configurable data retention windows.

  • ChatGPT Enterprise (App Confidence 4.3, GenAI Confidence 4) achieved the highest score, as training on input data is disabled by default while retaining the enhanced controls from the Team tier.

A detailed look at our rubric

We now walk through the details of the rubric behind each of our Scores.

Application Confidence Score (5.0 Points Total)

This half evaluates the app’s overall maturity as a SaaS service, drawing from enterprise best practices.

  • Regulatory Compliance: Checks for key certifications that signal operational maturity. We selected these because they represent proven frameworks that demonstrate a commitment to widely-adopted security and data protection best practices.

  • Data Management Practices: Focuses on how data is retained and shared to minimize exposure. These criteria were chosen as they directly impact the risk of data leaks or misuse, based on common vulnerabilities we’ve observed in SaaS environments and our own legal/GRC team’s experience assessing third-party SaaS applications at Cloudflare.

    • Documented data retention window:  Shorter retention limits risk.

      • 0 day retention: .5 points

      • 30 day retention: .4 points

      • 60 day retention: .3 points

      • 90 day retention: .1 point

      • No documented retention window: 0 points

    • Third-party sharing: No sharing means less external exposure of enterprise data. Sharing for advertising purposes means high risk of third parties mining and using the data.

      • No third-party sharing: .5 points.

      • Sharing only for troubleshooting/support: .25 points

      • Sharing for other reasons like advertising or end user targeting: 0 points

  • Security Controls: We prioritized these because they form the foundational defenses against unauthorized access, drawing from best practices that have prevented incidents in cloud services.

    • MFA support: .2 points.

    • Role-based access: .2 points.

    • Session monitoring: .2 points.

    • TLS 1.3: .2 points.

    • SSO support: .2 points.

  • Security reports and incident history: Rewards transparency and deducts for recent issues. This was included to emphasize accountability, as a history of breaches or proactive transparency often indicates how seriously a provider takes security.

    • Published safety framework and bug bounty: 1 point.

      • To get full points the company needs to have both of the following: 

        • A publicly accessible page (e.g., security, trust, or safety) that includes a comprehensive whitepaper, framework overview, OR detailed security documentation that covers:

          • Encryption in transit and at rest

          • Authentication and authorization mechanisms

          • Network or infrastructure security design

        • Incident Response Transparency – Published vulnerability disclosure or bug bounty policy OR a documented incident response process and security advisory archive.

      • Example: Google has a bug bounty program, a whitepaper providing an overview of their security posture, as well as a transparency report

    • No commitments or weak security framework with the lack of any of the above criteria. If the company only has one of the criteria above but lacks the other they will also receive no credit: 0 points.

      • Example: Lovable who has a security page but seems to lack many other parts of the criteria: https://lovable.dev/security

    • If there has been a material breach in the last two years. If the company has experienced a material cybersecurity incident that resulted in the unauthorized disclosure of customer data to external parties (e.g., data posted, sold, or otherwise made accessible outside the organization). Incident must be publicly acknowledged by the company through a trust center update, press release, incident notification page, or an official regulatory filing: Full deduction to 0.

      • Example: 23andMe suffered credential stuffing attack in 2023 that resulted in the exposure of user data.

  • Financial Stability: Gauges long-term viability of the company behind the application. We added this because a company’s financial health affects its ability to invest in ongoing security and support, and reduces the risk of sudden disruptions, corner-cutting, bankruptcy or sudden sale of user data to unknown third parties.

    • Public company or private with >$300M raised: .8 points.

    • Private with >$100M raised: .5 points.

    • Private with <$100M raised: .2 point.

    • Recent bankruptcy/distress (e.g. recent bankruptcy filings, major layoffs tied to funding shortfalls, failure to meet debt obligations): 0 points.

Gen-AI Confidence Score (5.0 Points Total)

This Score zooms in on AI-specific risks, like data usage in training and input vulnerabilities.

  • Regulatory Compliance, ISO 42001: ISO 42001 is a new certification for AI management systems. We chose this emerging standard because it specifically addresses AI governance, filling a gap in traditional certifications and signaling forward-thinking risk management.

    • ISO 42001 Compliant: 1 point.

    • Not ISO 42001 Compliant: 0 points.

  • Deployment Security Model: Stronger access controls get higher points. Authentication not only controls access but also enables monitoring and logging. This makes it easier to detect misuse and investigate incidents. Public, unauthenticated access is a red flag for shadow IT risk.

    • Authenticated web portal or key-protected API with rate limiting: 1 point.

    • Unprotected public access: 0 points.

  • Model Card:  A model card is a concise document that provides essential information about an AI model, similar to a nutrition label for a food product. It is crucial for AI safety and security because it offers transparency into a model’s design, training data, limitations, and potential biases, enabling developers and users to understand its risks and use it responsibly. Some leading AI providers have committed to providing model cards as public documentation of safety evaluations. We included this in our rubric to encourage the industry to broadly adopt model cards as a best practice. As the practice of model cards is further developed and standardized across the industry, we hope to incorporate more fine-grained details from model cards into our own risk scores. But for now, we only include the existence (or lack thereof) of a model card in our score.

    • Has its own model card: 1 point.

    • Uses a model with a model card: .5 points.

    • None: 0 points.

  • Training on user prompts: This is one of the most important components of our score.  Models that train on user prompts are very risky because users might share sensitive corporate information in user prompts. We weighted this heavily because control over training data is central to preventing unintended data exposure, a core risk in generative AI that can lead to major incidents.

    • Explicit opt-in is required for training on user prompts: 2 points.

    • Opt-out of training on user prompts is explicitly available to users: 1 point.

    • No way to opt out of training on user prompts: 0 points.

Here’s an example of these Scores applied to a few popular AI providers.  As expected, enterprise tiers typically earn higher Confidence Scores than consumer tiers of the same AI provider.

Company Application Score Gen AI Score
Gemini Free 3.8 4.0
Gemini Pro 3.8 5.0
Gemini Ultra 4.1 5.0
Gemini Business 4.7 5.0
Gemini Enterprise 4.7 5.0
OpenAI Free 3.3 1.0
OpenAI Plus 3.3 3.0
OpenAI Pro 3.3 3.0
OpenAI Team 4.3 3.0
OpenAI Enterprise 4.3 4.0
Anthropic Free 3.9 5.0
Anthropic Pro 3.9 5.0
Anthropic Max 3.9 5.0
Anthropic Team 4.9 5.0
Anthropic Enterprise 4.9 5.0

Note: Confidence scores are provided “as is” for informational purposes only and should not be considered a substitute for independent analysis or decision-making. All actions taken based on the scores are the sole responsibility of the user.

We’re just getting started…

We’re actively refining our scoring methodology. To that end, we’re collaborating with a diverse group of experts in the AI ecosystem (including researchers, legal professionals, SOC teams, and more) to fine-tune our scores, optimize for transparency, accountability and extensibility. If you have insights, suggestions, or want to get involved testing new functionality, we’d love for you to express interest in our user research program. We’d very much welcome your feedback on this scoring rubric. 

Today, we’re just releasing our scoring rubric in order to solicit feedback from the community. But soon, you’ll start seeing these Cloudflare Application Confidence Scores integrated into the Application Library in our SASE platform. Customers can simply click or hover over any score to reveal a detailed breakdown of the rubric and underlying components of the score. Again, if you see any issues with our scoring, please submit your feedback to app-confidence-scores@cloudflare.com, and our team will review it and make adjustments if appropriate. 

Looking even further ahead, we plan to enable integration of these scores directly into Cloudflare Gateway and Access, allowing our customers to write policies that block or redirect traffic, apply data loss prevention (DLP) or remote browser isolation (RBI) or otherwise control access to sites based directly on their Cloudflare Application Confidence Score. 

This is just the beginning. By prioritizing transparency in our approach, we’re not only bridging a critical gap in SASE capabilities but also driving the industry toward stronger AI safety practices. Let us know what you think!

If you’re ready to manage risk more effectively with these Confidence Scores, reach out to Cloudflare experts for a conversation.

Tagged:

Leave a Reply

Your email address will not be published. Required fields are marked *