E017NDYFGQL-U4W9C40NL-51be4d627a6f-512

Ari is a seasoned Product-Led Growth (PLG) leader with a passion for architecting flawless customer journeys. As the VP of Developer-Led Growth at Auth0, he leads the Developer Marketing, Developer Relations, Activation, and Conversion teams, among others. His career journey includes a key role on the Microsoft Developer Evangelism team and a foundational period as a Software Engineer at Southworks and other firms, where he delivered mission-critical projects for Fortune 100 clients.

Vice President, DLG

Ari Schapiro

Explore Auth0's August 2025 product updates, featuring enhanced flexibility with passwordless connection switching and improved security through DPoP and TLS fingerprints.

August 2025 in Auth0: Non-Unique Emails, DPoP, and TLS Fingerprints

andres

I’ve been at Auth0 since 2017. I’m currently working as a Product Manager for the Auth0 FGA and OpenFGA products. Previously, I worked in the teams that owned the Login and MFA flows. <br>
I spent my entire 20+ year career building tools for developers, wearing different hats. When I'm not doing that, I enjoy spending time with my family, singing in a choir, cooking, or trying new kinds of local cheese.

Product Manager

Andrés Aguiar

Now, you can deploy Auth0 FGA in several AWS regions with high availability and requests per second.

We are thrilled to announce that [Okta Fine Grained Authorization](https://fga.dev/) (FGA) is now available as an AWS private cloud deployment option.

Authorization is where Authentication was a decade ago. We aim to simplify Authorization for developers, just as we did with Authentication. In early March, we made significant progress toward this goal by [launching Okta Fine Grained Authorization](https://www.okta.com/blog/2024/03/setting-the-bar-for-authz-okta-fine-grained-authorization-is-now-generally-available/). Auth0 FGA is an authorization service that enables developers to implement centralized, flexible, scalable, and easy-to-use authorization.

Auth0 FGA has two deployment options: public cloud and private cloud. The public cloud option is a multi-tenant SaaS service available in three geographies: the United States, Europe, and Australia, offering a highly available multi-region deployment. The private cloud option, on the other hand, is tailored for enterprises seeking dedicated resources. Auth0 FGA Private Cloud [leverages the same architecture principles](https://auth0.com/blog/securing-auth0s-new-private-cloud-platform/) that have been battle-tested with Auth0 for over two years.

**Benefits of Running Auth0 FGA in Private Cloud**

* **Higher RPS**: Private cloud instances are optimized for high request-per-second (RPS) performance, scaling up to five times the average RPS based on your application’s needs. 
* **Data Residency and Compliance**: Deploy your Private Cloud environment in any AWS region to help meet specific data residency and compliance requirements. Initial regions include the US, Germany, Ireland, the UK, France, Japan, India, Singapore, Australia, and Brazil.
* **Reduced Latency**: Choose the AWS region closest to your application servers, which will significantly reduce latency for faster access control checks.
* **Multi-Geography Deployments:** Businesses can replicate the same authorization data across multiple regions worldwide, allowing them to maintain low-latency authorization services even for globally distributed applications. For example, a company can have the same data in the US, EU, and Australia, have their authorization data replicated across all regions, and have their applications routed to the closest region.
* **Automated, Hardened Release:** Benefit from automated weekly releases that are previously validated in Okta’s public cloud deployments.
* **Centralized Management:** Manage both private and public cloud instances seamlessly from the Auth0 FGA dashboard.

The availability of Auth0 FGA in Private Cloud on AWS marks another significant milestone in our vision of running Auth0 FGA across all major cloud providers and regions. Stay tuned for further updates, and [Contact Us](https://auth0.com/contact-us?place=header&type=button&text=contact%20sales) to learn more.

Auth0 Fine Grained Authorization is now Available in Private Cloud on AWS

bernardo

Bernardo is an Engineering Manager at Auth0 focusing on improving its Performance and Resilience posture. He is a technical leader who is passionate about building quality software that is scalable and resilient to failure, so we can meet the rising expectations of our customers.

Engineering Manager (Performance and Resilience Testing)

Bernardo Guerreiro

How Auth0 uses Performance Engineering to balance infrastructure cost and performance.

Auth0 is providing a world-class service empowering the critical path of millions of logins around the world, so it’s important that we ensure our services are performing well while keeping our infrastructure efficient. The Performance and Resilience Testing team was established in late 2021, as Auth0 made a lot of improvements in its resilience posture. Together with Platform and Product teams, we work toward upholding that performance, reliability, and resilience posture that is such an important pillar of Auth0. 

During 2022, the [development of Auth0’s new Kubernetes-based platform ramped up](https://auth0.com/blog/the-architect-s-view-of-auth0-s-new-private-cloud-platform/), which allows for a more repeatable and automated deployment of public and private cloud environments. Since then, our team began a natural partnership with the Platform teams and started working on defining the Private Cloud offerings for our customers. 

One of our key responsibilities has been to define performance tiers for private cloud customers based on different RPS (requests per second) thresholds - 100 RPS, 500 RPS, and 1500 RPS (with more to come). This initiative, called Tier Right-Sizing, is essential to ensuring Auth0 is able to provide our customers with offerings whose infrastructure costs are efficient for their needs, allowing our sales teams to more accurately sell to customers based on their capacity and performance needs. This blog post outlines the process we followed to define these tiers.

## Our Performance Testing Tooling

As you’ll find out as you read the blog post, in order to define these tiers, our team (and many other individuals at Auth0) ran countless performance tests of varying durations. One of the key things our team works on is the tooling and infrastructure for Performance Testing for the whole Engineering organization.

For the last three years, we’ve used [Artillery](https://www.artillery.io/) to run Performance Tests at Auth0, and it has allowed us to scale easily as usage of our framework organically grew within the company. This was important for a few reasons:

- Firstly, because in the last year and a half, the adoption of our framework went from a few people running Performance Tests at Auth0 to almost every engineering team;
- Secondly, because the scope of our tests also grew. For example, in December 2022, as part of the Tier Sizing for a larger customer, we ran 6,000+ RPS tests, which Artillery was able to easily handle thanks to leveraging AWS Fargate’s horizontal scaling.

We’ve also written our own CLI (a wrapper around the Artillery CLI called Airstrike), which has allowed us to standardize Artillery usage across the organization and cater to Auth0-specific use cases. Developers can now run authentication flows they’re familiar with by name, point it at an environment of their choice and simply type a target RPS, all without having to use an Artillery config. This abstracts even further the work of running a Load Test, which is especially helpful for teams that might not be product experts but still need to understand the performance impact of their work, such as platform teams. For example, anyone can run the `authorizationCode` scenario in a specific internal environment at any given RPS, for the duration they need, and with their desired load profile. All they need to do is run this fully configurable command:

```
airstrike run authorizationCodeNew --env tus1 --targetRPS 100 --targetDuration 3600 --loadProfile spike
```

Additionally, the latest Artillery offering, Artillery Dashboard, was also an important part of this Tier Right Sizing effort. Since Airstrike programmatically adds the relevant tags to tests by default, we were able to leverage the dashboard as a centralized location to view all our Performance Test reports for this effort, by filtering by tags to get the relevant test executions. Any load test report is available there for easy historical reference with all its results, logs, and graphs, and we found it easy to consume for stakeholders and developers alike.


![Auth0 Artillery Dashboard](https://images.ctfassets.net/23aumh6u8s0i/1ZpGlYTzV4cNJShIKMk9eq/f1d22eb93f77fe897cd2d75f91cda7bd/auth0-artillery-dashboard.png)


## Tier Right Sizing

My team followed a thorough four-step process which I’ll outline in this section. This process was followed for each of the tiers we’ve defined (5 private cloud tiers total). For all phases, a subset of core authentication scenarios was chosen by our Product team, which represents key product patterns our customers use. 

### Phase 1 - Exploratory Investigations

During this phase, the team worked together with Platform experts on experimenting with several infrastructure sizings and configurations, including:

- Number of resources needed for main services;
- Database Instance Sizes;
- Burstable Instance types;
- Kubernetes Nodes Instance Sizes;
- Kubernetes resource allocation;
- etc.

Since there were a lot of permutations to try, each experiment was only a 10-15 minute load test. This phase can almost be compared to “Exploratory Testing”. Apply an infrastructure change, run a load test, analyze results, rinse and repeat. It gave my team a good sense of the “gotchas” of the new platform, and served to develop an instinct for what would work and what wouldn’t. 

Once the team had arrived at what they believed to be an optimal configuration, it was time to prove that.

### Phase 2 - 1-Hour Baseline Load

This phase was the main acceptance criteria given to us by Product. Once we reached a stable configuration in Phase 1, the intent was to prove that it could hold the Performance SLA for 1 Hour at the defined Max RPS for the tier, for each of the core scenarios (ran sequentially).

If it failed the SLA, it was then necessary to go back to Phase 1 with more experiments until the team narrowed down a working configuration again.

### Phase 3 - 6-Hour Synthetic Load 

The previous phase (1-Hour Baseline Load) gave us confidence in the performance of each individual scenario at peak load, but that’s not always a typical traffic pattern. Realizing this, we decided to borrow from a concept we have in our Staging environments we call “Synthetic Load”, which emulates production-like traffic by running multiple weighted scenarios at the same time. 

For this effort, we added to our existing core scenarios a few more scenarios that customers are likely to have in their environments (Management API, Extensibility, etc.), creating a 6-hour Synthetic Load test, with a total RPS of the Max RPS for the Tier. This gave us confidence that the Tier would perform under more normal production-like conditions, when a lot more services are being hit.

### Phase 4 - 72-Hour Soak Test

The final piece of due diligence was to ensure that we didn’t see a performance impact from any memory leaks, cron jobs, database background operations, etc. For that reason, after consulting with a few experts in the system, we decided to be safe and run long soak tests. These gave us confidence that we wouldn’t see a problem manifest itself over time.

This ended up being a crucial part of the process and actually revealed a few important performance improvements that had to be done in our platform.

### Rollout

After the new tier definitions were live, all new customers used the new tier definitions. However, in a few cases, we had customers already migrated to our new Private Cloud platform, but not yet in our new optimized tiers. In order to guarantee a seamless transition to the newly defined tiers, a dedicated Platform team worked on a release plan. 

All infrastructure changes were gradually rolled out to customers through our [automated release pipeline](https://auth0.com/blog/how-we-do-releases-in-auth0-s-new-private-platform/). Changes were broken down into small batches, and thanks to our automated testing and the soak time built into our release pipeline, Platform teams were able to monitor each change going in to ensure no issues had been introduced, therefore minimizing any risk during this transition. 

## Conclusion

Right Sizing infrastructure is a game of balancing infrastructure cost and performance. There is no right final answer, and while this effort has taken us quite a bit of time, it will likely never be fully done - we will continue to optimize and improve on this over time. In a few cases, we have also had to do this exercise for specific customers with more unique and large-scale use cases (adapting which scenarios were used), so we will be working on generalizing some of these customer patterns.

I want to leave a final note that we were very excited to support this effort on the road to providing a modern platform for our customers. On a personal note, I am incredibly proud of my team for their contributions to this project, from the work in the tooling that eventually supported this effort to the people who directly worked on the project. It’s exciting to take our new platform to higher limits and take on bigger and bigger customers!

Optimizing Cost and Performance of Auth0's Private Cloud

MatthewHughes

I'm a Liverpool-based technology writer with a passion for usability and security. When I'm not churning out copy, you'll find me walking my two dogs: Luna and Apollo

Business Content Writer

Matthew Hughes

When it comes to SaaS applications, performance matters. Private Cloud SaaS helps you scale the performance of your growing SasS applications.

Performance matters. You know this, of course. Every second of additional latency translates into [lower conversions and tougher customer retention](https://businessmondays.co.uk/tackling-cart-abandonment-with-improved-login-experiences/). It's why you agonize over every individual detail, why your cloud infrastructure vibrates with the well-oiled efficiency of a Formula One engine.

As your business scales, staying fast becomes much harder. Essential SaaS (software-as-a-service) components that worked well when you were a smaller company now struggle to contend with your higher traffic volumes. 

Fortunately, there's an alternative. Private cloud SaaS offers the flexibility of a self-hosted application with none of the downsides. And, for performance-sensitive applications, it offers the boost necessary to deliver an incredible, slick user experience. 


## The Need for Speed 

Virtually every company and public sector organization uses SaaS in some form. Why? It's often the most practical and affordable option. Hiring and training someone to operate and provision a system takes time, money, and resources. With SaaS, the vendor handles long-term maintenance and upgrades. As most use a subscription model, there are no major upfront costs.

Most SaaS applications, by design, are multi-tenant environments. Multiple customers — whether individuals or organizations — share the same computational resources. This model is cheaper but has several trade-offs. Customers have little choice in where the application is deployed. They lose control. They can't make the subtle tweaks that ultimately result in faster, more responsive applications. 

Private cloud SaaS applications are different. They combine the benefits of a SaaS app (low costs, low maintenance, faster time-to-market) with the flexibility and autonomy of a self-hosted application. Customers can choose where, how, and with whom to deploy their application. They are the sole tenants, and they set the rules.

This control helps organizations comply with the ever-increasing number of data residency laws worldwide. But it can also deliver meaningful performance benefits, resulting in more responsive applications and better user experience outcomes.


## Reducing Latency One Millisecond at a Time

Have you ever wondered why Google only shows you ten results for each page? It seems counterintuitive, doesn't it? Google is a search engine. Its job is to show you relevant results for your queries. More is better, surely?

Not necessarily. In 2006, Marissa Mayer, then the company's Vice President of Search, [shared the results of an internal experiment](http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html) at the Web 2.0 Conference. Google asked a select group of users if they wanted to see more results with their queries. Most said yes. 

In response, Google provided random users (not just those in the focus group) with 30 results per page — triple the normal amount. 

What happened next? Frustration. Load times more than doubled for those in the test cohort, soaring from 0.4 seconds to 0.9 seconds. That 0.5-second delay had a measurable effect on user activity and satisfaction. Traffic and revenue from the test group dropped 20%. As Mayer said at the time: "Users really respond to speed." 

How does this relate to private cloud SaaS? Well, there are limits to how far you can optimize an application for speed with efficient coding and generous resourcing alone. 

If your primary application is located in Amsterdam but relies upon a component hosted in Los Angeles, you'll find a measurable delay as packets make their long journey through a network of submarine cables. You add 135ms to your load times – give or take 10ms.

You can't speed this process up. You are constrained by the speed of light. That's fast, but when you are trying to deliver an incredible experience to your customers, it feels like an eternity. 

You have only one solution: bring that component physically closer to your core application. 

The private cloud SaaS model lets you deploy resources in the same continent, country, and even data center. By eliminating performance bottlenecks in your product, your customers will benefit from a faster, more responsive experience. 


## How Private Cloud SaaS Can Help You Scale 

As your business grows, you may feel constrained by the limitations of multi-tenant, public cloud SaaS applications. Many vendors limit the number of requests that can be made per second (RPS) in order to preserve the quality of the service. While most SaaS companies are prepared to oblige larger customers with bespoke plans, this isn't always a given. 

And you cannot always predict when demand will spike. Take lab-testing and diagnostic services provider (and Auth0 customer) [BioReference Labs](https://auth0.com/case-studies/bioreference), for example. BioReference had begun transitioning to a patient-centric model, where customers could access their lab results without going through an intermediary — like their physician. As a result, it invested in Auth0's multi-tenant customer identity and access management (CIAM) platform. 

At the start of the COVID-19 pandemic, demand for its services jumped 25 times. Its monthly active users increased from 8,000 to 148,000. In response, the company migrated its identity infrastructure to a private instance of Auth0's Identity Platform, giving it essential breathing room to meet existing demand and add more customers in the future.

By opting for a private cloud SaaS application, you can provision your environments with the resources necessary to meet existing demand. And you have the flexibility to change your environment as circumstances dictate. 


## Becoming Resilient with Private Cloud SaaS 

Nobody likes downtime. Not your customers, who rely on your product. And certainly not you. 

Performance issues can devastate your bottom line. As load times and instability increase, conversion rates drop. Unfortunately, these problems are often beyond your control. In a multi-tenant SaaS environment, seasonal demand spikes — like those during the holiday shopping season — can result in longer wait times and instability for your customers. 

The private cloud SaaS model can insulate your business from any disruption in the public cloud SaaS application. With control over the environment, you can structure your SaaS infrastructure to be highly resilient, with custom disaster recovery protocols and redundant failover systems. 

You can establish custom backup and restore procedures. Mitigating against data center-level disruption becomes easy. Should the underlying cloud infrastructure fail, you can automatically switch operations to an unaffected environment, keeping downtime to a minimum. And with bespoke update rules, you can deploy patches and new features at a time when your customers are least active. 


## Nimble SaaS, Happy Customers

For most companies, it makes sense to use a multi-tenant public cloud SaaS product. But as your needs increase, you may find it sensible to switch to a dedicated environment. 

Private Cloud SaaS offers the best of both worlds. It requires little maintenance but offers the flexibility of a self-hosted application. You become able to make the choices necessary to deliver a lightning-fast, highly resilient product.

The Auth0 Identity Platform is available in both multi-tenant and private cloud versions. When you opt for a private environment, you get all the benefits of our industry-leading CIAM platform, but with the flexibility to choose how and where it's deployed. To find out how it can benefit your company, reach out to our team.

How Private Cloud SaaS Can Boost Application Performance

Dan McCorriston

Senior Product Marketing Manager

Now, all Auth0 customers - including those on our private cloud platform - have access to all the offerings on the Auth0 Marketplace.

Auth0 is excited to announce a newly expanded offering for [our new private cloud platform](https://auth0.com/blog/a-technical-primer-of-auth0-s-new-private-cloud-platform/) customers - Actions Integrations are now available from the Auth0 Marketplace. What that means is all Auth0 customers that utilize our new private cloud platform have access to all the offerings on the Auth0 Marketplace - SSO, Social Connections, Log Streaming Integrations, and now, Actions Integrations.

Actions Integrations will start to appear in the Development/Staging space now and will be ready in the Production environment in the coming weeks dependent on various release cadences.

### What are Actions Integrations and the Auth0 Marketplace, and why should I care?

At Auth0, we realize that many customers have unique needs, and as a result, they want the ability to customize their identity product. In order to serve these additional customer needs, Auth0 built a variety of tools to extend the functionality of identity. This concept is called **extensibility.**

Auth0 Actions and Actions Integrations is the next generation of **extensibility**. Actions is the low-code offering, and Actions Integrations allows customers to use pre-written no-code, drag and drop options to add other functionality such as ID proofing and consent management. Auth0 provides this additional functionality through integrations with **our Ecosystem partners** offered in the Marketplace.

The Auth0 Marketplace and Auth0 Actions are the two complementary ways we extend Auth0. Customers and their end users need the ability to resolve their unique identity requirements — which may include payments, subscription management, signing terms of services, etc. — to complete a successful user access journey into their application or service.

### What is Private Cloud?

Auth0’s [Private Cloud offering](https://auth0.com/cloud) provides the flexibility and agility offered by cloud deployments but also allows a customer to maintain the control and security they need to protect their customers and their businesses. Many customers that have high security and regulatory concerns - those in the financial services industry, for example - may require certain standards of uptimes, updates, failovers, and support for their deployments. The Auth0 Private Cloud deployment models come in both Standard and Managed levels, depending on the customer’s requirements.

Auth0 [recently invested in retooling the Private Cloud platform](https://auth0.com/blog/a-technical-primer-of-auth0-s-new-private-cloud-platform/) to support more automation, adopt current technologies and help scale the Private Cloud to thousands of environments.

### Benefit of Private Cloud and Actions Integrations

Private Cloud has a strong appeal for customers who have high volume or who are from certain verticals. For example, verticals like banking, where shared databases are a concern. Or customers who have data residency requirements or even those who prefer to deploy in a specific region close to their customers. Private Cloud customers do not have to be concerned with sharing space, public cloud request limits, and multiple customer’s data being present in the same deployment. 

Many of the same customers that have a need for Private Cloud also have extensive use cases for extensibility and the no-code integrations available in the Auth0 Marketplace. Easy-to-use integrations such as Identity Proofing and Customer Data Management. With Actions Integrations, developers no longer need to build or maintain custom code or make changes when we alter programming models or release new versions of Node. These Actions Integrations are built and maintained by our Ecosystem partners and are available right in the Auth0 Marketplace. 

A couple of examples of Actions Integrations available in the Auth0 Marketplace:

* [**Deduce Intelligent MFA**](https://marketplace.auth0.com/integrations/deduce-intelligent-mfa): Deduce prevents consumer-facing Account Takeover (ATO) and registration fraud via the constantly growing Deduce Identity Network, with over 400M profiles and in excess of 1.2B daily activities. Deduce risk signals significantly reduces false positive MFA challenges.
* [**Tealium User Login**](https://marketplace.auth0.com/integrations/tealium): Tealium connects customer data - web, mobile, offline, Internet of Things (IoT) - so brands can connect with their customers. Tealium empowers companies to create a unified, real-time customer data infrastructure and makes customer data more valuable, actionable, and secure.

### Is this an increased opportunity for Ecosystem Partners?

Now that Private Cloud customers have access to Actions Integrations in the Auth0 Marketplace, our Ecosystem Partners have a new and expanded pool of potential customers to reach through building and publishing Actions Integrations into the Marketplace. There’s never been a better time to become an Ecosystem Partner and market your Actions Integrations through the Auth0 Marketplace.

Make sure to visit the [Auth0 Marketplace](https://marketplace.auth0.com/) to see if there is an integration that optimizes your customer’s identity experience. To find out more about becoming a [Private Cloud](https://auth0.com/docs/deploy-monitor/deploy-private-cloud) customer, please [send an inquiry](https://auth0.com/get-started?place=documentation%20post&type=link&text=contact%20auth0%20sales) and someone from Auth0 will contact you about whether it’s a good fit for your business.

Actions Integrations Now Available in Private Cloud

Uploaded from Data Pipelines on Auth0’s New Private Cloud Platform

Andrew is a Staff Engineer at Auth0 focusing on bytes that move for the new Auth0 platform. He is a startup co-founder with over a decade of experience delivering developer focused products at scale. When not at the computer, you can find him hiking up mountains, watching improv comedy, or spending quality time with his wife and son.

Staff Engineer

Andrew Hawker

“Data goes in; data goes out. You can’t explain that.”

**TL;DR:** Auth0 has replatformed our private cloud offering onto Kubernetes. As part of this new architecture, we’ve had the opportunity to apply modern practices to the data infrastructure that powers near real-time features and provides the necessary knobs for data governance.

_Hello friends! Before we get started, I recommend that you check out “[A Technical Primer of Auth0’s New Private Cloud Platform](https://auth0.com/blog/a-technical-primer-of-auth0-s-new-private-cloud-platform/)”, “[Architecture of the New Platform](https://auth0.com/blog/the-architect-s-view-of-auth0-s-new-private-cloud-platform/)”, and “[Security of the New Platform](https://auth0.com/blog/securing-auth0s-new-private-cloud-platform/)”. These wonderful posts cover foundational aspects of the platform that I will build upon here._

## Introduction

“Data is the new oil,” quipped Clive Humby in 2006, and the industry has been chanting “Drill, baby, drill” ever since. Just like oil, data needs to be refined/processed/transformed to unlock its full value. As the saying goes, “Necessity is the mother of invention,” so the industry has birthed an entire specialization around Data (Science, Engineering, Platform, Warehouse) to extract that value. The analogy remains apt because, just like oil, data is now being respected as a potentially hazardous material. With GDPR, CCPA, and their future brethren, we’ve come to understand the importance of proper packaging and safe keeping of our data assets. No more ziplock bags of data goop in the trunk of my Dodge Neon. Neatly stacked steel crates with big padlocks under 24/7 surveillance that can be crypto shredded at a moment's notice is the future of data management. With this New Platform initiative at Auth0, we’ve had the opportunity to apply some modern practices to our data infrastructure and take another step towards that dystopian data future.

![Introduction](https://images.ctfassets.net/23aumh6u8s0i/5GGILFTY6KNSabp6Ixn2dA/1b6a7923fa10b614f320c94cf436c2b9/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

Taking a page from Leo’s 10k foot view in “[Architecture of the New Platform](https://auth0.com/blog/the-architect-s-view-of-auth0-s-new-private-cloud-platform/)”, we’ll be touching on the highlighted sections of the platform in this post. We’ll start with some introductory context, discuss our guiding principles, dive into the tools we chose, and then exercise them with a few sample scenarios commonly seen at Auth0.

Put down that bag of data goop, and let’s get started!

## What is a Data Pipeline?

_“In a world where SaaS and microservices have taken over...”_ — Movie Voiceover Guy

*Looks around*

“Hey, he’s talking about us!”

Software is on a hockey stick growth curve of complexity. We’ve got an ever-increasing number of disparate systems hoarding data and even higher demand on us to unlock value from it.

1. “Show me the region where credential stuffing attacks are originating from” — A customer dashboard
2. “Let me run custom code in an Auth0 Action immediately after a user is deleted” — A feature request
3. “Show me failed login counts from the past 60 days for all tenants on the Performance tier that have been a paying customer for more than one year” — An upsell opportunity

Before we can begin to actually answer any of those questions, we need the connective tissue to ship data from Point A to Point B enriched with E and transformed by T. That’s the role of the Data Pipeline.

That’s not to say that Data Pipelines just carry simple domain/product events. Application logs, exception stack traces, Stripe purchase orders, Salesforce billing updates, etc., are all events that can be emitted and flow through a pipeline.

_“The Data Pipeline is a series of tubes.”_ — Michael Scott

Eh, close enough.

## Our Guiding Principles

Before jumping into implementation, it’s important to understand that _our_ requirements are not necessarily _your_ requirements. The decisions we’ve made might not be the correct decisions for you! When evaluating solutions and discussing tradeoffs, these are some of the key points that helped guide us.

### Event Durability

Event durability is a supremely important requirement for us; missing and out-of-order events are a non-starter.

If we imagine emitting a simple event, e.g., “UserCreated,” there are a few approaches:

1. Integration/domain event from application code
2. [Outbox Pattern](https://microservices.io/patterns/data/transactional-outbox.html)
3. [Change Data Capture (CDC)](https://en.wikipedia.org/wiki/Change_data_capture)

Discussing tradeoffs between these is a blog post in itself, so I’ll just simply say that they’re complementary and fit different needs, with the main tradeoff being durability vs. coupling.

Events emitted by application code can get lost with failures during dual writes. Outbox Pattern struggles if you don’t have strong transactional guarantees with your database of choice and doubles your write load. Change Data Capture (CDC) is the most durable of the three but “ships” the database table schema in each event, promoting unnecessarily tight coupling and having limited access to only the modified row.

### Data Governance (O Data, Where art thou?)

Data Pipelines come in different shapes and sizes.

* Are messages serialized as Avro, JSON, or Protobuf? XML? _Crickets._
* Are they encrypted or plain text?
* What is the expected throughput? Sustained or burst?
* Streaming or batch processing?
* Do producers retry or drop events during downtime?
* What enrichments/transforms do we need to apply?

Before discussing these technical details of a pipeline, we must first consider what is **_in_** the pipeline.

1. Do messages within the pipeline contain Customer Data?
2. Do messages within the pipeline need to leave the region where they originated?

If you answered “Yes” to one or more of the previous questions, calmly go outside and light your computer on fire. Joking aside, by asking and answering those questions, you can begin to understand the security/compliance posture you must take. This is likely to lead you down the road of even more questions.

* What region is data being shipped to relative to where it’s generated?
* Are we sending aggregated data or individual records?
* How is this data encrypted in transit and at rest?
* What Customer Data (down to each individual field) are we processing?
* Are there treatments that are sufficient for each field to be compliant while still fulfilling the business need? Masking, partial redaction, hashing, encryption, etc.

Remember that data is a valuable resource but can also be a hazardous material and must be handled properly. If you discover cases of processing Customer Data and shipping it across geographic boundaries, find yourself a buddy in the compliance department to help sort it out.

### Inertia

Event streaming technology is an active area of development, with many new companies and products entering the space. This is great because we can stand on the shoulders of giants, but if you’re not careful, it can leave you prone to painful churn, deprecation, incompatibility, or entire products going bust.

While we don’t want to fully jump into the [Boring Technology Club](https://boringtechnology.club/) pool, we _do_ want to build on robust tools that will survive a hype cycle. If a project doesn’t have enough inertia and community buy-in to survive the departure of the original stewards, it might be best to consider an alternative.

### Composition

The only constant in life is changing, especially technological and business requirements. We need to consider how our technology choices can be composed, merged, bent, broken, and smeared together to get the job done. If we choose a solution too rigid, it’ll be re-written often. A solution too flexible is unlikely to be fully understood. We should try and use tools that provide a healthy balance.

## It's Dangerous to Go Alone! Take This

We’re not the only company building on data pipeline technologies, thankfully. We can stand on the shoulders of those before us. Using our guiding principles outlined above, here’s the alphabet soup of open source technologies we chose to plug together to make our data go.

#### Kafka

The original Auth0 platform was built to run only on AWS and heavily leveraged Kinesis, SQS, and RabbitMQ. When evaluating cloud-agnostic solutions for the new platform, it was difficult to say no to [Kafka](https://kafka.apache.org/). With most cloud providers offering a hosted solution (or API-compatible layer), it’s becoming the lingua franca for durable event backends. With relatively sane approaches to implementing encryption at rest or in-flight, it meets most of our needs right out of the gate.

#### Kafka Connect

[Kafka Connect](https://docs.confluent.io/platform/current/connect/index.html) has a solid ecosystem of integrations that is relatively plug-and-play (although schema vs. schema-less can be a pain). The [AWS S3](https://docs.confluent.io/kafka-connect-s3-sink/current/overview.html) sink connector was available to us immediately, and an open-source [Opensearch](https://github.com/aiven/opensearch-connector-for-apache-kafka) connector popped up quickly after the Elasticsearch license fork snafu.

[Single Message Transforms (SMT)](https://docs.confluent.io/platform/current/connect/transforms/overview.html) within a connector are simple mechanisms for stateless transformation/enrichment of all messages. This works well for us to inject static metadata from the new platform like Space, POP, Cloud, current deployment version, etc.

#### Schema Registry

Kafka Connect heavily utilizes the[ Confluent Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html) for message serialization and validation. While some connectors support “schemaless” messages, you’ll find yourself, as we have, swimming upstream if you don’t leverage the registry with these tools. We haven’t fully realized our vision here and plan further investment over the next few quarters.

#### Debezium

As I mentioned before, event durability was an extremely important requirement for us, so we started with Change Data Capture (CDC) powered by [Debezium](https://debezium.io/). With their [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html) and [Postgres](https://debezium.io/documentation/reference/stable/connectors/postgresql.html) source connectors, we’re immediately plugged into the Kafka Connect ecosystem.

#### Data Shield

When it comes to data policy enforcement for streaming messages, there didn’t seem to be many free/open-source solutions we could take off the shelf. For this, we wrote our own in-house solution in [golang](https://go.dev/).

Data Shield handles data policy enforcement with schemas, data mutation, and topic muxing. With it, we can define events flowing through a topic using an IDL like [protobuf](https://developers.google.com/protocol-buffers). The IDL is not tied to the serialization format in the topic, e.g., we can use a protobuf IDL for JSON events.

Let’s walk through a quick example. Let’s say we emit events of Company records that have a field for their EIN.

```
message Company {
  string name = 1;
  string address = 2;
  string ein = 3;
  string url = 4;
}
```

Field enforcement is zero-trust by default, so any fields not defined in the IDL are automatically excluded. This is a critical capability for us since we receive many JSON/BSON-based events with loose schemas where fields can easily be added/removed by the upstream producer. This minimizes the blast radius of an upstream service accidentally or intentionally producing new fields with Customer Data on streams that leave the region.

Using the IDL, we annotate fields with policy details like classification and treatment rules. From our Company example, let's assume the **_ein_** field is classified as Personally Identifiable Information (PII). We could model this as:

```
  string ein = 3 [
    json_name = "ein",
    (auth0.proto.options.fields.v1.classification) = {
      classification: PII,
    },
    (auth0.proto.options.fields.v1.transform) = {
      processor: {
        name: OPENSEARCH,
        transforms: [
          { mask: { length: 4, character: "*", reverse: true } }
        ]
      },
      processor: {
        name: FRAUD_DETECTION,
        transforms: []
      },
      processor: {
        name: EU_DATA_LAKE,
        transforms: [
          { drop: { } }
        ]
      },
      }
    }
  ];
```

Each separate data processor of the stream can apply their own independent rules based on their business function. In this case, our Observability stack in Opensearch has the last four characters masked, our Fraud Detection feature receives the value in plain text, and the Data Lake processor in the EU doesn’t receive it at all.

With this IDL definition and some minor service configuration, we can tell Data Shield to enforce the policy on one or more topics. In the simple case, it will take the incoming “dirty” topics and output “clean” topics, one for each data processor with their specific rules applied.

![IDL](https://images.ctfassets.net/23aumh6u8s0i/3ka0jensdx70UoBfYmdnrd/c85e55476cde1771e397710a63058d61/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

This multiplexing increases our storage costs but allows us to leverage different retention policies and Kafka ACLs so consumers can only access data from their assigned topics. While this is a cost we are (currently) willing to pay for strict data isolation, it is an area for future investment with a potential “decryption and schema enforcement on reading” strategy.

As we’ve iterated on the Data Shield concept, we’ve added a few more features to make our lives easier.

* One-to-many topic to IDL schema mapping for topics with heterogeneous message types. This is very helpful for topics that contain app/service logs.
* Schema format conversions, e.g., read JSON, write Avro
* Extend IDL to support field enrichment with generated runtime values
* Golang API for pluggable actions to modify processing pipeline as events flow through
* Rate-based sampling for sinks that can be overwhelmed

## Scenarios

With our guiding principles leading us to those tools, let’s diagram out a few simplified versions of scenarios we run on the new Auth0 platform.

### Application logs for incident response

In this scenario, we’re using [Fluent Bit](https://fluentbit.io/) to consume service logs from all pods within the Space Kubernetes cluster. They’re shipped to Kafka, where Data Shield consumes the raw log messages, applies its enforcement rules, and produces them back to Kafka. They’re consumed by Opensearch connectors, who ship them across the private link to the POP, where they’re accessible to Operations teams for incident response.

![Application logs for incident response](https://images.ctfassets.net/23aumh6u8s0i/5LkE9v6OyZJpJA85G1lJM7/1e3a523222efba8065f4359b47e456c4/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

### Events for autocomplete search

In this scenario, we’re using Debezium to consume MongoDB change streams to emit events for Auth0 Users into Kafka. Data Shield consumes them, applies its enforcement rules, transforms the structure, and produces them back to Kafka. They’re consumed by another service that processes them and writes them into Postgres running in the same region.

![Events for autocomplete search](https://images.ctfassets.net/23aumh6u8s0i/6WCxg7eRdAuOdGm0MslthE/2886847192f02ae1b1c1cdef18514760/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

### Event counts to a data lake

In this scenario, we’re using Debezium to consume the Postgres logical replication protocol to emit events for Auth0 Connections into Kafka. Data Shield consumes them, applies its enforcement rules, and ships them back to Kafka. They’re consumed by S3 connectors that ship them to an S3 bucket running in a different region.

![Event counts to a data lake](https://images.ctfassets.net/23aumh6u8s0i/4BlUDX70CAZfZ0wl1JOU39/5ce6cf28061d37de489e51bede949ea6/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

### User requested backups 

In this scenario, we're emitting product events for Auth0 Tenants into Kafka. Prior to processing by Data Shield, they’re backed up in-region to object storage. The backups are then periodically rolled up and offered as a data download upon user request in a dashboard.

![User requested backups ](https://images.ctfassets.net/23aumh6u8s0i/6PzHrtwKWMgRzKsp7Hun2V/ee40263dbe880df9975304e07d517945/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

## We’re Just Getting Started

Although these sample scenarios aren’t terribly complex, they really show the power of composability. We can mix and match a few Kafka connectors with Data Shield to meet the needs of our customers while providing the data controls we require. With more data-intensive use cases on the horizon needing stream processing and OLAP databases, this composability means our pipelines can easily evolve to support them. More importantly, even with a team experienced in stream processing and OLAP databases, we’ll still plan an R&D phase to make sure our choices once again align with our guiding principles.

Interested in hearing more about streaming joins, windowing, and OLAP databases? Well, you’ll have to wait for my next blog post then! Muahaha.

I would be remiss if I did not tell you that Auth0 (and specifically my team) are [hiring](https://auth0.com/careers)! Join me in building the next Auth0 data platform solving real problems for real people. Interested in solving problems for fake people instead? Visit the social network nearest you.

Data Pipelines on Auth0’s New Private Cloud Platform

Jon May

Jon May is a Principal Security Engineer at Auth0 where he is responsible for securing the platform and infrastructure of the Auth0 Identity Platform. He has over 10 years experience in cybersecurity and technology and holds a bachelor’s degree in economics from Brown University. Prior to joining Auth0, Jon was a Security Architect at Bridgewater Associates where he was responsible for designing and implementing cloud and enterprise security controls.

Principal Security Engineer, Platform Security

Architecting and building a new platform that is secure by design.

**TL;DR:** Auth0 replatformed our private cloud offering into a multi-cloud Kubernetes architecture. This large architectural change brings with it new threats and new security challenges. We took the opportunity to design and integrate security into the platform from the start, as opposed to "bolting it on" at the end.

In this post, we'll discuss how we approached securely replatforming the Auth0 Identity Platform into a large, multi-cloud architecture. We'll walk through the risks that most concern Auth0's Security team and the threats that could result in those risks becoming real, tangible, negative outcomes. After that, we'll discuss the challenges our team encountered while addressing those threats in a multi-cloud environment alongside the principles we applied to address them. Finally, we'll review the controls Auth0 put in place to mitigate the identified threats.

If you haven't read the previous posts yet, [A Technical Primer of Auth0's New Private Cloud Platform](https://auth0.com/blog/a-technical-primer-of-auth0-s-new-private-cloud-platform/) and [The Architect's View of Auth0's New Private Cloud Platform](https://auth0.com/blog/the-architect-s-view-of-auth0-s-new-private-cloud-platform/), I encourage you to check them out first. They provide foundational knowledge and context which will help you better understand this post.  

## Understanding What Risks Affect Us

At the highest level, Auth0's Security team is responsible for identifying, assessing, and mitigating risks to our business and, in effect, our customers. It's our job to reduce the impact and/or likelihood of undesirable business outcomes caused by security events. Specifically, we're interested in mitigating security and privacy risks, such as those involving:

1. **Product/platform compromises**: bad actors gaining control of a product or platform component
2. **Customer or Auth0 data loss**: unintended deletion or unauthorized access to customer or Auth0 data
3. **Service disruptions and outages**: disrupting the Auth0 service and causing an outage for our customers

There are more risks than the three listed above and more nuance as well, but we'll use them for brevity. These high-level risks help us enumerate our list of threats.

## Identifying How the Platform Will be Attacked

This new Kubernetes-native, multi-cloud architecture is large and complex, which makes it challenging to secure. Our approach is to start by enumerating the areas of the architecture most likely to be targeted and how they can be attacked. This process, called threat modeling, helps us determine where to focus our resources and implement countermeasures (also known as security controls). These "areas to focus on," or threats, are circumstances or events that could cause one of the risks highlighted above to occur.

![Platform](https://images.ctfassets.net/23aumh6u8s0i/3kEf7N3gMeLfxUtIOqqEZP/5a7ee4be2c16df0549cdf2528eb1749e/01_private_cloud.png)

The threat modeling process results in a long list of threats of varying severity. For the purposes of this post, we'll focus on a short list of significant ones:

1. **Employee account takeover**: an attacker compromises or gains access to employee credentials and utilizes them to access Auth0 systems
2. **Malicious insiders**: an internal employee abuses existing authorized access to Auth0 systems 
3. **Service or application compromise**: an attacker gains control of an Auth0 product or platform service, third-party managed service, or a cloud provider service
4. **Software supply chain compromise**: an attacker tampers with or exploits misconfigurations/vulnerabilities in software dependencies used by the Auth0 product or platform

This list of threats informs what secure design patterns and security controls we deploy into the platform's architecture.

## Securing Multi-Cloud Architecture is Hard

Now that we have defined the list of threats we're protecting against (the "what"), we can start determining the design changes, secure design patterns, and security controls that will address those threats (the "how"). But first, before jumping into that, let's take a step back and understand the challenges with securing a platform that utilizes multiple cloud providers. In Auth0's case, we use both Amazon Web Services (AWS) and Microsoft Azure.

### Increased complexity

Using multiple cloud providers increases the architecture's overall complexity. Each cloud provider has similar general capabilities such as computing, networking, storage, and logging. However, each cloud provider implements these capabilities in their own ways. What we end up with are multiple implementations of the same core capability, like virtual machines or blob storage. These nuances between cloud providers introduce complexities that are challenging to secure.

Let's use an example to illustrate these differences. The Auth0 product requires [blob storage](https://auth0.com/blog/azure-storage-introduction/), so let's compare AWS S3 and Azure Storage Accounts and only focus on the different access methods.

![AWS S3 and Azure](https://images.ctfassets.net/23aumh6u8s0i/6DKlibDUAMicDy2oQaUKez/a66838c5086a3e1203d513dc7c3d1eef/table.png)

Another example is AWS Security Groups (SG) vs. Azure Network Security Groups (NSG) and how each handles rule evaluation. AWS SG rules are additive (i.e., allow the superset of all applicable rules), while Azure NSG rules are assigned a priority value and evaluated in that order. These are only two examples of the many differences between AWS and Azure we accounted for in the new platform's architectural design. 

### Larger attack surface

Using multiple clouds means potentially securing multiple, similar services for each cloud provider, e.g., AWS S3 _and_ Azure Storage Accounts. The complexity mentioned above and an increased number of services result in a larger attack surface. Today, for instance, we need to secure two cloud service providers (AWS and Azure), and that means securing two sets of each cloud-specific service required to run Auth0. It also means our Detection and Response team needs to ingest and monitor multiple sets of cloud-specific log sources to properly monitor and investigate the activity. That's a large surface area to secure.

### Cross-cloud connectivity

Communication between services within the same cloud is straightforward. We simply need to use the cloud provider's own mechanisms, and it should "just work." However, there are different points in our new platform's architecture that require some form of communication between AWS and Azure. Network connectivity, for example, needs to be established between Azure Points-of-Presence (PoPs) and Auth0 Central (which runs in AWS). We don't have the ability to use either cloud's PrivateLink service in this scenario (which would be preferred). In another example, Azure Spaces need to upload usage metrics into Auth0 Central (AWS). In single cloud architecture, we could simply use Azure RBAC configured with the proper role assignments. However, that's not possible in this scenario; the Azure Kubernetes Service (AKS) cluster/pod will need to authenticate using AWS S3 authentication methods.

## Addressing Multi-Cloud Challenges

So far, we reviewed the risks Auth0 is most worried about, enumerated a short list of threats to help guide our security decision-making, and discussed some of the challenges Auth0 experienced securing a multi-cloud architecture. Next, let's walk through the principles we applied to determine how we address those threats in the context of those multi-cloud challenges.

### View all security-related decisions through a risk-based and threat-centric lens.

That means all secure design decisions or security controls implemented were directly related to the risk and threats we enumerated during the previous sections. Prioritization is key, and using this lens influences where we need to spend available resources and which countermeasures would be most effective.

### Apply secure design principles and integrate security into the platform early and often. 

The Auth0 Security team is involved throughout the product life cycle, from ideation to design to build to operation and finally to deprecation. We are heavily involved in the design and build phases. This is important because it's easiest to make architectural changes before the technology is built.

### Establish clear, enforceable trust boundaries.

These trust boundaries inform how we implement various countermeasures such as access controls. They also encapsulate areas of risk, making them easier to manage. For example, our development and production environments are completely isolated and separate from one another. Operators cannot use development access to affect any part of the production environment.

### Balance cloud-native and cloud-agnostic capabilities.

This is a tricky one because each has its own benefits and challenges. Cloud-native capabilities (e.g., AWS managed databases) are tightly integrated with the cloud provider, which means you'll get those integrations — like GuardDuty or Microsoft Defender — out-of-the-box. On the other hand, cloud-native services are tightly coupled to their cloud, and deploying them for _both_ AWS and Azure increases the complexity and attack surface (as we discussed above). Compare that to cloud-agnostic capabilities, which can provide a single pattern and enable a simpler architecture with less attack surface. However, integrating with cloud provider services tends to require more work or "glue" to stitch them together. For example, our Auth0 product consists of Kubernetes workloads (cloud agnostic). We use managed Kubernetes services (AKS and EKS), and it's very easy to use the same services across both clouds. That means less complexity (i.e., one change to the Auth0 product will be applied to both clouds) and less attack surface (i.e., one service to secure).

## Selecting the Right Set of Security Capabilities

Finally, let's review the various security capabilities Auth0 implemented into the new platform to address the threats and multi-cloud challenges mentioned above. In addition to our own security capabilities, Auth0 takes advantage of the cloud's shared responsibility model. The shared responsibility model allows us to focus on securing other areas of the platform instead of worrying about the underlying cloud service provider infrastructure.

### Cloud Governance

All Auth0 Spaces, PoPs, Central, and Control Plane technology stacks are isolated into their own AWS Account or Azure Subscription. This provides clean and clear resource isolation, especially for single-tenant Spaces. We used AWS Organizations to create an Organization Unit (OU) hierarchy based on environment and functionality and add each account to a specific OU. We used Azure Management Groups to create a similar hierarchy for Azure Subscriptions. The hierarchies we created enable scalable, granular access controls and policy enforcement; we grant access or apply policies (Azure Policy and AWS Service Control Policies) to specific Management Groups / OUs or Spaces. For AWS, we grant access based on OU conditions and assign SCPs to OUs. For Azure, we assign roles and policies scoped to specific Management Groups. The access and policies are then inherited by the intended accounts or subscriptions.

![Cloud Governance](https://images.ctfassets.net/23aumh6u8s0i/ovvm1JDueXMwiN8ZKQThA/8f1684125fcc76f224b5e9329d5aa0fb/02_private_cloud.png)

### Identity and Access Management

Auth0 was built to secure identities, so we leverage Auth0 + Okta to secure authentication and authorization to our platform tooling. We manage all cloud providers and support application access using a least privileged RBAC model. We also require Multi-factor authentication (MFA) for all operator access to the platform; this includes cloud providers and all supporting applications. Operators are required to use Azure Privileged Identity Management (PIM) - which provides just-in-time access - for any administrative activities. Azure AD Identity Protection identifies and blocks any risky Azure tenant sign-ins.

### Network Security

All single-tenant and multi-tenant Spaces are logically isolated from one another with no interconnectivity between them. This strong isolation drastically reduces an attacker's ability to pivot into other Spaces or PoPs and further their attack. We further restrict inbound connectivity to spaces by using AWS and Azure PrivateLink to connect outbound from Spaces to a PoP. All traffic is encrypted in transit using TLS 1.2+. This includes network traffic between all cloud services, Kubernetes pod-to-pod traffic, and Kubernetes cluster-to-cluster traffic. For some workloads, we utilize mutual TLS (mTLS). On the network edge, we protect against DDoS attacks on our infrastructure, identify and block bots, enforce rate limiting, and block malicious traffic with web application firewall (WAF) rules. Operators must use VPN to access platform infrastructure, adding another layer to our defense-in-depth strategy.

![Network Security](https://images.ctfassets.net/23aumh6u8s0i/2DQcFgwaIkylGFaN479rmz/4a28c57f312f42a25052f84ae788b7b1/03_private_cloud.jpeg)

### Data Protection

All data is encrypted at rest using AES-256 or better, and encryption occurs at multiple layers (e.g., disk, file, database, and application layers). All data in transit is encrypted using TLS 1.2+. Auth0 encrypts platform secrets using HashiCorp Vault and takes full advantage of Azure Key Vault and AWS KMS for encrypting data in cloud services.

![Data Protection](https://images.ctfassets.net/23aumh6u8s0i/1XOTfQ1STNV4y9rTo7FcHf/3fd90c1fd4c3317f63804562e892d5e5/04_private_cloud.png)

### Detection and Response

Auth0 monitors the platform using a combination of cloud services and our own monitoring tools. We take advantage of AWS GuardDuty and Microsoft Defender for Cloud in addition to our own tooling. Auth0 ingests a vast amount of AWS and Azure platform logs for all of the cloud services used by the platform. These logs are ingested into a security data lake and monitored with detections defined as code.

![Detection and Response](https://images.ctfassets.net/23aumh6u8s0i/3aH0JsipzpNkdNubtfYwWN/6123b09fe51ae344fd4537e777081a22/05_private_cloud.png)

### Kubernetes Security

The [4C's of Cloud Native security](https://kubernetes.io/docs/concepts/security/overview/) is a good framework to use when thinking through Kubernetes security. We are, in varying aspects, responsible for security in the clouds we're using, the clusters we're running, the containers running on those clusters, and the code running within the containers. Auth0 uses the managed Kubernetes services Azure Kubernetes Service (AKS) and AWS' Elastic Kubernetes Service (EKS). These offerings from Azure and AWS are security feature-rich. Our worker nodes meet Kubernetes CIS Level 1 Benchmarks. Cluster access is restricted to operators, and least privilege is enforced using specific RBAC roles. We use Kubernetes NetworkPolicy objects to restrict pod network ingress and egress to authorized sources and destinations.

![Kubernetes Security](https://images.ctfassets.net/23aumh6u8s0i/48mIINcAPiqFzHe39D2I53/eeb1592c93f41b01ed465d6abd7bf349/06_private_cloud.png)

### Vulnerability Management

Our build and deploy pipelines continuously scan our cluster, containers, and code for vulnerabilities. Our security capabilities include the usual controls: static application security testing (SAST) on both code and infrastructure-as-code, dynamic application security testing (DAST) on our application and API endpoints, and software composition analysis (SCA) on our software dependencies and containers. Cloud Security Posture Management (CSPM) tooling identifies and responds to cloud misconfigurations (e.g., publicly accessible S3 buckets). Patching is a foundational security control; we continuously upgrade our containers and software dependencies. In the shared responsibility model, Azure/AWS are responsible for building upgraded node images, and Auth0 is responsible for deploying the images to our nodes. We accomplish this by rebuilding our Kubernetes nodes with the latest image during each Auth0 product release cycle.

### Automation is a must

We are dealing with a scale that makes it impossible to configure all of these security capabilities manually. To address his challenge, the Auth0 Security team created an in-house service that automates AWS account and Azure subscription creation and configuration. Similar to a production line, when new Spaces and PoPs are needed, this service creates AWS accounts and Azure subscriptions to meet our strict security standards using predefined templates. It effectively "stamps out" completely configured new accounts/subscriptions on-demand. When the account or subscription is no longer needed, the service will remove all resources, revoke all access, and decommission it.

## Continuously Improving Auth0's Security Posture

All of this work has culminated into a more secure, scalable platform that continues to meet or exceed the many requirements needed to achieve ISO 27001/27018, SOC 2, PCI, STAR, and HIPAA compliance.

The threat landscape is ever-changing and constantly evolving. The Auth0 Security team's job is never done, and a core principle is a continuous improvement. There are always ways to mitigate more risk and enhance our security posture, and we continually investigate additional security capabilities to make our platform and product even more secure for our customers.

We are publishing a series of technical posts on how and why we re-platformed. They'll be linked at the bottom of this post as they publish and are also findable on our blog. If you have technical questions or ideas for future posts, please reach out to us via the [Auth0 Community](https://community.auth0.com/). (Our marketing team also tells us we would be remiss if we didn't include a link to talk to Sales, so that is [here](https://auth0.com/contact-us), too.)

<include src="asides/AboutAuth0" />

Securing Auth0's New Private Cloud Platform

leonardo

Leonardo is a Senior Software Architect at Okta, working on the new Auth0 platform and cutting-edge product features. He is a seasoned technical leader with 17 years of experience in a wide range of segments, from enterprise to SaaS systems. As an Apache committer, he has contributed to various open-source projects. He is a self-learner, passionate about new technology, and has spoken at several developer events.

Senior Software Architect

Leonardo Zanivan

Replatforming to enhance agility, productivity, scalability, reliability, and security.

**TL;DR:** Auth0’s private and public cloud offerings are converging on a single platform to deliver an increase in business agility, developer productivity, scalability, reliability, and security for our customers and Auth0 by using industry-leading tools and fully automated operations.

Initially, Auth0 evolved our public and private cloud offerings as separate platforms because we could go fast and innovate on the public cloud and then work through the greater performance, compliance, and control expected by our private cloud customers. But over time, this led to feature discrepancies and, frankly, just wasn’t that efficient to maintain, scale, or innovate.

So we began an ambitious replatforming project. How could we re-architect all of Auth0 into a single platform? Could we support deployments on both Amazon Web Services (AWS) and Microsoft Azure? What about increasing geo-failover options? And we needed to continue our 99.99% SLA across both public and private clouds.

Let’s not forget that private cloud customers may also require extra add-ons like data residency, private links, and dedicated performance. This post will walk you through our approach to rearchitecting Auth0 to address those challenges.

## The Convergence Starting Point

While both platforms worked exclusively on AWS cloud, the private cloud was not using cloud-native solutions. It had three tiers — Basic, Perf, and Perf+ — available as environment sizes. These were selected based on the number of users and requests per second, versus an elastic scalable public cloud environment shared by multiple customers. The main difference is that private deployment usefully isolated and dedicated resources in any region. In comparison, public deployment uses shared resources for multiple customers in specific regions (United States, Europe, Japan, and Australia).

Our public cloud saw new features delivered on a weekly basis, but due to limited automation, our private cloud required high-touch operations for upgrading to new releases that rolled out on a bi-weekly cadence. 

Requirements for private cloud deployments set it up as a superset of the public cloud offering that already guarantees a 99.99% SLA, with a track record of 99.99+% over the last three years. This was across three availability zones and a geo-failover for regional redundancy and availability.

But we knew we also wanted to include extra add-ons like private links, [PCI compliance](), and meet the customer requests for [Azure Cloud]() support.

## New Cloud-Native Architecture

On a high level, the new cloud-native architecture delivers an increase in business agility, developer productivity, scalability, reliability, and security for our customers and Auth0 by using industry-leading tools and fully automated operations.

It closes the gap between Auth0 public and private deployments by packaging all the product features in a single software version and making them available depending on the customer subscription and add-ons. 

The ideation for the new platform is based on the following premises:

* **Multi-cloud: **AWS and Azure cloud support through a common interface
* **Container orchestrated:** use of container images instead of installing packages
* **Stateless**: scalable and replaceable by design
* **Immutable**: run only trusted code and configuration
* **Fully automated:** provisioning, upgrades, and decommissioning are done by machines
* **GitOps:** enhanced DevOps, push code and safely reproduce it thousands of times  

This removes the worry about the underlying infrastructure, allowing Auth0 developers to place their focus on our customers. This can be represented by the layers in the box below:

![Aplication Layer](https://images.ctfassets.net/23aumh6u8s0i/3yX9TOMT8XC0UPPKK6N57h/41dbdb93833d8c2e20e7c2be1bb445ef/88133_Private_Cloud_Campaign_Auth0.png)

* **Infrastructure layer**: It builds the infrastructure using AWS and Azure cloud services such as networking, virtual machines, key management, load balancers, container repositories, Kubernetes, and managed datastores (Mongo, PostgreSQL, Redis, Kafka).
* **Interface layer**: [Terraform](https://www.terraform.io/) is used for bootstrapping the infrastructure using code on different clouds. [Vault](https://www.vaultproject.io/) is used for platform secrets management and [Kubernetes](https://kubernetes.io/) as de-facto technology for application runtime along with custom plugins to describe services manifest, configuration, and secrets using the platform control plane component. [Argo CD](https://argoproj.github.io/cd/) is a GitOps continuous delivery tool used for applying for new software releases while [Argo Workflows](https://argoproj.github.io/workflows) is used for orchestrating the provisioning, upgrades, and decommissioning steps. This layer also contains integrations and agents for managing service discovery/DNS, logs, metrics, security audit, and datastores.
* **Application layer**: It is where the containers run. The Auth0 Identity Platform is built to allow multiple stacks with different sets of containers deployed accordingly to their usage. Auth0 products, PoP, Control Plane, and Central Services use a promotion pipeline for validating new releases and configurable release schedules. Everything is managed by control plane API and developers interact with the new platform CLI or Hub (web interface). One might wonder how identity is managed to access Kubernetes resources, it uses [Pomerium with Auth0 Identity Provider](https://www.pomerium.com/docs/identity-providers/auth0.html), dogfooding our own product.

Before we go deep into the technical details, the blog [A Primer on the New Private Cloud Platform](https://auth0.com/blog/a-technical-primer-of-auth0-s-new-private-cloud-platform/) provides more details about the Space, PoP, Control Plane and Central components of the new platform.

## The Components—A Ten Thousand Foot View

The diagram below demonstrates the main components of the platform and its links, including data flows and cloud regions. Internal services and network details are omitted.

![The Components—A Ten Thousand Foot View](https://images.ctfassets.net/23aumh6u8s0i/2bmBmIoidTViCeyw2dInNB/26a1c8a5cd9ac8a556488585edbb35d4/Arquitecture.png)

**Edge Network**: The global network provider, used by Auth0 for DNS, edge computing, CDN, DDoS protection, web application firewall, custom domain certificates, and speed-up end-user experience.

**Space**: Customer environment containing the dedicated application and datastores resources, deployed in any cloud region, maintaining security and data residency through tight access control for operators. Datastores use dedicated private networks isolated from the applications. New application deployments spin a whole new application cluster containing the new version and then slowly switch the traffic using a 'canary deployment' strategy. In addition to that, 'Data Shield' component is responsible for receiving a variety of data such as application logs and then scrubbing it, removing any PII or sensitive information. Only the clean logs are ingested and clean data is forwarded to Auth0 data analytics and detection & response teams for monitoring the system. The customer data never leaves the cloud region (e.g. Azure EastUS).

**PoP**: Point of presence of the platform in a given cloud region (same as the Space), required to host the container registry of software stacks used by the Kubernetes cluster. This component runs a set of shared services to support the platform, including the OpenSearch stack used by operation engineers to troubleshoot issues. It also runs regional services, shared between Spaces to support platform operations, such as forwarding data analytics, and internal Auth0 system supporting services requiring low network latency for fulfilling requests. It is important to note though that PoP doesn't have access to the Spaces network, it acts as a regional endpoint.

**Central & Control Plane**: A region in the European Union (EU) where the control plane services are deployed. They are internal to Auth0 operators and are used to interact with the customer Spaces and PoPs. It is secured by Zero Trust model and manages platform secrets and metadata using Terraform, Vault, and Argo. In addition to the control plane services, the central component is also designed to run other completely isolated application stacks to host global services accessible by customers and support engineers.

**Auth0**: The analytics aggregated data is sent to Auth0 headquarters and other global systems such as DataDog and Snowflake to be consumed by internal teams in many different departments, such as business analytics, customer support, and engineering teams. The data is categorized into different levels of access and doesn't identify any customer-sensitive information such as personally identifiable information (PII). The Detection & Response team is an exception and runs security software to analyze production traffic and usage patterns to actively detect & mitigate attacks.

## High Availability and Geo-failover

Auth0 guarantees a 99.99% SLA for production deployments. On a high level, the infrastructure is fully redundant, and all components are deployed into multiple availability zones, using at least 3 availability zones (AZ) combined primary and secondary regions, either in clustered mode or standby replicas. 

For web applications, the traffic is distributed across multiple Kubernetes nodes using load balancing. For datastores, Mongo and PostgreSQL are considered the system of record, i.e. the authoritative source of data, deployed in the primary region with a writer and at least a reader node for splitting write/read traffic and replicating the data to a secondary (failover) region in real-time. The caching (Redis) and stream (Kafka) components are deployed using sharding and failover capabilities, always providing high availability for runtime and data.

As for central services, control plane, and PoPs, they also run on high availability infrastructure and have the ability to failover, while this process is executing or in case of temporary or permanent network connectivity issues, the Spaces continue functioning as expected without any disruption. Spaces are isolated from each other and are fault-tolerant to other tiers' failures. 

The geo-failover is a fully automated process and is manually triggered by an operator during incident response for disaster recovery in case a cloud region is fully or partially unavailable. The process redirects all the traffic to the secondary region (determined by the customer) which switches the DNS of the cluster ingress and promotes the secondary region replicas to become the primary writer; there's no dependency on the primary region for the disaster recovery to successfully execute. 

The target RPO (Recovery Point Objective) is less than a minute, the time period in which data and events can be lost due to the disaster recovery procedure. At the same time, the RTO (Recovery Time Objective) is less than 15 minutes for the amount of downtime tolerated during the disaster recovery process.

The process of failback (revert the failover) from the secondary to the primary region is the same and can be triggered once the situation resolves.

The diagram below shows the Auth0 deployment containing a primary and a secondary warm standby region with high available datastores, and for simplifying purposes other AZs running only applications are omitted:

![Azure servers](https://images.ctfassets.net/23aumh6u8s0i/7f96vQ2UJukSjYJYc5flNt/a406ca7bac6ad54dc5e2bfc75c754dd8/Azure_servers.png)

We are publishing a series of technical posts on how and why we replatformed. They’ll be linked at the bottom of this post as they publish and are also findable on our blog. If you have technical questions or ideas for future posts, please reach out to us via the [Auth0 Community](https://community.auth0.com/). (Our marketing team also tells us we would be remiss if we didn’t include a link to talk to Sales so that is [here](https://auth0.com/contact-us), too.)

The Architect’s View of Auth0’s New Private Cloud Platform

kaycie-east

As PR Manager, Kaycie supports public relations and global communications efforts for Auth0 in the AMER region. When not promoting the Auth0 brand, Kaycie can be found hiking, traveling, or searching for the next best bite to eat.

PR Manager, AMER Region

Kaycie East

New private cloud deployment option offers customers greater choice, scalability, and reliability

**BELLEVUE, Wash. – Oct. 13, 2021 –** The [Auth0 Identity Platform](https://auth0.com/), a product unit within Okta (NASDAQ: OKTA), today announced the launch of the [Auth0 Identity Platform on Microsoft Azure](https://auth0.com/cloud/azure). The launch gives customers the ability to choose Azure as their cloud deployment option from Auth0. The addition of Azure as an Auth0 private cloud platform option unlocks a secure cloud deployment option for organizations seeking strategic fit with their technology stack, supporting regional data residency capabilities and higher control over customer data.

Today, organizations moving to the cloud and selecting a [Customer Identity and Access Management (CIAM)](https://auth0.com/blog/what-is-ciam/) solution require significant control over sensitive customer data and may require local data residency capabilities. They often have to make a hard choice of selecting either a CIAM solution that supports their product differentiation goals or one that meets their technical requirements and regulatory needs. Auth0’s modern identity platform eliminates the need to choose by providing organizations with tools designed to support security, compliance, and data residency needs that may apply to them across multiple regions globally. The platform’s design can accelerate their time to value and go-to-market strategies to meet their long-term growth goals. 

Key benefits of the Auth0 Identity Platform on Azure include:

* **Technology Stack Alignment:** Organizations have the flexibility to choose Azure as their preferred cloud vendor for greater technology stack alignment with existing investments in Azure. 
* **Data Residency:** Improved data residency capabilities for Auth0 customers with Azure’s widespread geographic infrastructure.
* **Streamlined Experience with Private Cloud:** A modern, cloud-native platform built with a focus on security and compliance at every level. Organizations requiring data residency options can take advantage of periodic feature updates while maintaining control over the timing and the ability to roll back without the hassle of having to maintain them — resulting in faster time to market and a more seamless experience for organizations and their customers.

“As more applications move to the cloud and expand globally, organizations depend on their identity service to protect and manage access for their most critical users, applications, and data,” said Shiv Ramji, Chief Product Officer at Auth0. “The Auth0 Identity Platform on Azure can help unlock a secure and compliant cloud deployment experience for organizations seeking support across CIAM use cases. This is especially true for organizations in industries like retail or financial services where selecting vendors that are not competitive to their business is top priority, organizations operating in regions where certain cloud providers have higher geographic availability, and those that have existing technology investments in Azure.”

“Companies like Auth0 add value by integrating their solutions with Microsoft Azure,” said Mandana Javaheri, Global Senior Director, SCI Business Development at Microsoft Corp. “With Azure, Auth0 customers have ultimate performance capabilities and control over their environment without the hassle of maintaining it, allowing customers to focus more on their core business proposition.”

More detailed information on the Auth0 Identity Platform on Azure can be found on [Auth0’s blog](https://auth0.com/blog/auth0-on-microsoft-azure-as-a-private-cloud-deployment-option). Auth0 will discuss this offering in greater detail with Okta at Showcase21 taking place today, Oct. 13, 2021. To learn more, visit: [https://www.okta.com/showcase/register](https://www.okta.com/showcase/register)  

<include src="asides/AboutAuth0" />
### Media Contacts:

Global Communications<br />
Auth0<br />
[press@auth0.com](mailto:press@auth0.com)<br />

Matter Communications for Auth0<br />
[auth0@matternow.com](mailto:auth0@matternow.com)<br />

### Investor Contact:
Investor Relations<br />
[investor@okta.com](mailto:investor@okta.com)<br />

Auth0 Identity Platform Now Available on Microsoft Azure

auziros:jeana-tahnk

Jeana has been in the tech industry for 15+ years, both as a communications professional and writer. As Director of Global Communications at Auth0, she works with her internal team and external agencies to amplify the company's visibility worldwide and establish its leadership in the identity industry. When she is not working, she can most likely be found on a field watching one of her three kids playing sports, or cleaning up after her one-year-old retriever who chews everything.

Senior Director, Global Communications

Jeana Tahnk

Recognition highlights Auth0 as a top private cloud company for third consecutive year



**BELLEVUE, Wash. – September 16, 2020 –** [Auth0](http://www.auth0.com/), the identity platform for application teams, has been named to the [2020 Forbes Cloud 100](http://www.forbes.com/cloud100), ranking 19th out of 100 companies listed. The Forbes Cloud 100 is the definitive list ranking of the top 100 private cloud companies in the world, published by Forbes in partnership with [Bessemer Venture Partners](https://www.bvp.com/) and [Salesforce Ventures](https://www.salesforce.com/company/ventures/). Auth0 was also included on the list in [2019](https://auth0.com/blog/auth0-is-named-to-the-forbes-2019-cloud-100/) and [2018](https://auth0.com/blog/auth0-named-to-forbes-2018-cloud-100/).

As part of the rigorous selection process for the 2020 Forbes Cloud 100, Bessemer Venture Partners received submissions from hundreds of cloud startups. The Cloud 100 Judging Panel, made up of public cloud company CEOs, reviewed the data to select, score, and rank the top 100 private cloud companies from all over the world. The evaluation process involved ranking companies across four factors: market leadership (35%), estimated valuation (30%), operating metrics (20%), and people & culture (15%).

"Being included on the Forbes Cloud 100 for the third year in a row is an honor and an accolade we are incredibly grateful for. Thank you to Forbes, Bessemer Venture Partners, and Salesforce Ventures for their continual recognition of Auth0," said Eugenio Pace, CEO and co-founder of Auth0. "As companies and end-users give more of themselves away online, protecting personal data and user credentials is of paramount importance. We are proud to secure the digital world and serve as a key enabler in our customers' digital transformation and identity management efforts." 

Auth0 recently closed a [$120 million round of Series F funding](https://auth0.com/blog/auth0-helping-navigate-through-times-of-transformation/) in July, bringing the company's valuation to $1.92 billion and total capital raised to more than $330 million. Auth0's Series F round comes a little more than a year after the close of its [$103 million Series E funding](https://auth0.com/blog/auth0-closes-103m-in-funding-passes-1b-valuation/) last May, which launched the company into 'unicorn' status. After growing 70% in 2019, Auth0's rapid growth has continued through 2020 despite the uncertainty of the current global economic environment.

"The private cloud ecosystem continues to mature in light of rapid digital transformations, making the competition to land one of the coveted spots on the Cloud 100 list steeper than ever," said Byron Deeter, a top cloud investor, and partner at Bessemer Venture Partners. "Private cloud valuations are getting bigger as the market's appetite for cloud continues to grow. Over the past five years, the average Cloud 100 valuation has grown by a tremendous 2.5x, from $1 billion in 2016 to $2.7 billion in 2020. In fact, our 2020 Cloud 100 includes over 87 private cloud unicorns. These founders represent the absolute best in cloud computing today, and they will most certainly follow in the footsteps of our esteemed Cloud 100 alumni. More than a third of the companies of the Cloud 100 have exited either via strategic M&A, financial M&A or Initial Public Offering. Congratulations to these cloud leaders."

"In the face of this unprecedented time, digital is an urgent imperative, and the pace of innovation is accelerating. Companies around the globe, across all industries must adopt cloud technologies to enable their digital transformation," said Matt Garratt, Managing Partner, Salesforce Ventures. "The opportunity is massive for the cloud – there are now more than 25 public SaaS companies valued at over $10 billion whereas even five years ago there were only three. Sectors from collaboration and security to retail and healthcare, are being disrupted and becoming more digital. Salesforce Ventures is excited to be partnering with Bessemer Venture Partners and Forbes for the fifth year in a row to recognize those who are not just predicting what's coming but working to create the future."

"For five years now, we have ranked the best and brightest emerging companies in the cloud sector," said Alex Konrad, Forbes editor of The Cloud 100. "With so many businesses growing fast in the cloud, from data infrastructure to marketing, it's harder than ever to make the Cloud 100 list – but with more elite company if you do. Congratulations to each of the 2020 Cloud 100 honorees and the 20 Rising Stars honorees poised to join their ranks."

The 2020 Forbes Cloud 100 and 20 Rising Stars lists are published online at [www.forbes.com/cloud100](http://www.forbes.com/cloud100) and will appear in the September 2020 issue of Forbes magazine.

<include src="asides/AboutAuth0" />

## About Bessemer Venture Partners
Bessemer Venture Partners is the world's most experienced early-stage venture capital firm. With a portfolio of more than 200 companies, Bessemer helps visionary entrepreneurs lay strong foundations to create companies that matter, and supports them through every stage of their growth. The firm has backed more than 120 IPOs, including Pinterest, Shopify, Yelp, LinkedIn, Skype, LifeLock, Twilio, SendGrid, PagerDuty, DocuSign, Wix, and MindBody. Bessemer's 15 partners operate from offices in Silicon Valley, San Francisco, New York City, Boston, Israel, and India. For more information, please visit [www.bvp.com](http://www.bvp.com/).

## About Forbes
The defining voice of entrepreneurial capitalism, Forbes champions success by celebrating those who have made it, and those who aspire to make it. Forbes convenes and curates the most influential leaders and entrepreneurs who are driving change, transforming business and making a significant impact on the world. The Forbes brand today reaches more than 160 million people worldwide through its trusted journalism, signature LIVE and Forbes Virtual events, custom marketing programs and 40 licensed local editions in 70 countries. Forbes Media's brand extensions include real estate, education and financial services license agreements. For more information, visit the [Forbes News Hub](https://www.forbes.com/pr/#47fadf455a1c) or [Forbes Connect](https://www.forbes.com/connect/).

## About Salesforce Ventures
Salesforce is the global leader in Customer Relationship Management (CRM), bringing companies closer to their customers in the digital age. Salesforce Ventures, the global investment arm of Salesforce, invests in the next generation of enterprise technology that extends the power of the Salesforce Platform. Salesforce Ventures is building the world's largest ecosystem of enterprise cloud companies and extending that technology to customers. Portfolio companies receive funding, strategic advisory, and operating support, and can easily join [Pledge 1%](https://pledge1percent.org/) to make giving back part of their business model. Salesforce Ventures has invested in more than 400 companies, including DocuSign, GoCardless, Guild Education, nCino, Twilio, Zoom, and others across 22 countries since 2009. For more information, please visit [www.salesforce.com/ventures](http://www.salesforce.com/ventures).

### Media Contacts
Jeana Tahnk<br />
Global Communications<br />
Auth0<br />
[press@auth0.com](mailto:press@auth0.com)<br />
 
Hannah Carroll<br />
Matter for Auth0<br />
[auth0@matternow.com](mailto:auth0@matternow.com)


Auth0 is Named to the 2020 Forbes Cloud 100

jenny-obrien

I'm a former business journalist who's fascinated by the intersection of tech and business strategy — in particular how the right solution can transform an idea into reality. I specialize in ROI, data privacy, and identity.

Business Content Manager

Jenny O'Brien

Gain cloud flexibility and agility while maintaining security and control


Amazon famously brought [book publishing “to its knees”](https://mashable.com/2014/07/30/amazon-has-killed-publishers-they-just-dont-know-it-yet/) by dominating distribution, creating eBooks, and transforming the publishing world with its own titles and opportunity for authors to self publish. 

In 2018, [Amazon entered the insurance market in India](https://www.insurancebusinessmag.com/us/news/breaking-news/amazon-set-to-file-for-insurance-licence-111581.aspx). [And in 2019, the U.S.](https://www.geekwire.com/2019/amazon-jpmorgan-roll-new-health-insurance-plans-part-haven-joint-venture/) 

After killing off [car insurance comparison tools](https://www.businessinsider.com/google-kills-auto-insurance-comparison-tool-2016-2), Google’s invested in [insurance agency software](https://www.insurancejournal.com/news/national/2019/01/02/513324.htm) and signaled further interest in the market. 

In other words, digital transformation is already here. 

While industry complexities afford some protection, they offer greater opportunity for insiders, says TechCrunch. 

A recent [J.D. Power survey](https://www.jdpower.com/business/press-releases/2019-us-home-insurance-study) on customer satisfaction with home insurance noted that the home insurance business model was “primed for change” and that “weak points in customer experience offered an opportunity for startups.” 

The other side of the desire for customer experiences that are real-time and online is the fact that FinServs are tasked with protecting hackers’ favorite data targets. And the [global rise in data privacy regulation](https://auth0.com/explore/data-privacy/) makes stronger protections important from customer, legal, and compliance perspectives.

While digital transformation can seem daunting for incumbents relying on decades of legacy code, the right identity solution can create seamless customer experiences while increasing your data protection — even if you’re an incumbent partnering with (or acquiring an) up and coming Insurtech startup.

Forbes expects [XaaS (everything as a service), UX, CX (User/Customer Experience), and digital privacy](https://www.forbes.com/sites/danielnewman/2019/07/14/top-10-digital-transformation-trends-for-2020/#3dfc8bdc76be) to “take center stage” in as a digital transformation trend 2020. Like other industries, the need to streamline services, reduce capital expenditures, and be increasingly agile and innovative is fueling FinServ’s change. But FinServ’s need for security, stability, and high availability kept many systems in On-Prem solutions that have proven expensive to scale and maintain — putting the ability to keep up with evolving customer expectations at risk.

For FinServ’s looking to innovate (or even just keep pace), Public Cloud is often touted as an option, but it isn’t the best choice. Public Cloud outages and uptimes provide a strong reason to go private. Auth0 Private Cloud helps (or eliminate) that risk, by providing a secure, flexible, extensible solution. 

![Deployment Options Diagram](https://images.ctfassets.net/23aumh6u8s0i/23TVeU9N9cKlxxrenpT6hA/ae7991f61be42885833856bed91a74f7/deployment_options_diagram)

## Why Is Private Cloud Particularly Suited for Finserv Environments

For many years, incumbent FinServs have been buffered by deep compliance capabilities and an established customer base. But FinTech disruptors have eroded that edge says a recent PWC [report](https://www.pwc.com/gx/en/financial-services/assets/pdf/technology2020-and-beyond.pdf), in part because successful disruptors are able to offer a stronger customer experience at a lower cost to customers and with less investment. 

To solidify standing and increase the customer base in this environment, FinServ’s need to:

* Improve customer experience
* Keep ahead of evolving data privacy regulations
* Reduce costs without sacrificing cybersecurity

The data that FinServs hold makes them a prime target for hackers — and the sheer company scale usually means millions of breached accounts, rather than thousands. Globally, the increase in data privacy laws like GDPR and CCPA demonstrates that regulators have lowered tolerance for breaches and expect greater protections to be in place. When not implementing features like multi-factor authentication (MFA) can hit an insurer with a large fine, like what happened to [Dutch public insurance company UWV](https://auth0.com/blog/gdpr-fines-hit-these-companies-hard-heres-how-to-avoid-them/), the need for greater security while still ensuring flexibility becomes more appealing.

<include src="TweetQuote" quoteText="The data that FinServs hold makes them a prime target for hackers — how Private Cloud can help."/>

From an identity perspective, let’s take a closer look at what Auth0 Private Cloud can mean for key digital transformation considerations.

### Performance at Scale
The most innovative solution in the world does zero good if it doesn’t offer you the performance you need.  Auth0 Private Cloud offers request-per-second (rps) baseline is higher than public cloud, with the option of adding on capacity up to 1500 rps.

### Reliable High Availability
FinServ is more aware than most industries the dramatic impact of service interruptions, especially given the pressures of changing global conditions. High availability helps guard against those challenges, especially since Private Cloud can be deployed between two regions to provide failover.

### Control & Segregation
Managed Private Cloud offers a fully-isolated development environment that is updated independently from the production environment. Standard Private Cloud is a single instance production environment. Both Standard Private Cloud and Managed Private Cloud meet standards that require all data to be housed within specific regions.

### Free Developers to Improve Customer Experience
For a developer coming up with an innovative solution while working only On-Prem can feel like trying to code while wearing handcuffs — it’s hard to even get your hands on the keyboard. 

To keep up with the pace of customer’s hunger for real-time innovations, streams of data from connected vehicles and smart homes, and the data privacy compliance requirements from laws like GDPR and CCPA, developers need not only the flexibility but also the scale of a [cloud deployment](https://auth0.com/cloud). And they need that deployment to be secure and reliable enough to handle the demands of enterprises whose customers are often interacting during moments of high stress or need.

### Unify Legacy and New Data Stores
Unifying legacy and new data stores can be a giant headache, as well as a source of weeks of downtime without a plug-n-play identity solution that allows bulk imports, but can also streamline user migration without expensive forced password resets – whether unifying stores at a single organization or when an incumbent partners or acquires an insurance startup.

### Support for Different Languages/Architectures (SDK)
Aside from unifying data stores, incumbent companies often rely on a maze of programming languages and architectures. Shutting down one part of the maze can have unexpected consequences if all the interrelationships aren’t fully mapped out or even understood due to poorly commented code or loss of institutional knowledge due to attrition or staffing turnover.

Your devs will be a lot happier (and more productive) when working with an identity solution that can handle multiple programming languages and architectures and already has SDKs (software development kits) that can be dropped into place to provide rapid access to functionality. And the right solution gives you the flexibility to map the maze at your own pace, reducing costs by removing unnecessary middleware, maintenance, and security as you deem appropriate.

### Extensibility
<iframe width="560" height="315" src="https://www.youtube.com/embed/kCmU4EMMPf0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

The real world is a messy place where you might have an out-of-the-box tool that needs to be integrated with custom code. The ability to inject code into the Auth0 Identity solution means that developers can handle the last-mile customizations that execute on the C-suite’s business strategy without weeks of downtime or missed opportunities. To delve into this topic in greater technical detail, you might also check out Auth0 Principal Architect Vittorio Bertocci’s whitepaper “[The Tao of Extensibility.](https://auth0.com/resources/whitepapers/tao-of-extensibility)”

## Support and Failover
Uptimes, updates, failovers, and support can have a big impact on the success of your deployment. 

With Public Cloud coming in with an uptime SLA of 99.90%, Private Cloud offers a stronger uptime SLA of 99.95% with the option to upgrade to maxim uptime guarantees of 99.99%. Geo redundancy can provide resiliency against catastrophic events. As mentioned above, Private Cloud offers failover between two geographic regions. 

For teams that would appreciate additional support, Auth0 offers a support SLA as an add-on, as well as the add-on option to work with the skilled identity experts on our Professional Services team.

## Managed Private Cloud vs. Standard Private Cloud
Let’s take a look at the different deployment Private Cloud models and benefits, including a checklist to help you assess deployment needs.

**Standard Private Cloud** is a low touch engagement model that offers additional power, security and compliance* benefits through an isolated tenant just for a single customer. Uptime SLA is 99.95% with an upgrade possible to 99.99%. PCI compliance is available as an add-on.

**Managed Private Cloud** is a high-touch engagement model where customers are much more involved with all aspects of this deployment model. Our team will work closely with the customer to ensure all aspects of their Auth0 environment are tuned for what best suits their business needs. Uptime is 99.99%. PCI compliance is available as an add-on.

**Reasons to choose Managed Private Cloud**                                                                                                          * **Choice of update frequency**<br>Coordinated with Auth0 (required once per 3 months). The update cycle begins with the Pre-Production Environment.
* **A separate DEV (Pre-Production) environment**<br>Fully-isolated and independently updated instance for development and testing (Single Node)
* **On-demand scaling**<br>WIll scale customer environment for any spikes                                                                              * **Load Testing**<br>4x Load Testing per year with dedicated Auth0 MSE Support                                                 * **Customer-hosted AWS deployment**<br>More **control** over the environment                                                                              * **GEO-HA deployment option**<br>Datacenter **redundancy** and automatic **failure handling**.                                                               
<br>
**Reasons to choose Standard Private Cloud**
* ***Doesn’t require any of the Managed Private Cloud reasons***
* **More cost-effective**
* **Less involved with managing the deployment**<br>The customer is looking for an offering that feels similar to a standard Public Cloud SaaS but has isolation and the other advantages of our Private SaaS Deployments. |

<include src="TweetQuote" quoteText="Legacy code can make #digitaltransformation feel like a high hurdle for FinServs. Find out how Private Cloud can help securely smooth the jum"/>

## Reduce Risk, Increase Innovation With Private Cloud
Like identity, digital transformation is not a one-and-done solution, but an ongoing need to keep pace with the evolution of technological change. Auth0 Private Cloud provides the flexibility, agility offered by cloud deployments but allows FinServ’s to maintain the control and security they need to protect their customers and their businesses. Your IAM solution is a critical component of your security strategy, but even if you’re dealing with decades of legacy code, you don’t need to sit down and build it in-house. The more secure option is to let identity experts protect those critical credentials, freeing your team (and resources) to focus on what you do best. Identity no longer needs to be a roadblock or an afterthought. Identity Access Management (IAM) is an essential component of customer security and privacy. If you’d like to learn more about how Auth0 Private Cloud might work for your particular situation, please reach out to an [Auth0 resource](https://auth0.com/).


How FinServs Can Reduce Risk, Increase Innovation with Private Cloud

List showcases private cloud companies with highest levels of employee satisfaction


**BELLEVUE, Wash. -- April 29, 2019 --** [Auth0](https://auth0.com/), a global leader in Identity-as-a-Service (IDaaS), has been recognized as one of the **50 Highest Rated Private Cloud Computing Companies To Work For** in a list released by [Battery Ventures](https://www.battery.com/), a global investment firm and cloud investor, with data provided by [Glassdoor](https://www.glassdoor.com/index.htm)*, one of the world's largest job and recruiting sites. The list highlights 50 privately-held companies -- all business-to-business, cloud-computing companies -- where employees report the highest levels of satisfaction at work, according to employee feedback shared on Glassdoor.

The distinction placed Auth0 at number 19 (out of 50 total) with an overall company rating of 4.7.  The broader average across Glassdoor is 3.4. Auth0's CEO, Eugenio Pace, boasts a 100 percent approval rating on Glassdoor -- compared to an average of 69 percent for all approximately 900,000 employers on the site. The company has an 88 percent positive business-outlook rating, again based on feedback shared by employees. The broader Glassdoor average is 49 percent. A positive business outlook means employees believe business will improve in the next six months.

This is the third year Battery has issued the list, along with a related ranking of the 25 Highest Rated Public Cloud Computing Companies to Work For. The rankings highlight the broader trend of businesses increasingly turning to the cloud to run critical technology systems and software, instead of using on-premise systems.

They also highlight the increasing importance of cohesive culture and employee happiness in running a successful business, said Neeraj Agrawal, a Battery general partner who specializes in cloud investing.

"The private companies on this list have not only scaled their products, teams, and business functions -- but they've managed to scale culture," Agrawal said. "We view these rankings as a key indicator of company health and longevity, and we hope all companies on this list view it as an honor to be included." It was also more difficult to make the list this year, compared with last year, Agrawal added. A Glassdoor economic research [study](https://www.glassdoor.com/research/satisfied-employees-drive-business-results/), as well as other third party [studies](https://www.glassdoor.com/research/satisfied-employees-drive-business-results/), show that companies with high employee satisfaction often post stronger financial performance.

"Culture is one of our primary pillars and is something we invest a lot of resources into," said Melinda Starbird, VP of People and Culture at Auth0. "This recognition is very rewarding because it is based on feedback from employees and former employees, and is a testament to how richly-infused culture is for everyone at the company, regardless of their position or location."

Glassdoor noted that employees at these highly rated companies commonly mention in online reviews that they enjoy working for mission-driven companies with strong and unique company cultures; employers that promote transparency; and companies with experienced senior leaders who regularly and clearly communicate with employees. For instance, according to one anonymous employee review of Auth0 on Glassdoor:

*I never knew such a great workplace could exist. People work hard and they care about each other -- and this starts at the very top.*

Full lists of the highest rated 50 private cloud companies and 25 public cloud companies to work for can be found [here](https://www.battery.com/powered/cloudculture-2019/).

<include src="asides/AboutAuth0" />

### About Battery Ventures

Battery strives to invest in cutting-edge, category-defining businesses in markets including software and services, Web infrastructure, consumer Internet, mobile and industrial technologies. Founded in 1983, the firm backs companies at stages ranging from seed to private equity and invests globally from offices in Boston, the San Francisco Bay Area, London, New York and Israel. Follow the firm on Twitter @BatteryVentures, visit our website at www.battery.com and find a full list of Battery's portfolio companies [here](https://www.battery.com/our-companies/list/).

*By a company name, denotes a Battery investment. For a full list of all Battery investments and exits, please click [here](https://www.battery.com/our-companies/list/).*

### **Media Contacts:**

Kasia Hall<br/>
Matter for Auth0<br/>
[khall@matternow.com](mailto:khall@matternow.com)<br/>
971-246-7898


Auth0 Named One of 50 Highest Rated Private Cloud Computing Companies to Work For

diegopoza

System Engineer, geek, foodie, technology lover, speaker.

Sr Manager, Developer Advocacy

Diego Poza

Tavis Ormandy, a vulnerability researcher at Google, discovered that Cloudflare was accidentally leaking sensitive data including passwords, private messages, and more. Learn how this may affect you and your customers and what to do next.



On February 17th, Tavis Ormandy, a vulnerabilities researcher at Google, sent the tweet that kicked off 2017's biggest security story yet:

![Tweet](https://images.ctfassets.net/23aumh6u8s0i/70lrHor3JHJ2mpMhcIYmD0/a8ae61fe8062a2abad060c9707f51819/tweet)

*Could someone from cloudflare security urgently contact me.* That's not something anyone wants to read on a Friday afternoon, but especially not when it's coming from one of the world's top infosec researchers.

The problem? While doing some routine bug checking, [Ormandy](https://twitter.com/taviso?lang=en) saw some data that did not match at all what he expected to find.

"It's not unusual to find garbage, corrupt data, mislabeled data or just ... non-conforming data...but the format of the data this time was confusing enough that I spent some time trying to debug what had gone wrong, wondering if it was a bug in my code," [he wrote](https://bugs.chromium.org/p/project-zero/issues/detail?id=1139), "In fact, the data was bizarre enough that some colleagues around the Project Zero office even got intrigued."

The rest of the Google Project Zero team (a team of security researchers employed specifically to find new [zero-day exploits](https://en.wikipedia.org/wiki/Project_Zero_(Google))) probably expected Google's announcement of the first [practical SHA-1 collision](https://auth0.com/blog/sha-1-collision-attack/) to be the big news of the week.

As they looked closer, they found “encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests” for all sorts of different sites using Cloudflare—private messages, reservations, plaintext API requests from a password manager, hotel bookings.

How did all of this happen?

## Cloudflare was dumping memory across the web

It begins with a simple bug in the code to Cloudflare's highly popular reverse proxy service. HTTP requests to certain types of sites signed up with Cloudflare would trigger the bug, which would then "leak" data from random* *Cloudflare customers' sites (those that happened to be in memory at the time).

![Cloudflare](https://images.ctfassets.net/23aumh6u8s0i/SFApl9rEVZFr1bZEQnm2k/1df7cc080c9c802b225f4610a0daee12/cloudflare)

*A simple depiction of how a reverse proxy like Cloudflare's works. Since reverse proxies are shared between customers, data from all was at risk of being disclosed.*

Since September, random bits of data have been coming out of initialized memory and leaking across random Cloudflare customer's sites. In other words, if you went to visit a site that used Cloudflare, you could have had other people's sensitive information “leaking” into your own browsing session.

When search engines “crawled” these pages to index them, the same kind of information “leaked.” Because search engines cache the output they receive when they visit a page, all these leaked tokens, secrets, and messages were indexed, split up and spread across *millions* of pages of search results.

![Search Results](https://images.ctfassets.net/23aumh6u8s0i/569KQNvP3RuTSc2z9FCZy/83586ba930635d2bace66281330bc291/SERPs)

(Source: [Wordfence](https://www.wordfence.com/blog/2017/02/cloudflare-data-leak/))

While Cloudflare is working with the major search engines to purge their caches of sensitive information now, it's hard to avoid thinking about the paranoid version of events.

Paranoid version: state actors, or state-sponsored actors with significant resources, discovered this vulnerability before Google did. They found a way to send manipulated HTTP requests to Cloudflare sites that would output a predictable stream of user data—and have either packaged everything they found for resale or set about figuring out how to crack what could be billions of passwords, credit card numbers, and secret tokens.  

Cloudflare claims they can rule out this scenario with access logs, but whether or not that's true has been subject to [some debate](https://news.ycombinator.com/item?id=13721452).

> Auth0 can improve software security with the breached password detection feature. [This feature protects and notifies your users when their credentials are leaked by a data breach of a third party](https://auth0.com/breached-passwords). You can optionally prevent access until the user has reset their password.

## What you should do now

The way that data was disclosed means that any site using Cloudflare could potentially have had its secrets and tokens compromised. Patreon, Yelp, Uber, Medium, Fitbit, and OKCupid all use Cloudflare—for a full list of sites you can [click here](https://github.com/pirate/sites-using-cloudflare). Several tools have also been [built to assess your risk](http://www.doesitusecloudflare.com/), like this script that [crawls your Chrome history](https://gist.github.com/kamaljoshi/2cce5f6d35cd28de8f6dbb27d586f064).

Though many, many sites use Cloudflare, at least to some degree. It's still quite unlikely that your personal information has been exposed. Many of the individual sites mentioned have also come out and written posts to explain whether you should be concerned—perhaps most notably, 1Password, which was initially mentioned in Tavis's tweets as a [potentially compromised site](https://blog.agilebits.com/2017/02/23/three-layers-of-encryption-keeps-you-safe-when-ssltls-fails/).

The first thing you want to do to protect yourself from this and future incidents is to enable [multifactor authentication](https://auth0.com/multifactor-authentication). The most important sites to enable MFA on are the ones you use as an SSO iDP—if you're logging into 20 different sites using your Google credentials, then you definitely want to enable some form of MFA there.

While it's unlikely your passwords were exposed, this is as good a reminder as any that you should be using strong, unique passwords and organizing them using a password manager.

Resetting individual passwords for sites doesn't make much sense—the list of compromised sites is simply so vast. The only solution is to do a mass reset and make sure you're using proper password hygiene going forward.

## What you should do for your customers

As a site operator, you're probably wondering 1) if you're at risk, and 2) if any of your users' data was exposed in this incident.

Cloudflare has already reached out to domain owners that they have proactively identified as being at risk in this leak. However, the absence of notification does not mean that your domain (and customers) are safe.

First off, enable multifactor authentication [if you haven't already](https://auth0.com/multifactor-authentication). It's the easiest way to get a huge boost to the security of your site.

Many site operators have begun forcing password resets for all of their users. If you run a consumer site, it may not be worth the trouble and inconvenience. If your customers are in the enterprise, then it is probably a very good idea to force a change. If you're not going to make your users change their passwords (and from a cursory browse of the web this Monday morning, it appears that this is mostly the case), then you should at least recommend it.

As far as data invalidation, any secrets which you can easily rotate—session identifiers, tokens, keys—should immediately be changed. Customer SSL keys do not appear to have been compromised in this incident, but it would be prudent to change as much data as you can which may have passed through Cloudflare.  

If you or any of your customers are regulated by the Health Insurance Portability and Accountability Act (HIPAA), then you'll [definitely want](https://www.aptible.com/blog/aptible-was-not-affected-by-cloudbleed/) to have your security/compliance teams get in touch with your lawyers and discuss whether this is a breach that needs to be reported.


"private cloud"