Uploaded from Data Pipelines on Auth0’s New Private Cloud Platform

Andrew is a Staff Engineer at Auth0 focusing on bytes that move for the new Auth0 platform. He is a startup co-founder with over a decade of experience delivering developer focused products at scale. When not at the computer, you can find him hiking up mountains, watching improv comedy, or spending quality time with his wife and son.

Staff Engineer

Andrew Hawker

“Data goes in; data goes out. You can’t explain that.”

**TL;DR:** Auth0 has replatformed our private cloud offering onto Kubernetes. As part of this new architecture, we’ve had the opportunity to apply modern practices to the data infrastructure that powers near real-time features and provides the necessary knobs for data governance.

_Hello friends! Before we get started, I recommend that you check out “[A Technical Primer of Auth0’s New Private Cloud Platform](https://auth0.com/blog/a-technical-primer-of-auth0-s-new-private-cloud-platform/)”, “[Architecture of the New Platform](https://auth0.com/blog/the-architect-s-view-of-auth0-s-new-private-cloud-platform/)”, and “[Security of the New Platform](https://auth0.com/blog/securing-auth0s-new-private-cloud-platform/)”. These wonderful posts cover foundational aspects of the platform that I will build upon here._

## Introduction

“Data is the new oil,” quipped Clive Humby in 2006, and the industry has been chanting “Drill, baby, drill” ever since. Just like oil, data needs to be refined/processed/transformed to unlock its full value. As the saying goes, “Necessity is the mother of invention,” so the industry has birthed an entire specialization around Data (Science, Engineering, Platform, Warehouse) to extract that value. The analogy remains apt because, just like oil, data is now being respected as a potentially hazardous material. With GDPR, CCPA, and their future brethren, we’ve come to understand the importance of proper packaging and safe keeping of our data assets. No more ziplock bags of data goop in the trunk of my Dodge Neon. Neatly stacked steel crates with big padlocks under 24/7 surveillance that can be crypto shredded at a moment's notice is the future of data management. With this New Platform initiative at Auth0, we’ve had the opportunity to apply some modern practices to our data infrastructure and take another step towards that dystopian data future.

![Introduction](https://images.ctfassets.net/23aumh6u8s0i/5GGILFTY6KNSabp6Ixn2dA/1b6a7923fa10b614f320c94cf436c2b9/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

Taking a page from Leo’s 10k foot view in “[Architecture of the New Platform](https://auth0.com/blog/the-architect-s-view-of-auth0-s-new-private-cloud-platform/)”, we’ll be touching on the highlighted sections of the platform in this post. We’ll start with some introductory context, discuss our guiding principles, dive into the tools we chose, and then exercise them with a few sample scenarios commonly seen at Auth0.

Put down that bag of data goop, and let’s get started!

## What is a Data Pipeline?

_“In a world where SaaS and microservices have taken over...”_ — Movie Voiceover Guy

*Looks around*

“Hey, he’s talking about us!”

Software is on a hockey stick growth curve of complexity. We’ve got an ever-increasing number of disparate systems hoarding data and even higher demand on us to unlock value from it.

1. “Show me the region where credential stuffing attacks are originating from” — A customer dashboard
2. “Let me run custom code in an Auth0 Action immediately after a user is deleted” — A feature request
3. “Show me failed login counts from the past 60 days for all tenants on the Performance tier that have been a paying customer for more than one year” — An upsell opportunity

Before we can begin to actually answer any of those questions, we need the connective tissue to ship data from Point A to Point B enriched with E and transformed by T. That’s the role of the Data Pipeline.

That’s not to say that Data Pipelines just carry simple domain/product events. Application logs, exception stack traces, Stripe purchase orders, Salesforce billing updates, etc., are all events that can be emitted and flow through a pipeline.

_“The Data Pipeline is a series of tubes.”_ — Michael Scott

Eh, close enough.

## Our Guiding Principles

Before jumping into implementation, it’s important to understand that _our_ requirements are not necessarily _your_ requirements. The decisions we’ve made might not be the correct decisions for you! When evaluating solutions and discussing tradeoffs, these are some of the key points that helped guide us.

### Event Durability

Event durability is a supremely important requirement for us; missing and out-of-order events are a non-starter.

If we imagine emitting a simple event, e.g., “UserCreated,” there are a few approaches:

1. Integration/domain event from application code
2. [Outbox Pattern](https://microservices.io/patterns/data/transactional-outbox.html)
3. [Change Data Capture (CDC)](https://en.wikipedia.org/wiki/Change_data_capture)

Discussing tradeoffs between these is a blog post in itself, so I’ll just simply say that they’re complementary and fit different needs, with the main tradeoff being durability vs. coupling.

Events emitted by application code can get lost with failures during dual writes. Outbox Pattern struggles if you don’t have strong transactional guarantees with your database of choice and doubles your write load. Change Data Capture (CDC) is the most durable of the three but “ships” the database table schema in each event, promoting unnecessarily tight coupling and having limited access to only the modified row.

### Data Governance (O Data, Where art thou?)

Data Pipelines come in different shapes and sizes.

* Are messages serialized as Avro, JSON, or Protobuf? XML? _Crickets._
* Are they encrypted or plain text?
* What is the expected throughput? Sustained or burst?
* Streaming or batch processing?
* Do producers retry or drop events during downtime?
* What enrichments/transforms do we need to apply?

Before discussing these technical details of a pipeline, we must first consider what is **_in_** the pipeline.

1. Do messages within the pipeline contain Customer Data?
2. Do messages within the pipeline need to leave the region where they originated?

If you answered “Yes” to one or more of the previous questions, calmly go outside and light your computer on fire. Joking aside, by asking and answering those questions, you can begin to understand the security/compliance posture you must take. This is likely to lead you down the road of even more questions.

* What region is data being shipped to relative to where it’s generated?
* Are we sending aggregated data or individual records?
* How is this data encrypted in transit and at rest?
* What Customer Data (down to each individual field) are we processing?
* Are there treatments that are sufficient for each field to be compliant while still fulfilling the business need? Masking, partial redaction, hashing, encryption, etc.

Remember that data is a valuable resource but can also be a hazardous material and must be handled properly. If you discover cases of processing Customer Data and shipping it across geographic boundaries, find yourself a buddy in the compliance department to help sort it out.

### Inertia

Event streaming technology is an active area of development, with many new companies and products entering the space. This is great because we can stand on the shoulders of giants, but if you’re not careful, it can leave you prone to painful churn, deprecation, incompatibility, or entire products going bust.

While we don’t want to fully jump into the [Boring Technology Club](https://boringtechnology.club/) pool, we _do_ want to build on robust tools that will survive a hype cycle. If a project doesn’t have enough inertia and community buy-in to survive the departure of the original stewards, it might be best to consider an alternative.

### Composition

The only constant in life is changing, especially technological and business requirements. We need to consider how our technology choices can be composed, merged, bent, broken, and smeared together to get the job done. If we choose a solution too rigid, it’ll be re-written often. A solution too flexible is unlikely to be fully understood. We should try and use tools that provide a healthy balance.

## It's Dangerous to Go Alone! Take This

We’re not the only company building on data pipeline technologies, thankfully. We can stand on the shoulders of those before us. Using our guiding principles outlined above, here’s the alphabet soup of open source technologies we chose to plug together to make our data go.

#### Kafka

The original Auth0 platform was built to run only on AWS and heavily leveraged Kinesis, SQS, and RabbitMQ. When evaluating cloud-agnostic solutions for the new platform, it was difficult to say no to [Kafka](https://kafka.apache.org/). With most cloud providers offering a hosted solution (or API-compatible layer), it’s becoming the lingua franca for durable event backends. With relatively sane approaches to implementing encryption at rest or in-flight, it meets most of our needs right out of the gate.

#### Kafka Connect

[Kafka Connect](https://docs.confluent.io/platform/current/connect/index.html) has a solid ecosystem of integrations that is relatively plug-and-play (although schema vs. schema-less can be a pain). The [AWS S3](https://docs.confluent.io/kafka-connect-s3-sink/current/overview.html) sink connector was available to us immediately, and an open-source [Opensearch](https://github.com/aiven/opensearch-connector-for-apache-kafka) connector popped up quickly after the Elasticsearch license fork snafu.

[Single Message Transforms (SMT)](https://docs.confluent.io/platform/current/connect/transforms/overview.html) within a connector are simple mechanisms for stateless transformation/enrichment of all messages. This works well for us to inject static metadata from the new platform like Space, POP, Cloud, current deployment version, etc.

#### Schema Registry

Kafka Connect heavily utilizes the[ Confluent Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html) for message serialization and validation. While some connectors support “schemaless” messages, you’ll find yourself, as we have, swimming upstream if you don’t leverage the registry with these tools. We haven’t fully realized our vision here and plan further investment over the next few quarters.

#### Debezium

As I mentioned before, event durability was an extremely important requirement for us, so we started with Change Data Capture (CDC) powered by [Debezium](https://debezium.io/). With their [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html) and [Postgres](https://debezium.io/documentation/reference/stable/connectors/postgresql.html) source connectors, we’re immediately plugged into the Kafka Connect ecosystem.

#### Data Shield

When it comes to data policy enforcement for streaming messages, there didn’t seem to be many free/open-source solutions we could take off the shelf. For this, we wrote our own in-house solution in [golang](https://go.dev/).

Data Shield handles data policy enforcement with schemas, data mutation, and topic muxing. With it, we can define events flowing through a topic using an IDL like [protobuf](https://developers.google.com/protocol-buffers). The IDL is not tied to the serialization format in the topic, e.g., we can use a protobuf IDL for JSON events.

Let’s walk through a quick example. Let’s say we emit events of Company records that have a field for their EIN.

```
message Company {
  string name = 1;
  string address = 2;
  string ein = 3;
  string url = 4;
}
```

Field enforcement is zero-trust by default, so any fields not defined in the IDL are automatically excluded. This is a critical capability for us since we receive many JSON/BSON-based events with loose schemas where fields can easily be added/removed by the upstream producer. This minimizes the blast radius of an upstream service accidentally or intentionally producing new fields with Customer Data on streams that leave the region.

Using the IDL, we annotate fields with policy details like classification and treatment rules. From our Company example, let's assume the **_ein_** field is classified as Personally Identifiable Information (PII). We could model this as:

```
  string ein = 3 [
    json_name = "ein",
    (auth0.proto.options.fields.v1.classification) = {
      classification: PII,
    },
    (auth0.proto.options.fields.v1.transform) = {
      processor: {
        name: OPENSEARCH,
        transforms: [
          { mask: { length: 4, character: "*", reverse: true } }
        ]
      },
      processor: {
        name: FRAUD_DETECTION,
        transforms: []
      },
      processor: {
        name: EU_DATA_LAKE,
        transforms: [
          { drop: { } }
        ]
      },
      }
    }
  ];
```

Each separate data processor of the stream can apply their own independent rules based on their business function. In this case, our Observability stack in Opensearch has the last four characters masked, our Fraud Detection feature receives the value in plain text, and the Data Lake processor in the EU doesn’t receive it at all.

With this IDL definition and some minor service configuration, we can tell Data Shield to enforce the policy on one or more topics. In the simple case, it will take the incoming “dirty” topics and output “clean” topics, one for each data processor with their specific rules applied.

![IDL](https://images.ctfassets.net/23aumh6u8s0i/3ka0jensdx70UoBfYmdnrd/c85e55476cde1771e397710a63058d61/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

This multiplexing increases our storage costs but allows us to leverage different retention policies and Kafka ACLs so consumers can only access data from their assigned topics. While this is a cost we are (currently) willing to pay for strict data isolation, it is an area for future investment with a potential “decryption and schema enforcement on reading” strategy.

As we’ve iterated on the Data Shield concept, we’ve added a few more features to make our lives easier.

* One-to-many topic to IDL schema mapping for topics with heterogeneous message types. This is very helpful for topics that contain app/service logs.
* Schema format conversions, e.g., read JSON, write Avro
* Extend IDL to support field enrichment with generated runtime values
* Golang API for pluggable actions to modify processing pipeline as events flow through
* Rate-based sampling for sinks that can be overwhelmed

## Scenarios

With our guiding principles leading us to those tools, let’s diagram out a few simplified versions of scenarios we run on the new Auth0 platform.

### Application logs for incident response

In this scenario, we’re using [Fluent Bit](https://fluentbit.io/) to consume service logs from all pods within the Space Kubernetes cluster. They’re shipped to Kafka, where Data Shield consumes the raw log messages, applies its enforcement rules, and produces them back to Kafka. They’re consumed by Opensearch connectors, who ship them across the private link to the POP, where they’re accessible to Operations teams for incident response.

![Application logs for incident response](https://images.ctfassets.net/23aumh6u8s0i/5LkE9v6OyZJpJA85G1lJM7/1e3a523222efba8065f4359b47e456c4/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

### Events for autocomplete search

In this scenario, we’re using Debezium to consume MongoDB change streams to emit events for Auth0 Users into Kafka. Data Shield consumes them, applies its enforcement rules, transforms the structure, and produces them back to Kafka. They’re consumed by another service that processes them and writes them into Postgres running in the same region.

![Events for autocomplete search](https://images.ctfassets.net/23aumh6u8s0i/6WCxg7eRdAuOdGm0MslthE/2886847192f02ae1b1c1cdef18514760/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

### Event counts to a data lake

In this scenario, we’re using Debezium to consume the Postgres logical replication protocol to emit events for Auth0 Connections into Kafka. Data Shield consumes them, applies its enforcement rules, and ships them back to Kafka. They’re consumed by S3 connectors that ship them to an S3 bucket running in a different region.

![Event counts to a data lake](https://images.ctfassets.net/23aumh6u8s0i/4BlUDX70CAZfZ0wl1JOU39/5ce6cf28061d37de489e51bede949ea6/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

### User requested backups 

In this scenario, we're emitting product events for Auth0 Tenants into Kafka. Prior to processing by Data Shield, they’re backed up in-region to object storage. The backups are then periodically rolled up and offered as a data download upon user request in a dashboard.

![User requested backups ](https://images.ctfassets.net/23aumh6u8s0i/6PzHrtwKWMgRzKsp7Hun2V/ee40263dbe880df9975304e07d517945/Uploaded_from_Data_Pipelines_on_Auth0_s_New_Private_Cloud_Platform)

## We’re Just Getting Started

Although these sample scenarios aren’t terribly complex, they really show the power of composability. We can mix and match a few Kafka connectors with Data Shield to meet the needs of our customers while providing the data controls we require. With more data-intensive use cases on the horizon needing stream processing and OLAP databases, this composability means our pipelines can easily evolve to support them. More importantly, even with a team experienced in stream processing and OLAP databases, we’ll still plan an R&D phase to make sure our choices once again align with our guiding principles.

Interested in hearing more about streaming joins, windowing, and OLAP databases? Well, you’ll have to wait for my next blog post then! Muahaha.

I would be remiss if I did not tell you that Auth0 (and specifically my team) are [hiring](https://auth0.com/careers)! Join me in building the next Auth0 data platform solving real problems for real people. Interested in solving problems for fake people instead? Visit the social network nearest you.

Data Pipelines on Auth0’s New Private Cloud Platform

mayank

Guest Author

Mayank Raj

Learn how to use Jenkins to add a Continuous Integration (CI) pipeline to your applications.

Managing Continuous Integration Pipelines with Jenkins

idee

Software Engineer building with Kotlin and Flutter.

Software Engineer at Busha

Idorenyin Obong

How to configure a Continuous Deployment pipeline utilizing Now.sh, GitHub, and Travis CI to automate your open-source Node.js web apps.

Continuous Deployment Pipelines and Open-Source Node.js Web Apps

99c4080f412ccf46b9b564db7f482907?s=200 avatar

99c4080f412ccf46b9b564db7f482907?s=200

I'm currently a Senior Developer Advocate at MongoDB. I'm also a Google Developer Expert for Web Technologies and spend most my time giving talks at conferences and meetups, mentoring developers and running workshops, and creating online content to help technology professionals.

Former Auth0 Employee

Ado Kukic

Day two of ng-conf 2016 was Fair Day. We got to deep dive and learn more about what's in the pipeline for Angular 2.



Day two of ng-conf was titled **Fair Day**. Fair day consisted of a brief keynote in the morning and then a series of breakout sessions and workshops allowing developers to deep dive into all the new technologies announced on day one. Check out our [recap of day one](https://auth0.com/blog/2016/05/05/ng-conf-summary-day-1/) to learn about these technologies which include Angular Universal, Material, CLI, Augury and Mobile Toolkit. Day 1 was all about big announcements, [day three](https://auth0.com/blog/2016/05/07/ng-conf-summary-day-3) went hands-on on how to work with Angular 2 and associated projects.

## The Future of Angular 2

It may be a bit early to discuss the future of Angular 2 when it's not even out yet officially, but a key theme of today's keynote was the future of Angular 2. Angular offline compiling, advanced Material Design components and effortless upgrades for the future were discussed.

<include src="TweetQuote" quoteText="It's early to discuss the future of Angular 2, but the theme of the day 2 keynote was looking ahead."/>

### Offline Compiler

Offline compiling is what allows the Angular 2 library to become smaller than Angular 1. This is done through a custom compiler that the Angular team built, which is invoked through the `ngc` command once installed, which examines your apps code and does tree shaking to remove any unnecessary modules. The ng-compiler is one of the key features of Angular 2 that is required before Angular 2.0 can be delivered but as of today it is not ready.

### Advanced Material Components
The Material Design team gave a brief talk about the future of Material Design with Angular 2. The team has already committed to parity with the existing Material 1 for Angular 1.x, but talked about future potential with components that could include components such as `<md-map>` which would embed a Google Map or `<md-chart>` which would be used for charting UI. The coolest potential component discussed was `<md-text-editor>` which would create a WYSIWYG text-editor which have traditionally been difficult to implement, especially with Angular 1 applications.

### Seamless Upgrades

Finally, the team discussed effortless future upgrades and even joked about Angular 3. The goal here is to give developers a seamless path to upgrading once future 2.x versions of Angular are released. The Angular team is already doing extensive testing to ensure that any change to the core Angular libraries does not break compatibility with existing modules. 
 
## Angular 2 Workshops
  
There were plenty of Angular 2 workshops that took a deep dive into some of the technologies announced on day one of the keynote. Ionic had a great presentation on building hybrid mobile apps with Ionic 2 and Angular 2. The Telerik team took to the stage again and gave a detailed walkthrough of how to build native mobile applications with Angular 2 and NativeScript. Sean Larkin did a great workshop on integrating Angular 2 with Webpack.
 
 Auth0 hosted a workshop on integrating JSON Web Token (JWT) authentication with both Angular 1.x and Angular 2. The workshop highlighted the benefits of JWT authentication as well as differences in implementation between the two frameworks. If you would like to learn how to implement JWT authentication in your app check our guide for both [Angular 1](https://auth0.com/docs/client-platforms/angularjs) and [Angular 2](https://auth0.com/docs/client-platforms/angular2). If you don't already have an Auth0 account, <a href="https://auth0.com/signup" data-amp-replace="CLIENT_ID" data-amp-addparams="anonId=CLIENT_ID(cid-scope-cookie-fallback-name)">sign up</a> for free and get started today.
 
## Conclusion - Day 3 Preview
 
 Tomorrow is the final day of ng-conf 2016. TypeScript, Animations, React Native and AngularFire are just some of the talks in store. We can't wait to learn more about Angular 2 and will be sure to give a detailed summary of day three. You can follow the live stream tomorrow on the ng-conf official [website](https://www.ng-conf.org/). 

"pipeline"