deepu-sasidharan

Deepu is a polyglot developer, Java Champion, and OSS aficionado. He mainly works with Java, JS, Rust, and Golang. He co-leads JHipster and created the JDL Studio and KDash. He's a Staff Developer Advocate at Okta. He is also an international speaker and a published author.

Staff Developer Advocate

Deepu K Sasidharan

Discover how AI agents are taking center stage by seamlessly integrating with multiple digital tools like Gmail, Calendar, Slack, and Google Drive to be more efficient. Learn why it is important to secure them and how to do it.

**AI agents** or **Operators** are the latest trend in AI. At their core, agents are autonomous systems that not only process natural language using Large Language Models (LLMs) but also decide when and how to act by interfacing with external tools and APIs. These smart entities blend machine learning with decision-making capabilities, enabling them to perform complex tasks, interact in real-time, use the most efficient tools to perform certain tasks, and even learn from their environment—all without constant human intervention.

<AmpContent>
<amp-youtube
    data-videoid="r7wEinUG0ns"
    layout="responsive"
    width="480" height="270">
</amp-youtube>
</AmpContent>

<NonAmpContent>
<div
  className='embed-container'
  style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%;margin-bottom:40px;">
    <iframe
      style="position: absolute; top: 0; left: 0; width: 100%; height: 100%;"
      src='https://www.youtube.com/embed/r7wEinUG0ns'
      frameborder='0' allowfullscreen>
    </iframe>
</div>
</NonAmpContent>

## Tool Calling Agents

LLMs on their own are still susceptible to hallucinations and mistakes, but the ability to call tools opens up a whole new world of possibilities where error-prone tasks can be offloaded to a more traditional non-AI tool, like using a calculator to do math rather than relying on the LLM for it. Similarly, tool calling allows AI agents to interface with other applications and services, like Gmail, Calendar, Slack, and Google Drive, to do more for you than just answering questions.

Tool calling can be generally classified into these:

1. **Unauthenticated Tools:** These are tools that don't require authentication and are generally simpler to configure, like a calculator, a weather API, or a unit converter.
2. **Authenticated Tools:** These are tools that require authentication and are generally complex to configure, such as an API provided by another internal application or services like Gmail, Calendar, Slack, and Google Drive.

In an enterprise environment, the authenticated tools can be further classified into:

1. **First-Party Tools:** These are services that are part of the same infrastructure and require authentication, like an API provided by another internal application or a microservice.
2. **Third-Party Tools:** These are external services that require more complex authentication. They could be services that are authenticated using service accounts or API keys, like a payment API, or services that need to be authenticated by the end user, such as Gmail, Calendar, Slack, and Google Drive.

Tool calling provides many key advantages to an AI agent:

- **Autonomous Decision-Making:** AI agents analyze context and determine the next best action, reducing the need for manual oversight.
- **Optimized Task Execution:** They can connect to various services and APIs, enabling them to perform specialized tasks like data retrieval, scheduling, and communication.
- **Scalability and Adaptability:** As systems evolve, agents can easily incorporate new functionalities, adapting to changes in workflow or priorities.
- **Improved User Experience:** Their ability to tackle diverse tasks leads to more personalized and efficient interactions, making technology feel both intuitive and proactive.

## Tool Calling Agent: A Conceptual Example

Imagine an AI personal assistant that consolidates your digital life by dynamically accessing multiple tools to help you stay organized and efficient. Here’s how it could work:

1. **Gmail Integration:** The assistant regularly scans your inbox to generate concise summaries. It highlights urgent emails, categorizes conversations by importance, and even suggests drafts for quick replies.
2. **Calendar Management:** By interfacing with your calendar, it can remind you of upcoming meetings, check for scheduling conflicts, and even propose the best time slots for new appointments based on your availability.
3. **Slack Notifications:** For team communications, the assistant monitors Slack channels. It identifies key messages and creates action items, ensuring you never miss an important update from your colleagues.
4. **Google Drive Access:** Whether you need immediate access to the latest project document or a file related to a current task, the assistant retrieves pertinent documents from Google Drive on demand. It can create document summaries and even create new documents based on your instructions.

With tool-calling capabilities, the possibilities are endless. In this conceptual scenario, the AI agent embodies a digital personal secretary—one that not only processes information but also proactively collates data from connected services to provide comprehensive task management. This level of integration not only enhances efficiency but also ushers in a new era of intelligent automation, where digital assistants serve as reliable, all-in-one solutions that tailor themselves to your personal and professional needs.

## Security Challenges with Tool Calling AI Agents

Building such an assistant is not that difficult. Thanks to frameworks like [LangChain](https://www.langchain.com/), [LlamaIndex](https://www.llamaindex.ai/), and [Vercel AI](https://vercel.com/ai), you can get started quickly. The difficult part is doing it securely so that you can protect the user's data and credentials.

Many current solutions involve storing credentials and secrets in the AI agent application’s environment or letting the agent impersonate the user. Storing credentials securely in the environment might work for APIs that require API keys or service accounts. However, this is not a good idea for cases where the end users need to authenticate to the service, as it can lead to security vulnerabilities and excessive scope and access for the AI agent.

<br>

<div class="alert alert-info alert-icon">
  <i class="icon-budicon-487 icon-info"></i>
  <b>To learn more about other security concerns and vulnerabilities in AI check out this blog post on <a href="https://auth0.com/blog/identity-challenges-for-ai-powered-apps/">Identity Challenges for AI-Powered Applications</a>.</b><br> 
</div>

When you build an AI agent, you need to consider all its other security implications as well. You don't want an LLM to have unlimited access to your personal data like email and documents, and more importantly, you don't want to provide your credentials to the LLM to access these tools. Let's face it, regardless of how secure the LLM is, there is always a possibility of it getting manipulated into divulging sensitive information or doing something you don't want it to do.

## Tool Calling with the Help of Auth0

This is where Auth0 comes to the rescue. As the leading identity provider (IdP) for modern applications, our upcoming product, [Auth for GenAI](https://auth0.com/blog/auth-for-genai/), provides standardized ways built on top of OAuth and OpenID Connect to call APIs of tools on behalf of the end user from your AI agent.

Auth0 brokers a secure and controlled handshake between the AI agents and the services you want the agent to interact with on your behalf – in the form of scoped access tokens. This way, the agent and LLM do not have access to the credentials and can only call the tools with the permissions you have defined in Auth0. This also means your AI agent only needs to talk to Auth0 for authentication and not the tools directly, making integrations easier.

### Call first-party APIs on users' behalf

When your AI agent, secured with Auth0, needs to call another application also secured with Auth0, you can use standard OAuth 2.0 flows, like the [Authorization Code Flow](https://auth0.com/docs/get-started/authentication-and-authorization-flow/device-authorization-flow), to get an API token from that application with user consent. In this case, Auth0 will manage the retrieval of new access tokens using cached refresh tokens when the API token expires.

### Call third-party APIs on users' behalf

When your AI agent, secured with Auth0, wants to call external services like Gmail, Calendar, Slack, and Google Drive, Auth0 can help the agent get access tokens for the external service on behalf of the end user. In this case, Auth0 brokers the API access token from the external service to the AI agent.

![Tool calling with Federated API token exchange](https://images.ctfassets.net/23aumh6u8s0i/1gY1jvDgZHSfRloc4qVumu/d44bb7102c1e858e5ac64dea324478fe/tool-calling-with-federated-api-token-exchange.jpg)

This is made possible by Federated API token exchange, which is a way to obtain an access token from an external identity provider without the need for the user to re-authenticate every time. The end user authenticates and connects the external service once, and Auth0 intermediates the authentication and authorization process and provides the API access token to the AI agent. Auth0 takes care of storing refresh tokens and getting new access tokens when the current access token expires.

## Security Best Practices for Tool Calling AI Agents

Here is a summary of security best practices to consider when building a tool-calling AI agent:

- **Least Privilege**: Grant the minimum permissions needed (e.g., read-only access).
- **Audit Logs**: Track API calls and credential usage.
- **Proxying**: Use an IdP like Auth0 as a broker between the AI agent and the tools to protect the user's credentials.
- **Encryption**: Encrypt credentials at rest and in transit (HTTPS/TLS).
- **Token Rotation**: Regularly rotate API keys and tokens.
- **Sandboxing**: Run tools in isolated environments when possible to limit damage from leaks.

## Learn More about AI Agents and Auth for GenAI

This post offers a glimpse into the transformative potential of AI agents and highlights how secure and intelligent tool calling can empower AI agents to deliver a more connected and personalized experience. Stay tuned for a series of posts on tool-calling Agents with [Auth for GenAI](https://a0.to/ai-content), where we will build the conceptual personal assistant from the previous section in Node.js and Python, showing how to integrate with different types of services securely using your favorite AI frameworks and Auth0.

Before you go, we have some great news to share: we are working on more content and sample apps in collaboration with amazing GenAI frameworks like [LlamaIndex](https://www.llamaindex.ai/), [LangChain](https://www.langchain.com/), [CrewAI](https://www.crewai.com/), [Vercel AI](https://vercel.com/ai), and [GenKit](https://firebase.google.com/docs/genkit). [Auth for GenAI](https://auth0.com/blog/auth-for-genai/) is our upcoming product to help you protect your user's information in GenAI-powered applications. Make sure to join the [Auth0 Lab Discord server](http://a0.to/auth0-lab-discord) to hear more and ask questions.

Tool Calling in AI Agents: Empowering Intelligent Automation Securely

Ramona-Schwering_square

Ramona, a Developer Advocate at Auth0, bridges roles as a tester and developer. She's a Google Dev Expert in Web Tech, Women Techmaker Ambassador, and Cypress Ambassador, building trust in test automation and identity topics.

Developer Advocate

Ramona Schwering

Many applications utilize Auth0 in their login procedures. However, how do you deal with that in testing? Let's learn how to cover the default login process in test automation with Cypress.

In this post, you’ll explore leveraging the Cypress testing framework to enhance security and user experience. This article is the first post of a series, which will closely examine one of the most crucial aspects: the login form and the standard login procedure.

## Why Do We Need to Mind Auth0 in Testing?

You might wonder why you should write login tests. After all, you log in to your application daily. In addition, evem more significant, Auth0's Univeral Login, thus its Login form, is already tested by the responsible teams, right? 

Well, while that's true, it's important to mind the Auth0 integration inside your application - so you're confident it's working with your application's code flawlessly. Moreover, testing the login functionality is crucial for quality assurance due to its frequent use and the significant damage that login failures can cause. Everything protected behind the login needs to be covered by tests, too. And to reach those parts of your application, you need to get through authentication via your tests as well. 

> Disclaimer: When testing Auth0’s functionality and running automated tests in a pipeline, be aware that you might use actual users, affecting your plan’s [monthly active users (MAUs)](https://community.auth0.com/t/understanding-monthly-active-users/116379). This fact is particularly relevant when registering users via automated tests. Use one dedicated test user to avoid creating too many test users and consuming your plan’s capacity.

### API Testing and Password Grant 

So, we agree that we need a solution to test the login—or at least to have a command to log in to test the behavior of your application afterward. You can imagine that this process of logging in is something you’ll do a lot when writing tests for your application, correct? In the usual case, I’d recommend including repetitive steps, such as logging in inside a [custom command](https://docs.cypress.io/api/cypress-api/custom-commands) and avoiding clicking through it, to save time. Cypress advocated for it as well. As a result, in such cases, I’d normally advocate **logging in via API and avoiding using UI through such shortcuts**. An excellent example of this approach is provided by [Cypress itself](https://docs.cypress.io/guides/end-to-end-testing/auth0-authentication#Programmatic-Login). 

However, this recommendation would require using the Password grant, which is no longer defined as OAuth 2.1, as OAuth discourages its use. These changes have significant implications for API testing, as it may no longer be a practical approach. Despite the potential time investment, we are advised to use the UI method, as we aim to avoid endorsing practices disapproved by OAuth.

## Setting Up Your Example Project

For this blog post, you'll use Cypress as the framework for your examples because it offers a robust, user-friendly solution for end-to-end testing. With its real-time reloading, detailed error messages, and intuitive API, Cypress simplifies the process of writing and running tests. Its ability to operate directly in the browser provides a clear view of how your application behaves in real-world scenarios, making it an excellent choice for ensuring your login functionality is reliable and secure. One critical approach is guiding Cypress through login forms, which is the typical method for testing this crucial feature. Fortunately, Cypress provides built-in support for handling such tests, making the process straightforward and efficient.

To help test login functionality, I recommend using a GitHub repository that hosts a clean Cypress project on the main branch. I prepared a [sample repository](https://github.com/oktadev/auth0-cypress-sample) for this blog post, which you can use for reference. 

This project is built using Cypress Version 13, ensuring you have access to the latest features and improvements. You can set up the project by cloning the repository and following the setup instructions provided, making it easy to get started with your login testing.

### Choosing a Test Base

To write a test, you naturally need some test base—something to write a test against. For this blog post, I decided to use the demo app for our virtual developer day event—[dev_day](https://developerday.com/). It took place on September 24 and offered a variety of talks, games, and other sessions, especially [labs](https://developerday.com/events/devday24/labs). Those labs are a workshop-like format, actually a guided experience to teach you Auth0's features. 

![Dev_day Demo App](https://images.ctfassets.net/23aumh6u8s0i/5uKOqkX3aDOvdODdNxMkqQ/758129b7f53b3235d9676d20b4e33bdb/Dev_day_Demo_App.png)

For those labs, we provide a Demo App showcasing all its features, which makes it perfect as a test base for our blog post.

<div className="alert alert-info alert-icon">
  <i className="icon-budicon-487"></i>
  <strong>Our Demo App's Tech Stack:</strong><br> 
The Demo app, built in React, uses NextJS to power  Frontend and Backend. The application is kept as simple as possible, e.g., by running a database based on JSON files. Of course we'll use Auth0 inside this application, together with [OpenFGA](https://openfga.dev/) <br> 
</div>

## The Standard Way of Testing Login UI

When it comes to testing your login processes, whether you’re using Auth0 or not, what comes to mind? It’s probably filling out a login form and submitting it, as this is how you’re used to login and, thus, testing it manually while coding. It might not be the most efficient way of automated testing, but let’s use it as our first example. Spoiler alert: there will be a more efficient way, but it’s essential to understand the gist of it. 

As a result, before diving into a "default" login test, let's define what we mean by "default." The typical login process consists of a form containing two input fields: an email address or username and a password. These credentials are single-factor authentication. They are sent to the server for verification; the user is granted access if successful. This is what we consider the "default" login. In testing, the user is replaced by a computer in a typical end-to-end test that clicks through a login form as a human would.

<div className="alert alert-info alert-icon">
  <i className="icon-budicon-487"></i>
  <strong>Hint:</strong><br> 
When covering a login form with tests and running them in a pipeline, know rate limits to avoid triggering them and hindering your testing efforts. Mocking responses or increasing the delay between login attempts can help circumvent this issue.<br> 
</div>

Let's start writing the test! Like any Cypress test, you'll begin with a basic test skeleton as a foundation. The following code snippet defines a test suite using Cypress to verify your application's default login flow.

```javascript
describe('Default Login Flow', () => {
    it('logs in', () => {
        cy.visit('/');
        cy.contains('Log in').click();
        cy.get('input#username').type(username);
    })
})
```

This snippet outlines a basic login flow test, ensuring the login button can be clicked and the username can be entered into the appropriate field. However, this marked line won’t work because Auth0 changes to a new domain, requiring [multi-domain testing](https://www.cypress.io/blog/cypress-9-6-0-easily-test-multi-domain-workflows-with-cy-origin) for Cypress to proceed. This kind of testing allows you to handle authentication flows that span multiple domains, which is helpful for modern web applications with third-party authentication providers.

First, set environment variables to work with Auth0’s credentials more comfortably. For this, we’ll use [dotenv](https://www.npmjs.com/package/dotenv): Install `dotenv`, and create a `.env` file in your root folder, as follows:

```
AUTH0_USERNAME=exampleusername
AUTH0_PASSWORD=examplepassword
DOMAIN=exampledomain
CLIENT_ID=examplecliendid
```

Then, you can use your environment variables and set them in `cypress.config.ts` as follows:

```javascript
import { defineConfig } from 'cypress';
import dotenv from "dotenv";

dotenv.config({ path: `.env.local` });

export default defineConfig({
    env: {
        auth0_username: process.env.AUTH0_USERNAME,
        auth0_password: process.env.AUTH0_PASSWORD,
        auth0_domain: process.env.DOMAIN,
        auth0_client_id: process.env.CLIENT_ID,
    },
    e2e: {
        // other configurations
    }
});
```

After these steps, you’re ready to start! As login steps are probably used in multiple tests, using Custom Commands to avoid duplication and keep code clean is a good practice. Create the custom command as follows:

```javascript
Cypress.Commands.add('loginToAuth0', (username, password) => {
    cy.origin(
        Cypress.env('auth0_domain'),
        { args: { username, password } },
        ({ username, password }) => {
            cy.get('input#username').type(username);
            cy.get('input#password').type(password, { log: false });
            cy.contains('button[value=default]', 'Continue').click();
        }
    );
    cy.url().should('contain', Cypress.config('baseUrl'));
});
```

Next, you'll write the custom command - where the magic's happening. 🪄 Therefore, go to `cypress/support/e2e.ts`. In it, let's create the following command stub:

```javascript
Cypress.Commands.add('loginToAuth0', (username: string, password: string) => {
  // Your custom command's steps go here...
})
```

The `cy.origin` command is used for [multi-domain testing](https://docs.cypress.io/api/commands/origin). This command will enable Cypress to handle the switch to the domain handling the login, even if it's the default Auth0 one. Let's include it inside your custom command:

```javascript
Cypress.Commands.add('loginToAuth0', (username: string, password: string) => {
  // Origin for Multidomain
  cy.origin(
    Cypress.env('auth0_domain'),
    { args: { username, password } },
    ({ username, password }) => {
      // There goes the login part
    }
  );

  cy.url().should('contain', Cypress.config('baseUrl'));
})
```

This way, we can treat the login page like our own domain - so we can make Cypress interact with it as we're used to. So finally, let's continue with our login steps we tried before:

```javascript
Cypress.Commands.add('loginToAuth0', (username: string, password: string) => {
  // Origin for Multidomain
  cy.origin(
    Cypress.env('auth0_domain'),
    { args: { username, password } },
    ({ username, password }) => {
      cy.get('input#username').type(username);
      cy.get('input#password').type(password, { log: false });
      cy.contains('button[value=default]', 'Continue').click();
    }
  );

  cy.url().should('contain', Cypress.config('baseUrl'));
})
```

Returning to your test file, use the new custom command and add assertions to check if the test logs in correctly. Here’s an example:

```javascript
describe('Default Login Flow', () => {
    it('logs in', () => {
        cy.visit('/');

        // Make sure the Login button is completely rendered and visible before interacting
        cy.get('.bg-secondary').should('be.visible');
        cy.get('.bg-secondary').click();

        cy.loginToAuth0(
            Cypress.env('auth0_username'),
            Cypress.env('auth0_password')
        );

        // Validate if the user is completely logged in
        cy.get('[href="/my-courses"]').should('be.visible');
        cy.get('.aspect-square').should('be.visible');
    });
});
```

Did you notice? I added some more assertions to make the test more stable. I'll assert on things a real user would pay attention to:
* You make sure the Login button is wholly rendered and visible before interacting to avoid flakiness
* At the end of the test, you assert two elements that are only visible if you're logged in.

Now, let's run our test and see if it's passing. 🥁

![Test is passing](https://images.ctfassets.net/23aumh6u8s0i/43XLZDtl4CX61ew9YZFdLI/21a78acffd2af12a9444dcb89bae9045/Test_is_passing.png)

That’s it! You've written a multi-domain test covering a default login process with Auth0. 🚀

### Common Issue: Hydration Errors in Cypress

As of September 2024, there's still an open [GitHub issue](https://github.com/cypress-io/cypress/issues/27204) on an error I encountered myself when working with Cypress and our dev_day Demo App:

```bash
(uncaught exception)Error: Hydration failed because the initial UI does not match what was rendered on the server. See more info here: https://nextjs.org/docs/messages/react-hydration-error.

(uncaught exception)Error: There was an error while hydrating. Because the error happened outside of a Suspense boundary, the entire root will switch to client rendering.
```

This error is thus a known issue in Cypress, and we'll adjust the test to include a workaround to execute our test. It's partly based on [this response](https://github.com/cypress-io/cypress/issues/27204#issuecomment-1894217901), providing a test-wise workaround. You start by adding the following code snippet to your `cypress/support/e2e.ts` file:

```javascript
Cypress.on("uncaught:exception", (err) => {
  if (
    /hydrat/i.test(err.message) ||
    /Minified React error #418/.test(err.message) ||
    /Minified React error #423/.test(err.message)
  ) {
    return false;
  }
});
```
<div className="alert alert-danger alert-icon">
  <i className="icon-budicon-487"></i>
  <strong>Hint:</strong><br> 
It's important to mention again that it's a workaround and not an ideal one, as it conceals an error message completely - thus, it could also hide valid errors. As soon as the issue is fixed, it's highly recommended that this workaround be removed.<br> 
</div>

After applying the snippet to `cypress/support/e2e.ts`, there are still some edits we need to do in our test itself: 
We need to wait for the error to be present. This way, we time the test execution just right to have the click on the right time, targeting this error.

> (uncaught exception)Error: Hydration failed because the initial UI does not match what was rendered on the server.

To get the timing right, you can intercept the error's request and wait for it just before you click "Log In": 

```javascript
    it('logs in', () => {
      cy.intercept('/oauth/token').as('token');

      // Intercept the error message's request
      cy.intercept('/__nextjs_original-stack-frame?*').as('nextjs');
    
      cy.visit('/');

      // Wait for the error message to pop up
      cy.wait('@nextjs').its('response.statusCode').should('equal', 200);

      // Proceed with your test...
    });
```

## Some Words on Mocking

So, is there only the possibility of using UI-based end-to-end (e2e) tests, you might wonder? This does look very bleak, doesn't it? You could look at another alternative when working with third-party dependencies inside your test, including Auth0. Right, I'm talking about mocking.

Mocking in e2e tests is a practice that isolates the system under test from external dependencies. It ensures your tests run reliably, quickly, and independently of external services. However, it has a significant downside: Overuse of mocks in e2e tests could defeat the purpose of end-to-end verification by not truly reflecting real-world scenarios. Because of that, a balanced approach is vital. You need to weigh those options accordingly; it always depends on your project to decide if mocking is okay. I might answer this question in a later article regarding mocking Auth0.

## Recap

That's it. You wrote a test on Auth0's login form and workflow. Well, to be more precise, you've learned...
* ... why it's essential to consider your login inside your test, even if covered by a third party like Auth0 or you'll use it all the time
* ... About the discouragement of using password grants and the implications on API testing
* ... what Multi-Domain Testing is and how to utilize it to get through Auth0's login workflow 
* How to deal with Hydration issues inside your Cypress test in case they arise

Thanks for reading, and may your tests always stay green. 💚

How to Cover Auth0’s Login Form with Tests

Samuel Frank

Sam Frank is Auth0's Sr. Product Manager for new initiatives. He works within the Auth0 Lab to commercialize the team's new ideas so they're available for builders everywhere. As a former platform product manager and startup product lead, he knows the challenges that developers face when creating new apps and he leverages those experiences to focus our teams efforts delivering the next generation of identity tools.

Senior Product Manager

DataGrail’s Auth0 Marketplace Integration is ready to solve Data Subject Requests (DSRs)

Auth0 has done the research to help you keep up with changing privacy regulations. Our docs and blogs provide the overviews you need to understand the [General Data Protection Regulation](https://auth0.com/gdpr) (GDPR), [California Consumer Privacy Act](https://auth0.com/ccpa) (CCPA), [compliance](https://auth0.com/docs/secure/data-privacy-and-compliance), [data privacy](https://auth0.com/blog/data-privacy-101/), and [consent management](https://auth0.com/resources/whitepapers/consent-management/) topics.

To keep up with evolving privacy regulations, the [Auth0 Lab team](https://twitter.com/auth0lab) actively explored how to help organizations deal with [data subject access requests (DSARs)](https://www.datagrail.io/glossary/data-subject-access-request-dsar/) in 2021. The more we researched, the more we realized that solving this problem was complex. Shortly after the [acquisition of Auth0](https://auth0.com/blog/okta-acquisition-announcement/), [the Okta Ventures team](https://www.okta.com/okta-ventures/) introduced us to [DataGrail](https://www.datagrail.io/) which is a portfolio company and who already had a DSAR solution ready to deploy.

## The Complexity of Data Subject Requests

The European Union (EU) Commission was tasked with unifying multiple approaches to data privacy. This need to find a solution that would work for all the countries within the EU helped drive a simplified set of regulations for data ecosystems under the GDPR. A clear dichotomy emerged between a data controller and a data processor. An organization that determined how and why to use personal data (a “controller”) would hire a company to process personal data on its behalf and under its instructions (a “processor”). If an individual wanted to exercise a DSAR, that individual would simply ask the controller about what personal data they had, to delete their personal data, or another request about their personal data, and the controller would just send that request to the processor to obtain the information and share it with the individual.

![Data Subject Requests](https://images.ctfassets.net/23aumh6u8s0i/7eRRavWS0Jd5Fvj5M6hfiP/a66842e40c97dc7e3b761b648a58b09f/01.png)

Easy! Except only the simplest organizations have a technology stack that is this uncomplex.

In reality, the problem is far more complex. An organization might have data stored in application databases, CRMs (customer relationship management), logfiles, ad placement platforms, and a host of other business applications. So what seems like a simple relationship turns into a mess. Here’s what an example could look like (these relationships don’t necessarily exist).

![Data Subject Requests](https://images.ctfassets.net/23aumh6u8s0i/4rueDMrZ8PBEEFmN6QWfCs/c12f5761cc4c38de2e430e98ebfb2655/02.png)

We have spoken to organizations with data ecosystems that were far more complicated than this!

Suddenly, a DSAR could involve a dozen applications and the administrators of each. A privacy operations professional needs to keep up-to-date with the various applications used across an organization to make sure personal data is accessed or removed from each. Failure to respond to a single request could result in a massive penalty for an organization, as this example illustrates:

![DSAR](https://images.ctfassets.net/23aumh6u8s0i/3VVuK1C6DADnasBMFmxPS9/ec5cf821128663da4c3f88294cd7e9d6/03.png)

## The Solution, a Data Privacy Platform like DataGrail

DataGrail is a data privacy platform that maps where Personally Identifiable Information (PII) lives across your organization and then tracks, automates, and delivers DSRs. We use DataGrail’s Request Manager at Auth0 and Okta, and it helps us manage requests across our own complex marketing stack.

### How it works

When built with intention, holistic privacy programs ensure consumers are able to receive transparent and timely responses about their personal data, while internal teams can produce a single audit trail and avoid extensive manual effort. DataGrail takes four primary steps to create both outcomes:

1. **DSR forms.** DataGrail provides branded, user-friendly, accessible forms that can be embedded directly on your website. These forms are simple and comprehensive, so consumers spend less time submitting their DSRs, and internal teams receive all the information needed to fulfill the request quickly. Here’s [Okta’s form](https://preferences.okta.com/privacy/) powered by DataGrail.

2. **Identity verification.** Before submissions make their way to our team, DataGrail’s Smart Verification tech checks to ensure the request isn’t fraudulent, so personal data stays private until the individual verifies their identity. Through the DataGrail integration, customers can configure their setup to leverage personal data that is already held within Auth0 to authenticate a submitter’s identity, so they don’t have to volunteer additional personal information like IDs passports, or photographic evidence.

3. **Request automation.** All DSRs are automatically and centrally organized. Whether the incoming request specifies a desire for access, deletion, or to update personal data, DataGrail connects to tools across your organization (internal and third party) so that you can avoid managing manual processes in a spreadsheet. The direct connection to your tools helps avoid security risks and gaps that are introduced by manual efforts.

4. **Audits and compliance.** DataGrail’s dashboard tracks the full request lifecycle in one place. You can see up-to-date logs, ensure requests are tracked and enable consumers to choose to receive their DSRs in PDF and other machine-readable formats. 

DataGrail makes it possible for consumers to effortlessly and safely request their personal data while reducing the manual efforts by our operational teams in carrying out DSRs.

### Auth0 Marketplace Integration

You can find the DataGrail marketplace integration here: [https://marketplace.auth0.com/integrations/datagrail](https://marketplace.auth0.com/integrations/datagrail). You can set it up in as little as 15 minutes!

### 1. Add a DataGrail Application

Head over to your [Applications in your management dashboard](https://manage.auth0.com/#/applications) and create a new application.

![DataGrail](https://images.ctfassets.net/23aumh6u8s0i/uLnbm41DaFmmR2GIARnHp/e4ccf55034e120209b1abff9c93d2d4f/04.png)

Name it something that represents the integration and select “Machine to Machine Applications.”

![Machine to Machine Applications](https://images.ctfassets.net/23aumh6u8s0i/3nbS4aSV0KsiUyVxMKTqFS/8d94e2396e7aff3f82049a8ca86843d6/05.png)

### 2. Authorize Access

The next step in the flow will prompt you to select which API and scopes will have access. Select the Auth0 Management API, and select the following:

![Authorize Access](https://images.ctfassets.net/23aumh6u8s0i/1hPwPiemNgvGK58mD52V5u/7f38b1297a639b3318b020d94bc95917/06.png)

### 3. Configure Application and Gather Connection Data

In the application screen for your new application, head to the settings tab, and then scroll down to Application URIs. 

![Gather Connection Data](https://images.ctfassets.net/23aumh6u8s0i/1kL3x4fq5TTabikQNpvOlB/278fdbf08d58b096d97b1ebbeb3f2a25/07.png)

Add [https://datagrail.com/oauth/auth](https://datagrail.com/oauth/auth) to the “Allowed Callback URLs”:

![Callback URL](https://images.ctfassets.net/23aumh6u8s0i/5Oq4P7CxSeR73K3GJnhGif/42ca0318316b7c0c9f2948c4186dab38/08.png)

Save changes and move to the Basic Information form.

Here you’ll find the necessary information to enter into your DataGrail integration page.

![DataGrail](https://images.ctfassets.net/23aumh6u8s0i/23zl9z58BMEM4bgZnX2pxI/bfc9659dc0f707b3dea389884e7670d0/09.png)

### 4. Enter Necessary Information in DataGrail

   1. Return to the DataGrail Portal (window should be open in a separate tab).
   2. Click **Connect** within the Integration page for Auth0.
   ![Auth0](https://images.ctfassets.net/23aumh6u8s0i/ujn8bo54AkBkQ5SS5deZn/4d5dffda3b92aab198eef286981abb81/10.png)
   3. Enter **_Subdomain_**, **_Client ID_** and **_Client Secret_** obtained from **Step 1.**
   ![Client ID](https://images.ctfassets.net/23aumh6u8s0i/2TteGBUGRfhgY9FF9YKwL7/5d3d350364ac37d246d53c68ec5e5dbe/11.png)
   4. Click **Connect Auth0**
   ![Connect Auth0](https://images.ctfassets.net/23aumh6u8s0i/7acBkw7IwxYhNcz6chdaK8/e0480ea75af593fdc3d1c1153eb8b60d/12.png)
   5. If there are additional Auth0 accounts to integrate, follow these instructions: 

   a. Click **Edit Connection** within the Integration page for Auth0.
   ![Auth0](https://images.ctfassets.net/23aumh6u8s0i/zOcQPCDx8d6jPXrc4HAuQ/237717bb3c2e27f17a828da2548816cd/13.png)
   b. From the drop-down, select **+Add New Connection.** <br>
   c. Under **Connection Name**, enter a new name to identify this separate account (ex: _Auth0 Training Account_). <br>
   d. Repeat **Step 1** to copy **_Subdomain_**,_ **Client ID **_and** _Client Secret_ **for this new account. <br>
   e. Enter **_Subdomain_**,_ **Client ID **_and** _Client Secret_**. <br>
   f. Click **Connect.** 
   ![Connect](https://images.ctfassets.net/23aumh6u8s0i/1SKFhcCzxlrhrN5Yhnf55y/2375424f747946f77fbd44404f3bd48d/14.png)

   6. In the DataGrail Portal, if you have any other integrations not connected (no green check mark), please click **Connect** to complete the configuration and reference the Help Docs link for instructions

You can always email DataGrail for help at [support@datagrail.io](mailto:support@datagrail.io) with any issues or questions regarding your integrations.

### Connecting with Auth0 and Okta Ventures

DataGrail is integrated with multiple teams at Okta. You can start your partnership journey by reaching out to any one of these teams:

The [Auth0 Marketplace](https://marketplace.auth0.com/) is always looking for partners to build new extensions onto Auth0. Visit [https://auth0.com/integrate](https://auth0.com/integrate) to learn more.

The [Okta Ventures](https://www.okta.com/okta-ventures/) team is actively seeking new innovative organizations. You can apply for funding here: [https://www.okta.com/okta-ventures-application](https://www.okta.com/okta-ventures-application/). 

You can follow the research and experiments of the Auth0 Lab team on [Twitter](https://twitter.com/auth0lab) and start a conversation with us on [Discord](https://discord.gg/xkbkz9rZcD).

<include src="asides/AboutAuth0" />

Privacy Automation is an Integration Away

facundo-giuliani

Facundo is a Systems Engineer from Buenos Aires, Argentina with 15+ years of experience in software development. Full Stack Developer. Auth0 Ambassador. GitKraken Ambassador. Cloudinary Media Developer Expert. He is also an open-source contributor.

Systems Engineer, Full Stack Developer

Facundo Giuliani

Besides surfing the web, you can use web browsers programmatically to automate tasks. Learn how to drive a Chromium browser using Puppeteer Sharp, a browser automation library for C#.

One of the goals every developer has is to automate as many actions and tasks as possible. Some of the reasons why we should try to automate our workflows are:

- We save work time.
- We depend less on human assistance.
- We make things faster.
- We spend less money.
- We get consistent results over iterations.

Besides the benefits that automation brings us, there are some scenarios that couldn't be achieved if we don't execute an automatic process. One example of this is the dynamic generation of files and images. In this article, we will see how to implement automatic workflows interacting with a web browser instance.

## Browser Automation

"Browser automation" is the action of automating the usage of a web browser. Its purpose is to achieve efficiency and speed at a certain level that wouldn't be possible if we rely on human interaction.

With a browser automation tool, we can simulate interactions using input devices (keyboard, mouse, etc.), or we can verify that visual elements are working and displayed correctly. We also have different elements inside a web page that we can interact with: dialogs, downloads, authentication forms, videos, etc.

With browser automation, we can do most of the things we do manually with a web browser. Some examples where we could use a browser automation API are:

- Automate form submission.
- Develop and implement a UI testing strategy using the latest version of Google Chrome Browser.
- Automate and validate keyboard input responses.
- Generate screenshot images.
- Generate PDF files.
- Scrape a website.
- Crawl a Single-Page Application (SPA) and generate pre-rendered content.
- Diagnose performance issues.

There are various browser automation tools in the market, implemented for several programming languages and with different features or purposes. One of the first to be released was [Puppeteer](https://pptr.dev/).

## What Is Puppeteer?

According to the official definition, [Puppeteer](https://pptr.dev/) is "A Node library which provides a high-level API to control headless Chrome or Chromium over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/)".

A [headless browser](https://developers.google.com/web/updates/2017/04/headless-chrome) is basically a web browser without a graphic user interface. It renders HTML documents and executes JavaScript code, but without displaying it on the screen. We can interact with a headless browser through a command line.

Even though Puppeteer runs mainly as a headless Chromium instance, we have the possibility to set it as "non-headless" and display all the interactions on the screen.

## Puppeteer Sharp

Puppeteer Sharp is a .NET port of the official Puppeteer API. It can run on .NET Framework or .NET Core projects. As the library was developed with a C# focus, we can use it with features like dependency injection, Async suffixes, etc.

One cool thing is that Puppeteer Sharp supports Chromium-based browsers (Google Chrome, Microsoft Edge), but also Mozilla Firefox. So, we can automate interactions with those web browsers.

Let's play with Puppeteer Sharp and leverage it to generate content in PDF.

## PDF Generation App Using Puppeteer Sharp

Let's see how to generate PDF files with a C# console app using the [Puppeteer Sharp NuGet package](https://www.nuget.org/packages/PuppeteerSharp/). The application will receive a URL from the user; it will generate a PDF with the content of the website and will save the generated file in a defined directory.

If you want to see the final code of the application, download it from [this GitHub repository](https://github.com/auth0-blog/puppeteer-sharp-pdf-generator).

First of all, we'll create a new Console App.

![Create Console App in Visual Studio for Mac](https://images.ctfassets.net/23aumh6u8s0i/3AmUM3S7DrcgTm6o5hzv5E/640c7747f2e4922b1b1c4266c75d22a0/console-app.png)

After we named our application and generated the boilerplate code, we will add the "PuppeteerSharp" NuGet package to our solution.

![Add PuppeteerSharp NuGet package in Visual Studio for Mac](https://images.ctfassets.net/23aumh6u8s0i/1NumDfbWC4x7bLBB3fVdmx/20aa0a63fb1634341cc44a3d203c7ce8/puppeteer-sharp-nuget-package.png)

We will add the following code to the `Program.cs` file:

```csharp
using System;
using System.Threading.Tasks;
using PuppeteerSharp;

namespace PuppeteerSharpPdfGenerator
{
    class Program
    {
        static async Task Main(string[] args)
        {
            Console.WriteLine("Enter URL to generate PDF file:");

            string url = Console.ReadLine();

            var browserFetcher = new BrowserFetcher();
            await browserFetcher.DownloadAsync();

            await using var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = true });

            await using var page = await browser.NewPageAsync();
            await page.GoToAsync(url);
            await page.PdfAsync($"{DateTime.Today.ToShortDateString().Replace("/", "-")}.pdf");

            Console.WriteLine("PDF File generated successfully.");
            Console.ReadLine();
        }
    }
}
```

Let's detail what this code is doing:

- We start requesting the URL that we want to browse and print as a PDF file. We store that in the string variable `url`.

- We create an instance of BrowserFetcher. This class helps us to download the Chromium browser that we will use to automate our tasks.

- We call `browserFetcher.DownloadAsync` to download all the files related to the Chromium browser that we will use. You can see that the files are downloaded only the first time that we execute our app. In subsequent executions, we won't download anything.

- `DownloadAsync` method can receive a specific version of Chromium and run Puppeteer against it: for example, `await browserFetcher.DownloadAsync(533271)`.

- We launch the Chromium browser. `LaunchAsync` method has several parameters. We send `Headless = true` because we don't want to see how the browser operates while generating the PDF file. In case we want to see how it runs, setting `Headless = false` would be helpful.

- We open a new page in the browser and go to the URL entered by the user.

- We call ` page.PdfAsync` to generate the PDF file. The parameter is the filename we will assign to the newly created file.

- In case we want to change the browser's viewport before we generate the PDF file, we can call the `SetViewportAsync` method.

```csharp
await page.SetViewportAsync(new ViewPortOptions
{
    Width = 500,
    Height = 500
});
```

So, if we run our application, we will get this message in the Terminal:

![Terminal Requesting URL](https://images.ctfassets.net/23aumh6u8s0i/2IqgX4LqMijRA2lAneIRoG/931b63a0c3f136b20a88efe9972b0ff3/terminal-begin.png)

We will type a URL like "[https://jwt.io/introduction](https://jwt.io/introduction)" and press `Enter`. We should wait until the `PDF File generated successfully.` message appears on the screen and press `Enter` to exit from our app.

![Terminal PDF File Generated](https://images.ctfassets.net/23aumh6u8s0i/17k87SXzvMm7BNr1zDbQmA/5da168f22ac528b5ed7affc7a6324de2/terminal-pdf-generated.png)

After that, let's go to the directory where our app is located (if you are running or debugging it from Visual Studio, take a look at the `bin` directory inside the solution folder). We will see the generated PDF file, and if we open it, we will see the JWT tutorial page.

![PDF file showing the JWT tutorial](https://images.ctfassets.net/23aumh6u8s0i/46zAVG7OwOSf76GOOaCOEe/a20787ec65fc53592b0c9ae0ce9a4385/pdf-file.png)

## Summary

In this article, we talked about browser automation and how to implement it using Puppeteer Sharp, a .NET port of the Node.js library Puppeteer. We talked about different scenarios where we could automate browser tasks. We also talked about the "headless browser" concept. Finally, we created a sample application to show how to generate PDF files using Puppeteer Sharp from a C# console application.

If you want to learn more, explore other features and methods of Puppeteer Sharp, or see example code for other scenarios and usages, you can visit the [official GitHub repository](https://github.com/hardkoded/puppeteer-sharp).

<include src="asides/DotNetCore" />

PDF Generation with Puppeteer Sharp

Automating web browser interactions is a great way to test the functionalities of our web application. Let's see how to create UI tests in C# using Playwright Sharp, a browser automation library.

While creating web applications, we want to release them with as few errors as possible. We also want to make sure that the new functionalities we add to our apps don't break the existing behavior and workflows. The only way we can do that is by testing the different scenarios our apps support.

While our code base gets bigger and bigger, it gets pretty hard to manually go through the different paths our app offers every time we apply changes and edits. And even if we are coding a small web application, manual testing executed by a person may not always be the best solution. That is why having automated tests that run periodically through our apps is always a recommended idea.

We talk about User Interface testing when we want to test the visual aspects of an application. We do this to verify that the parts of our product that users will interact with are working as expected, and they meet the performance and functionality requirements.

We can create automatic processes and algorithms, probably using mock data, that execute [unit and integration tests](https://auth0.com/blog/xunit-to-test-csharp-code/) across our codebase. Doing this, we verify that everything is working as expected, and we are supporting all the logical scenarios and business rules our application handles. But if we want to test visual elements from our apps, we will need the help of a web browser.

## What Is Browser Automation?

As developers, we (should) always try to automate things. We make things faster, we save work time, we spend less money, we depend less on human assistance, we get consistent results over iterations.

If we want to create and run UI tests for our web apps, we should automate actions related to web browsers. We will need to simulate interactions using input devices (keyboard, mouse, etc.), and we will verify that visual elements are working and displayed correctly.
"Browser automation" is automating the usage of a web browser. The goal is to achieve efficiency and speed at a certain level that wouldn't be possible if we depend on human interaction.

There are different tools available to accomplish browser automation, implemented in different programming languages and with different features. Today we will talk about Playwright and the .NET implementation: Playwright Sharp.

## What Is Playwright?

[Playwright](https://playwright.dev/) is an [open-source](https://github.com/microsoft/playwright) browser automation library. It is a JavaScript-based library created to be used with Node.js. One of the main differences with other browser automation tools is that Playwright is cross-browser. It can automate tasks using Chromium (Google Chrome, Microsoft Edge), WebKit (Apple Safari), and Mozilla Firefox browsers, all with the same API.

Playwright was created by Microsoft. It was developed based on [Puppeteer](https://pptr.dev/), and it was improved to perform automatic UI testing.

Some of the benefits Playwright offers are:

- Support for all browsers
- Test for mobile using device emulation
- Cross-platform WebKit testing
- Headless and headful execution: while automation is being executed, you can hide (headless) or show (headful) the web browser UI. Headful is usually selected for debugging, while headless is the preferred option for continuous integration or cloud executions.

<include src="TweetQuote" quoteText="Playwright Sharp ports the power of browser automation to .NET." />

## What Is Playwright Sharp?

[Playwright Sharp](https://github.com/microsoft/playwright-sharp) is a .NET porting for Playwright. It works close to the product, so we have all the features from the original library available to be used in .NET.

Playwright Sharp has been built with C#, so developers can get the advantage of .NET features such as dependency injection, "Async" suffixes, etc.

The NuGet package has two required dependencies: the Playwright driver and the browsers that we can use. Playwright drivers will be copied to the bin folder at build time. The browsers are installed in different ways depending on the use we give to the library. If we want to use Playwright Sharp in test projects, browsers will be installed at build time.

Let's create an example project to use some of the features Playwright Sharp offers.

## Playwright Sharp in Action

To get acquainted with Playwright Sharp methods and features, we will create a small end-to-end test. End-to-end testing is a technique that tests a complete workflow, from beginning to end, to ensure the flow behaves as expected. The idea is to test real scenarios to verify that everything is working as expected.
For this article, we will test part of the [Auth0 website](https://auth0.com/). Let's say that we want to test if a particular Quickstart guide is working, and the link to the GitHub repository where the sample code lives is correct. Let's pick ["Regular Web App" for ASP.NET Core](https://auth0.com/docs/quickstart/webapp/aspnet-core).

If you want to see the final code of the project we are going to create, download it from [this GitHub repo](https://github.com/auth0-blog/playwright-sharp-auth0).

First of all, we need to create a project using a testing library to code and run our tests. You can choose any testing library you want that works with .NET Core and C#. For this example, let's create a new [NUnit](https://nunit.org/) Test Project.

![NUnit Test Project in Visual Studio for Mac](https://images.ctfassets.net/23aumh6u8s0i/5I32SOweWF4ndmTeapbNP9/82198c8a017365acd1e69073c86f1f77/nunit-test-project.png)

Once the project is created and the standard files are generated, let's add the "PlaywrightSharp" NuGet package to the project.

![Playwright Sharp Nuget Package in Visual Studio for Mac](https://images.ctfassets.net/23aumh6u8s0i/6DO9jJ8OSwyKPMGDrtTSNa/2e6740b35a1e5c562619b5b3af37abfb/playwright-sharp-nuget.png)

We will create only one test, so we will use the `UnitTest1.cs` file already generated and replace its content with the following:

```csharp
// UnitTest1.cs

using System.Threading.Tasks;
using NUnit.Framework;
using PlaywrightSharp;

namespace PlaywrightSharpAuth0
{
    public class Tests
    {
        [Test]
        public async Task Auth0RegularWebAppASPNETCoreLinkWorks()
        {
            using var playwright = await Playwright.CreateAsync();
            await using var browser = await playwright.Chromium.LaunchAsync(headless: true);

            // Go to Auth0 home page
            var page = await browser.NewPageAsync();
            await page.GoToAsync("https://auth0.com/");

            // Click on the cookie policy acceptance button if it exists
            if ((await page.QuerySelectorAsync("#truste-consent-button")) != null)
            {
                await page.ClickAsync("#truste-consent-button");
            }

            // Hover on Developers
            await page.HoverAsync("li[aria-labelledby='nav-item-Developers']");

            // Click on Quickstarts
            await page.ClickAsync("a[href='https://auth0.com/docs/quickstarts/']");

            // Hover on Regular Web App
            await page.HoverAsync("div[data-type='webapp']");

            // Click on Launch Quickstart
            await page.ClickAsync("div[data-type='webapp'] button.btn-success");

            // Search for ASP.NET Core
            await page.TypeAsync("div.quickstart-search-input input", "ASP.NET Core");

            // Click on ASP.NET Core logo
            await page.ClickAsync("div[data-name='aspnet-core']");

            // Look for "View on Github" button and get the href property
            var gitHubLink = await page.EvalOnSelectorAsync<string>("a.btn-transparent.btn-block", "e => e.getAttribute('href')");

            Assert.AreEqual(gitHubLink, "https://github.com/auth0-samples/auth0-aspnetcore-mvc-samples/tree/master/Quickstart/01-Login");
        }
    }
}

```

Let's talk about the different methods used in the test.

- We create a Playwright instance that allows us to launch a web browser.
- For this example, we launch Chromium, the core of browsers like Google Chrome and Microsoft Edge. `LaunchAsync` method has several parameters. We use `headless: true` because we don't want to see how the browser operates while executing the test. If we set `headless: false` instead, a new browser window will be opened, and we will see how each step of the test is executed. This could be helpful in case we want to debug our tests.
- We open a new page in the browser, and we go to a specific URL. In this case, the Auth0 homepage.
- We hover some elements to display different menus. We also click on links to go to other pages. `HoverAsync` and `ClickAsync` accept [CSS selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors) to look for specific elements in the web page.
- We can also type text inside an input with `TypeAsync`.
- With `EvalOnSelectorAsync`, we execute an expression and get its result. In this case, we look for the `href` property value of an `<a>` element with the specified CSS selector. We compare that value with the URL we expect, so we can say the test passed correctly.

If we run the test, it should pass (if the Auth0 page didn't change since this article was written).

![Successful Test](https://images.ctfassets.net/23aumh6u8s0i/8IlLz1hcJmRQRkMtRJX2z/d2b1104b3c134af53b4ecd19fdb5b44e/playwright-successful-test.png)

## What's Next?

We created a simple end-to-end test, so we saw how to use some of Playwright Sharp main features. If we want to create more complex UI tests or to automate another kind of browser-based task, we can take a look at the [Playwright GitHub repository](https://github.com/microsoft/playwright-sharp) for some examples. We also have the [official Playwright documentation site](https://playwright.dev/docs/intro), where we can read about different concepts and functions we have available to create our automation processes. There are various elements from a web page that we can interact with: dialogs, downloads, authentication, videos, etc. We can also generate screenshots of what's displayed in the browser or even generate PDF files.

## Summary

In this article, we talked about UI testing and how to implement that through browser automation using Playwright Sharp. If we want to create tests that cover the possible workflows of our web application and analyze the visual elements of them, a library like Playwright Sharp is a powerful tool. With it, we can create end-to-end tests supporting all modern web browsers.

<include src="asides/DotNetCore" />

End-To-End Testing With Playwright Sharp

kaycie-east

As PR Manager, Kaycie supports public relations and global communications efforts for Auth0 in the AMER region. When not promoting the Auth0 brand, Kaycie can be found hiking, traveling, or searching for the next best bite to eat.

PR Manager, AMER Region

Kaycie East

Auth0 replaces in-house authentication solution to support rapid global adoption of tax automation and technology solutions



**BELLEVUE, Wash. – Jan. 26, 2021** – The [Auth0](https://auth0.com/) Identity Platform, a product unit within Okta, today announced that it has been selected as the authentication provider for [Vertex](https://www.vertexinc.com/), Inc. (Nasdaq: VERX), a pioneer in tax automation and a leading global provider of indirect tax software and solutions. Having recently [completed an IPO](https://www.vertexinc.com/company/news/vertex-announces-pricing-initial-public-offering), Vertex will use Auth0’s identity and authentication solutions to manage and secure access to the Vertex platform.

Vertex helps more than 4,000 customers across 130 countries worldwide simplify tax complexities and digitally advance their business. The Vertex platform provides a comprehensive, integrated suite of tax technology solutions, including tax determination, compliance and reporting, data management, and document management. Vertex serves more than half of the Fortune 500 companies, along with the top tax, accounting, and consulting firms.   
 
To aid the company’s strategic priorities and digital transformation efforts, Vertex replaced its in-house identity management solution with the Auth0 identity platform. Auth0’s solutions help secure the authentication and authorization to Vertex’s applications&mdash;which help businesses efficiently manage end-to-end tax operations for customer transaction events&mdash;and help Vertex manage its user authentication experience. 
 
“The rapid changes taking place in today’s global business, technology, and regulatory environments demand intelligent solutions that are secure and scalable,” said Steve Wasserman, Principal Architect at Vertex, Inc. “Identity management is a critical component to any transformation process, and having a proven identity expert like Auth0 allows us to confidently serve our growing customer base.”

“As a trusted leader in tax technology, Vertex is instrumental in calculating the tax for businesses while removing the many hurdles surrounding tax obligations and global commerce,” said Ghazi Masood, SVP of Americas for Auth0. “The security of these transactions is paramount, and Vertex understands the role authentication plays in that process. We look forward to aiding the Vertex team and its customers with their digital priorities to drive long-term success.”

<include src="asides/AboutAuth0" />

### Media Contacts
Auth0 Global Communications<br />
[press@auth0.com](mailto:press@auth0.com)<br />

Matter for Auth0<br />
[auth0@matternow.com](mailto:auth0@matternow.com)<br />


Vertex Selects Auth0 to Help Businesses Deliver Secure Tax Solutions

alejandro-ortuno

Alejandro is a Security Engineer in Auth0's Detection & Response team with a passion for automation and the cloud. Initiated as a software developer, he has experience working on AppSec, network security and cloud security for startups and enterprise companies such as Cisco or Rackspace. He spends his spare time learning newer technologies, playing padel and breaking things.

Security Engineer

Alejandro Ortuno

Discover how Auth0 uses security automation to respond to GuardDuty events at scale and our learnings in the process



Cloud adoption is occurring at a rapid pace and so are the risks and challenges in maintaining those secured environments. Doing it at scale requires the extensive use of automation and tooling. DevSecOps methodologies can also be applied to the detection and response of threats, decreasing the time to respond and preventing potential breaches. Logging pipelines, Slack bots, orchestration tools, and serverless technologies empower operation teams in a way that traditional security operations centers (SOC) can’t. Services such as AWS GuardDuty provide indispensable threat detection to continuously monitor for malicious activity and unauthorized access on cloud resources.

At Auth0, we have embraced the cloud since our inception, and with cloud resources in numerous AWS accounts in our organization, strong defenses and timely response to security events in the cloud are an absolute priority for us. That is why our Detection and Response team has embraced technologies to automate the response to security events in the cloud to scale for our [future organizational growth](https://techcrunch.com/2019/05/20/auth0-billion-dollar-unicorn/).

In this blog, we will provide an overview of the implemented architecture for the alert analysis, triage, user notification, and automated response of the AWS GuardDuty security findings on all of our AWS accounts.

## AWS GuardDuty Service

[_Amazon GuardDuty_](https://aws.amazon.com/guardduty/) is a threat detection paid service that continuously monitors for malicious activity and unauthorized behavior on AWS accounts and workloads. The service consumes CloudTrail, VPC Flow, and DNS logs and then enriches them with threat intel from AWS and other third-party services (such as CrowdStrike). GuardDuty also uses machine learning techniques to detect malicious activity on an AWS account. The service is regional (needs to be enabled on a per-region basis) and has a 30-day free trial. GuardDuty findings, if any, will start populating just a few minutes after the service is enabled.

At Auth0, we enable the GuardDuty service at account creation in all regions. Configuration follows a [_master/member_](https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_accounts.html) setup where every new AWS account becomes a new member of the master GuardDuty account.
From there, a combination of regional CloudWatch event rules and AWS Lambda functions ships all GuardDuty events into our security information and event management (SIEM) system.

We also monitor the health of this ecosystem closely as any failure would mean a critical gap in our alerting systems. As an example, we monitor specific [_GuardDuty API calls_](https://docs.aws.amazon.com/guardduty/latest/ug/logging-using-cloudtrail.html) (such as `Disassociate`, `StopMonitoring`, `DeleteDetector`, and many more) that could tamper with our infrastructure resulting in a page notification to our team.

Also, we have subscribed to the [_GuardDuty Announcements SNS Topic_](https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_sns.html) and integrated the alerts with our bot, so we get Slack notifications when there are new GuardDuty findings or updates to the existing ones that require our attention.

## SecurityBot

At Auth0, we rely heavily on Slack as our main communication channel. On top of it, [we have created Slack Bots](https://auth0.com/blog/auth0-internal-dev-tools-unleashing-engineering-potential/) that can automate our common tasks and help us scale and better respond to security events.

One of those bots is the SecurityBot, developed by the Detection and Response team on top of our Security Orchestration, Automation, and Response (SOAR) system. Auth0’s SecurityBot was inspired by the awesome work of the security teams at both [_Slack_](https://i736c61636bo656e67696e656572696e67z.oszar.com/distributed-security-alerting-c89414c992d6) and [_Dropbox_](https://github.com/dropbox/securitybot).
The SecurityBot helps us in two main tasks:

- As a ChatOps function, we can ask the bot to create an incident response channel, add all necessary members to it, initiate the documentation for an incident, or perform an investigation on specific IPs and Domains.

- As our main channel to communicate with employees. The bot can send notifications or inquiry messages (that require a confirmation) to our employees, ask for MFA confirmation, or even page our team if an employee responds negatively to a specific event.

![A message notification from SecurityBot](https://images.ctfassets.net/23aumh6u8s0i/3heWkQLOzq6ehvjVSDE5a1/967048353b36f46ce89be46c3ea7be00/a-message-notification-from-securitybot)

## AWS Incident Response Tool (aws-ir)

Especially important when dealing with an incident is to have a timely response. For that reason, our team has developed an internal tool to make investigation and response on AWS a much easier and repeatable task.

AWS IR is an API built on a serverless infrastructure that allows our team to take necessary forensic artifacts and response options on a specific machine in any of our AWS accounts. Those actions are mainly:

- Take a snapshot of the instance volume.

- Run a set of [_osquery_](https://osquery.io/) commands.

- Take a memory dump.

- Take a `tcpdump` of the network traffic.

- Isolate an instance.

We have integrated this tool with the response to GuardDuty events so we can collect artifacts and perform response actions depending on the severity of the alerts.

## The Hive Project

Keeping records for the actions taken and the artifacts collected is sometimes a “boring” but necessary task to do while performing a response to an incident. [_The Hive Project_](https://thehive-project.org/) is our main ticketing and artifacts collection platform that we use to perform our investigations.

We rely on its APIs to create alerts, promote them to cases, and to upload the artifacts that we automatically collect.

## Automating the Response Like a Pro

Now that we have explained all the different elements that compose our architecture let’s see how we glue them together.

We partner with the security automation company [_Tines_](https://tines.io/) in this regard. Every alert from our detection stack, whether it’s from SIEM, GuardDuty, Phishing, or custom tooling we’ve built, first goes through Tines where it is enriched, prioritized, and ticketed. We also use Tines to automate the response to the GuardDuty alerts.

One of the most common observations after enabling GuardDuty is that it can be daunting at the beginning and that it requires a significant amount of time to tune it to make it serviceable for your environment. Let’s take a look at a few examples of what this tuning looks like and some associated caveats:

### Network Load Balancer: Port Probe Alert False Positives 

Receiving a GuardDuty event of an instance which have a random port that is being probed by a malicious IP from the internet can be concerning. But the story changes if the instance has no public IP, and is behind a load balancer which has port 443 open to the internet. The reason for this is that the Network Load Balancer, by default, offers "Source IP Preservation" and the source IP addresses of the clients are preserved and provided to your backend applications. This is documented in the following [_public article_](https://mysteriouscode.io/blog/dont-panic-false-positives-from-guardduty-and-network-load-balancer-nlb/) as well. Later we will discuss how we have overcome this limitation when making changes on the load balancer is not an option.

### Alerting on harmless customer provided domains

If your service takes or ingests domains supplied by your customers and queries them (as is the case for us), GuardDuty can flag some of this legit activity as malicious. We encountered false positive "Backdoor and Trojan" alerts that happened due to customers providing flagged domains into our service.

### Detecting anomalies while employees are on the move

Machine learning is great, and we definitely believe in this technology for detection purposes (we want to further expand on this technology for our own sake). Still, it doesn’t cope well on a remote-friendly company where employees are on the move, continuously changing their locations and source IPs.

### GuardDuty events for already terminated EC2 Instances

We have received GuardDuty events where the instance state is “terminated” (this information is contained on the GuardDuty event). On an ephemeral infrastructure where instances are spun up and torn down on demand, and there are no long-running services, this can limit our capabilities to respond to incidents or to perform live investigations.

### Visibility gaps in cloud environments

There is a lack of visibility for Digital Forensics and Incident Response (DFIR) with GuardDuty for containers/microservices, especially if they are running on the same EC2 instance. Alerts will come from the same EC2, and additional logging/tooling will be needed to respond to those alerts.

<include src="TweetQuote" quoteText="Learn how to tune GuardDuty to make it serviceable for your environment and the associated caveats."/>

So let’s dig deeper into how we have automated the response for those GuardDuty findings and overcome some of those challenges. We have grouped the different GuardDuty events and their responses mostly into four different categories:

1. EC2 PortProbe alerts

2. IAM User non-critical alerts

3. IAM User critical alerts

4. EC2 Other alerts

Let's go over the response for each of them.

## PortProbe Alerts

The first group of alerts involves inbound port probing, which means potentially malicious IPs probing ports that are open on the internet. We do have PortProbe type of events (_Recon:EC2/PortProbeUnprotectedPort_) and also SSH and RDP brute force events included in this group.

Initially, we filter out common ports that we know are open and are commonly attacked (for example, port 443). Also, as mentioned before, we do not want instances that are behind a network load balancer to trigger events if the load balancer is opening common ports such as 443. For that reason, we have built an [_AWS Step function_](https://aws.amazon.com/step-functions/) that maps network load balancer ports against instance ports and dumps this information on a DynamoDB database. From Tines, we query the database to filter out alerts on instances behind a network load balancer from which the listening port is 80 or 443.

![AWS Step function to collect network load balancer information](https://images.ctfassets.net/23aumh6u8s0i/6ApTwxb12qBo36xQ3tlpVc/a22e10551f48604894d5ef8326394160/aws-step-function-to-collect-network-load-balancer-information)

Finally, if the event is not filtered and has not been notified before, we send a notification via SecurityBot to the resource owner (based on Instance tags).

## IAM User Non-Critical Alerts

In this group, we cover GuardDuty `IAMUser` events that we have deemed non-critical. These events typically suggest suspicious API calls made by AWS principals. Our main action with those alerts is to first automatically determine the mapping of the Principal to an employee and their endpoint machine. From there, we check their device IP and compare it to the IP in the alert. If there is any mismatch, we trigger a SecurityBot Inquiry message to ask the employee for confirmation that they made the AWS API call.

![SecurityBot Inquiry Message](https://images.ctfassets.net/23aumh6u8s0i/3urYXdOgk6xG4F2nOp2L1B/53a2bf233268ff3b2f95a9c14ee28ab2/securitybot-inquiry-message)

## IAM User Critical Alerts

In this group, we cover the GuardDuty IAMUser events that we have deemed critical, and as such, they trigger a PagerDuty alert to our team. These events include modification to CloudTrail, password policy changes, credential exfiltration, or the [_latest S3 specific alerts_](https://aws.amazon.com/about-aws/whats-new/2019/10/amazon-guardduty-adds-three-new-threat-detections/) among others.

We also enrich the alerts with the following information:

- List of employees seen under the same offending IP over the last 24 hours

- CloudTrail API calls made by the offending IP

## EC2 Alerts

Within this group, we have the following event types:

- EC2 Outbound network connection

- EC2 DNS request

- EC2 PortScanning

There is an important filtering layer that we needed to implement for our network scanner, and also for services that query customer provided domains which caused false positives to trigger. We initiate the AWS Incident Response tool to collect specific artifacts on the instance so we can better perform timely forensics investigation on the machines.

<include src="TweetQuote" quoteText="Dig deep into how Auth0 has automated the response to GuardDuty events and the associated challenges."/>

## Conclusions

We have no doubt as to the incredible value and visibility the AWS GuardDuty service provides to our team. Enabling it is incredibly simple, and we have decided to include it on each of our AWS accounts.

Automating our response to the GuardDuty alerts has helped us focus on what is important to us, filtering out some of the noise it generates and customizing it to our environments. We have evolved from processing hundreds of GuardDuty events per week (that would completely consume our team’s time to review and respond to these alerts) into just a few events per week needing our manual review. To add to this, with SecurityBot, we empower our employees to provide valuable real-time data relating to potential security incidents, making them aware of the alerts and also adding them to the front line of our security team.

For future work, we will continue to improve the automation for our incident response process. We also want to include other sources of information into our alerts, such as the events provided by agents like [_osquery_](https://osquery.io/) or information specific to containers and Kubernetes infrastructures. On top of that, we will expand our threat model to include attacks vectors such as compromised employee machines, malicious insiders, etc.

<include src="asides/AboutAuth0" />


Guardians of the Cloud: Automating the Response to Security Events

brandon-maxwell

Brandon is a Senior Security Engineer in Auth0's Detection & Response team. As a proponent of automation, he enjoys automating as many tasks as possible to spend time focusing on solving difficult security problems.

Senior Security Engineer

Brandon Maxwell

Learn how Auth0 uses security automation to generate phishing response at scale



Phishing continues to be a popular attack vector used by attackers with all sorts of motivations — from [_political_](https://www.fireeye.com/blog/threat-research/2019/03/apt40-examining-a-china-nexus-espionage-actor.html) to [_financial_](https://www.wired.co.uk/article/hr-email-scam-phishing-impersonating-employees) ones. According to Verizon’s [_2019 Data Breach Investigations Report_](https://enterprise.verizon.com/resources/reports/dbir/), 32% of the investigated breaches involved phishing. Most organizations deal with this threat on an almost daily basis, which makes automating the triage for these events an important part of maturing a security program.

In this post we talk about the approach Auth0 has taken to automating phishing in the hopes that it will help other organizations tackle this near-universal threat vector.

## phishing@auth0.com

Auth0 uses the email address `phishing at auth0 dot com` as an "abuse" inbox. We encourage customers and members of the public to report any suspicious Auth0-related emails they receive to this address. In addition, all Auth0 employees are trained to report suspicious emails they receive to this abuse inbox.

In Auth0’s infancy, every email that was sent to the abuse inbox was analyzed manually. A security engineer would review the email body and headers, copy URLs from the body of the emails and analyze them in various intelligence tools, and upload suspicious attachments (or their hashes) to various sandboxes. If the reported email represented a real threat, a remediation process was performed.

This worked great; however, as Auth0 started to grow, the volume of reports to `phishing at auth0 dot com` increased. Email analysis work became time-consuming and was detracting our attention from other higher-impact activities. In other words, it was a perfect use-case for security automation.

## Security Automation

At Auth0, we rely heavily on security automation to ensure we’re not wasting valuable cycles performing repetitive, manual tasks. We partner with the [security automation company Tines](https://tines.io/) in this regard. Every alert from our detection stack, whether it’s from SIEM, GuardDuty, or custom tooling we’ve built, first goes through Tines where it is enriched, prioritized, and ticketed. We also use Tines to automate phishing responses.

## Automated Phishing Response

For the most part, our automated phishing response story maps directly to our previous, manual process and consists of the following high-level steps:

1. Fetch email data and extract indicators
2. Create tracking tickets
3. Analyse indicators
4. Determine if the email represents a real threat
5. Close case if benign
6. Remediate if true positive

### Step 1: Fetch email data and extract IOCs

The first step is to read all emails sent to our abuse inbox. We do this with GSuite’s Gmail API. Once we have the email content, we can extract all URLs and attachments and begin to analyse them. We also recursively open and extract Indicators of Compromise (IOCs) from EML files attached to the originally reported email.

![Flowchart to fetch email data and extract IOCs](https://images.ctfassets.net/23aumh6u8s0i/4kB2PfHEaRZZswUn4bRU6M/d71c7ec982dcde9f05fd943244b2ed13/how-auth0-automates-phishing-response_3Afetch-email-data-and-extract-iocs)

### Step 2: Create tracking tickets

We track and record every single email that’s reported to our abuse inbox in our case management system. Not only does this allows us to track the volume of emails reported over time, but it also allows us to store indicators for future reference and later threat-hunting.

### Step 3 - Analyze indicators

One of the key benefits of automating analysis of indicators is that we can leverage many different intelligence sources without having to worry about wasting an analyst’s valuable time. In our phishing automation story we analyse URLs in:

- VirusTotal
- Urlscan.io
- URLhaus
- PhishTank
- Phish.AI

> **Protip**: When submitting URLs to public sandboxes, remember to first [_redact sensitive information_](https://tines.io/blog/automated-information-leakage/) from URL parameters and never submit links that could be used to reset passwords, etc.

Tines allows us to easily add and remove intelligence sources and experiment with different combinations of sandboxes, without having to change the core flow of the automation story.

![Flowchart to analyze indicators](https://images.ctfassets.net/23aumh6u8s0i/4XqCWGvVyGwl1ttZ140Y7C/8d4b825d58a6cf9b2f4e89089cdf2758/how-auth0-automates-phishing-response_3Aflowchart-to-analyze-indicators)


### Step 4: Determine if the email represents a real threat

Once we have analysed a URL or attachment in all the various intelligence sources and sandboxes, we use the combined results to determine whether the URL represents a legitimate threat and should be remediated/escalated to a human engineer or whether it’s a false positive and we can ignore it. There’s some secret sauce involved in making this determination and we regularly change the criteria used as our requirements adapt.

### Step 5: Close case if benign

If we have determined that all indicators associated with an email are benign, we update the ticket in our case management system and no further action is taken.

### Step 6: Remediate if true positive

If, however, we have determined that the reported email represents a real threat, we perform a number of automated remediation actions. Again, these actions change regularly but include the following:

- Purge the email from all employee inboxes
- Sweep endpoint logs to determine if any employees visited the malicious domain/URL, or downloaded/executed a malicious file
- Proactively share indicators with trusted peers in the InfoSec community
- Quarantine the devices we believe may have been infected
- Contact users who visited and/or interacted with malicious content and provide them with the next steps needed to mitigate any damage
- Reset potentially compromised credentials

Again, for every step that’s taken, evidence is collected and added to the relevant ticket in our case management system. This ensures a complete paper trail is available for every action that’s taken.

## Summary

Security awareness, email sandboxes, and strong multi-factor authentication all play a part in our *defense-in-depth* approach to phishing response. However, an abuse inbox, where employees and customers can report suspicious emails, is an indispensable resource, providing vital insight into on-going and emerging phishing attacks.

By automating triage of our abuse inbox, our security engineers can focus on higher-impact activities such as threat hunting and refinement or detection logic.

<include src="asides/AboutAuth0" />


How Auth0 Automates Phishing Response

37590801?s=460&v=4

A software engineer who dreams about non-electric sheep and autonomous vehicles and likes to champion security.

Software Engineer

Dan Arias

Leverage Docker images and containers to create an isolated Node development environment that runs a server


In this tutorial, instead of creating and running a Node app locally, you'll to take advantage of the Debian Linux operating system that official Docker Node images are based on. You'll create a portable Node development environment that solves the **"But it runs on my machine"** problem that constantly trolls developers since containers are created predictably from the execution of Docker images on any platform.

Throughout this tutorial, you'll be working in two realms:

- **Local Operating System**: Using a CLI application, such as Terminal or PowerShell, you'll use a local installation of Docker to build images and run them as containers.

- **Container Operating System**: Using Docker commands, you'll access the base operating system of a running container. Within that context, you'll use a container shell to issue commands to create and run a Node app.

The container operating system runs in isolation from the local operating system. Any files created within the container won't be accessible locally. Any servers running within the container can't listen to requests made from a local web browser. **This is not ideal for local development**. To overcome these limitations, you'll bridge these two systems by doing the following:

- **Mount a local folder to the container filesystem**: Using this mount point as your container working directory, you'll persist locally any files created within the container and you'll make the container aware of any local changes made to project files.

- **Allow the host to interact with the container network**: By mapping a local port to a container port, any HTTP requests made to the local port will be redirected by Docker to the container port.

To see this Docker-based Node development strategy in action, you'll create a basic Node Express web server. Let's get started!

## <a class="toc-target" id="removing-the-burden-of-installing-node"></a>Removing the Burden of Installing Node

To run a simple "Hello World!" Node app, the typical tutorial asks to:

- Download and install Node
- Download and install Yarn
- To use different versions of Node, uninstall Node and install [`nvm`](https://github.com/creationix/nvm)
- Install NPM packages globally

![Ain't nobody got time for that!](https://images.ctfassets.net/23aumh6u8s0i/4xVUHMWNLagUIrrnQDuE46/24abba17c53d63744bfb84e235796432/Aint-Nobody-Got-Time)

Each operating system has its own quirks making the aforementioned installations non-standard. However, access to the Node ecosystem can be standardized using Docker images. **The only installation requirement for this tutorial is Docker**. If you need to install Docker, choose your operating system from [this Docker installation document](https://docs.docker.com/install/#supported-platforms) and follow the steps provided.

Similar to how NPM works, Docker gives us access to a large registry of Docker images called [Docker Hub](https://hub.docker.com/). From Docker Hub, you can pull and run different versions of Node as images. You can then run these images as local processes that don't overlap or conflict with each other. You can simultaneously create a cross-platform project that depends on Node 8 with NPM and another one that depends on Node 11 with Yarn.

## <a class="toc-target" id="creating-the-project-foundation"></a>Creating the Project Foundation

To start, anywhere in your system, create a `node-docker` folder. This is the project directory.

With the goal of running a Node Express server, under the `node-docker` project directory, create a `server.js` file, populate it as follows and save it:

```javascript
// server.js
const express = require("express");
const app = express();

const PORT = process.env.PORT || 8080;

app.get("/", (req, res) => {
  res.send(`
    <h1>Docker + Node</h1>
    <span>A match made in the cloud</span>
  `);
});

app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}...`);
});
```

A Node project needs a `package.json` file and a `node_modules` folder. _Assuming that Node is not installed in your system_, you'll use Docker to create those files following a structured workflow.

## <a class="toc-target" id="accessing-the-container-operating-system"></a>Accessing the Container Operating System

You can gain access to the container OS with any of the following methods:

- <a href="#single-command" target="_self">Using a single but long `docker run` command</a>.
- <a href="#dockerfile" target="_self">Using a `Dockerfile` combined with a short `docker run` command</a>.
- <a href="#docker-compose" target="_self">Using a `Dockerfile` in combination with Docker Compose</a>.

### <a id="single-command"></a> Using a single `docker run` command

Execute the following command:

```
docker run --rm -it --name node-docker \
-v $PWD:/home/app -w /home/app \
-e "PORT=3000" -p 8080:3000  \
-u node node:latest /bin/bash
```

Let's breakdown this `docker run` command to understand how helps you access the container shell:

```bash
docker run --rm -it
```

`docker run` creates a new container instance. The [`--rm` flag](https://docs.docker.com/engine/reference/run/#clean-up---rm) automatically stops and removes the container once the container exits. The combined `-i` and `-t` flags run [interactive processes such as a shell](https://docs.docker.com/engine/reference/run/#foreground). The `-i` flag keeps the STDIN (Standard Input) open while the `-t` flag lets the process pretend it is a text terminal and pass along signals.

> Think of `--rm` as "out of sight, out of mind".

> Without the `-it` team, you won't see anything on the screen.

![Hello, IT, have you tried turning it off and on again?](https://images.ctfassets.net/23aumh6u8s0i/3nakiU8KELbKSpyeyDXaPQ/b226e7664c58d45971fd708ee36c8117/giphy)

```
docker run --rm -it --name node-docker
```

The [`--name` flag](https://docs.docker.com/engine/reference/run/#name---name) assigns a friendly name to the container to easily identify it in logs and tables. For example when you run [`docker ps`](https://docs.docker.com/engine/reference/commandline/ps/).

```
docker run --rm -it --name node-docker \
-v $PWD:/home/app -w /home/app
```

The [`-v` flag](https://docs.docker.com/engine/reference/run/#volume-shared-filesystems) mounts a local folder into a container folder using this mapping as its argument:

`<HOST FOLDER RELATIVE PATH>:<CONTAINER FOLDER ABSOLUTE PATH>`

An environmental variable can print the current working directory when the command is executed: `$PWD` on Mac and Linux and `$CD` on Windows. The [`-w` flag](https://docs.docker.com/engine/reference/run/#workdir) sets the mounting point as the container working directory.

```
docker run --rm -it --name node-docker \
-v $PWD:/home/app -w /home/app \
-e "PORT=3000" -p 8080:3000
```

The [`-e` flag](https://docs.docker.com/engine/reference/run/#env-environment-variables) sets an environmental variable `PORT` with a value of `3000`. The [`-p` flag](https://docs.docker.com/engine/reference/run/#expose-incoming-ports) maps a local port `8080` to a container port `3000` to match the environmental variable `PORT` that is consumed within `server.js`:

```javascript
const PORT = process.env.PORT || 8080;
```

```
docker run --rm -it --name node-docker \
-v $PWD:/home/app -w /home/app \
-e "PORT=3000" -p 8080:3000  \
-u node node:latest /bin/bash
```

For security and to avoid file permission problems, the [`-u` flag](https://docs.docker.com/engine/reference/run/#user) sets the non-root user `node` available in the Node image as the user that runs the container processes. After setting the flags, the image to execute is specified: `node:latest`. The last argument is a command to execute inside the container once it's running. `/bin/bash` invokes the container shell.

> If the image is not present locally, Docker issues [`docker pull`](https://docs.docker.com/engine/reference/commandline/pull/) in the background to download it from [Docker Hub](https://hub.docker.com/).

Once the command executes, you'll see the container shell prompt:

```bash
node@<CONTAINER ID>:/home/app$
```

Before moving to the next method, exit the container terminal by typing `exit` and pressing `<ENTER>`.

### <a id="dockerfile"></a> Using a `Dockerfile`

The `docker run` command from the previous section is made of image build time and container runtime flags and elements: 

```
docker run --rm -it --name node-docker \
-v $PWD:/home/app -w /home/app \
-e "PORT=3000" -p 8080:3000  \
-u node node:latest /bin/bash
```

Anything related to image build time can be defined as a custom image using a `Dockerfile` as follows:

* `FROM` specifies the container base image: `node:latest`
* `WORKDIR` defines `-w`
* `USER` defines `-u`
* `ENV` defines `-e`
* `ENTRYPOINT` specifies to execute `/bin/bash` once the container runs

Based on this, under the `node-docker` project directory, create a file named `Dockerfile`, populate it as follows, and save it:

```dockerfile
FROM node:latest

WORKDIR /home/app
USER node
ENV PORT 3000

EXPOSE 3000

ENTRYPOINT /bin/bash
```

`EXPOSE 3000` documents the port to expose at runtime. However, container runtime flags that define container name, port mapping, and volume mounting still need to be specified with `docker run`.

The custom image defined within `Dockerfile` needs to be built using [`docker build`](https://docs.docker.com/engine/reference/commandline/build/) before it can be run. In your local terminal, execute:

```
docker build -t node-docker .
```

`docker build` provides your image the friendly name `node-docker` using the [`-t` flag](https://docs.docker.com/engine/reference/commandline/build/#tag-an-image--t). This is different than the container name. To verify that the image was created, run `docker images`.

With the image built, execute this shorter command to run the server:

```bash
docker run --rm -it --name node-docker \
-v $PWD:/home/app -p 8080:3000 \
node-docker
```

The container shell prompts comes up with the following format:

`node@<CONTAINER ID>:/home/app$`

Once again, before moving to the next method, exit the container terminal by typing `exit` and pressing `<ENTER>`.

### <a id="docker-compose"></a> Using Docker Compose

> For Linux, [Docker Compose is installed separately](https://docs.docker.com/compose/install/).

Based on the `Dockerfile` and the shorter `docker run` command of the previous section, you can create a Docker Compose YAML file to define your Node development environment as a service:

**`Dockerfile`**:

```dockerfile
FROM node:latest

WORKDIR /home/app
USER node
ENV PORT 3000

EXPOSE 3000

ENTRYPOINT /bin/bash
```

**Command**

```bash
docker run --rm -it --name node-docker \
-v $PWD:/home/app -p 8080:3000 \
node-docker
```

The only elements left to abstract from the `docker run` command are the container name, the volume mounting, and the port mapping.

Under the `node-docker` project directory, create a file named `docker-compose.yml`, populate it with the following content, and save it:

```yaml
version: "3"
services:
  nod_dev_env:
    build: .
    container_name: node-docker
    ports:
      - 8080:3000
    volumes:
      - ./:/home/app
```

* `nod_dev_env` gives the service a name to easily identify it
* `build` specifies the path to the `Dockerfile`
* `container_name` provides a friendly name to the container
* `ports` configures host-to-container port mapping
* `volumes` defines the mounting point of a local folder into a container folder

To start and run this service, execute the following command:

```
docker-compose up
```

`up` builds its own images and containers separate from those created by the `docker run` and `docker build` commands used before. To verify this run:

```bash
docker image
# Notice the image named <project-folder>_nod_dev_env
docker ps -a
# Notice the container named <project-folder>_nod_dev_env_<number>
```
 
`up` created an image and a container but the container shell prompt didn't come up. What happened? `up` starts the full service composition defined in `docker-compose.yml`. However, it doesn't present interactive output; instead, it only presents static service logs. To get interactive output, you use `docker-compose run` instead to run `nod_dev_env` individually. 

First, to clean the images and containers created by `up`, execute the following command in your local terminal:
       
```bash
docker-compose down
```

Then, execute the following command to run the service:

```bash
docker-compose run --rm --service-ports nod_dev_env
```

The `run` command acts like `docker run -it`; however, it doesn't map and expose any container ports to the host. In order to use the port mapping configured in the Docker Compose file, you use the `--service-ports` flag. The container shell prompt comes up once again with the following format:

`node@<CONTAINER ID>:/home/app$`

If for any reason the ports specified in the Docker Compose file are already in use, you can use the `--publish`, (`-p`) flag to manually specify a different port mapping. For example, the following command maps the host port `4000` to the container port `3000`:

```bash
docker-compose run --rm -p 4000:3000 nod_dev_env
```

## <a class="toc-target" id="installing-dependencies-and-running-the-server"></a>Installing Dependencies and Running the Server

> If you don't have an active container shell, using any of the previous section methods to access it.
 
In the container shell, initialize the Node project and install dependencies by issuing the following commands (if you prefer, use `npm`):

```bash
yarn init -y
yarn add express
yarn add -D nodemon
```

Verify that `package.json` and `node_modules` are now present under your local `node-docker` project directory.

[`nodemon`](https://github.com/remy/nodemon) streamlines your development workflow by restarting the server automatically anytime you make changes to source code. To configure `nodemon`, update `package.json` as follows:

```json
{
  // Other properties...
  "scripts": {
    "start": "nodemon server.js"
  }
}
```

In the container shell, execute `yarn start` to run the Node server.

To test the server, visit [`http://localhost:8080/`](http://localhost:8080/) in your local browser. Docker intelligently redirects the request from the host port `8080` to the container port `3000`.

To test the connection of local file content and server, open `server.js` locally, update the response as follows and save the changes:

```javascript
// server.js

// package and constant definitions...

app.get("/", (req, res) => {
  res.send(`
    <h1>Hello From Node Running Inside Docker</h1>
  `);
});

// server listening...
```

Refresh the browser window and observe the new response.

## <a class="toc-target" id="modifying-and-extending-the-project"></a>Modifying and Extending the Project


Assuming that Node is not installed in your local system, you can use the local terminal to modify project structure and file content but you can't issue any Node-related commands, such as `yarn add`. As the server runs within the container, you are also not able to make server requests to the internal container port `3000`.

In the event that you want to interact with the server within the container or modify the Node project, you need to execute commands against the running container using [`docker exec`](https://docs.docker.com/engine/reference/commandline/exec/) and the running container ID. You don't use `docker run` as that command would create a new isolated container.

Getting the running container ID is easy.

* If you already have a container shell open, the container ID is present in the shell prompt:

`node@<CONTAINER ID>:/home/app$`

* You can also get the container ID programmatically using [`docker ps`](https://docs.docker.com/engine/reference/commandline/ps/) to filter containers based on name and return the `CONTAINER ID` of any match:

```bash
docker ps -qf "name=node-docker"
```

> The `-f` flag filters containers by the `name=node-docker` condition. The `-q` (`--quiet`) limits the output to only display the ID of the match, effectively plugging the `CONTAINER ID` of `node-docker` into the `docker exec` command.

Once you have the container ID, you can use `docker exec` to:

* Open a new instance of the running container shell:

```bash
docker exec -it $(docker ps -qf "name=node-docker") /bin/bash
```

* Make a server request using the internal port `3000`:

```bash
docker exec -it $(docker ps -qf "name=node-docker") curl localhost:3000
```

* Install or remove dependencies:

```bash
docker exec -it $(docker ps -qf "name=node-docker") yarn add body-parser
```

> One you have another active container shell, you can easily run `curl` and `yarn add` there instead.

## <a class="toc-target" id="recap-and-uncovering-little-lies"></a>Recap... and Uncovering Little Lies

You've learned how to create an isolated Node development environment through different levels of complexity: by running a single `docker run` command, using a `Dockerfile` to build and run a custom image, and using Docker Compose to run a container as a Docker service.

Each level requires more file configuration but a shorter command to run the container. This is a worthy trade-off as encapsulating configuration in files makes the environment portable and easier to maintain. Additionally, you learned how to interact with a running container to extend your project.

You may still need to install Node locally for IDEs to provide syntax assistance, or you can use a CLI editor like `vim` within the container.

![Harry Potter is angry that he needs to download Node locally for his code editor to provide syntax assistance.](https://images.ctfassets.net/23aumh6u8s0i/fiXwLKWUXoYnHnCvlqgWL/735189d25b0f454bad861c84c67f50f2/giphy)

Even so, you can still benefit from this isolated development environment. If you restrict project setup, installation, and runtime steps to be executed within the container, you can standardize those steps for your team as everyone would be executing commands using the same version of Linux. Also, all the cache and hidden files created by Node tools stay within the container and don't pollute your local system. Oh, and you also get `yarn` for free!

[JetBrains](https://www.jetbrains.com/) is starting to offer the capability to use Docker images as remote interpreters for Node and Python when running and debugging applications. In the future, we may become entirely free from downloading and installing these tools directly in our systems. Stay tuned to see what the industry provides us to make our developer environments standard and portable.

<include src="asides/AboutAuth0" />


Use Docker to Create a Node Development Environment

Dirceu-Tiegs

What I enjoy the most about being an SRE at Auth0 is that it allows me to exercise different areas such as coding, architecture, writing, training, investigations, debugging, and incident response. It's great to be able to think deeply about all aspects related to reliability and then apply it to large-scale, real-world projects.

Site Reliability Engineering Team

Dirceu Tiegs

Learn more about how Auth0 architects its services to ensure scalability and high availability to handle more than 1.5 billion logins per month.



Auth0 provides authentication, authorization, and single sign-on services for apps of any type (mobile, web, native) on any stack. Authentication is critical for the vast majority of apps. We designed Auth0 from the beginning so that it could run anywhere: on our cloud, on your cloud, or even on your own private infrastructure.

In this post, we'll talk more about our public SaaS deployments and provide a brief introduction to the infrastructure behind [auth0.com](https://auth0.com/) and the strategies we use to keep it up and running with high availability. This is an updated version of the [2014 High Scalability post about Auth0's architecture](http://highscalability.com/blog/2014/12/1/auth0-architecture-running-in-multiple-cloud-providers-and-r.html).

A lot has changed since then in Auth0. These are some of the highlights:

- We went from processing a couple of million logins per month to 2.5 billion logins per month, serving thousands of customers, including [FuboTV](https://auth0.com/learn/sports-centric-streaming-service-fubotv-sees-50-roi-just-auth0s-security/), [Mozilla](https://auth0.com/blog/auth0-mozilla-partnership/), [InterMiles](https://auth0.com/learn/jetprivilege-case-study/), and more.

- We implemented new features like [custom domains](https://auth0.com/docs/custom-domains), [scaled bcrypt operations](https://i6175746830o656e67696e656572696e67z.oszar.com/bcrypt-as-a-service-9e71707bda47), vastly [improved user search](https://auth0.com/docs/users/search/v3), and much more.

- The number of services that compose our product in order to scale our organization and handle the increases in traffic went from under 10 to over 30 services.

- The number of cloud resources grew immensely as well; we used to have a couple dozen nodes in one environment (US), now we have more than a thousand over four environments (US, US-2, EU, AU).

- We doubled-down decided to use a single cloud provider for each of our environments and moved all our public cloud infrastructure to AWS.

## Core service architecture

![Auth0.com core service architecture](https://images.ctfassets.net/23aumh6u8s0i/2mden7W7brrYGv8LHDK2Lp/5347634d4197ae655555cf93b8209c2d/auth0-core-service-architecture)

The core service is composed of different layers:

- Core applications: Auto-scaling groups of servers running different services of our stack (authentication, management API, multi-factor authentication API, and much more).

- Data storage: Clusters of MongoDB, Elasticsearch, Redis, and PostgreSQL storing a variety of datasets for different applications and features.

- Transports/Queues: Kinesis streams and RabbitMQ, SNS, and SQS queues.

- Base services: Services for rate-limiting, bcrypt clusters, feature-flags, and more.

- Routing: AWS load balancers (ALB, NLB, and ELB from AWS) and some nodes running NGINX as a proxy.

## High Availability

In 2014 we used a multi-cloud architecture (using Azure and AWS, with some extra resources on Google Cloud) and that served us well for years. As our usage (and load) rapidly increased, we found ourselves relying on AWS resources more and more.

At first, we switched our primary region in our environment to be in AWS, keeping Azure as failover. As we began using more AWS resources like Kinesis and SQS, we started having trouble keeping the same feature set in both providers. As our need to move (and scale) faster grew, we opted to keep supporting Azure with a limited feature parity: if everything went down on AWS, we could still support core authentication features using the Azure clusters, but not much of the new stuff we had been developing.

After some bad outages in 2016, we decided to finally converge on AWS. We stopped all efforts related to keeping the services and automation platform-independent and instead focused on:

- Providing a better failover story inside AWS, using multiple regions and at least 3 Availability Zones per region.

- Increasing usage of AWS-specific resources like auto-scaling groups (instead of using fixed clusters of nodes), application load balancers (ALBs), and more.

- Writing better automation: we revamped our automation, completely embracing infrastructure as code using TerraForm and SaltStack to provision new Auth0 environments (and also to replace existing ones). This allowed us to grow from partially automated environments doing ~300 logins per second to fully automated environments doing more than ~3.4 thousand logins per second; using the new tooling makes it easier to scale up (and down, whenever it makes sense). The level of automation we achieved is not perfect but is allowing us to grow to new regions and create new environments in a much more convenient way.

- Writing better playbooks: with more time and focus, we saw that besides automation we also needed better playbooks in order to understand, manage, and respond to incidents related to our ever-growing mesh of services. This vastly improved scalability and reliability while also allowing us to onboard new employees faster.

<include src="TweetQuote" quoteText="Writing better automation let us grow from partially automated environments doing ~300 logins per second to fully automated environments doing more than ~3.4 thousand logins per second"/>

Let's take a look at our US environment architecture, for instance. We have this general structure:

![Auth0 US Environment Architecture](https://images.ctfassets.net/23aumh6u8s0i/4hGMwY5n4jk2aoJyyRAhjF/0ca131b596da7068aced34ca69fbbc5c/auth0-us-environment-architecture)

And this is the structure inside a single AZ:

![Auth0 Single Availablity Zone](https://images.ctfassets.net/23aumh6u8s0i/4EtOG7to5lHmyYnU5Q3aaC/a22b74fe7b0a58d0f54a78a6b801bbd2/auth0-single-availability-zone)

In this case, we use two AWS regions: us-west-2 (our primary) and us-west-1 (our failover). Under normal circumstances, all requests will go to us-west-2, being served by three separate availability zones.

This is how we achieve high availability: all services (including databases) have running instances on every availability zone (AZ). If one AZ is down due to a data center failure, we still have two AZs to serve requests from. If the entire region is down or having errors, we can update Route53 to failover to us-west-1 and resume operations.

<include src="TweetQuote" quoteText="We achieve high availability by running all services instances on every AWS availability zone"/>

We have different maturity levels for service failover: some services, like user search v2 (that builds a cache on Elasticsearch) might work but with slightly stale data; still, core functionality keeps working as expected.

In the data layer, we use:

- A cross-region cluster for MongoDB.

- RDS replication for PostgreSQL.

- Clusters per region for Elasticsearch with automated snapshot and restores running regularly to account for the lack of cross-region clustering.

We exercise failover at least once per year, and we have playbooks and automation to help new infrastructure engineers get up to speed on how to do it and what are the implications.

Our deployments are usually triggered by a Jenkins node; depending on the service we either use Puppet, SaltStack, and/or Ansible to update individual or groups of nodes, or we update our AMIs and create new autoscaling groups for immutable deployments. We have different types of deployments for new and old services, and this has been shown to be largely ineffective as we need to maintain automation, docs, and monitoring for something that should be unified.

We are currently rolling out blue/green deployments for some of the core services, and we intend to implement the same for every core and supporting service.

## Automated Testing

Besides unit test coverage on every project, we have multiple functional test suites that run in every environment; we run it on a staging environment before we deploy to production, and we run them again in production after finishing a deployment to ensure that everything works.

The highlights:

- We have thousands of unit tests in different projects.

- We use Pingdom probes that run every minute to check core functionality.

- We use a mix of Selenium and CodeceptJS-based functional tests before and after every deployment. The functional test suites test different API endpoints, authentication flows, identity providers, and much more.

<include src="TweetQuote" quoteText="Besides unit test coverage on every project, we have multiple functional test suites that run in every environment: staging before deploying to production and again in production after finishing deployment."/>

## CDN

Until 2017 we ran our own, custom-built CDN using NGINX, Varnish, and EC2 nodes in multiple regions. Since then, we transitioned to CloudFront, which has given us several benefits that include:

- More edge locations, which means less latency for our customers.

- Lower maintenance costs.

- Easier configuration.

There are a few downsides, like the fact that we need to run Lambdas to perform some configurations (like adding custom headers to PDF files and things like that). Still, the upsides definitely make up for that.

## Extend

One of the features we provide is the ability to run custom code as part of the login transaction, either via [_authentication rules_](https://auth0.com/docs/rules/current) or [_custom database connections_](https://auth0.com/docs/connections/database/custom-db). These features are powered by [_Extend_](https://goextend.io/), an extensibility platform that grew out of Auth0 and is now being used by other companies as well. With Extend, our customers can write anything they want in those scripts and rules, allowing them to extend profiles, normalize attributes, send notifications, and much more.

We have Extend clusters specifically for Auth0; they use a combination of EC2 auto-scaling groups, Docker containers, and custom proxies to handle requests from our tenants, processing thousands of requests per second and responding fast to variations of load. For more details about how this is built and run, check out this post on [how to build your own serverless platform](https://tomasz.janczuk.org/2018/03/how-to-build-your-own-serverless-platform.html).

## Monitoring

We use a combination of different tools for monitoring and debugging issues:

- CloudWatch

- DataDog

- Pingdom

- SENTINL

The vast majority of our alerts come from CloudWatch and DataDog.

We tend to configure CloudWatch alarms via TerraForm, and the main monitors we keep on CloudWatch are:

- HTTP errors from our main load balancers.

- Unhealthy instances in a target group.

- SQS processing delays.

CloudWatch is the best tool for alarms based on AWS-generated metrics (like ones from load balancers or autoscaling groups). CloudWatch alerts usually go to PagerDuty, and from PagerDuty to Slack/phones.

DataDog is a service we use to store and act on time-series metrics. We send metrics from Linux boxes, AWS resources, off-the-shelf services (like NGINX or MongoDB), and also custom services we have built (like our Management API).

We have many DataDog monitors. A few examples:

- Increase in response times in `$service` on `$environment`.

- Running out of space on `$volume` in `$instance` (`$ip_address`).

- Issue with `$process` on `$environment` / `$host` (`$ip_address`).

- Increase in processing time for `$service` on `$environment`.

- NTP drifting off / clock issue on `$host` (`$ip_address`).

- MongoDB replica-set change on `$environment`.

As you can see from the examples above, we have monitors on low-level metrics (like disk space) and high-level metrics (like MongoDB replica-set change, which alerts us if there was a change in the primary node definition, for example). We do much more and have some pretty sophisticated monitors around some services.

DataDog alerts are pretty flexible in their outputs and we usually send them all to Slack, sending to PagerDuty only those who should "wake people up" (like spikes of errors, or most things that we are sure that are having an effect on customers).

For logging we use Kibana and SumoLogic; we are using SumoLogic to record audit trails and many AWS-generated logs, and we use Kibana to store application logs from our own services and other "off-the-shelf" services like NGINX and MongoDB.

## The Future

Our platform evolved quite a bit in order to handle the extra load and the huge variety of use cases that are important to our customers, but we still have more room for optimizations.

Not only our platform grew, but our engineering organization increased in size: we have many new teams building their own services and are in need of automation, tooling, and guidance around scalability.
With that in mind, these are the initiatives in place for us to scale not only our platform but also our engineering practice:

- Build a Platform, PaaS style: As mentioned before, today we have different automation and deployment flows; this causes confusion and creates a barrier of entry for engineers as it's hard to experiment and scale up without touching too many repositories. We are developing a PoC for a platform (currently running on top of ECS) in which engineers can configure a YAML file and just deploy it to get computing resources, monitoring, logging, backups, and much more. All of that without having to configure it explicitly. This effort is in an early stage and might still change a lot (EKS?). However, we feel we are taking the right direction given our growing size and scalability constraints.

- Implement smoke tests for every new pull request: besides unit testing (that already runs on each new PR), we want to run integration tests on ephemeral environments wherever that's possible.

- Centralize our logging solution into just one provider. This might mean moving away from Kibana and using just SumoLogic, but we still need to evaluate feature set, data volume, and more.

- Automate metrics: Too much of our metric story is manual now: adding metrics-related calls to the code while deploying, and building dashboards and monitors using the DataDog interface. If we use a standard format and naming, we could do things like building dashboards/monitors automatically, extracting metrics from logs instead of explicitly adding calls to the code, and more.

- Ensure we have auto-scaling and blue/green deployments on every core service. This should come out-of-the-box from our new Platform, but while that is being built and tested we need to improve the scaling/deployment/rollback story for the core services that are still lacking in that regard.

<include src="asides/AboutAuth0" />


Auth0 Architecture: 5 Years In

1097492785caf9ffeebffeb624202d8f?s=200 avatar

1097492785caf9ffeebffeb624202d8f?s=200

Prosper is a great speaker, community leader, open source hacker, technical consultant, and a fervent Developer Advocate. He is a full-stack software engineer who has worked on biometric, health and developer tools. Prosper recently co-founded a startup called Eden that’s focused on improving the quality of lives in Nigeria and currently leads the engineering team. He also co-founded forloop, the largest developer community in Africa.

Former Auth0 Employee

Prosper Otemuyiwa

Webpack 4.0 has officially been released. What's new? What improvements were made? Learn how to build faster web applications with this new release.



---

**TL;DR:** On Sunday, February 25, 2018, **Webpack 4.0**, was released to the public. **Legato** _(Weboack 4.0's codename)_ is a major release to the JavaScript module bundler. It has taken about eight months since the release of 3.0. In this article, I'll cover the new features in **Webpack 4.0** and several other changes and deprecations.

---

Webpack is a capable module bundler for JavaScript applications. It bundles every one of the modules in your application into at least one file (frequently, only one) and serves it to the browser. In any case, Webpack is something beyond a module bundler. With the assistance of loaders and plugins, it can change, minify and optimize a wide range of files before serving them as a bundle to the browser. It takes in different resources, for example, JavaScript, CSS, Fonts, Images, and HTML, and afterwards changes these assets into a configuration that is helpful to use through a browser. The genuine energy of Webpack is the whole of its parts.

Alright, that's enough, [this article](https://auth0.com/blog/webpack-a-gentle-introduction/) provides a gentle introduction to webpack. Now, what's new in Webpack 4.0?

# What's new in Webpack 4.0?

## 1. Node.js 4 Support Dropped

Node.js 4 is no longer supported. The source code was upgraded to a higher ECMAScript version.

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Make sure to use node &gt;= 8.9.4 when using <a href="https://twitter.com/hashtag/webpack?src=hash&amp;ref_src=twsrc%5Etfw">#webpack</a> 4 legato.<br><br>Older versions will have reduced performance, because we started using newer JS features, which are less optimized in older V8 versions.<br><br>Btw. awesome work of the v8 team regarding performance.</p>&mdash; Tobias Koppers (@wSokra) <a href="https://twitter.com/wSokra/status/967852475918274561?ref_src=twsrc%5Etfw">February 25, 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

The creator of Webpack, Tobias, warns users to use Node >= 8.9.4 for optimal performance because of newer JavaScript features now used in the codebase.

## 2. Farewell CommonsChunkPlugin, Hello SplitChunksPlugin

It's sad to see you go, dear `CommonsChunkPlugin`. The infamous _CommonsChunkPlugin_ has been removed and replaced with two new smooth APIs called `optimize.splitChunks` and `optimization.runtimeChunk`. Let me explain!

Webpack 4 shipped with huge improvements to the chunk graph and a new optimization technique for chunk splitting. A new plugin was born in the process of improvements called `SplitChunksPlugin`. The `SplitChunksPlugin` automatically identifies modules which should be split by heuristics and splits the chunks. And it has some other great features such as working efficiently on async chunks, and handling vendor splitting with multiple vendor chunks.

By default, shared chunks will be automatically generated for you out of the box in Webpack 4. And it is configurable via `optimize.splitChunks`. The `optimization.runtimeChunk: true` option adds an additional chunk to each entry point containing only the runtime.

Tobias has [compelling examples of how the new optimization technique works!](https://gist.github.com/sokra/1522d586b8e5c0f5072d7565c2bee693).

## 3. WebAssembly Support Out of the Box

WebAssembly (wasm) is a new portable, and load-time-efficient format suitable for compilation to the web. It's really fast and the developer community is embracing it rapidly. For this reason, Webpack 4 now supports WebAssembly out of the box. In Webpack 4, you can import and export any local WebAssembly module. And you can write loaders that allow you to import C++, C and Rust directly.

**Note:** The WebAssembly modules can only be used in async chunks.

## 4. Support for various Module Types

Webpack 4 now supports five module types. These module types are:

* **javascript/auto:** Webpack 3 shipped with support for this module type with all module systems enabled, `CommonJS`, `AMD`, `ESM`.
* **javascript/esm:** Supports only ECMAScript modules.
* **javascript/dynamic:** Supports only CommonJS and AMD.
* **json:** Supports only JSON data, It's available via `require` and `import`.
* **webassembly/experimental:** Supports only WebAssembly modules. For now, it is currently experimental.

<include src="TweetQuote" quoteText="Webpack 4 now supports five module types."/>

## 5. Lighting the Fire with Mode

I am more excited about this feature than everyone in the universe at the moment. The Webpack team has introduced a new config property called `mode` all in the quest to achieve a zero-config (#0CJS) module bundler. The `mode` option can be set to either of these two values; `development` or `production`. Out of the box, it defaults to `production`.

The `production` option provides a set of sensible defaults that allows for:

* Small output size.
* Fast code at runtime.
* Omitting development-only code.
* Not exposing source code or file paths.
* Easy to use output assets.

The `development` option provides a set of sensible defaults that allows for:

* Better tooling for in-browser debugging.
* Fast incremental compilation for a fast development cycle.
* Better error messages at runtime.

Check out the [configuration options affected by `mode`](https://medium.com/webpack/webpack-4-mode-and-optimization-5423a6bc597a).

**Note:** Without the `mode` option in your Webpack config, a warning will be thrown at you.

![Webpack 4 Mode warning](https://images.ctfassets.net/23aumh6u8s0i/1pywqp48M17uEI2pAdt8aC/850297e32eb039c6ce38b0633ab2bef5/modewarning)

## 6. Zero Config Module Bundler #OCJS

Before now, an entry point had to be defined inside a `webpack.config.js` file for Webpack to bundle your app. However, with Webpack 4, there is no need to define the entry point, it will take `./src/index.js` as the default.

Furthermore, there's also no need to define the output file, it emits the bundle to `/.dist/main.js`.

<include src="TweetQuote" quoteText="Webpack 4 supports zero-config (#0CJS) setups. For example, it will use ./src/index.js as the default entry point."/>

The significance of this slick feature comes to play when spinning up small projects. No need for a configuration file. Just webpack away!

![Webpack 4 without entry](https://images.ctfassets.net/23aumh6u8s0i/742oY1MfAFwJlXc72Qi3yL/cbc244952e08ff17567369038d5c3864/withoutindex)
_Webpack 4 without any config and entry file._

![Webpack 4 with index.js](https://images.ctfassets.net/23aumh6u8s0i/1UR6XUVw2VmNivcIfLPM6Y/f2b8f238d4d5b7b04a73d7f9f5bca1e1/withindex)
_Webpack 4 without any config file. Just a src/index.js file._

## 7. Faster Build Times

Webpack 4's build times are now up to about 98% faster than the previous major version. Don't take my word for it. Try it out yourself. You can get started with Webpack 4 like so:

```bash
npm i webpack --save-dev
```

or use yarn like so:

```bash
yarn add webpack --dev
```

In your package.json's `devDependencies` section, you should now have:

```js
"webpack": "^4.0.0"
 ```

Webpack 3: 2405ms
![Webpack 3's build time](https://images.ctfassets.net/23aumh6u8s0i/pjcYwgfXqTnJk3qI3Qcj3/e9e3e62728d25b6707028206cd3e1a20/buildtime)
_Webpack 3's build time_

Webpack 4: 1388ms
![Webpack 4's build time](https://images.ctfassets.net/23aumh6u8s0i/7ojTTm9RKywkGWRxpoyL7v/6e2e4db21a718228b6d360eca8a10121/buildtime)
_Webpack 4's build time_

Webpack v4 is almost 2x faster than v3 in build time!

Other performance tests from the community can be found below:

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">6 entries, dev mode, source maps off, using a bunch of loaders and plugins. dat speed ⚡️ <a href="https://t.co/fgzYPN24k1">pic.twitter.com/fgzYPN24k1</a></p>&mdash; Evan Scott (@probablyup) <a href="https://twitter.com/probablyup/status/965128389307846657?ref_src=twsrc%5Etfw">February 18, 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

## 8. Fresh Plugin System

Webpack 4 ships with a complete overhaul of the plugin system. There is a new simple API for plugins and hooks which comes with the following configuration:

* The `hooks` object houses all hooks as property of the extensible class.
* Multiple `Hook` classes now exist depending on the hook type: `sync`, `async`, `normal`, `bailing`, `waterfall`, `looping`.
* You must provide a name when adding plugins.
* You can choose the type of the plugin (sync/callback/promise) when adding plugins.
* `this.hooks = { myHook: new SyncHook(...) }` is the new way of registering hooks. Creating a new Hook object as a property of the `hooks` object.

Tobias wrote a [comprehensive note on how the new plugin system works](https://medium.com/webpack/the-new-plugin-system-week-22-23-c24e3b22e95). The plugin method is backward-compatible.

## 9. Giant Move to webpack-cli

The Webpack's command line interface has been moved to [webpack-cli](https://github.com/webpack/webpack-cli), you need to install `webpack-cli` to use the CLI.

The [Webpack CLI documentation](https://webpack.js.org/api/cli/) can get you started on using the `webpack-cli` effectively.

## Deprecations and Other Updates

* These plugins have been deprecated, `NoEmitOnErrorsPlugin`, `ModuleConcatenationPlugin`, `NamedModulesPlugin`. You can now use `optimization.noEmitOnErrors`, `optimization.concatenateModules`, and `optimization.namedModules` respectively.
* `import()` always returns a namespace object.
* Webpack now removes dead branches by default.
* The use of `System.import()` now emits a warning.
* `webpackInclude` and `webpackExclude` are supported by the magic comment for `import()`. They allow to filter files when using a dynamic expression.

## Road to Webpack 5

There are several plans underway for Webpack 5. Some of these plans include:

* Stable WebAssembly support.
* The ability to create a custom module type and have Webpack support it.
* Eliminating `ExtractTextWebpackPlugin` and supporting a CSS module type out of the box.
* Supporting HTML module type out of the box.
* Persistent Caching

## Aside: Webpack and JavaScript at Auth0

At [Auth0](https://auth0.com/) we use JavaScript heavily in development and automate tasks using build tools such as Webpack. Using our authentication and authorization server from your JavaScript web apps is a piece of cake.

> [Auth0 offers a **free tier**](https://auth0.com/pricing) to get started with modern authentication.

It's as easy as installing the `auth0-js` and `jwt-decode` node modules like so:

```bash
npm install jwt-decode auth0-js --save
```

```js
import auth0 from 'auth0-js';

const auth0 = new auth0.WebAuth({
    clientID: "YOUR-AUTH0-CLIENT-ID",
    domain: "YOUR-AUTH0-DOMAIN",
    scope: "openid email profile YOUR-ADDITIONAL-SCOPES",
    audience: "YOUR-API-AUDIENCES", // See https://auth0.com/docs/api-auth
    responseType: "token id_token",
    redirectUri: "http://localhost:9000" //YOUR-REDIRECT-URL
});

function logout() {
    localStorage.removeItem('id_token');
    localStorage.removeItem('access_token');
    window.location.href = "/";
}

function showProfileInfo(profile) {
    var btnLogin = document.getElementById('btn-login');
    var btnLogout = document.getElementById('btn-logout');
    var avatar = document.getElementById('avatar');
    document.getElementById('nickname').textContent = profile.nickname;
    btnLogin.style.display = "none";
    avatar.src = profile.picture;
    avatar.style.display = "block";
    btnLogout.style.display = "block";
}

function retrieveProfile() {
    var idToken = localStorage.getItem('id_token');
    if (idToken) {
        try {
            const profile = jwt_decode(idToken);
            showProfileInfo(profile);
        } catch (err) {
            alert('There was an error getting the profile: ' + err.message);
        }
    }
}

auth0.parseHash(window.location.hash, (err, result) => {
    if(err || !result) {
        // Handle error
        return;
    }

    // You can use the ID token to get user information in the frontend.
    localStorage.setItem('id_token', result.idToken);
    // You can use this token to interact with server-side APIs.
    localStorage.setItem('access_token', result.accessToken);
    retrieveProfile();
});

function afterLoad() {
    // buttons
    var btnLogin = document.getElementById('btn-login');
    var btnLogout = document.getElementById('btn-logout');

    btnLogin.addEventListener('click', function () {
        auth0.authorize();
    });

    btnLogout.addEventListener('click', function () {
        logout();
    });

    retrieveProfile();
}

window.addEventListener('load', afterLoad);
```

Get the [full example using this code](https://github.com/auth0-blog/es2015-rundown-example).

Go ahead and check out our [quickstarts](https://auth0.com/docs/quickstarts) for how to implement authentication using different languages and frameworks in your apps.

## Conclusion

You have now learned the new additions to Webpack. Webpack 4.0 shipped with a lot of new features and improvements. The Webpack team is close to having a complete migration and additions guide to Webpack 4. In the upcoming weeks and months, they'll be out. New to Webpack? I wrote a [gentle introduction to the module bundler](https://auth0.com/blog/webpack-a-gentle-introduction). [Sean Larkin](https://twitter.com/thelarkinn) also has some awesome Webpack courses on [Webpack Academy](https://webpack.academy/).

Have you started working with Webpack 4 yet? What are your thoughts? Let me know in the comments section! 😊

Webpack 4.0 Release: What's New?

Learn how to use GulpJS to automate time-consuming tasks in your development workflow.



---

**TL;DR:** In this tutorial, I'll introduce you to GulpJS and show you how to set it up in your application for task automation.

---

**[GulpJS](https://gulpjs.com)** is a JavaScript task runner that lets you automate several tasks during development. These tasks involve minifying JavaScript and CSS files, automatically refreshing the browser once a file has been edited, compiling CSS preprocessors, running tests, compressing images, and several others. GulpJS solves the problem of repetition.

<include src="TweetQuote" quoteText="GulpJS solves the problem of repetition."/>

**GulpJS** is a build tool that helps you as a developer get rid of manually running such tasks as mentioned above during development for every project. Once you set up GulpJS, it automates the most frustrating tasks, supercharges your performance, and lets you focus on building your app logic.

## GulpJS Requirements

In order to use GulpJS, you need to have the following tools installed on your machine.

* **Node.js**: Navigate to the [Node.js website](https://nodejs.org/en/download/) and install the latest version on your machine.
* **GulpJS**: Install Gulp globally in your terminal so that the `gulp` command can be run from any directory.

```bash
npm install gulp-cli -g
```

## GulpJS Features

GulpJS provides a standardized API. The API methods allow you accomplish whatever work you want done with Gulp. See below a quick run down of the API methods:

* `gulp.task` - Define a task.

    ```js
    gulp.task('dosometask', () => {
      // Do some task
    });
    ```
* `gulp.src` - Read a file or set of files in.

    ```js
      gulp.src('client/templates/*.jade')
      .pipe(jade())
      .pipe(minify())
      .pipe(gulp.dest('build/minified_templates'));
    ```

* `gulp.dest` - Write files out to a directory.

    ```js
    gulp.src('./client/templates/*.jade')
    .pipe(jade())
    .pipe(gulp.dest('./build/templates'))
    .pipe(minify())
    .pipe(gulp.dest('./build/minified_templates'));
    ```

    ```js
    gulp.src('client/js/**/*.js') // Matches 'client/js/somedir/somefile.js' and resolves `base` to `client/js/`
    .pipe(minify())
    .pipe(gulp.dest('build'));  // Writes 'build/somedir/somefile.js'

    gulp.src('client/js/**/*.js', { base: 'client' })
      .pipe(minify())
      .pipe(gulp.dest('build'));  // Writes 'build/js/somedir/somefile.js'
    ```

* `gulp.watch` - Watch files and do something when a file changes.

    ```js
    gulp.watch('js/**/*.js', function(event) {
      console.log('File ' + event.path + ' was ' + event.type + ', running tasks...');
    });
    ```

Next, let's look at how we can actually use GulpJS in an application. We need to create a gulpfile.

## Setting Up GulpFile in an Application

In a typical project, GulpJS needs some form of instructions to guide it in automating your tasks. These instructions are written in a file called the `gulpfile`. Let's go ahead and create this file.

Make sure you are in a new project directory. Inside the directory, create a `gulpfile.js` file.

In the `gulpfile.js` file, add this code to it like so:

```js
var gulp = require('gulp');

gulp.task('run-this-common-task', function(){
    console.log('This is a simple task. Now, run it');
});
```

We just added a very simple task to the gulpfile. You need to always require the `gulp` node module before you can define and run tasks.

Before running the task, go ahead and install gulp locally in your project like so:

```bash
npm install --save-dev gulp
```

Now, run your task with gulp:

```bash
gulp run-this-common-task
```

![GulpJS - Run common task](https://images.ctfassets.net/23aumh6u8s0i/y7nFTaRBpjkZbMgcjgY8g/57f7541bbbe19ec53607ccea2183b5a1/run-common-task)
_GulpJS: Run common task_

The task had just one job - Log a sentence to the console. And it successfully executed this task.

Let's take a stab at something that gives us a better overview of the different kind of operations gulp can handle in your application.

## GulpJS Use Case: Project Lingo

This is a practical use case. We are going to paint a scenario of the tasks we need to accomplish in _Project Lingo_. Before we define such tasks, what the heck is _Project Lingo_?

**Lingo** is a project that we have embarked upon in Company TechnoVille. _Project Lingo_ is set to take over the world in ways you can't imagine. One of such ways is allowing a mere human click a button, input a destination and Lingo automatically teleports the human to their preferred destination in approximately 5 minutes.

In _Project Lingo_, we have a bunch of JavaScript, Less and Image files. We need to compress and optimize these files so that Lingo can be super-fast. I mean, you don't want Lingo taking 3 hours to teleport a human because of our in-efficient developer work-flow. We also don't want investors running way & Project Lingo dying when we could have done something about it.

This is what we need to do:

* Concatenate all our JavaScript files into just one file. This will make sure our app makes one HTTP request while serving JavaScript rather than multiple HTTP requests.
* Compile our Less files into CSS. The browser understands CSS, not LESS.
* Compress all our image files.
* Detect errors and potential problems in our JavaScript code. We need to make sure Lingo developers are writing quality JavaScript code.

We have these requirements listed above. How do we approach tackling these tasks with GulpJS?

## Project Lingo - Install GulpJS Plugins

Yes, GulpJS has an amazing [ecosystem of plugins](https://gulpjs.com/plugins). For every task a developer tries to accomplish, there is probably a GulpJS plugin out there for automating such task. Go ahead and install the following plugins via your terminal:

```bash
npm install gulp-imagemin gulp-less gulp-jshint gulp-concat gulp-uglify gulp-rename --save-dev
```

<include src="TweetQuote" quoteText="GulpJS has an amazing ecosystem of plugins."/>

A quick breakdown of what each of these plugins aim to accomplish.

* **[gulp-imagemin](https://www.npmjs.com/package/gulp-imagemin)** - Minify PNG, JPEG, GIF and SVG images.
* **[gulp-less](https://www.npmjs.com/package/gulp-less)** - Compile Less files to CSS.
* **[gulp-jshint](https://www.npmjs.com/package/gulp-jshint)** - Automatically detect errors and problems in our JavaScript code. Lint it!
* **[gulp-uglify](https://www.npmjs.com/package/gulp-uglify)** - Minify JavaScript files.
* **[gulp-rename](https://www.npmjs.com/package/gulp-rename)** - Rename a file.
* **[gulp-concat](https://www.npmjs.com/package/gulp-concat)** - Concatenate all the JavaScript files into one file.

## Project Lingo - Rewrite GulpFile

Open up the `gulpfile.js` file and modify it to this code below:

```js
const gulp = require('gulp');

const imagemin = require('gulp-imagemin');
const jshint = require('gulp-jshint');
const concat = require('gulp-concat');
const less = require('gulp-less');
const uglify = require('gulp-uglify');
const rename = require('gulp-rename');

// Lint Task
gulp.task('lint', () => {
    return gulp.src('js/*.js')
        .pipe(jshint())
        .pipe(jshint.reporter('default'));
});

// Compile Project Lingo Less files
gulp.task('less', () => {
    return gulp.src('less/*.less')
        .pipe(less())
        .pipe(gulp.dest('dist/css'));
});

// Concatenate & Minify Project Lingo JS files
gulp.task('scripts', () => {
    return gulp.src('js/*.js')
        .pipe(concat('build.js'))
        .pipe(gulp.dest('dist'))
        .pipe(rename('build.min.js'))
        .pipe(uglify())
        .pipe(gulp.dest('dist/js'));
});

// Compress all Project Lingo image files
gulp.task('compress-images', () => {
     gulp.src('images/*')
        .pipe(imagemin())
        .pipe(gulp.dest('dist/images'))
});

// Watch Files For Changes
gulp.task('watch', () => {

    // JavaScript changes
    gulp.watch('js/*.js', ['lint', 'scripts']);

    // Less changes
    gulp.watch('less/*.less', ['less']);

    // Image changes
    gulp.watch('images/*', ['compress-images']);
});

// Run Project Lingo Task
gulp.task('lingo', ['less', 'scripts', 'compress-images', 'watch']);
```

In the code above, we have six tasks. Let's examine what each of these tasks do.

* **lint task** - This task checks all the JavaScript files in our `js` directory and runs them through `jshint`. Jshint ensures that the JavaScript code is well-written and rid of errors. If there are any errors, it reports them to the console.

    ```js
    gulp.task('lint', () => {
    return gulp.src('js/*.js')
        .pipe(jshint())
        .pipe(jshint.reporter('default'));
    });
    ```
* **less task** - This task checks all the Less files in our `less` directory, compiles them to CSS and copies them to a `dist/css` directory. If you use sass more often, there is a [gulp-sass](https://www.npmjs.com/package/gulp-sass) plugin available to compile your sass files to CSS.

    ```js
    gulp.task('less', () => {
        return gulp.src('less/*.less')
            .pipe(less())
            .pipe(gulp.dest('dist/css'));
    });
    ```
* **scripts task** - This task checks all the JavaScript files in our `js` directory, concatenates them into a single file, `build.js`, copies the file into a `dist/js` directory, renames the file to `build.min.js` and uglifies it.

    ```js
    gulp.task('scripts', () => {
        return gulp.src('js/*.js')
            .pipe(concat('build.js'))
            .pipe(gulp.dest('dist'))
            .pipe(rename('build.min.js'))
            .pipe(uglify())
            .pipe(gulp.dest('dist/js'));
    });
    ```
* **compress-images task** - This task checks all the files in our `images` directory, compresses them and copies them to a `dist/images` directory.

    ```js
    gulp.task('compress-images', () => {
         gulp.src('images/*')
            .pipe(imagemin())
            .pipe(gulp.dest('dist/images'))
    });
    ```
* **watch task** - This task listen for changes made in our files and automatically run all our tasks again.

    ```js
    gulp.task('watch', () => {

        // JavaScript changes
        gulp.watch('js/*.js', ['lint', 'scripts']);

        // Less changes
        gulp.watch('less/*.less', ['less']);

        // Image changes
        gulp.watch('images/*, ['compress-images']);
    });
    ```
* **lingo task** - This task is the father of all tasks. At the terminal, all we need to do is run `gulp lingo` and all the tasks defined in our gulfile will run in one command!

    ```js
    gulp.task('lingo', ['less', 'scripts', 'compress-images', 'watch']);
    ```

So, head over to the terminal, and try `gulp lingo` and watch all the tasks run. It's that simple.

**Note:** Ensure you have a `js`, `less` and `images` folder with multiple JavaScript, Less and image files respectively.

## Gulp Plugins

As demonstrated in _Project Lingo_, we took advantage of the Gulp Plugins ecosystem. Gulp Plugins are building blocks for your gulpfile.

There are several plugins available at [https://gulpjs.com/plugins](https://gulpjs.com/plugins/). Head over there, search for any plugin that might suit your use case and take advantage of it.

Check out other popular plugins that you might immediately find useful.

* **gulp-util** - Contains all sorts of utility functions such as color-coding and logging.
* **gulp-nodemon** - Automatically restarts your Node.js server using `nodemon`.
* **gulp-strip-debug** - Removes all `console` and `debugging` statements.
* **gulp-htmlclean** - Minify HTML code.

## Gulp Recipes

There are some tasks that you might want to implement, but not necessarily know the right way to go about it. Apart from the Gulp Plugins that exist, there are recipes that you can take advantage of. Some of the recipes you can use directly in your project include:

* [Running tasks in series](https://github.com/gulpjs/gulp/blob/master/docs/recipes/running-tasks-in-series.md)
* [Browserify builds with Watchify](https://github.com/gulpjs/gulp/blob/master/docs/recipes/fast-browserify-builds-with-watchify.md)
* [Templating with Swig and YAML front-matter](https://github.com/gulpjs/gulp/blob/master/docs/recipes/templating-with-swig-and-yaml-front-matter.md)
* [Deleting files and folders before running builds](https://github.com/gulpjs/gulp/blob/master/docs/recipes/delete-files-folder.md)
* [Run Grunt Tasks from Gulp](https://github.com/gulpjs/gulp/blob/master/docs/recipes/run-grunt-tasks-from-gulp.md)

Check out a great list of [Gulp Recipes](https://github.com/gulpjs/gulp/tree/master/docs/recipes).

<include src="asides/JavascriptAtAuth0" />

## Conclusion

You have now learned how to automate your development workflow with GulpJS. This GulpJS tutorial simply covers how Gulp - the task runner and automation tool works and how it can automate common tasks in your application. I am rest assured _Project Lingo_ is now highly performant. All thanks to the optimization techniques we deployed using GulpJS.

There are other tasks runners and automation tools you should check out. Examples are [Grunt](https://gruntjs.com), [Broccoli](https://github.com/broccolijs/broccoli), [Brunch](https://brunch.io) and [Webpack](https://github.com/webpack/webpack). As at the time of this writing, Webpack is the most popular build tool/task runner amongst web developers.

Do you still use GulpJS? Are you now in the league of Webpackers? Are there specific reasons you would choose one over the other? Please let me know in the comment section. 😊


Automate Your Development Workflow With GulpJS

df6c864847fba9687d962cb80b482764?size=200 avatar

df6c864847fba9687d962cb80b482764?size=200

Martin Gontovnikas, a.k.a Gonto, is a software engineer at heart who moved to the ‘dark side’ and became VP of Marketing at Auth0. He considers himself lucky to have found a way to combine his two passions and apply his engineering thinking model to marketing.<br/><br/> Gonto’s analytical thinking is a huge driver of his data-driven approach to marketing strategy and experimental design. He is based in the Bay area, and in his spare time, can be found eating gourmet food at the best new restaurants, visiting every local brewery he can find, or traveling the globe in search of new experiences.

Former SVP of Marketing and Growth at Auth0

Martin Gontovnikas

Amazon's bold play for Whole Foods isn't just about the impending automation of the retail sector, it's about the increasing importance of IAM in redefining consumer behavior.



Amazon is poised to reinvent the grocery store from the ground up. They've been experimenting with cutting-edge retail automation through their Amazon Go experiment, leading to strong speculation about how they're going to enter the brick and mortar grocery market. Now we've got our answer: with Whole Foods.

The high-end organic retailer is likely going to be a pilot for applying the highly anticipated Amazon Go model: a way for consumers to shop without ever waiting in line for a register — a massive shift that brings identity to the center of the retail experience. This is a bold play that is going to change the way [IAM works in retail](https://auth0.com/blog/jda-standardizes-iam-with-auth0/), and forever change consumer expectations and behavior.

In this article, we're going to break down the underlying architecture of Amazon's retail automation model, its impact on consumers, and highlight the ways in which [identity management](https://auth0.com/b2c-customer-identity-management) stands as the keystone of this entire [retail revolution](https://auth0.com/retail/).

## How Amazon Rethinks Shopper's Identities

Amazon has well-established forays into the grocery space with its Prime Pantry and Amazon Fresh services, but its first brick and mortar entry is the Amazon Go beta store in Seattle. As an entirely cashierless store, customers scan their phones as they enter the store, shop for the items they want, and leave the store without waiting.

![Wholefoods](https://images.ctfassets.net/23aumh6u8s0i/0rAvRGZ0WdOucQf3XEUnf/d9f816ec8e6bf2c9a3f9929bdd41deab/new)
_[Source](https://www.engadget.com/2016/12/05/amazon-go-grocery-store/)_

The technology is an ingenious combination of computer vision, sensor fusion, and Amazon's ability to assign these datapoints to specific customers. Cameras observe, track, and identify customers as they make purchases, and these are matched up with their Amazon accounts, allowing them to simply leave the store knowing that their purchases have been attributed to their accounts.

This is possible due to Amazon's 2014 [patent filing](http://bit.ly/2tWF8gB) for “Transitioning Items From a Materials Handling Facility”, which reveals the underlying systems at play that enable sensor data to be applied to specific customer identities.

According to the patent, an item identifier list — a customer's cart, so to speak is said to use skin tone identification to keep a customer's carts distinct from one another while in-store. Upon exiting the store, the user is identified so this internal store shopping list can be attributed to a customer's Amazon account. This reveals that identity is the key to seamlessly merging back-end stock logistics to front-facing customer billing.

Amazon Go is clearly a new benchmark for IoT retail. With its system, a store can at once know the real-time status of every item, attribute these items to individuals, and then instantly bill them for it.

The behavioral analytics potential of linking all stages of the supply and purchasing funnel is immense, allowing Amazon to pursue a more agile and responsive brick and mortar retail strategy. Customers might even be able to get real-time suggestions based on their shopping lists, or even get coupons for items they're mulling over — all because of identity.

## Implications for Whole Foods

Amazon's announcement of its intent to purchase Whole Foods not only has the potential to upend the grocery sector, it also promises to radically alter the retail sector in terms of identity management and employment policy.

![Whole foods Market](https://images.ctfassets.net/23aumh6u8s0i/2GiMCCBpLnwQt0sdgLnhLl/7dc8751c4acb1a46c19bea59cd9c5157/foods)
_[Source](http://www.wholefoodsmarket.com/stores/southlakeunion)_

Amazon's blueprint for a hyperconnected identity and authentication management system is in many ways unchartered territory. Grocers already do track shopping analytics through [reward card incentives](https://www.theguardian.com/money/2013/jun/08/supermarkets-get-your-data), which allow them to develop a complex view of their customer's behaviors. However, Amazon's novel unification of these backend analytics with specific customer identities increases the value of this data collection.

The consolidation of both front-end analytics and back-end logistics may be able to offer a whole host of new efficiencies in the grocery sector. Whether it be through a more agile inventory management system that reduces waste and spoilage, or by automation-led workforce reduction, Whole Foods may be able to pursue a more targeted and lean approach to its operations.

Additionally, given Whole Food's affluent and aspirational demographic reach, the experimental appeal of a new, automated shopping experience could prove to be a huge draw, not only for those who are excited to try the latest tech, but for those who will benefit from a more effortless shopping experience.

## Implications for Customers

It's still unclear as to how Amazon intends to scale its Amazon Go technology across hundreds of much larger stores across the US. Even murkier is how customers will react to these large, systemic changes in consumer behavior. Amazon will have to work to ensure that the friction of getting customers up and running in an Amazon Go-style store isn't prohibitive.

### Advantages for Customers:

The advantages of this shift to identity-centric IoT retail are pretty clear:

* Eliminations of lines, shoplifting, and cashiers could lead to savings that are passed along to consumers.
* Customers can access new insights into their spending and consumption patterns.
* The personalized experience of online shopping with hyper-targeted discounts and suggestions can now be experienced offline.
* Store stock can be more responsive to the needs and preferences of customers.

### Disadvantages for Customers

With all this flexibility that comes with identity-centric automation, there also comes the sense that Amazon is seeking to expand its walled garden ever outward.

* Consumers may not be comfortable with the accuracy of automated cashierless stores, or they might reject attempts to customize their shopping experience based on identification.
* Given Amazon's recent patent filing for [blocking comparison shopping-in store](http://tu9srvbirvvtmtukd3d3lnbhddjwzgyub3jn.g00.pcmag.com/g00/2_d3d3LnBjbWFnLmNvbQ%3D%3D_/TU9SRVBIRVVTMTUkaHR0cHM6Ly93d3cucGF0MnBkZi5vcmcvcGF0ZW50cy9wYXQ5NjY1ODgxLnBkZg%3D%3D_$/$/$), it seems like the overall approach isn't simply to make shopping more effortless for its customers, but also to proactively shape buyer behavior.
* While the convenience and hyper-targeting afforded by the Amazon Go model is a draw, there's also a chance that it alienates customers who already feel overly reliant on Amazon's retail ecosystem.

As retail automation becomes more reliant on customer identity to function, retailers should keep in mind that customers are willing to play along if there's a net benefit to them. Once they start to feel walled in or beholden to a monopolized retail experience, things can go awry, so it is in the best interest of retailers to not visibly commodify customer identity when developing IAM-centric automated retail platforms.

## The Road Ahead

Amazon's bold acquisition of Whole Foods indicates its willingness to radically upend one of the most staid and reliable sectors of consumer spending. While Amazon Go is a clear model for what Amazon wants to do in the grocery space, it's still unclear as to how it will apply this model to Whole Foods.

Regardless, it is clear that the future of retail is just as much about identity management as it is about the waning days of the cashier age. Amazon is betting that consumers are willing opt-into the necessary identity procedures that enable them to take advantage of seamless checkout and billing procedures. With Whole Foods now in its portfolio, Amazon will surely have plenty of room to experiment and refine this new paradigm of the retail experience.

## Aside: Securing Applications with Auth0

Are you building a [B2C](https://auth0.com/b2c-customer-identity-management), [B2B](https://auth0.com/b2b-enterprise-identity-management), or [B2E](https://auth0.com/b2e-identity-management-for-employees) tool? Auth0, can help you focus on what matters the most to you, the special features of your product. [Auth0](https://auth0.com/) can improve your product's security with state-of-the-art features like [passwordless](https://auth0.com/passwordless), [breached password surveillance](https://auth0.com/breached-passwords), and [multifactor authentication](https://auth0.com/multifactor-authentication).

[We offer a generous **free tier**](https://auth0.com/pricing) so you can get started with modern authentication.


Why Amazon and Whole Foods Will Change How You Shop

54f3fecf3ce3bbef2d7a62b4f0496de2

Thameera Senanayaka

Learn how to create a bot that posts mentions in Zendesk to relevant Slack users using the Auth0 Webtask platform



If you’re like me, whenever you get a cool idea to create an app, the mere thought of preparing and maintaining a server always gets you down—and then there’s all the security considerations you have to think about. The advent of PaaS services like Heroku solved this problem to some extent. Now that serverless technologies have come into the mainstream, making your ideas reality is easier than ever.

Serverless platforms allow you to focus more on the code and get real work done without worrying about managing servers. Many people prefer the term function-as-a-service (FaaS) instead of serverless to serverless, because what essentially functions are the unit of application logic in this realm. Auth0's [Webtask](https://webtask.io/) is a hassle-free serverless platform you can use to deploy and run functions.

At Auth0, we tend to solve most of our problems using Webtasks. Here's an example of how we used them to create a Slack bot to tackle a pressing need we had.

## Go Beyond Basic Slack and Zendesk Integration

We’re not trying to brag, but Auth0 is known for its wonderful customer service. We use Zendesk to manage the hundreds of customer queries we get every day. Since this is an integral part of our day-to-day work, we use multiple Zendesk-to-Slack integrations to make sure we respond to every customer query in a timely manner.

For a long time, we needed the ability to mention someone else in a ticket. The only out-of-the box option was to cc that person’s email, but this had its own drawbacks. We wanted to achieve an integration that would go hand-in-hand with our primary method of communication: Slack. Zendesk has some basic Slack integration options, but it’s not programmable enough to handle mentions.

Then we thought, “Can we use a webtask to achieve this?”

## Webtasks to the rescue

Of course we can! Any time you want to automate something, you can use a webtask. We quickly implemented a Zendesk-mentions-to-Slack integration in a webtask. We didn’t need to procure any servers or worry about how much computing power or uptime we would need. All we had to do was write the code and create a trigger in Zendesk.

## Overview

Here’s an overview of the data flow in our app:

![Slack and Zendesk Integration](https://images.ctfassets.net/23aumh6u8s0i/4FPj0WOpBZklsSsdchQd0V/507cef8b51d949b8b61e23c83a55f5f5/slack-and-zendesk-integration)

Peter, a Zendesk agent, mentions his colleague Rodrigo as `<@rodrigo>`. That’s Rodrigo’s Slack username. A Zendesk trigger picks up the comment with a mention and does an HTTP POST request to our webtask. The webtask analyzes the comment and determines the exact Slack user (`@rodrigo` in this case). It then calls the Slack API to send a direct message to `@rodrigo` with details about the comment. If Peter mentioned a Slack channel name instead of a username, it would be sent to that specific Slack channel.

To set this up, we needed to do the following:

- Create a Slackbot
- Write and deploy the webtask
- Create a trigger in Zendesk

## 1. Create a Slackbot

Simply visit [this page](https://my.slack.com/services/new/bot) and enter a name for the bot. A new Slack bot will be created. After it has been created, an API token will be assigned to the bot. Make a note of this because we will need it in the next step.

## 2. Create the webtask

Now the fun begins. First, if you haven’t set up the webtask CLI, follow the three simple steps on this [webtask.io](https://webtask.io/cli) page. Once you’re done, it’s time to write the code.

Here's the code we used at Auth0. You may edit it to your liking.

```javascript
'use latest';

const request = require('request');
const _ = require('lodash@4.8.2');

const baseURL = 'https://slack.com/api/';
const usersListEndpoint = 'users.list';
const imOpenEndpoint = 'im.open';
const chatPostEndpoint = 'chat.postMessage';
let token;

/* Call the given endpoing in Slack API */
const callAPI = (endpoint, form, cb) => {
  request.post(baseURL + endpoint, {form}, (err, res, body) => {
    if (err) return cb(err);

    body = JSON.parse(body);
    if (!body.ok) return cb(body.error);

    return cb(null, body);
  });
};

/* Find Slack ID of the user with given username */
const findUser = (username, cb) => {
  callAPI(usersListEndpoint, {token}, (err, body) => {
    if (err) return cb(err);

    const user = _.find(body.members, {name: username});

    if (!user) return cb(`User ${username} not found`);
    cb(null, user.id);
  });
};

/* Open a direct msg channel with given Slack user id */
const openIM = (user, cb) => {
  callAPI(imOpenEndpoint, {token, user}, (err, body) => {
    if (err) return cb(err);
    cb(null, body.channel.id);
  });
};

/* Post message to specified Slack channel */
const postMsg = (channel, data, cb) => {
  const obj = {
    title: `#${data.id} | ${data.title}`,
    title_link: 'https://auth0.zendesk.com/agent/tickets/' + data.id,
    fields: [
      {title: 'Mentioned by', value: data.author},
      {title: 'Comment', value: data.comment},
      {title: 'Tags', value: data.tags}
    ]
  };
  callAPI(chatPostEndpoint, {
    token,
    channel,
    as_user: false,
    username: 'Zendesk Mentions Bot',
    icon_url: 'http://i.imgur.com/IhN4IzR.png?1',
    text: 'You were mentioned in this ticket:',
    attachments: JSON.stringify([obj])
  }, (err, body) => {
    if (err) { console.log(err); return cb(err); }
    cb(null);
  });
};

/* Extract mentioned user's name from comment */
const extractName = (comment) => {
  const start = comment.indexOf('<@') + 2;
  const end = comment.substr(start).indexOf('>');
  return comment.substr(start, end);
};

module.exports = (context, cb) => {

  // Slack bot token
  token = context.data.BOT_TOKEN;

  const name = extractName(context.data.comment).toLowerCase();

  findUser(name, (err, id) => {
    if (err) {
      // If no such user, assume it's a channel
      return postMsg(name, context.data, cb);
    }

    return openIM(id, (err, channelId) => {
      if (err) { console.log(err); return cb(); }
      else postMsg(channelId, context.data, cb);
    });
  });

};
```

That’s less than 100 lines of code!

The entry point to the code is the line starting with `module.exports = (context, cb) => {`. When the webtask is triggered, this function will be executed, which will in turn call the rest of the functions when necessary.

For security reasons, we add the bot’s token as a configuration (as described below) instead of in the code. Then we find the mentioned name from the comment text. We check Slack to see if such a user exists. If not, we assume it’s a channel name and post it into the channel with that name. If a user is found, we open an IM channel with that user and post the message.

Save this code in a file named `slackmention.js` and enter the following command to deploy it as a webtask:

```
wt create --secret BOT_TOKEN=<Slack bot token from step 1 goes here> slackmention.js
```

After the webtask is deployed, the CLI will present you with its unique URL. We will use this in the next step.

## 3. Create a trigger in Zendesk

The last step is to tell Zendesk to call our webtask when it sees a comment that might have a mention in it. You need to be an admin in Zendesk to do this. If you are not, reach out to your friendly Zendesk admin (and don’t forget to give them a cookie!).

In Zendesk, go to **Settings > Extensions** and create a new HTTP target with the webtask’s URL as the URL (obtained in Step 2), POST as the method, and JSON as the content type.


Now go to **Settings > Triggers** and create a new trigger with _‘Ticket: Comment text...’_ as the condition, “Contains the following string” as the op, and `<@` as the string. The performed action should be _‘Notifications: Notify target’_ set to the HTTP target created above. Set the JSON body as follows:

```json

{ "id": "{{ticket.id}}", "title": "{{ticket.title}}", "comment": "{{ticket.latest_comment}}", "author": "{{ticket.latest_comment.author.name}}", "tags": "{{ticket.tags}}" }

```

## Test it!

Now we are all done. To test our new setup, add the following comment in a new (or existing) Zendesk ticket. You can add it to a public comment or an internal comment.


```
Hello <@thameera>!
```

Replace `thameera` with your Slack username, of course. If you performed all the steps in order, you will get a Slack notification like this:

![Slack mention in the Zendesk ticket](https://images.ctfassets.net/23aumh6u8s0i/6vYCYmGihKFIEtHrecV9Kd/04d132295f04d63293749edb6f7b0f0a/zendesk-bot-with-slack-mention)

That’s it! You can customize the trigger and the webtask to suit your company’s needs and deploy your own Zendesk-to-Slack-mentions implementation in a few minutes. For example, you can modify it to send an email or send a DM via HipChat if that’s your chat application of choice.

## Conclusion

The next time you need to create a bot, automate something, or write a program, remember that you can use a webtask to say no to [yak shaving](http://www.hanselman.com/blog/YakShavingDefinedIllGetThatDoneAsSoonAsIShaveThisYak.aspx) and focus on your code.

You can start experimenting with webtasks by creating a free Auth0 account below or heading directly to [https://webtask.io](https://webtask.io). Enjoy!


Build a Serverless Slack Bot with Webtask.io

71313b369e8f23b7ec7b4614acdf3031?s=200 avatar

71313b369e8f23b7ec7b4614acdf3031?s=200

Staff Security Engineer

Radek Karpowicz

Automation which helps you monitor the multi-factor authentication status of various services.



Imagine a situation where your company hires more and more people and uses a lot of external services. The number of available assets you need to track is always growing. It's hard to keep up with all the services that employees at your company use on a day-to-day basis. Are you sure that you have appropriate security checks in place?

## Proactive protection

Monitoring your assets can be a manual or an automated job. If you implement the right solution, it could give you almost an instant notification when there's a problem. It's often a missing feature on the service provider side to notify you when someone disables MFA protection for one of your organization's accounts.

If you go the manual route, that just creates more routine work. More routine at work (and sometimes in life) doesn't lead to good things. Also, with shifting priorities everyday, it's easy to skip a manual process or even forget about them. Automation is a better solution.

By being more proactive and productive, we mean automation and an appropriate solution to the problem. At Auth0, we want to automate as many things as we can and decrease the number of alerts we receive to get only the real threats and not noise. It gives us the best overview of our infrastructure's security status.

## The MFA Monitoring tool

The [MFA monitor](https://github.com/radekk/webtask-mfa-monitor) uses an API to fetch information about the MFA status for every user under your organization, checks the previous state of MFA, and alerts you when that status changed (enabled or disabled state).

![Slack MFA alert](https://images.ctfassets.net/23aumh6u8s0i/5OJnSCaBGuhks7eZ8ue5RV/3e2fa24f916f501cdce48819cd42b7ff/slack)

Possibilities for implementing additional monitors or notifiers which will receive your alerts are endless as long as there's an API on the provider side. For now, we have implemented checks for GitHub and Amazon Web Services (AWS), we added Slack as a notification provider. If you use Hipchat or Twitter for notifications, you can quickly add them as an additional module. Feel free to send pull requests!

### Implementation

The MFA Monitor was implemented in JavaScript with Node.JS as a target platform. The toolkit consists of two main parts - notification and monitor modules.

#### Monitors

The monitor module was implemented to provide the possibility to inspect the status of MFA settings of different services. You can quickly add new controls as far as the service you want to monitor allows you to fetch MFA status over an API or another channel. What monitor module requires is to receive the status of protection from the external service.

As an example, the GitHub monitor was implemented using publicly available API to fetch information about users assigned to your organization. Requirements could be different for every service as GitHub requires an API token with access to the list of members of your organization. Always remember to choose privileges for access tokens wisely.

The important part of the monitor module is the callback that returns information about all user logins with MFA disabled:

```javascript
  client.orgs.getMembers({
    org: ctx.secrets.ORGANIZATION,
    per_page: 100,
    filter: '2fa_disabled'
  }, (err, res) => {
    if (err) return cb(err);
    if (!res) return cb(new Error('Empty response received from Github API!'));

    cb(null, res.map(data => data.login));
  });
```

#### Notifier

Notifiers are modules that were created to notify users about the security event that occurred on the 3rd party service - it could be information about enabled or disabled MFA on the particular account.

The first module we implemented was Slack, but it doesn't mean you can only use chat applications. It's able to send notifications directly to a URL, Twitter, or even TCP socket; it depends on what you need and want to implement.

To implement notifier modules and secure required tokens you can use secrets that could be stored by the [Webtask](https://webtask.io) platform. After that you can call them using ``ctx.secrets`` prefix.

```javascript
  const slack = require('slack-notify')(ctx.secrets.SLACK_WEBHOOK_URL);
  const mfaStatus = !!ctx.body.mfaStatus;
  slack.onError = (err) => cb(err);

  ctx.body.data.forEach(service => {
    let accounts = service.accounts.map(account => `*${account}*`).join(', ');
    let message  = getMessage(service.name, accounts, mfaStatus);

    slack.send({
      channel: ctx.secrets.SLACK_CHANNEL_NAME,
      username: 'MFA Agent',
      text: message
    });
  });
```

#### Storage

An early version of this tool was using MongoDB as a storage. It generated many additional steps to setup this database. As for the latest release, it's using built-in storage into the Webtask service. In short it means that you don't need to use any external database and worry about the security of your data, now it's safe, encrypted, and stored locally.

```javascript
db.init(ctx.storage);
```

Using the [Webtask storage](https://webtask.io/docs/storage) is easy, to write or read data you need to call methods on the ``ctx.storage`` object. Below is the implementation of the storage wrapper which helps to handle storing and receiving data using Promises.

```javascript
'use strict';

const Promise = require('bluebird').Promise;
let db;

module.exports = {
  init: (storage) =>
    db = storage,

  getStoredData: () =>
    new Promise((resolve, reject) =>
      db.get((err, data) => {
        if (err) return reject(err);
        resolve(data || []);
      })
  ),

  storeData: (data) =>
    new Promise((resolve, reject) =>
      db.set(data, {force: 1}, err => {
        if (err) return reject(err);
        resolve();
      })
  )
};
```

You can easily implement additional storage modules if needed. We wanted to keep it as simple as possible, that's why we decided to use built-in storage which is also the best way to work with the Webtask platform.

### Installation

Setting up the MFA monitoring in place requires creating the Webtask account for free ([https://webtask.io](https://webtask.io)). It's a server-less environment, and it means that you don't need any additional infrastructure to monitor your assets. Fewer assets means less work for you.

![MFA monitor configuration process](https://images.ctfassets.net/23aumh6u8s0i/5a6kFSy2amtxyZ94pFc4cR/7067346475056cac6d9cb16c5eea2aad/mfa)

After creating the Webtask account you need to clone the MFA monitor repository from GitHub, install required npm packages (``npm install``) and you're ready to go. The configuration process is rather straightforward, and it asks you for all necessary data like GitHub token, Slack token, etc.

By default, it installs GitHub monitoring and sends alerts to Slack. In case if you want to modify it, you need to write additional monitors or notifiers. You can find more details on the official repository [README](https://github.com/radekk/webtask-mfa-monitor/blob/master/README.md) file.

### Usage

Once installed MFA monitor shouldn't require any maintenance steps. It's working as a Webtask cron job which means that it runs periodically to pull out the data from your services and checks the status of MFA setting.

You can periodically check the log stream to verify that MFA monitor is working or verify if the status is monitored.

## Awareness

Keep in mind that monitoring the MFA status will not protect you against other malicious activities that could occur. The purpose of this particular tool is to alert you every time your users forget to enable MFA.

<include src="TweetQuote" quoteText="The MFA monitoring tool will alert you every time your users forget to enable MFA."/>


This tool works well for us; it fits into our methodology of being a proactive team. Setup your monitors and start protecting your accounts now.


"automation"