Ask any AI agent to implement Auth0.
Choose the LLM, choose the framework, and see how those work with your Auth0 implementation.
Compare stacksWithout Auth0 tools
SDK setup & imports
Tokens stored securely
75%
avg. success rate
With Auth0
SkillsMCP
SDK setup & imports
Tokens stored securely
91%
avg. success rate16%
Tested across 5 models · 12 frameworks
Agent Experience Score for Auth0
Validated results for every model, framework, and configuration tested with Auth0
Select a framework to filter results for your stack.
#
Model
Framework
Without tools
With Auth0 tools *
Grade
Est. Cost
Est. Time *
1
Claude 4.6 Sonnet
Express
87%13/15
100%+13%
22/22$0.10
1m 40s
With Auth0 Tools·100%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (22/22 passed)
L6 · Holistic(22/22)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) sto
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile rout
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: doe
Does the solution correctly integrate Auth0 into an Express web app using expres
2
Claude 4.6 Opus
Fastify API
57%8/14
100%+43%
19/19$0.12
49s
With Auth0 Tools·100%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /ap
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: d
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth
3
Gemini 3.1 Pro
SPA JS
93%13/14
99%+6%
20/20$0.22
4m 40s
With Auth0 Tools·99%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
13.3/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render use
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using
4
Claude Opus 4.7
Express
87%13/15
99%+12%
22/22$0.62
1m 7s
With Auth0 Tools·99%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
13.9/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
24/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (22/22 passed)
L6 · Holistic(22/22)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) sto
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile rout
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: doe
Does the solution correctly integrate Auth0 into an Express web app using expres
5
Claude Opus 4.7
Fastify API
50%7/14
99%+49%
19/19$0.59
1m 3s
With Auth0 Tools·99%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /ap
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: d
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth
6
Claude 4.6 Opus
SPA JS
100%14/14
98%-2%
20/20$0.17
1m 8s
With Auth0 Tools·98%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.6/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render use
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using
7
Gemini 3.1 Pro
Vue
100%13/13
98%-2%
19/19$0.25
2m 31s
With Auth0 Tools·98%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.6/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticat
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/aut
8
Claude 4.6 Opus
Express
87%13/15
98%+11%
22/22$0.15
1m 5s
With Auth0 Tools·98%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.9/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
24.3/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (22/22 passed)
L6 · Holistic(22/22)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) sto
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile rout
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: doe
Does the solution correctly integrate Auth0 into an Express web app using expres
9
Gemini 3.1 Pro
Fastify API
43%6/14
98%+55%
19/19$0.18
2m 54s
With Auth0 Tools·98%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /ap
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: d
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth
10
Claude 4.6 Sonnet
SPA JS
100%14/14
97%-3%
20/20$0.12
1m 58s
With Auth0 Tools·97%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
13.3/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
7.2/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render use
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using
11
GPT-5.4
SPA JS
100%14/14
97%-3%
20/20$0.39
1m 44s
With Auth0 Tools·97%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render use
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using
12
Gemini 3.1 Pro
React
100%13/13
97%-3%
19/19$0.22
2m 31s
With Auth0 Tools·97%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.3/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticat
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: doe
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider,
13
Claude 4.6 Opus
React
92%12/13
97%+5%
19/19$0.16
1m 12s
With Auth0 Tools·97%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticat
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: doe
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider,
14
GPT-5.4
React
92%12/13
97%+5%
19/19$0.52
2m 9s
With Auth0 Tools·97%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticat
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: doe
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider,
15
Claude 4.6 Sonnet
React
100%13/13
96%-4%
19/19$0.17
2m 2s
With Auth0 Tools·96%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
9.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticat
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: doe
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider,
16
Claude Opus 4.7
SPA JS
100%14/14
96%-4%
20/20$0.66
1m 6s
With Auth0 Tools·96%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
13.3/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.6/14 pts
Error Recovery
When something broke, did the agent fix it?
7.2/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render use
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using
17
Claude Opus 4.7
React
92%12/13
96%+4%
19/19$0.84
1m 33s
With Auth0 Tools·96%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
9.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticat
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: doe
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider,
18
Gemini 3.1 Pro
Express
80%12/15
96%+16%
21/22$0.27
4m 20s
With Auth0 Tools·96%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.3/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (21/22 passed)
L6 · Holistic(21/22)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) sto
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile rout
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: doe
Does the solution correctly integrate Auth0 into an Express web app using expres
19
Claude Opus 4.7
Android
67%6/9
96%+29%
14/14$0.38
1m 21s
With Auth0 Tools·96%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.1/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (14/14 passed)
L6 · Holistic(14/14)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
No auth0-java (server-side SDK, not for Android)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvid
Is the Auth0 callback URL scheme correctly handled? Check that build.gradle defa
Does the solution correctly integrate Auth0 into an Android app with WebAuthProv
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(co
20
Claude 4.6 Sonnet
Fastify API
50%7/14
96%+46%
19/19$0.19
1m 35s
With Auth0 Tools·96%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
13.3/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.6/14 pts
Error Recovery
When something broke, did the agent fix it?
7.2/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /ap
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: d
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth
21
GPT-5.4
Fastify API
50%7/14
96%+46%
19/19$0.47
1m 55s
With Auth0 Tools·96%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.1/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /ap
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: d
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth
22
Gemini 3.1 Pro
FastAPI
40%6/15
96%+56%
20/20$0.49
4m 10s
With Auth0 Tools·96%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
9.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
96/100 ptsA
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages wi
Does the solution use current auth0-fastapi-api patterns? Specifically: does it
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fasta
23
Claude 4.6 Opus
Express API
85%11/13
95%+10%
20/21$0.23
1m 9s
With Auth0 Tools·95%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
13.3/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
7.2/8 pts
Correctness
Does the generated code actually work?
22.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
95/100 ptsA
Grader Results (20/21 passed)
L6 · Holistic(20/21)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically:
Does the solution correctly protect an Express.js API using express-oauth2-jwt-b
24
GPT-5.4
Vue
100%13/13
94%-6%
19/19$0.69
2m 20s
With Auth0 Tools·94%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.6/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticat
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/aut
25
Claude 4.6 Opus
Android
67%6/9
94%+27%
14/14$0.41
1m 21s
With Auth0 Tools·94%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (14/14 passed)
L6 · Holistic(14/14)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
No auth0-java (server-side SDK, not for Android)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvid
Is the Auth0 callback URL scheme correctly handled? Check that build.gradle defa
Does the solution correctly integrate Auth0 into an Android app with WebAuthProv
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(co
26
Gemini 3.1 Pro
Android
56%5/9
94%+38%
14/14$0.49
3m 30s
With Auth0 Tools·94%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.5/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (14/14 passed)
L6 · Holistic(14/14)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
No auth0-java (server-side SDK, not for Android)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvid
Is the Auth0 callback URL scheme correctly handled? Check that build.gradle defa
Does the solution correctly integrate Auth0 into an Android app with WebAuthProv
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(co
27
Claude 4.6 Sonnet
Vue
100%13/13
93%-7%
19/19$0.21
3m 9s
With Auth0 Tools·93%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.6/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
24/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
93/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticat
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/aut
28
Gemini 3.1 Pro
Angular
100%12/12
93%-7%
17/17$0.47
5m 41s
With Auth0 Tools·93%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
13/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
93/100 ptsA
Grader Results (17/17 passed)
L6 · Holistic(17/17)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/aut
29
Claude 4.6 Opus
Vue
100%13/13
92%-8%
19/19$0.20
1m 20s
With Auth0 Tools·92%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
92/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticat
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/aut
30
Gemini 3.1 Pro
Swift
80%8/10
92%+12%
13/14$0.67
5m 13s
With Auth0 Tools·92%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.1/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
92/100 ptsA
Grader Results (13/14 passed)
L6 · Holistic(13/14)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error hand
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Sp
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth()
31
GPT-5.4
Express
87%13/15
91%+4%
22/22$1.08
3m 11s
With Auth0 Tools·91%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
13.3/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.2/14 pts
Error Recovery
When something broke, did the agent fix it?
7.2/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
91/100 ptsA
Grader Results (22/22 passed)
L6 · Holistic(22/22)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) sto
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile rout
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: doe
Does the solution correctly integrate Auth0 into an Express web app using expres
32
GPT-5.4
FastAPI
47%7/15
91%+44%
19/20$0.74
1m 57s
With Auth0 Tools·91%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.5/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
91/100 ptsA
Grader Results (19/20 passed)
L6 · Holistic(19/20)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages wi
Does the solution use current auth0-fastapi-api patterns? Specifically: does it
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fasta
33
Claude Opus 4.7
Angular
100%12/12
90%-10%
17/17$1.38
2m 32s
With Auth0 Tools·90%·Tool Calls 28
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
12.7/14 pts
Efficiency
How many steps did it take? Fewer is better.
5/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
90/100 ptsA
Grader Results (17/17 passed)
L6 · Holistic(17/17)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/aut
34
Claude Opus 4.7
Vue
100%13/13
90%-10%
18/19$1.03
1m 29s
With Auth0 Tools·90%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.2/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
90/100 ptsA
Grader Results (18/19 passed)
L6 · Holistic(18/19)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticat
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/aut
35
Claude 4.6 Sonnet
Express API
77%10/13
90%+13%
20/21$0.15
2m 3s
With Auth0 Tools·90%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12.6/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
6.4/8 pts
Correctness
Does the generated code actually work?
22.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
8.3/10 pts
Total weighted score
90/100 ptsA
Grader Results (20/21 passed)
L6 · Holistic(20/21)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically:
Does the solution correctly protect an Express.js API using express-oauth2-jwt-b
36
Gemini 3.1 Pro
Express API
77%10/13
90%+13%
20/21$0.35
3m 15s
With Auth0 Tools·90%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.1/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
5/10 pts
Total weighted score
90/100 ptsA
Grader Results (20/21 passed)
L6 · Holistic(20/21)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically:
Does the solution correctly protect an Express.js API using express-oauth2-jwt-b
37
Claude 4.6 Opus
Swift
70%7/10
90%+20%
12/14$0.16
1m 31s
With Auth0 Tools·90%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12.6/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.3/14 pts
Error Recovery
When something broke, did the agent fix it?
6.4/8 pts
Correctness
Does the generated code actually work?
20.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
90/100 ptsA
Grader Results (12/14 passed)
L6 · Holistic(12/14)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error hand
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Sp
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth()
38
Claude 4.6 Sonnet
Nuxt
35%7/20
90%+55%
30/30$0.18
3m 52s
With Auth0 Tools·90%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
11.3/14 pts
Efficiency
How many steps did it take? Fewer is better.
7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
24.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
90/100 ptsA
Grader Results (30/30 passed)
L6 · Holistic(30/30)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) sto
No tokens stored in localStorage
No tokens stored in sessionStorage
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via defin
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Does the solution correctly integrate Auth0 into a Nuxt application using the @a
39
Claude 4.6 Sonnet
Android
67%6/9
88%+21%
13/14$0.26
3m 45s
With Auth0 Tools·88%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.5/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
7.5/10 pts
Total weighted score
88/100 ptsB
Grader Results (13/14 passed)
L6 · Holistic(13/14)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
No auth0-java (server-side SDK, not for Android)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvid
Is the Auth0 callback URL scheme correctly handled? Check that build.gradle defa
Does the solution correctly integrate Auth0 into an Android app with WebAuthProv
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(co
40
GPT-5.4
Android
67%6/9
88%+21%
14/14$0.65
1m 50s
With Auth0 Tools·88%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
13.3/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
5.9/14 pts
Error Recovery
When something broke, did the agent fix it?
7.2/8 pts
Correctness
Does the generated code actually work?
22.3/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
88/100 ptsB
Grader Results (14/14 passed)
L6 · Holistic(14/14)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
No auth0-java (server-side SDK, not for Android)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvid
Is the Auth0 callback URL scheme correctly handled? Check that build.gradle defa
Does the solution correctly integrate Auth0 into an Android app with WebAuthProv
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(co
41
GPT-5.4
Angular
100%12/12
87%-13%
17/17$1.47
4m 48s
With Auth0 Tools·87%·Tool Calls 32
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
13.3/14 pts
Setup Speed
How fast did the agent complete the task?
12.3/14 pts
Efficiency
How many steps did it take? Fewer is better.
4.3/14 pts
Error Recovery
When something broke, did the agent fix it?
7.2/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
87/100 ptsB
Grader Results (17/17 passed)
L6 · Holistic(17/17)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/aut
42
GPT-5.4
Swift
80%8/10
86%+6%
12/14$1.14
4m 1s
With Auth0 Tools·86%·Tool Calls 27
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
13.3/14 pts
Efficiency
How many steps did it take? Fewer is better.
5.3/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
20.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
86/100 ptsB
Grader Results (12/14 passed)
L6 · Holistic(12/14)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error hand
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Sp
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth()
43
Claude Opus 4.7
Nuxt
55%11/20
86%+31%
30/30$0.82
2m 56s
With Auth0 Tools·86%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
13.3/14 pts
Setup Speed
How fast did the agent complete the task?
9/14 pts
Efficiency
How many steps did it take? Fewer is better.
7/14 pts
Error Recovery
When something broke, did the agent fix it?
7.2/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
86/100 ptsB
Grader Results (30/30 passed)
L6 · Holistic(30/30)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) sto
No tokens stored in localStorage
No tokens stored in sessionStorage
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via defin
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Does the solution correctly integrate Auth0 into a Nuxt application using the @a
44
Claude 4.6 Opus
Angular
100%12/12
85%-15%
17/17$0.34
2m 57s
With Auth0 Tools·85%·Tool Calls 32
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
13.3/14 pts
Setup Speed
How fast did the agent complete the task?
9.8/14 pts
Efficiency
How many steps did it take? Fewer is better.
4.5/14 pts
Error Recovery
When something broke, did the agent fix it?
7.2/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
85/100 ptsB
Grader Results (17/17 passed)
L6 · Holistic(17/17)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/aut
45
Claude Opus 4.7
Express API
85%11/13
85%
20/21$1.08
1m 33s
With Auth0 Tools·85%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
9.8/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
9.7/14 pts
Error Recovery
When something broke, did the agent fix it?
3.2/8 pts
Correctness
Does the generated code actually work?
22.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
85/100 ptsB
Grader Results (20/21 passed)
L6 · Holistic(20/21)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically:
Does the solution correctly protect an Express.js API using express-oauth2-jwt-b
46
Claude 4.6 Sonnet
Swift
70%7/10
85%+15%
13/14$0.20
2m 5s
With Auth0 Tools·85%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
10.5/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
4/8 pts
Correctness
Does the generated code actually work?
23.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
85/100 ptsB
Grader Results (13/14 passed)
L6 · Holistic(13/14)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error hand
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Sp
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth()
47
Claude 4.6 Sonnet
Angular
100%12/12
84%-16%
17/17$0.31
4m 4s
With Auth0 Tools·84%·Tool Calls 25
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
12.5/14 pts
Efficiency
How many steps did it take? Fewer is better.
5.6/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
84/100 ptsB
Grader Results (17/17 passed)
L6 · Holistic(17/17)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/aut
48
Claude 4.6 Opus
Next.js
71%10/14
84%+13%
22/25$0.24
1m 44s
With Auth0 Tools·84%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
13.7/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.3/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
10.1/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
84/100 ptsB
Grader Results (22/25 passed)
L6 · Holistic(22/25)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) store
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a cal
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLI
49
Gemini 3.1 Pro
Next.js
57%8/14
84%+27%
21/25$0.59
4m 48s
With Auth0 Tools·84%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.9/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21/25 pts
Hallucination
Did the agent invent fake packages or methods?
10.1/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
84/100 ptsB
Grader Results (21/25 passed)
L6 · Holistic(21/25)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) store
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a cal
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLI
50
Claude 4.6 Opus
Nuxt
55%11/20
84%+29%
29/30$0.20
3m 8s
With Auth0 Tools·84%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
7.6/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.3/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
84/100 ptsB
Grader Results (29/30 passed)
L6 · Holistic(29/30)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) sto
No tokens stored in localStorage
No tokens stored in sessionStorage
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via defin
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Does the solution correctly integrate Auth0 into a Nuxt application using the @a
51
Claude 4.6 Opus
FastAPI
47%7/15
84%+37%
19/20$0.30
1m 48s
With Auth0 Tools·84%·Tool Calls 24
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.3/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
22.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
84/100 ptsB
Grader Results (19/20 passed)
L6 · Holistic(19/20)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages wi
Does the solution use current auth0-fastapi-api patterns? Specifically: does it
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fasta
52
GPT-5.4
Nuxt
35%7/20
84%+49%
30/30$2.18
5m 41s
With Auth0 Tools·84%·Tool Calls 42
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12.6/14 pts
Setup Speed
How fast did the agent complete the task?
11.2/14 pts
Efficiency
How many steps did it take? Fewer is better.
3.5/14 pts
Error Recovery
When something broke, did the agent fix it?
6.4/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
84/100 ptsB
Grader Results (30/30 passed)
L6 · Holistic(30/30)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) sto
No tokens stored in localStorage
No tokens stored in sessionStorage
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via defin
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Does the solution correctly integrate Auth0 into a Nuxt application using the @a
53
GPT-5.4
Next.js
79%11/14
83%+4%
21/25$0.64
2m 33s
With Auth0 Tools·83%·Tool Calls 23
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.2/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21/25 pts
Hallucination
Did the agent invent fake packages or methods?
10.1/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
83/100 ptsB
Grader Results (21/25 passed)
L6 · Holistic(21/25)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) store
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a cal
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLI
54
Claude Opus 4.7
Next.js
71%10/14
83%+12%
21/25$1.02
2m 10s
With Auth0 Tools·83%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
13.2/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.6/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21/25 pts
Hallucination
Did the agent invent fake packages or methods?
10.1/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
83/100 ptsB
Grader Results (21/25 passed)
L6 · Holistic(21/25)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) store
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a cal
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLI
55
GPT-5.4
Express API
77%10/13
82%+5%
20/21$0.74
2m 13s
With Auth0 Tools·82%·Tool Calls 19
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.9/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8/14 pts
Error Recovery
When something broke, did the agent fix it?
5.6/8 pts
Correctness
Does the generated code actually work?
22.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
5/10 pts
Total weighted score
82/100 ptsB
Grader Results (20/21 passed)
L6 · Holistic(20/21)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically:
Does the solution correctly protect an Express.js API using express-oauth2-jwt-b
56
Claude Opus 4.7
Swift
60%6/10
82%+22%
13/14$0.89
1m 28s
With Auth0 Tools·82%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
9.8/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
4/8 pts
Correctness
Does the generated code actually work?
21.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
82/100 ptsB
Grader Results (13/14 passed)
L6 · Holistic(13/14)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error hand
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Sp
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth()
57
Claude Opus 4.7
FastAPI
47%7/15
82%+35%
20/20$1.33
2m 18s
With Auth0 Tools·82%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
9.1/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.4/14 pts
Error Recovery
When something broke, did the agent fix it?
2.4/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
82/100 ptsB
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages wi
Does the solution use current auth0-fastapi-api patterns? Specifically: does it
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fasta
58
Claude 4.6 Sonnet
FastAPI
40%6/15
81%+41%
20/20$0.20
2m 2s
With Auth0 Tools·81%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
8.4/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
1.6/8 pts
Correctness
Does the generated code actually work?
24/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
81/100 ptsB
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages wi
Does the solution use current auth0-fastapi-api patterns? Specifically: does it
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fasta
59
Gemini 3.1 Pro
Nuxt
30%6/20
78%+48%
29/30$0.50
15m 49s
With Auth0 Tools·78%·Tool Calls 19
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
0/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.6/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
78/100 ptsC
Grader Results (29/30 passed)
L6 · Holistic(29/30)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) sto
No tokens stored in localStorage
No tokens stored in sessionStorage
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via defin
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Does the solution correctly integrate Auth0 into a Nuxt application using the @a
60
Claude 4.6 Sonnet
Next.js
50%7/14
76%+26%
21/25$0.27
3m 8s
With Auth0 Tools·76%·Tool Calls 26
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
13.4/14 pts
Efficiency
How many steps did it take? Fewer is better.
5.6/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
21/25 pts
Hallucination
Did the agent invent fake packages or methods?
10.1/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
76/100 ptsC
Grader Results (21/25 passed)
L6 · Holistic(21/25)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) store
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a cal
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLI
* With Auth0 MCP + Skills enabledLast updated on April 2026
How we validate results
Every score is reproducible. Real integration tasks, automated grading, transparent methodology.
1
Real integration tasks
Real AI agents run identical Auth0 SDK prompts end-to-end. No synthetic benchmarks.
2
Automated grading
Pattern matching, security checks, and correctness validators score each response.
3
Tool comparison
Each model tested with and without MCP + Skills to measure improvement.
Start building with confidence
These tools are how we achieved the benchmark results above. Enable them in your AI agent to integrate Auth0 reliably.
MCP Server
API access for agents
Gives AI agents access to Auth0 Management APIs and current documentation.
https://auth0.com/docs/mcpSkills
Best practices for agents
Teaches agents Auth0 integration patterns so they follow security best practices.
npx skills add auth0/agent-skills