[TRAIN-3456] Add healthchecks. Add functionality to Makefile. Refactor Puppeteer. by briannadelvalle-datadog · Pull Request #122 · DataDog/storedog

briannadelvalle-datadog · 2025-10-15T20:42:04Z

Description

Use healthchecks instead of wait-for-it -- reduces chance of bad race conditions breaking storedog in labs
Add Makefile functionality
Add scripts to generate docker-compose.yml files from docker-compose.dev.yaml. (One is intended for pre-release, another is for testing the frontend prod build target while using the dev docker compose file.)
Update readme and add service-level readmes (and a FEATURES.md file)
Improve frontend service image: multilayer builds, use build targets
Add a simple env.development.template for TCD use.
Migrate puppeteer service from rum-retention-filters branch — replaces the monolithic puppeteer.js (841-line single script + shell wrapper) with a modular traffic generator architecture. Key changes:
- Upgrade Puppeteer base image from v20 to v24
- Add package.json with proper npm dependency management
- Modular session system: SessionManager, BrowserPool, BaseSession class, and 10 individual session types (browsing, taxonomy, frustration, homePage, short, vip, bot, cartAbandonment, earlyError, lateError)
- Memory-aware concurrency profiles (8GB/16GB/32GB) with configurable ramp-up
- New environment variables for session control, debugging, and concurrency tuning (see services/puppeteer/README.md)
- Updated docker-compose service definitions (both dev and prod) with all new env vars
- RUM credentials (RUM_APP_ID/RUM_CLIENT_TOKEN) mapped from existing NEXT_PUBLIC_DD_APPLICATION_ID/NEXT_PUBLIC_DD_CLIENT_TOKEN

Frontend build times

Tested with the help of Cursor/Claude.

Dockerfile	Clean build	Build env vars change (second build)	Source code change (third build)
Production build target	39.35s	8.01s	9.67s
Old Dockerfile	47.43s	27.48s	18.90s

Image Sizes

Tested with the help of Cursor/Claude.

Dockerfile	Size
Production (Multi-stage)	301.4 MB
Old Dockerfile (Single-stage)	812.8 MB

How to test

Test the healthchecks

Run make up to build the dev containers.

Test the frontend prod build target

In the Makefile, set ENV=dev-frontend-prod.
Run make up clean.

Test the dockerfile generation script

Run make prepare-release.
Check the docker-compose.generated.yml file and answer Y/N in the terminal.

Test the Makefile updates

Run make help to see the commands.

Test the puppeteer migration

Run make up (or docker compose -f docker-compose.dev.yml up --build puppeteer)
Set PUPPETEER_LOOP=single and PUPPETEER_DEBUG=true in .env for a quick smoke test
Check logs for the session manager startup table and session execution

briannadelvalle-datadog · 2025-10-20T21:12:21Z

      console.error(e)
      setLoading(false)
    }
-  }, [adsPath, getRandomArbitrary, setData, setLoading])


React said that this wasn't needed here.

Makes sense, it's not a dependency

briannadelvalle-datadog · 2025-10-20T21:12:42Z

@@ -1,29 +1,134 @@
-FROM node:20 AS builder
+FROM node:lts-alpine AS base


To mitigate a security vulnerability in other image

briannadelvalle-datadog · 2025-10-21T19:27:06Z

@@ -1,4 +1,5 @@
 module.exports = {
+  output: 'standalone',


This is for the optimized production builds (production build target)

arosenkranz · 2025-10-21T20:23:35Z

+
+settings.json
+
+docker-compose.dev.frontend-prod.yml


Was this last one left here on purpose?

Yeah, this is only for testing the frontend prod build target.
Can't test it with the regular docker-compose.yml because it's not in the storedog image.
Can't test it with the docker-compose.dev.yml because of the volume mounts.
I'm adding a script and a make command to generate this file, but there's no reason to add it to source control.

For more context, the prod build creates a server.js file and the command is node server.js. If we're mounting the entire service directory, we end up losing that file.

…tation; add transform_compose.py script and unit tests for compose transformations.

…ddprofrb for profiling

…nction for generated files in transform_compose.py

…put setting in next.js for prod build

…production frontend build

…rget, and build args in docker-compose template; refine regex pattern for build section replacement in transformation logic.

…opment and production environments, including environment variable configurations and dependency installations.

… function for selecting the appropriate Docker Compose file based on the ENV variable. Simplify commands for starting, stopping, and managing containers in both development and production environments.

…ariables for frontend service; add port mapping for local development.

…th NEXT_PUBLIC_DD variables for frontend service configuration.

…ment to production

…nv var stuff to env templates, move feature stuff to feature.md

…BLIC_DD variables for consistency in frontend service configuration.

… with NEXT_PUBLIC_DD variables for consistency in frontend service configuration.

…ckerfile

… NEXT_PUBLIC prefix and enhance transformation logic for development to production changes.

…g, and improve documentation for better usability.

…og configurations for cleaner logging setup.

…, simplifying the setup for Datadog logging.

…et to simplify usage instructions.

…filters branch Replaces the monolithic puppeteer.js script with a modular traffic generator featuring a SessionManager, BrowserPool, and 10 configurable session types. Upgrades Puppeteer from v20 to v24 and adds memory-aware concurrency profiles. Updates docker-compose service definitions with new environment variables. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…vices These tags are unnecessary since DD_SERVICE, DD_VERSION, and DD_ENV are already set as environment variables on these containers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This reverts commit a68ce35.

- Fix NEXT_PUBLIC_CLIENT_TOKEN -> NEXT_PUBLIC_DD_CLIENT_TOKEN in .env.development.template - Remove unused getRandomArbitrary function and leftover console.logs in Ad.tsx - Fix .PHONY to match actual Makefile targets (remove dd-dev, dd-prod, prepare-frontend-prod; add frontend-prod) - Deduplicate node_modules and .env entries in .gitignore, add trailing newline - Fix copy-paste error in discounts README (ads.py -> discounts.py) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

briannadelvalle-datadog commented Oct 20, 2025

View reviewed changes

briannadelvalle-datadog marked this pull request as ready for review October 21, 2025 19:25

briannadelvalle-datadog requested review from a team as code owners October 21, 2025 19:25

briannadelvalle-datadog commented Oct 21, 2025

View reviewed changes

briannadelvalle-datadog marked this pull request as draft October 21, 2025 19:52

arosenkranz reviewed Oct 21, 2025

View reviewed changes

Comment thread docker-compose.dev.yml Outdated

arosenkranz reviewed Oct 21, 2025

View reviewed changes

Comment thread .gitignore Outdated

briannadelvalle-datadog changed the title ~~[TRAIN-3456] Add healthchecks. Add functionality to Makefile.~~ [TRAIN-3456] Add healthchecks. Add functionality to Makefile. Refactor Puppeteer. Apr 7, 2026

briannadelvalle-datadog added 19 commits April 7, 2026 14:38

Remove wait-for-it, add healthchecks. Add functionality to Makefile.

99c1298

Enhance Makefile with prepare-release target and improve help documen…

c8a5716

…tation; add transform_compose.py script and unit tests for compose transformations.

add healthchecks to docker compose.dev

74cbdf7

Update Dockerfile to change command for starting Rails server to use …

581b8bb

…ddprofrb for profiling

Add .gradle/ and settings.json to .gitignore

9116d67

Refine regex pattern for service build replacement and add cleanup fu…

0917c01

…nction for generated files in transform_compose.py

Update docker-compose.yml

fcfbdbe

Use dev and prod Dockerfiles for frontend service, use standalone out…

005df10

…put setting in next.js for prod build

Add basic dev env template for faster setup

a73b7ad

Update frontend dependencies

b7f544f

rename frontend Dockerfile

aa8a710

Update frontend README with info about different Dockerfiles

8f995ee

Use build targets instead for frontend

0023ab6

Add frontend build target to env template for development testing of …

0a8b886

…production frontend build

Enhance frontend service configuration by adding dockerfile, build ta…

90fe213

…rget, and build args in docker-compose template; refine regex pattern for build section replacement in transformation logic.

Add Dockerfile for frontend service with multi-stage builds for devel…

8577336

…opment and production environments, including environment variable configurations and dependency installations.

Refactor Makefile to streamline environment handling by introducing a…

477e693

… function for selecting the appropriate Docker Compose file based on the ENV variable. Simplify commands for starting, stopping, and managing containers in both development and production environments.

Update docker-compose files to include build target and environment v…

777acbf

…ariables for frontend service; add port mapping for local development.

Update default values of development stage variables

bd5e8b6

briannadelvalle-datadog and others added 25 commits April 7, 2026 14:38

Add or update readmes for services

acbae18

Update env templates

ca20513

Update docker-compose.dev.yml to replace RUM environment variables wi…

5f0c19f

…th NEXT_PUBLIC_DD variables for frontend service configuration.

Move features info to separate .md file

60801a9

Update Makefile to support no-cache builds and change default environ…

2131bde

…ment to production

Update readme. Move service specific stuff to service readmes, move e…

077968a

…nv var stuff to env templates, move feature stuff to feature.md

Update README to include a blank line for improved readability

195756c

Remove some console.logs

6e60a7b

Refactor Dockerfile to replace RUM environment variables with NEXT_PU…

a6dfec1

…BLIC_DD variables for consistency in frontend service configuration.

Update test_transform_compose.py to replace RUM environment variables…

d4581f1

… with NEXT_PUBLIC_DD variables for consistency in frontend service configuration.

Adjust frontend Dockerfile to match vercel example

6e972ed

Frontend command var made obsolete by build targets and multistage Do…

a685b85

…ckerfile

Add unit tests for frontend and compose transformation scripts

7c80990

Improve makefile

b76e88f

Update dockerfiles

e9eb126

Update .gitignore

8fa6bda

Update some documentation

b3c244c

Remove variable that's no longer needed

728682b

Align frontend variables, don't use args in docker compose

ae6f2ef

Refactor environment variable handling in Docker Compose files to use…

5787a2c

… NEXT_PUBLIC prefix and enhance transformation logic for development to production changes.

Refactor Makefile to streamline commands, enhance environment handlin…

f877027

…g, and improve documentation for better usability.

Refactor Docker Compose labels to remove service names from Datadog l…

c195201

…og configurations for cleaner logging setup.

Refactor Docker Compose log configurations to eliminate service names…

072a1e2

…, simplifying the setup for Datadog logging.

Remove one-time override instruction from Makefile frontend-prod targ…

811cd17

…et to simplify usage instructions.

briannadelvalle-datadog force-pushed the TRAIN-3546-Update-Storedog branch from d3e964e to bb45864 Compare April 7, 2026 18:38

briannadelvalle-datadog and others added 4 commits April 7, 2026 14:50

Remove redundant com.datadoghq.tags labels from discounts and ads ser…

798f6cf

…vices These tags are unnecessary since DD_SERVICE, DD_VERSION, and DD_ENV are already set as environment variables on these containers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove Gemfile.lock

a68ce35

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Revert "Remove Gemfile.lock"

1aff261

This reverts commit a68ce35.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[TRAIN-3456] Add healthchecks. Add functionality to Makefile. Refactor Puppeteer.#122

[TRAIN-3456] Add healthchecks. Add functionality to Makefile. Refactor Puppeteer.#122
briannadelvalle-datadog wants to merge 49 commits into
mainfrom
TRAIN-3546-Update-Storedog

briannadelvalle-datadog commented Oct 15, 2025 •

edited

Loading

Uh oh!

briannadelvalle-datadog Oct 20, 2025

Uh oh!

arosenkranz Oct 21, 2025

Uh oh!

briannadelvalle-datadog Oct 20, 2025

Uh oh!

briannadelvalle-datadog Oct 21, 2025

Uh oh!

arosenkranz Oct 21, 2025

Uh oh!

briannadelvalle-datadog Oct 22, 2025

Uh oh!

briannadelvalle-datadog Oct 22, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -1,29 +1,134 @@
		FROM node:20 AS builder
		FROM node:lts-alpine AS base


		settings.json

		docker-compose.dev.frontend-prod.yml No newline at end of file

Uh oh!

Conversation

briannadelvalle-datadog commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Frontend build times

Image Sizes

How to test

Test the healthchecks

Test the frontend prod build target

Test the dockerfile generation script

Test the Makefile updates

Test the puppeteer migration

Uh oh!

briannadelvalle-datadog Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

arosenkranz Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

briannadelvalle-datadog Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

briannadelvalle-datadog Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

arosenkranz Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

briannadelvalle-datadog Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

briannadelvalle-datadog Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

briannadelvalle-datadog commented Oct 15, 2025 •

edited

Loading