From 1baf8210eda9494ccc19a6f915a14a2cb39effd3 Mon Sep 17 00:00:00 2001 From: Ian Carroll Date: Thu, 4 Dec 2025 06:43:07 -0800 Subject: [PATCH 1/5] Create practices/build-local-dev-monorepo --- practices/build-local-dev-monorepo.md | 80 +++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 practices/build-local-dev-monorepo.md diff --git a/practices/build-local-dev-monorepo.md b/practices/build-local-dev-monorepo.md new file mode 100644 index 0000000..0721ad1 --- /dev/null +++ b/practices/build-local-dev-monorepo.md @@ -0,0 +1,80 @@ +# Migrate to a Monorepo + +Developing with dozens of local micro-repos that need to work together and may also have database access or cloud permissions needs can feel like you've only got the choice of flying blind or waiting to get off the ground. This practice helps teams consolidate scattered projects into a single monorepo so they can see the system at work locally and start coding anywhere (even on a plane) without a need for access approvals or an unused cloud environment. Many popular build tools support this functionality: + +- [pnpm](https://pnpm.io/workspaces) +- [Nx](https://nx.dev/concepts/decisions/why-monorepos) +- [Bazel](https://bazel.build/concepts/build-ref#workspace) +- [Turborepo](https://turborepo.com/docs#what-is-turborepo) +- [Pants](https://www.pantsbuild.org/stable/docs/using-pants/environments#in-workspace-execution-experimental_workspace_environment) +- etc (this isn't by any means a definitive list) + +## When to Experiment + +- You are a Developer working on features that involve multiple interdependent repositories and need to get fast feedback on changes without needing access to managed cloud environments so you can start coding right away instead of waiting for resources to be made available or accessible to you. + +## How to Gain Traction + +Treat it as an experiment, not a mandate. + +### Start by bringing together the interdependent repos + +Choose as few services as you can get away with (2-3 ideally) that already depend on each other and form a complete system. Clone them under a single directory called something like `\services`. + +### Create a monorepo to wrap the services + +In the parent directory, create a repo that uses a build tool that supports workspaces or monorepos. Develop shell scripts that clone the interdependent repos into `/services` from the monorepo root, build each service, run each service, etc. + +### Add an integration testing layer + +Use this monorepo to test the seams of interaction between the services, ensuring that contracts and connections still function. This can be a harness to gain more confidence that each service built in isolation previously will still work well with the rest of the system. Snapshot testing via a framework like jest, while not ideal in the longterm, can fill this role in a starting capacity. Do not add unit tests here, instead develop a script that can run through the tests on each service the monorepo houses. + +### Make a local docker container with a dummy database + +Use this pilot to work through defining the interfaces to the "outside world". Use docker-compose to orchestrate fake versions of the various databases your services depend on. Dump a working copy of the data from a lower environment (if none is available, scrub a production dump of PII and use that). Start by populating the local database via the dump, then add migration scripts to keep the schema current, finally create data generators to quickly set the database to specific or random conditions the code will work against. + +### Fill in vendor dependecies with echo servers + +For additional services that call vendors that return values, create echo servers that take requests in an identical shape and return fake production-like repsonses. This is an opportunity to build in robustness into the system if vendor latency fluxuates - when there is additional time, simulate this as well. Do the same for services that don't directly talk to the services, but modify the Database on cron schedules. + +### Create a Migration Path + +Even before all the pieces allow for a tested and independently running local setupo, demonstrate how it works for you to gain team awareness. Instead of mandating use, allow individual developers to adopt the monorepo locally themselves and become advocates and champions for it. Pair program using the tooling, and help orther developers mirror your setup. You know the team has adopted the practice when developers reference it amongst eachother in work that doesn't directly involve you. + +### Continue Improving DevEx + +Listen to gripes and complaints and set asside time to improve those parts of the system. Note what still causes you confusion or wait times. Schedule time with developers working in the new repository to understand how they use it and where they still get friction. This can be done in the course of regular work if you pair program. Brainstorm potential solutions as a group. Keep an eye toward improving the items listed in the Goals, Metrics & Signals section of this page to ensure efforts are making a difference. You know it's simple to setup and use when you observe developers using it who haven't come to you with questions first. + + +## Lessons From The Field + +- If you find yourself in a company structured so that you must wait for resource allocations or a separate team to come in and build/grant you access, the wait time you are experiencing can be used to start building this local repo. It may be a matter of building it for you, then once its working, sharing it with the team. +- Remote teams might not make thier local developer setups visible. Just because no one mentions the tool or praises it doesn't mean it's not being used. If it saves them time, and shows them what they need to see more easlily than what they've cobbled together for themselves, they will use it. + +## Deciding to Polish or Pitch + +After experimenting with this practice for **one month**, bring the team together to determine whether the following metrics and/or signals have changed in a positive direction: + +### Fast & Measurable + +**Lead Time** for changes spanning multiple repositories should decrease. By consolidating services into a monorepo, cross-cutting changes can be made without waiting for access approvals, access re-approvals, test-deploying, or waiting for a clean tests enviornment, reducing coordination overhead and speeding up delivery. Tools like [DX](https://getdx.com/platform/data-lake/), [Jellyfish](https://jellyfish.co/platform/engineering-metrics/), or others can measure these sort of lead time changes. Custom solutions can also be built using data from project planning tools like [JIRA](https://community.atlassian.com/forums/App-Central-articles/Cycle-Time-and-Lead-Time-in-Jira-Productivity-Measurement-with/ba-p/1905845), [Monday](https://monday.com/blog/project-management/what-is-lead-time/), or others. + +### Slow & Intangible + +**Cleaner Service Boundaries** should evolve over time, because refactoring those boundaries becomes easier. Poor service boundaries can be removed with less friction when everything lives in one repo. On the flipside, teams can more quickly extract new services with the shared tooling, configuration, and build setup. + +**Better cross-team collaboration** The monorepo can be spun up fairly simply to collaborate and demonstrate system behavior. When database changes are considered, comparing what's set up locally to the new schema change can be a point of conversation between teams that manage the data and teams than manage the code. The same thing applies to cloud environment connections between services, and to thrid pary API vendor interactions. When change involves less uncertainty, confidence to experiment increases. + +## Supported Capabilities + +### [Code Maintainability](/capabilities/code-maintainability.md) + +A local monorepo can help expose duplicated scripts and configuration across micro-repos, making the codebase easier to maintain and evolve. + +### [Test Automation](/capabilities/test-automation.md) + +Being able to test the interactions between the services the team maintains means there is much greater confidence that the whole system will perform as expected. Knowing those interactions still work while developing instead of at deploy-time shortens feedback loops and speeds up delivery. + +### [Test Data Management](/capabilities/test-data-management.md) + +Having control of the data locally means we also gain knowledge of the data's schema. By knowing and maintaining this information within the monorepo, we can eliminate uncertainty caused by database changes that previously existed outside the team's control. The same can be said for outside service vendors that supply information or data transformations our system depends on. By designing echo servers, we can even start to build the codebase to be robust in the event these services don't perform flawlessly via principles of Chaos Engineering. From 940176d20e9b3bfd98120b04ecc4e7a4f5109ee9 Mon Sep 17 00:00:00 2001 From: Ian Carroll Date: Thu, 4 Dec 2025 06:44:55 -0800 Subject: [PATCH 2/5] Update the title --- practices/build-local-dev-monorepo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/practices/build-local-dev-monorepo.md b/practices/build-local-dev-monorepo.md index 0721ad1..1c6ee64 100644 --- a/practices/build-local-dev-monorepo.md +++ b/practices/build-local-dev-monorepo.md @@ -1,4 +1,4 @@ -# Migrate to a Monorepo +# Build a Monorepo for local development Developing with dozens of local micro-repos that need to work together and may also have database access or cloud permissions needs can feel like you've only got the choice of flying blind or waiting to get off the ground. This practice helps teams consolidate scattered projects into a single monorepo so they can see the system at work locally and start coding anywhere (even on a plane) without a need for access approvals or an unused cloud environment. Many popular build tools support this functionality: From 7287f98351d234e844330b897d6a4eabe6c200d6 Mon Sep 17 00:00:00 2001 From: nicoletache Date: Thu, 18 Dec 2025 12:35:05 -0600 Subject: [PATCH 3/5] edits to build local dev monorepo practice --- practices/build-local-dev-monorepo.md | 46 +++++++++++++-------------- 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/practices/build-local-dev-monorepo.md b/practices/build-local-dev-monorepo.md index 1c6ee64..bcd73e5 100644 --- a/practices/build-local-dev-monorepo.md +++ b/practices/build-local-dev-monorepo.md @@ -1,55 +1,53 @@ -# Build a Monorepo for local development +# Build a Monorepo for Local Development -Developing with dozens of local micro-repos that need to work together and may also have database access or cloud permissions needs can feel like you've only got the choice of flying blind or waiting to get off the ground. This practice helps teams consolidate scattered projects into a single monorepo so they can see the system at work locally and start coding anywhere (even on a plane) without a need for access approvals or an unused cloud environment. Many popular build tools support this functionality: +Developing with dozens of local micro-repos that need to work together (and may also have database access or cloud permissions needs) is both daunting and time-consuming -- it can feel like your choices are either flying blind or waiting to get off the ground. Teams may encounter duplicated scripts and configuration across these many micro-repos. They may not trust interactions between services will behave as expected. And database changes may be out of their control, leading to more uncertainty. But when scattered projects are consolidated into a single monorepo, teams can see the system at work locally and start coding anywhere (even on a plane), without needing access approvals or an unused cloud environment. Many popular build tools support this functionality: - [pnpm](https://pnpm.io/workspaces) - [Nx](https://nx.dev/concepts/decisions/why-monorepos) - [Bazel](https://bazel.build/concepts/build-ref#workspace) - [Turborepo](https://turborepo.com/docs#what-is-turborepo) - [Pants](https://www.pantsbuild.org/stable/docs/using-pants/environments#in-workspace-execution-experimental_workspace_environment) -- etc (this isn't by any means a definitive list) +- Etc. (this isn't by any means a definitive list) ## When to Experiment -- You are a Developer working on features that involve multiple interdependent repositories and need to get fast feedback on changes without needing access to managed cloud environments so you can start coding right away instead of waiting for resources to be made available or accessible to you. +- You are a developer working on features that involve multiple interdependent repos and you need fast feedback on changes (without needing access to managed cloud environments) so you can start coding right away (instead of waiting for resources to be made available or accessible to you). ## How to Gain Traction -Treat it as an experiment, not a mandate. - -### Start by bringing together the interdependent repos +### Start by Bringing Together the Interdependent Repos Choose as few services as you can get away with (2-3 ideally) that already depend on each other and form a complete system. Clone them under a single directory called something like `\services`. -### Create a monorepo to wrap the services +### Create a Monorepo to Wrap the Services -In the parent directory, create a repo that uses a build tool that supports workspaces or monorepos. Develop shell scripts that clone the interdependent repos into `/services` from the monorepo root, build each service, run each service, etc. +In the parent directory, create a repo that uses a build tool that supports workspaces or monorepos (see the list above). Develop shell scripts that clone the interdependent repos into `/services` from the monorepo root, build each service, run each service, etc. -### Add an integration testing layer +### Add an Integration Testing Layer -Use this monorepo to test the seams of interaction between the services, ensuring that contracts and connections still function. This can be a harness to gain more confidence that each service built in isolation previously will still work well with the rest of the system. Snapshot testing via a framework like jest, while not ideal in the longterm, can fill this role in a starting capacity. Do not add unit tests here, instead develop a script that can run through the tests on each service the monorepo houses. +Use this monorepo to test the seams of interaction between the services, ensuring that contracts and connections still function. This can be a harness to gain more confidence that each service previously built in isolation will still work well with the rest of the system. Snapshot testing via a framework like jest, while not ideal in the long term, can get the job done initially. Do not add unit tests here; instead, develop a script that can run through the tests on each service the monorepo houses. -### Make a local docker container with a dummy database +### Make a Local Docker Container With a Dummy Database -Use this pilot to work through defining the interfaces to the "outside world". Use docker-compose to orchestrate fake versions of the various databases your services depend on. Dump a working copy of the data from a lower environment (if none is available, scrub a production dump of PII and use that). Start by populating the local database via the dump, then add migration scripts to keep the schema current, finally create data generators to quickly set the database to specific or random conditions the code will work against. +Use this pilot to work through defining the interfaces to the "outside world." Use docker-compose to orchestrate fake versions of the various databases your services depend on. Dump a working copy of the data from a lower environment (if none is available, scrub a production dump of PII and use that). Start by populating the local database with the dump, then add migration scripts to keep the schema current. Finally, create data generators to quickly set the database to specific or random conditions the code will work against. -### Fill in vendor dependecies with echo servers +### Fill in Vendor Dependecies With Echo Servers -For additional services that call vendors that return values, create echo servers that take requests in an identical shape and return fake production-like repsonses. This is an opportunity to build in robustness into the system if vendor latency fluxuates - when there is additional time, simulate this as well. Do the same for services that don't directly talk to the services, but modify the Database on cron schedules. +For additional services that call vendors that return values, create echo servers that take requests in an identical shape and return fake production-like repsonses. This is an opportunity to build in robustness into the system if vendor latency fluxuates -- when there is additional time, simulate this as well. Do the same for services that don't directly talk to the services, but modify the database on cron schedules. ### Create a Migration Path -Even before all the pieces allow for a tested and independently running local setupo, demonstrate how it works for you to gain team awareness. Instead of mandating use, allow individual developers to adopt the monorepo locally themselves and become advocates and champions for it. Pair program using the tooling, and help orther developers mirror your setup. You know the team has adopted the practice when developers reference it amongst eachother in work that doesn't directly involve you. +Before you have a tested and independently running local setup, get your team involved and demonstrate how it works for you. Treat this as an experiment, not a mandate. Allow individual developers to adopt the monorepo locally and become advocates and champions for it. Pair program using the tooling and help orther developers mirror your setup. You know the team has adopted the practice when developers reference it in work that doesn't directly involve you. ### Continue Improving DevEx -Listen to gripes and complaints and set asside time to improve those parts of the system. Note what still causes you confusion or wait times. Schedule time with developers working in the new repository to understand how they use it and where they still get friction. This can be done in the course of regular work if you pair program. Brainstorm potential solutions as a group. Keep an eye toward improving the items listed in the Goals, Metrics & Signals section of this page to ensure efforts are making a difference. You know it's simple to setup and use when you observe developers using it who haven't come to you with questions first. +Schedule time with developers working in the new repo to understand how they use it and where they still encounter friction. This can be done during the course of regular work if you pair program. Listen to any gripes and complaints, and note what still causes you confusion or wait times. Brainstorm potential solutions as a group and set aside time to improve problematic parts of the system. Focus on improving the items listed in the Polish or Pitch section of this page to ensure efforts are making a difference. You know it's simple to set up and use when you observe developers using it who haven't come to you with questions first. ## Lessons From The Field -- If you find yourself in a company structured so that you must wait for resource allocations or a separate team to come in and build/grant you access, the wait time you are experiencing can be used to start building this local repo. It may be a matter of building it for you, then once its working, sharing it with the team. -- Remote teams might not make thier local developer setups visible. Just because no one mentions the tool or praises it doesn't mean it's not being used. If it saves them time, and shows them what they need to see more easlily than what they've cobbled together for themselves, they will use it. +- [*insert summary line*] If your company is structured in a way that you must wait for resource allocations or a separate team to build/grant you access, then use that time to start building this local repo. Once you've got an experimental version up and running, share it with the team. +- [*insert summary line*] Remote teams might not make thier local developer setups visible. Just because no one mentions the tool or praises it doesn't mean it's not being used. If it saves your developers time and easily shows them what they need to see, then they'll use it. ## Deciding to Polish or Pitch @@ -57,13 +55,13 @@ After experimenting with this practice for **one month**, bring the team togethe ### Fast & Measurable -**Lead Time** for changes spanning multiple repositories should decrease. By consolidating services into a monorepo, cross-cutting changes can be made without waiting for access approvals, access re-approvals, test-deploying, or waiting for a clean tests enviornment, reducing coordination overhead and speeding up delivery. Tools like [DX](https://getdx.com/platform/data-lake/), [Jellyfish](https://jellyfish.co/platform/engineering-metrics/), or others can measure these sort of lead time changes. Custom solutions can also be built using data from project planning tools like [JIRA](https://community.atlassian.com/forums/App-Central-articles/Cycle-Time-and-Lead-Time-in-Jira-Productivity-Measurement-with/ba-p/1905845), [Monday](https://monday.com/blog/project-management/what-is-lead-time/), or others. +**Lead Time for Changes Decreases.** Changes spanning multiple repos should happen faster. By consolidating services into a monorepo, cross-cutting changes can be made without waiting for access approvals, access re-approvals, test deploying, or a clean tests enviornment. This reduces coordination overhead and speeds up delivery. Tools like [DX](https://getdx.com/platform/data-lake/) and [Jellyfish](https://jellyfish.co/platform/engineering-metrics/) can measure lead time changes. Custom solutions can also be built using data from project-planning tools like [JIRA](https://community.atlassian.com/forums/App-Central-articles/Cycle-Time-and-Lead-Time-in-Jira-Productivity-Measurement-with/ba-p/1905845) and [Monday](https://monday.com/blog/project-management/what-is-lead-time/). ### Slow & Intangible -**Cleaner Service Boundaries** should evolve over time, because refactoring those boundaries becomes easier. Poor service boundaries can be removed with less friction when everything lives in one repo. On the flipside, teams can more quickly extract new services with the shared tooling, configuration, and build setup. +**Cleaner Service Boundaries.** Refactoring service boundaries should become easier. When everything lives in one repo, poor service boundaries can be removed with less friction. On the flipside, teams can more quickly extract new services with the shared tooling, configuration, and build setup. -**Better cross-team collaboration** The monorepo can be spun up fairly simply to collaborate and demonstrate system behavior. When database changes are considered, comparing what's set up locally to the new schema change can be a point of conversation between teams that manage the data and teams than manage the code. The same thing applies to cloud environment connections between services, and to thrid pary API vendor interactions. When change involves less uncertainty, confidence to experiment increases. +**Better cross-team collaboration.** The monorepo can be spun up fairly simply for team collaboration and to demonstrate system behavior. In terms of database changes, comparing what's set up locally to the new schema change can be a point of conversation between teams that manage the data and teams than manage the code. The same thing applies to cloud environment connections between services, and third-party API vendor interactions. When change involves less uncertainty, then confidence to experiment increases. ## Supported Capabilities @@ -73,8 +71,8 @@ A local monorepo can help expose duplicated scripts and configuration across mic ### [Test Automation](/capabilities/test-automation.md) -Being able to test the interactions between the services the team maintains means there is much greater confidence that the whole system will perform as expected. Knowing those interactions still work while developing instead of at deploy-time shortens feedback loops and speeds up delivery. +When the team can test interactions between services, there is greater confidence that the whole system will perform as expected. Knowing those interactions still work *while developing*, instead of at deploy time, shortens feedback loops and speeds up delivery. ### [Test Data Management](/capabilities/test-data-management.md) -Having control of the data locally means we also gain knowledge of the data's schema. By knowing and maintaining this information within the monorepo, we can eliminate uncertainty caused by database changes that previously existed outside the team's control. The same can be said for outside service vendors that supply information or data transformations our system depends on. By designing echo servers, we can even start to build the codebase to be robust in the event these services don't perform flawlessly via principles of Chaos Engineering. +Having control of the data locally means we also gain knowledge of the data's schema. Having and maintaining this information within the monorepo means we can eliminate uncertainty caused by database changes that previously existed outside the team's control. The same can be said for outside service vendors that supply information or data transformations the system depends on. By designing echo servers, teams can build more robust codebases (in the event these services don't perform flawlessly via principles of Chaos Engineering). From bd8fe4a690a580f5beb693fce70ec33a515e044b Mon Sep 17 00:00:00 2001 From: Ian Carroll Date: Fri, 26 Dec 2025 14:42:53 -0800 Subject: [PATCH 4/5] Clarify vagueries --- practices/build-local-dev-monorepo.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/practices/build-local-dev-monorepo.md b/practices/build-local-dev-monorepo.md index bcd73e5..8c739b9 100644 --- a/practices/build-local-dev-monorepo.md +++ b/practices/build-local-dev-monorepo.md @@ -2,6 +2,8 @@ Developing with dozens of local micro-repos that need to work together (and may also have database access or cloud permissions needs) is both daunting and time-consuming -- it can feel like your choices are either flying blind or waiting to get off the ground. Teams may encounter duplicated scripts and configuration across these many micro-repos. They may not trust interactions between services will behave as expected. And database changes may be out of their control, leading to more uncertainty. But when scattered projects are consolidated into a single monorepo, teams can see the system at work locally and start coding anywhere (even on a plane), without needing access approvals or an unused cloud environment. Many popular build tools support this functionality: +#### build tools that support this functionality + - [pnpm](https://pnpm.io/workspaces) - [Nx](https://nx.dev/concepts/decisions/why-monorepos) - [Bazel](https://bazel.build/concepts/build-ref#workspace) @@ -21,7 +23,7 @@ Choose as few services as you can get away with (2-3 ideally) that already depen ### Create a Monorepo to Wrap the Services -In the parent directory, create a repo that uses a build tool that supports workspaces or monorepos (see the list above). Develop shell scripts that clone the interdependent repos into `/services` from the monorepo root, build each service, run each service, etc. +In the parent directory, create a repo that uses a build tool that supports workspaces or monorepos ([see the list above](#build-tools-that-support-this-functionality)). Develop shell scripts that clone the interdependent repos into `/services` from the monorepo root, then extend the shell scripts to set up each of the services for local development as you would in isolation(install, build, run, etc.) ### Add an Integration Testing Layer @@ -29,15 +31,17 @@ Use this monorepo to test the seams of interaction between the services, ensurin ### Make a Local Docker Container With a Dummy Database -Use this pilot to work through defining the interfaces to the "outside world." Use docker-compose to orchestrate fake versions of the various databases your services depend on. Dump a working copy of the data from a lower environment (if none is available, scrub a production dump of PII and use that). Start by populating the local database with the dump, then add migration scripts to keep the schema current. Finally, create data generators to quickly set the database to specific or random conditions the code will work against. +Use this pilot to work through defining the interfaces to the "outside world." Use docker-compose to orchestrate fake versions of the various databases your services depend on. It's likely you'll need a local copy of data that exists in running cloud-based systems. Create a copied data file from a database dump of a working staging environment (if none is available, scrub a production dump of PII and use that). Start by populating the local database with the copied file, then add migration scripts to keep the schema current. Finally, create data generators to quickly set the database to specific or random conditions the code will work against. ### Fill in Vendor Dependecies With Echo Servers -For additional services that call vendors that return values, create echo servers that take requests in an identical shape and return fake production-like repsonses. This is an opportunity to build in robustness into the system if vendor latency fluxuates -- when there is additional time, simulate this as well. Do the same for services that don't directly talk to the services, but modify the database on cron schedules. +Echo servers are named because they simply "echo back" what you send them. The echo servers we need will only be slightly more complex. They'll accept requests shaped as your vendor expects and return responses shaped the same as how your vendor replies. There should be next to no business logic in them. + +Your services may depend on calls to third-party vendors, such as accounting software, a CRM, or even a Zip Code finder. Calling out to the vendors directly, whenther using their live APIs or development sandboxes can have technical limitations or throttle feedback loops. To isolate your development environment and speed up feedback loops, you can instead create these basic servers ("echo servers") that take requests in an identical shape and return fake production-like repsonses. This can also create an opportunity to build in robustness into the system if vendor latency fluxuates -- when there is additional time, simulate this as well. Do the same for services that don't directly talk to the services, but modify the database on cron schedules. ### Create a Migration Path -Before you have a tested and independently running local setup, get your team involved and demonstrate how it works for you. Treat this as an experiment, not a mandate. Allow individual developers to adopt the monorepo locally and become advocates and champions for it. Pair program using the tooling and help orther developers mirror your setup. You know the team has adopted the practice when developers reference it in work that doesn't directly involve you. +Before you have a tested and independently running local setup, get your team involved and demonstrate how it works for you. Treat this as an experiment, not a mandate. Allow individual developers to adopt the monorepo locally and become advocates and champions for it. Pair program using the tooling and help orther developers mirror your setup. You know the team has adopted the practice when developers reference the monorepo amongst each other when when discussing daily work that's independent of yours. ### Continue Improving DevEx @@ -46,8 +50,8 @@ Schedule time with developers working in the new repo to understand how they use ## Lessons From The Field -- [*insert summary line*] If your company is structured in a way that you must wait for resource allocations or a separate team to build/grant you access, then use that time to start building this local repo. Once you've got an experimental version up and running, share it with the team. -- [*insert summary line*] Remote teams might not make thier local developer setups visible. Just because no one mentions the tool or praises it doesn't mean it's not being used. If it saves your developers time and easily shows them what they need to see, then they'll use it. +- *Use Idle-time to develop tooling* - Your organization may be structured in a way that you must wait for a separate team to build/configure/grant you access to required resurces - such as cloud databases, vendor sandboxes, or development environments. Instead of just waiting, use that time to start building this local repo. Once you've got an experimental version up and running, share it with the team. You may discover that you don't need to rely on others as much and can deliver value faster, saving cross-team-collaboration time for when it really matters. +- *Don't be discouraged by silence* - Remote teams might not make thier local developer setups visible. Just because no one mentions the monorepo or praises it doesn't mean it's not being used. If it saves your developers time and easily shows them what they need to see, then they'll use it. ## Deciding to Polish or Pitch @@ -55,7 +59,7 @@ After experimenting with this practice for **one month**, bring the team togethe ### Fast & Measurable -**Lead Time for Changes Decreases.** Changes spanning multiple repos should happen faster. By consolidating services into a monorepo, cross-cutting changes can be made without waiting for access approvals, access re-approvals, test deploying, or a clean tests enviornment. This reduces coordination overhead and speeds up delivery. Tools like [DX](https://getdx.com/platform/data-lake/) and [Jellyfish](https://jellyfish.co/platform/engineering-metrics/) can measure lead time changes. Custom solutions can also be built using data from project-planning tools like [JIRA](https://community.atlassian.com/forums/App-Central-articles/Cycle-Time-and-Lead-Time-in-Jira-Productivity-Measurement-with/ba-p/1905845) and [Monday](https://monday.com/blog/project-management/what-is-lead-time/). +**Lead Time for Changes Decreases.** Changes spanning multiple services and applications should happen faster. By consolidating services into a monorepo, cross-cutting changes can be made without waiting for access approvals, access re-approvals, test deploying, or an available cloud staging enviornment. This reduces coordination overhead and speeds up delivery. Tools like [DX](https://getdx.com/platform/data-lake/) and [Jellyfish](https://jellyfish.co/platform/engineering-metrics/) can measure lead time changes. Custom solutions can also be built using data from project-planning tools like [JIRA](https://community.atlassian.com/forums/App-Central-articles/Cycle-Time-and-Lead-Time-in-Jira-Productivity-Measurement-with/ba-p/1905845) and [Monday](https://monday.com/blog/project-management/what-is-lead-time/). ### Slow & Intangible @@ -75,4 +79,6 @@ When the team can test interactions between services, there is greater confidenc ### [Test Data Management](/capabilities/test-data-management.md) -Having control of the data locally means we also gain knowledge of the data's schema. Having and maintaining this information within the monorepo means we can eliminate uncertainty caused by database changes that previously existed outside the team's control. The same can be said for outside service vendors that supply information or data transformations the system depends on. By designing echo servers, teams can build more robust codebases (in the event these services don't perform flawlessly via principles of Chaos Engineering). +Having control of the data locally means we also gain knowledge of the data's schema. Having and maintaining this information within the monorepo means we can have confidence our code works against a given database schema. Comparing that schema to ones in cloud environments allows developers to stay in step with any updates a DBA or Data Team may add, which means there are less surprises, unsceduled meetings, and debugging at deploy-time. + +The same can be said for outside service vendors that supply information or data transformations the system depends on. By designing echo servers, developers can get faster feedback. Additionally, teams can build more robust code when a vendor's system slows or goes down. Major cloud vendors have been in the news for bringing enterprise systems down with them when the vendor experiences an outage. Since echo servers can also be built to simulate an outage, developers can develop graceful failure states that keep our systems running. From 6c2f1cf73749b1d3249d04b86237881ae933fab4 Mon Sep 17 00:00:00 2001 From: nicoletache Date: Tue, 30 Dec 2025 14:45:56 -0600 Subject: [PATCH 5/5] final edits to build monorepo for local dev practice --- practices/build-local-dev-monorepo.md | 30 +++++++++++++-------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/practices/build-local-dev-monorepo.md b/practices/build-local-dev-monorepo.md index 8c739b9..2c72bab 100644 --- a/practices/build-local-dev-monorepo.md +++ b/practices/build-local-dev-monorepo.md @@ -1,8 +1,6 @@ # Build a Monorepo for Local Development -Developing with dozens of local micro-repos that need to work together (and may also have database access or cloud permissions needs) is both daunting and time-consuming -- it can feel like your choices are either flying blind or waiting to get off the ground. Teams may encounter duplicated scripts and configuration across these many micro-repos. They may not trust interactions between services will behave as expected. And database changes may be out of their control, leading to more uncertainty. But when scattered projects are consolidated into a single monorepo, teams can see the system at work locally and start coding anywhere (even on a plane), without needing access approvals or an unused cloud environment. Many popular build tools support this functionality: - -#### build tools that support this functionality +Developing with dozens of local micro-repos that need to work together (and may also have database access or cloud permissions needs) is both daunting and time-consuming. Sometimes it can feel like your choices are either flying blind or waiting to get off the ground. Teams may encounter duplicated scripts and configuration across these many micro-repos. They may not trust interactions between services will behave as expected. And database changes may be out of their control, leading to more uncertainty. But when scattered projects are consolidated into a single monorepo, teams can see the system at work locally and start coding anywhere (even on a plane), without needing access approvals or an unused cloud environment. Many popular build tools support this functionality, including: - [pnpm](https://pnpm.io/workspaces) - [Nx](https://nx.dev/concepts/decisions/why-monorepos) @@ -17,13 +15,13 @@ Developing with dozens of local micro-repos that need to work together (and may ## How to Gain Traction -### Start by Bringing Together the Interdependent Repos +### Start by Bringing Together Some Interdependent Repos Choose as few services as you can get away with (2-3 ideally) that already depend on each other and form a complete system. Clone them under a single directory called something like `\services`. ### Create a Monorepo to Wrap the Services -In the parent directory, create a repo that uses a build tool that supports workspaces or monorepos ([see the list above](#build-tools-that-support-this-functionality)). Develop shell scripts that clone the interdependent repos into `/services` from the monorepo root, then extend the shell scripts to set up each of the services for local development as you would in isolation(install, build, run, etc.) +In the parent directory, create a repo that uses a build tool that supports workspaces or monorepos (see list at top of page). Develop shell scripts that clone the interdependent repos into `/services` from the monorepo root, then extend the shell scripts to set up each of the services for local development as you would in isolation (install, build, run, etc.). ### Add an Integration Testing Layer @@ -35,23 +33,23 @@ Use this pilot to work through defining the interfaces to the "outside world." U ### Fill in Vendor Dependecies With Echo Servers -Echo servers are named because they simply "echo back" what you send them. The echo servers we need will only be slightly more complex. They'll accept requests shaped as your vendor expects and return responses shaped the same as how your vendor replies. There should be next to no business logic in them. +Echo servers are named because they simply "echo back" what you send them. The echo servers we need will only be slightly more complex. They'll accept requests shaped as your vendor expects and return responses shaped in the way your vendor replies. There should be next to no business logic in them. -Your services may depend on calls to third-party vendors, such as accounting software, a CRM, or even a Zip Code finder. Calling out to the vendors directly, whenther using their live APIs or development sandboxes can have technical limitations or throttle feedback loops. To isolate your development environment and speed up feedback loops, you can instead create these basic servers ("echo servers") that take requests in an identical shape and return fake production-like repsonses. This can also create an opportunity to build in robustness into the system if vendor latency fluxuates -- when there is additional time, simulate this as well. Do the same for services that don't directly talk to the services, but modify the database on cron schedules. +Your services may depend on calls to third-party vendors, such as accounting software, a CRM, or even a Zip Code finder. Calling out to the vendors directly, whenther using their live APIs or development sandboxes can have technical limitations or throttle feedback loops. To isolate your development environment and speed up feedback loops, you can instead create these basic echo servers that take requests in an identical shape and return fake production-like repsonses. This can also create an opportunity to build robustness into the system if vendor latency fluxuates -- when there is additional time, simulate this as well. Do the same for services that don't directly talk to vendors, but modify the database on cron schedules. ### Create a Migration Path -Before you have a tested and independently running local setup, get your team involved and demonstrate how it works for you. Treat this as an experiment, not a mandate. Allow individual developers to adopt the monorepo locally and become advocates and champions for it. Pair program using the tooling and help orther developers mirror your setup. You know the team has adopted the practice when developers reference the monorepo amongst each other when when discussing daily work that's independent of yours. +Before you have a tested and independently running local setup, get your team involved and demonstrate how it works for you. Treat this as an experiment, not a mandate. Allow individual developers to adopt the monorepo locally and become advocates and champions for it. Pair program using the tooling and help other developers mirror your setup. You know the team has adopted the practice when developers reference the monorepo when when discussing daily work that's independent of yours. ### Continue Improving DevEx -Schedule time with developers working in the new repo to understand how they use it and where they still encounter friction. This can be done during the course of regular work if you pair program. Listen to any gripes and complaints, and note what still causes you confusion or wait times. Brainstorm potential solutions as a group and set aside time to improve problematic parts of the system. Focus on improving the items listed in the Polish or Pitch section of this page to ensure efforts are making a difference. You know it's simple to set up and use when you observe developers using it who haven't come to you with questions first. +Schedule time with developers working in the new repo to understand how they use it and where they still encounter friction. This can be done during the course of regular work if you pair program. Listen to any gripes and complaints, and note what still causes confusion or wait times. Brainstorm potential solutions as a group and set aside time to improve problematic parts of the system. Focus on improving items listed in the Polish or Pitch section of this page to ensure efforts are making a difference. You know the repo is simple to set up and use when you observe developers using it who haven't come to you with questions first. ## Lessons From The Field -- *Use Idle-time to develop tooling* - Your organization may be structured in a way that you must wait for a separate team to build/configure/grant you access to required resurces - such as cloud databases, vendor sandboxes, or development environments. Instead of just waiting, use that time to start building this local repo. Once you've got an experimental version up and running, share it with the team. You may discover that you don't need to rely on others as much and can deliver value faster, saving cross-team-collaboration time for when it really matters. -- *Don't be discouraged by silence* - Remote teams might not make thier local developer setups visible. Just because no one mentions the monorepo or praises it doesn't mean it's not being used. If it saves your developers time and easily shows them what they need to see, then they'll use it. +- *Use Idle-time to Develop Tooling* - Your organization may be structured in a way that requires you to wait for a separate team to build/configure/grant you access to needed resources -- cloud databases, vendor sandboxes, or development environments. Instead of just waiting, use that time to start building this local repo. You may discover that you don't need to rely on others as much here and you can deliver value faster than you thought. Once you've got an experimental version up and running, share it with the team. This way, you save cross-team collaboration for when it really matters. +- *Don't Be Discouraged By Silence* - Remote teams might not make their local developer setups visible. Just because no one mentions the monorepo or praises it doesn't mean it's not being used. If it saves your developers time and easily shows them what they need to see, then they'll use it. ## Deciding to Polish or Pitch @@ -59,13 +57,13 @@ After experimenting with this practice for **one month**, bring the team togethe ### Fast & Measurable -**Lead Time for Changes Decreases.** Changes spanning multiple services and applications should happen faster. By consolidating services into a monorepo, cross-cutting changes can be made without waiting for access approvals, access re-approvals, test deploying, or an available cloud staging enviornment. This reduces coordination overhead and speeds up delivery. Tools like [DX](https://getdx.com/platform/data-lake/) and [Jellyfish](https://jellyfish.co/platform/engineering-metrics/) can measure lead time changes. Custom solutions can also be built using data from project-planning tools like [JIRA](https://community.atlassian.com/forums/App-Central-articles/Cycle-Time-and-Lead-Time-in-Jira-Productivity-Measurement-with/ba-p/1905845) and [Monday](https://monday.com/blog/project-management/what-is-lead-time/). +**Lead Time for Changes Decreases.** Changes spanning multiple services and applications should happen faster. By consolidating services into a monorepo, cross-cutting changes can be made without waiting for access approvals, access re-approvals, test deploying, or an available cloud staging environment. This reduces coordination overhead and speeds up delivery. Tools like [DX](https://getdx.com/platform/data-lake/) and [Jellyfish](https://jellyfish.co/platform/engineering-metrics/) can measure lead time changes. Custom solutions can also be built using data from project-planning tools like [JIRA](https://community.atlassian.com/forums/App-Central-articles/Cycle-Time-and-Lead-Time-in-Jira-Productivity-Measurement-with/ba-p/1905845) and [Monday](https://monday.com/blog/project-management/what-is-lead-time/). ### Slow & Intangible -**Cleaner Service Boundaries.** Refactoring service boundaries should become easier. When everything lives in one repo, poor service boundaries can be removed with less friction. On the flipside, teams can more quickly extract new services with the shared tooling, configuration, and build setup. +**Cleaner Service Boundaries.** Refactoring service boundaries should become easier. When everything lives in one repo, poor service boundaries can be removed with less friction. Teams can also quickly extract new services with the shared tooling, configuration, and build setup. -**Better cross-team collaboration.** The monorepo can be spun up fairly simply for team collaboration and to demonstrate system behavior. In terms of database changes, comparing what's set up locally to the new schema change can be a point of conversation between teams that manage the data and teams than manage the code. The same thing applies to cloud environment connections between services, and third-party API vendor interactions. When change involves less uncertainty, then confidence to experiment increases. +**Better Cross-team Collaboration.** The monorepo can be spun up fairly simply for team collaboration and to demonstrate system behavior. In terms of database changes, comparing what's set up locally to the new schema change can be a point of conversation between teams that manage the data and teams than manage the code. The same thing applies to cloud environment connections between services and third-party API vendor interactions. When change involves less uncertainty, the confidence to experiment increases. ## Supported Capabilities @@ -79,6 +77,6 @@ When the team can test interactions between services, there is greater confidenc ### [Test Data Management](/capabilities/test-data-management.md) -Having control of the data locally means we also gain knowledge of the data's schema. Having and maintaining this information within the monorepo means we can have confidence our code works against a given database schema. Comparing that schema to ones in cloud environments allows developers to stay in step with any updates a DBA or Data Team may add, which means there are less surprises, unsceduled meetings, and debugging at deploy-time. +Having control of the data locally means understanding the data's schema. By maintaining this information within the monorepo, we can feel confident our code works against a given database schema. Comparing that schema to ones in cloud environments allows developers to stay in step with any updates a DBA or data team may add, which means there are less surprises, unscheduled meetings, and debugging at deploy time. -The same can be said for outside service vendors that supply information or data transformations the system depends on. By designing echo servers, developers can get faster feedback. Additionally, teams can build more robust code when a vendor's system slows or goes down. Major cloud vendors have been in the news for bringing enterprise systems down with them when the vendor experiences an outage. Since echo servers can also be built to simulate an outage, developers can develop graceful failure states that keep our systems running. +The same can be said for outside service vendors that supply information or data transformations the system depends on. By designing echo servers, developers can get faster feedback. Additionally, teams can build more robust code when a vendor's system slows or goes down. Major cloud vendors have been in the news for bringing enterprise systems down with them when the vendor experiences an outage. Since echo servers can also be built to simulate an outage, developers can develop graceful failure states that keep systems running.