Addressing Challenges with Monorepos

Working with a large Git monorepo can introduce continuous integration challenges, but there are some techniques you and your team can employ to overcome common hurdles.

100,000 files

Microservices

CI/CD

As large monorepos continue to grow overtime, teams can experience continuous integration (CI) challenges. Running the entire test suite every day, for example, isn’t an option anymore.

Common continuous integration challenges with monorepos include:

The first step is finding the changed files in your Git monorepo. Next, you will need to find all the transitively dependent test and binary targets from changed files and repo test them. The latter step can get complex.

A structure for calculating changed targets for a monorepo can look something like this:

You have four changed targets: A -> B -> C <- D (A depends on B; B depends on C; and D depends on C)

Thus, if there is a change to file C; that will change targets A, B, C, & D; if there is a change to file B; that will change targets A & B; if there is a change to file A; that will change target A.

As an example, in the Oscar Health monorepo, every build or diff can potentially have 10,000 targets that need to be tested, but on average they only need to test a few hundred using this calculation model.

As you work with a monorepo, two types of objects will continue growing linearly over time: the monorepo itself and the third parties defined in the monorepo.

Techniques for caching these types of large objects can differ depending on the remote CI server type, whether long-running or ad hoc, with the latter being more complicated.

For long-running servers, you can employ the following strategies:

Ad hoc servers autoscale by nature, you will need to be more thoughtful about using them in your workflow and will need to use caching techniques.

Related posts:

This program will give our community members who want to build products, create content, and innovate new ideas within the SX Network ecosystem a chance to earn $SX for doing so! This program will…

It is easier to imagine the end of the world than the end of capitalism (Mark Fisher, 2009). As Mark Fisher claims through his whole life, we live in this highly capitalised society where many…

Collaboration is the key to success in today’s fast-paced and interconnected world. Whether you’re working on a team project, managing a remote workforce, or coordinating efforts across different…