By Matthew Mangion
Introduction
At Yieldstreet, we strive to be at the forefront of front-end development and improve our development delivery cadence, codebase quality, and maintainability. As front-end projects scale with new features, inevitably, complexity and maintenance grow exponentially.
One significant shift we made was moving from a multirepo structure to a monorepo structure. This change brings about a host of benefits that help us streamline our operations and improve our overall efficiency. In this article, we will explore the reasons behind this transition and the advantages it provides us.
What is a monorepo?
A monorepo, short for monolithic repository, is a software development strategy where multiple projects are stored in a single repository. This approach is becoming increasingly popular among developers due to its many benefits. With a monorepo, developers can easily share code between projects, ensure consistency across all projects, and simplify the build and deployment process. This way, monorepos help to streamline development workflows and promote collaboration among team members.
Monorepos should not be confused with monolithic architecture, which is a software development practice of self-contained applications.
The opposite of a monorepo is a multirepo, whereas every project has a separate version-controlled repository.
Why do we need a monorepo?
The consideration of the monorepo strategy kickstarts via a thorough analysis of its fit with the company’s technical architecture and future goals. Four factors are primarily considered:
- Codebase Size and Complexity: Monorepos are well-suited for larger projects with complex interdependencies and frequent code and asset sharing.
- Collaboration Requirements: Monorepos promote better collaboration and communication, while also reducing code and team friction.
- Build and Deployment Processes: Although monorepos might increase some build time for one application, passing through the whole merging process would be simpler; one pull request every task.
- Impact: The overall impact on efficiency on the development team is weighed against the cost of migration.
Why do we need to change?
Prior to the monorepo migration, the YS front end team was responsible for the curation of (i) five host applications, and (ii) three libraries for shared functionality. A simplified version of the relationships across these eight artifacts are captured below.
Pre-Monorepo Development flow (multirepo)
Syncronize latest changes
Consider a scenario that requires the addition of a new API call to the Yieldstreet application, with the logic being stored inside a hook (utility function) within the utils SDK.
This addition requires:
- Retrieving the latest changes from the Yieldstreet app, Platform SDK, and Utils SDK repositories.
- Creating a symlink across all repositories (assuming that no one else is merging any breaking changes).
- Building all the libraries.
- Starting the application, and commencing development.
Upon completion of development, opening a pull request for each repository, which awaits approval from the respective code owners.
Merging
The merging process has to proceed in a staged approach. In this case, the flow looks like:
- Merging Platform SDK, and waiting for internal CI to build and publish the package on our internal package manager.
- Updating the Platform SDK version on Utils SDK, rebuilding, merging, and waiting for the package to be published.
- Updating the Yieldstreet application with both Utils SDK and Platform SDK, rebuilding the application, and merging.
This process is evidently cumbersome, often consuming hours of development time due to the merging of conflict resolutions. Understandably, this also significantly raises developers' frustrations during their day-to-day experience.
Monorepo Development flow
Syncronize latest changes
Consider the same scenario described previously. Whilst leveraging a monorepo, the development process requires:
- Syncing to a single repository.
- Starting the application, and commencing development.
This process is evidently greatly simplified compared to the previous iteration. Furthermore, upon completion of development, only a single pull request to the repository is necessary, awaiting approval from the respective code owners.
Merging
A monorepo completely eliminates the need for a staged merging process. All the development context gets merged when the pull request is merged.
The evaluation process
Each project and library is evaluated, including their dependencies (such as react, redux, and react-native), as well as their limitations. Consider the scenario below:
The migration complexity evaluation includes:
- Identification of shared components: This encompasses shared libraries like react, redux, typescript, jest, and their version mismatches.
- Analyzing project relationships: Determining the interconnections between projects, including identifying potential challenges during monorepo migration. This process involves creating multiple rapid proof of concepts.
- Considering the build and deployment processes: Understanding the current build, testing, and deployment procedures. This understanding serves as the foundation for refining these processes in subsequent iterations to align with the new monorepo structure.
- Package manager migration: The classic NPM, Yarn, or Pnpm options are weighed. Classic npm or yarn v1 installations are generally inadequate for a monorepo architecture due to the excessive time they take to run. As a result, there is contemplation of migrating to yarn berry or Pnpm to enhance installation efficiency.
Plan the monorepo structure
Designing the structure of our monorepo is a critical step. The primary goals of this processing include
- ensuring a more straightforward and quicker development output,
- ensuring scalability and ease of code maintenance.
To achieve these objectives, the libraries need to be directly linked to one another, eliminating the need for a separate library building and publishing step. Refer to the image below.
These criteria are being considered:
- Shared code and dependencies: Deciding on an effective strategy for managing shared code and dependencies.
- Utilizing an actively maintained monorepo management tool like Nx or Turborepo.
- Upgrading the package manager to Pnpm or Yarn Berry to enhance installation speed.
- Defining boundaries: Planning how to group related code, taking into account ownership, functionality, and business domains.
- Managing versioning and releases: Planning the approach to versioning and releases within the monorepo.
- Testing: Leveraging the comprehensive codebase context offered by the monorepo to enhance the efficiency of unit and integration testing.
Planning the migration process
With a clear vision of the end goal in mind, the next crucial step involves creating a clear plan and estimating the effort required for migrating from the current codebases to a monorepo. It becomes evident that this effort is not trivial, and consolidating development resources would not necessarily expedite the migration timeline. Therefore, a meticulous plan is essential, along with considering contingency plans.
Migration timeline?
Considering the significant timeline and the crucial focus on maintaining continuous efforts throughout the migration, we brainstormed a solution. This solution involves synchronizing the existing repositories with the new monorepo repository. This synchronization is facilitated through the utilization of git subtrees (distinct from git submodules). By employing this approach, ongoing work on the monorepo can proceed concurrently, allowing the rest of the development team to concentrate on delivering product features and addressing bugs without any interruptions.
Consider the current folder structure, where apps, libs, and package.json reside at the root level of the monorepo.
Monorepo Root/
├── apps/
│ ├── Yieldstreet-App/
│ ├── Yieldstreet-Blog/
│ ├── Admin/
│ ├── E2E/
│ └── Native-App/
├── libs/
│ ├── Ui-Kit/
│ ├── Utils-SDK/
│ └── Platform-SDK/
├── package.json
├── jest.config.ts
├── turbo.json
├── pnpm-lock.yaml
└── pnpm-workspace.yaml
Linking Multi-Repo Repository to a Monorepo Folder
-
Start by adding a remote repository
git remote add-f WebApp1 git@github.com:yieldstreet/multi-repo-webapp1.git -
Create subtree into the folder apps/WebApp1 and use multi-repo develop branch
git subtree add --prefix apps/WebApp1 WebApp1 develop -
In the root package.json, add the following to the scripts. Run this command to sync with the multi-repo regularly.
{ "scripts": { "pull:webapp1": "git subtree pull --prefix apps/WebApp1 WebApp1 develop" } }
Execution
Package manager upgrade
It immediately becomes apparent that migrating to Yarn Berry won't function with react-native, leading to a strategy shift towards Pnpm. This strategy incorporates the utilization of the metro resolver:@rnx-kit/metro-resolver-symlinks. Thankfully, Pnpm has the capability to import yarn.lock through the pnpm import command, which streamlines the process of determining the correct versions, saving a few hours in the process.
Migrate to a workspace
In the multi-repo structure, the libraries depend on the version. In the monorepo structure, the version is no longer used; you simply use workspace:* for each dependent library.
{
"dependencies": {
"@yieldstreet/ui-kit": "workspace:*",
"@yieldstreet/utils-sdk": "workspace:*",
"@yieldstreet/platform-sdk": "workspace:*"
}
}
Monorepo Tooling
The monorepo tooling plays a crucial role in the project's success. In this regard, both Nx and TurboRepo are being evaluated. Nx stands out as a more mature tool. However, TurboRepo is ultimately preferred due to its concise and clear documentation that offers all the necessary functionality.
Docker Cleanup
Monorepo migration also implies the necessity of altering deployment scripts. As a result, the Docker scripts are being rewritten by adhering to the suggestions provided by TurboRepo. This adjustment also facilitates appropriate docker layer caching, leading to faster builds.
RUN pnpm fetch --prod
RUN pnpm install -r --offline --prod
Package.json Cleanup
Monorepo adds some complexity and installation time during ci/cd builds. To improve this, the global set of dependencies were split into dependencies, dev dependencies, and optional dependencies.
Optional dependencies are utilized to underscore potential dependency issues. The provided code snippet illustrates that axios-mock-adapter need not be required for building the application.
{
"name": "Yieldstreet App,"
"dependencies": {
"react": "18",
"typescript": "^4.9.x"
},
"optionalDependencies": {
"axios-mock-adapter": "^1.21.x"
},
"devDependencies": {
"@types/jest": "^29"
}
}
Unit testing cleanup
Migration
One primary challenge is migrating unit tests, and a key hurdle lies in dealing with varying versions utilized across different repositories. Substantial effort is being invested in unifying all versions to the latest jest version, resolving test-related issues along the way. This step is crucial, especially given that some repositories lag behind by several major versions. Furthermore, this process is uncovering actual bugs as well.
Project grouping
Jest supports grouping projects. The following snippet in jest.config.js is located in the root folder enabled jest to run all projects from the root directory.
module.exports = {
...baseConfig,
projects: [
'<rootDir>/apps/*/jest.config.[jt]s',
'<rootDir>/libs/*/jest.config.js',
]
};
Selective testing
One of the primary advantages of a monorepo is the consolidation of all file changes within a single pull request. To speed up testing, the Jest changedSince command is being considered, allowing the execution of only the pertinent files. This adjustment has minimized unit testing time, reducing it from around 15 minutes to a few minutes or even seconds, depending on the specific on the nature of the change.
Communicate the changes
Finally, it is crucial to communicate the changes to all team members and stakeholders. This include:
- Explaining the benefits of the monorepo.
- Outlining the new processes for code management.
- Provide training and support to ensure a smooth transition.
Summary
The decision to transition to a monorepo proves to be highly beneficial and valuable for our company. One of the significant advantages we are experiencing is a reduction in development time, leading to substantial time savings. By consolidating our codebase into a single repository, we eliminate the need to switch between multiple repositories and simplify the overall development process.
Moreover, this shift to a monorepo facilitates an improvement in code quality and standardization. Developers are encouraged to write comprehensive unit tests and actively address any failing tests. This emphasis on testing ensures that our codebase becomes more robust and reliable. Additionally, the practice of refactoring is promoted, enabling developers to continuously improve the codebase's structure and maintainability.
The monorepo setup also encourages the sharing of code within libraries. Developers can identify reusable components or functionalities and extract them into separate libraries, which can be easily shared across projects. This approach fosters code reusability, reducing duplication, and promoting consistency across different applications.
As a result of adopting the monorepo, our company also prioritizes the establishment of best practices. Developers are encouraged to advocate for standardized coding conventions, architectural patterns, and development workflows. This ensures that our projects follow consistent guidelines, leading to better collaboration among team members and easier onboarding of new developers.
In addition to the previously mentioned benefits, the transition to a monorepo prompts us to make further improvements in our development ecosystem. For example, we are revamping our Docker scripts to optimize performance. We are introducing a caching layer to prevent the need for reinstalling node modules with every build, resulting in faster build times and more efficient resource utilization.
Furthermore, we are taking the opportunity to upgrade and consolidate key tools such as Jest, EsLint, Node, and Storybook. This enables us to leverage the latest features and enhancements, improving the overall development experience. It also empowers us to take advantage of advanced testing capabilities and better documentation.
Overall, the decision to switch to a monorepo has a positive and far-reaching impact on our company. It not only reduces development time, improves code quality, and fosters code sharing, but also motivates us to enhance our development practices and optimize our tooling. As a result, we are experiencing easier bug fixes and hotfixes, smoother collaboration, and a more efficient and streamlined development workflow.
