diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index a5be29266d3e4f16a3edaba1c5412c45b367375a..fb6fbc5a37e577dc745e971cf8c0d01a536e5504 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -50,7 +50,9 @@ full list of reference architectures, see - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. -6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large + repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to + [Large repositories](index.md#large-repositories) for more information. NOTE: @@ -144,66 +146,7 @@ monitor .[#7FFFD4,norank]u--> elb ## Requirements -Before starting, you should take note of the following requirements / guidance for this reference architecture. - -### Supported CPUs - -This reference architecture was built and tested on Google Cloud Platform (GCP) using the -[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). - -Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported. - -NOTE: -Any "burstable" instance types are not recommended due to inconsistent performance. - -### Supported infrastructure - -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, -or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. -However, this does not constitute a guarantee for every potential permutation. - -See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - -### Additional workloads - -The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with -good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, -such as security software, you may still need to adjust the specs accordingly to compensate. - -This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). - -As a general rule, it's recommended to have robust monitoring in place to measure the impact of -any additional workloads to inform any changes needed to be made. - -### Large repositories - -The Reference Architectures were tested with repositories of varying sizes that follow best practices. - -However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance -of Git and in turn the environment itself if best practices aren't being followed such as not storing -binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging -when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) -taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and -CI pipelines alike. - -As such, large repositories come with notable cost and typically will require more resources to handle, -significantly so in some cases. It's therefore **strongly** recommended then to review large repositories -to ensure they maintain good health and reduce their size wherever possible. - -NOTE: -If best practices aren't followed and large repositories are present on the environment, -increased Gitaly specs may be required to ensure stable performance. - -Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) -for more information and guidance. - -### Praefect PostgreSQL - -It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and -that to achieve full High Availability a third-party PostgreSQL database solution will be required. -We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). +Before starting, see the [requirements](index.md#requirements) for reference architectures. ## Setup components @@ -1223,7 +1166,7 @@ NOTE: Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos not following these practices can significantly impact Gitaly performance and requirements. -Refer to the [Large Repositories](#large-repositories) for more info. +Refer to [Large repositories](index.md#large-repositories) for more information. The recommended cluster setup includes the following components: @@ -1533,14 +1476,14 @@ requirements that are dependent on data and load. NOTE: Increased specs for Gitaly nodes may be required in some circumstances such as -significantly large repositories or if any [additional workloads](#additional-workloads), +significantly large repositories or if any [additional workloads](index.md#additional-workloads), such as [server hooks](../server_hooks.md), have been added. NOTE: Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos not following these practices can significantly impact Gitaly performance and requirements. -Refer to the [Large Repositories](#large-repositories) for more info. +Refer to [large repositories](index.md#large-repositories) for more information. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -2346,7 +2289,9 @@ services where applicable): - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. -6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large + repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to + [Large repositories](index.md#large-repositories) for more information. NOTE: diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 2a9636b6e05e5981d3cb6440c8ab6aeba0462152..e63d3c0cc2335f6510fe3b26268606265784dd5a 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -67,47 +67,7 @@ The diagram above shows that while GitLab can be installed on a single server, i ## Requirements -Before starting, you should take note of the following requirements / guidance for this reference architecture. - -### Supported CPUs - -This reference architecture was built and tested on Google Cloud Platform (GCP) using the -[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). - -Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported. - -NOTE: -Any "burstable" instance types are not recommended due to inconsistent performance. - -### Supported infrastructure - -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, -or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. -However, this does not constitute a guarantee for every potential permutation. - -See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - -### Additional workloads - -The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with -good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, -such as security software, you may still need to adjust the specs accordingly to compensate. - -This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). - -As a general rule, it's recommended to have robust monitoring in place to measure the impact of -any additional workloads to inform any changes needed to be made. - -### Swap - -In addition to the stated configurations, we recommend having at least 2 GB of -swap on your server, even if you currently have enough available memory. Having -swap helps to reduce the chance of errors occurring if your available memory -changes. We also recommend configuring the kernel's swappiness setting to a -lower value (such as `10`) to make the most of your memory, while still having -the swap available when needed. View the -[Memory requirements](../../install/requirements.md#memory) for details. +Before starting, see the [requirements](index.md#requirements) for reference architectures. ## Setup instructions diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 3884b55b358a7f9eb9e9d53727bcc6ec351348c8..93985c0ef551030aa6b1ffc8a0d012ee7f650746 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -50,7 +50,9 @@ full list of reference architectures, see - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. -6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large + repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to + [Large repositories](index.md#large-repositories) for more information. NOTE: @@ -144,66 +146,7 @@ monitor .[#7FFFD4,norank]u--> elb ## Requirements -Before starting, you should take note of the following requirements / guidance for this reference architecture. - -### Supported CPUs - -This reference architecture was built and tested on Google Cloud Platform (GCP) using the -[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). - -Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported. - -NOTE: -Any "burstable" instance types are not recommended due to inconsistent performance. - -### Supported infrastructure - -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, -or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. -However, this does not constitute a guarantee for every potential permutation. - -See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - -### Additional workloads - -The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with -good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, -such as security software, you may still need to adjust the specs accordingly to compensate. - -This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). - -As a general rule, it's recommended to have robust monitoring in place to measure the impact of -any additional workloads to inform any changes needed to be made. - -### Large repositories - -The Reference Architectures were tested with repositories of varying sizes that follow best practices. - -However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance -of Git and in turn the environment itself if best practices aren't being followed such as not storing -binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging -when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) -taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and -CI pipelines alike. - -As such, large repositories come with notable cost and typically will require more resources to handle, -significantly so in some cases. It's therefore **strongly** recommended then to review large repositories -to ensure they maintain good health and reduce their size wherever possible. - -NOTE: -If best practices aren't followed and large repositories are present on the environment, -increased Gitaly specs may be required to ensure stable performance. - -Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) -for more information and guidance. - -### Praefect PostgreSQL - -It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and -that to achieve full High Availability a third-party PostgreSQL database solution will be required. -We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). +Before starting, see the [requirements](index.md#requirements) for reference architectures. ## Setup components @@ -1243,7 +1186,7 @@ NOTE: Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos not following these practices can significantly impact Gitaly performance and requirements. -Refer to the [Large Repositories](#large-repositories) for more info. +Refer to [Large repositories](index.md#large-repositories) for more information. The recommended cluster setup includes the following components: @@ -1551,14 +1494,14 @@ requirements that are dependent on data and load. NOTE: Increased specs for Gitaly nodes may be required in some circumstances such as -significantly large repositories or if any [additional workloads](#additional-workloads), +significantly large repositories or if any [additional workloads](index.md#additional-workloads), such as [server hooks](../server_hooks.md), have been added. NOTE: Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos not following these practices can significantly impact Gitaly performance and requirements. -Refer to the [Large Repositories](#large-repositories) for more info. +Refer to [Large repositories](index.md#large-repositories) for more information. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -2365,7 +2308,9 @@ services where applicable): - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. -6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large + repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to + [Large repositories](index.md#large-repositories) for more information. NOTE: diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 3d9f91b22c38b4421f5e1b09250d3aa152e34623..b102842068a0a56b6b02f9b285d08934df2a533e 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -42,7 +42,9 @@ For a full list of reference architectures, see 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. -5. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. +5. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large + repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to + [Large repositories](index.md#large-repositories) for more information. NOTE: @@ -80,59 +82,7 @@ monitor .[#7FFFD4,norank]u--> elb ## Requirements -Before starting, you should take note of the following requirements / guidance for this reference architecture. - -### Supported CPUs - -This reference architecture was built and tested on Google Cloud Platform (GCP) using the -[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). - -Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported. - -NOTE: -Any "burstable" instance types are not recommended due to inconsistent performance. - -### Supported infrastructure - -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, -or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. -However, this does not constitute a guarantee for every potential permutation. - -See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - -### Additional workloads - -The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with -good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, -such as security software, you may still need to adjust the specs accordingly to compensate. - -This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). - -As a general rule, it's recommended to have robust monitoring in place to measure the impact of -any additional workloads to inform any changes needed to be made. - -### Large repositories - -The Reference Architectures were tested with repositories of varying sizes that follow best practices. - -However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance -of Git and in turn the environment itself if best practices aren't being followed such as not storing -binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging -when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) -taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and -CI pipelines alike. - -As such, large repositories come with notable cost and typically will require more resources to handle, -significantly so in some cases. It's therefore **strongly** recommended then to review large repositories -to ensure they maintain good health and reduce their size wherever possible. - -NOTE: -If best practices aren't followed and large repositories are present on the environment, -increased Gitaly specs may be required to ensure stable performance. - -Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) -for more information and guidance. +Before starting, see the [requirements](index.md#requirements) for reference architectures. ## Setup components @@ -460,14 +410,14 @@ specifically the number of projects and those projects' sizes. NOTE: Increased specs for Gitaly nodes may be required in some circumstances such as -significantly large repositories or if any [additional workloads](#additional-workloads), +significantly large repositories or if any [additional workloads](index.md#additional-workloads), such as [server hooks](../server_hooks.md), have been added. NOTE: Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos not following these practices can significantly impact Gitaly performance and requirements. -Refer to the [Large Repositories](#large-repositories) for more info. +Refer to [Large repositories](index.md#large-repositories) for more information. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index eef7a9d08ce9eee57201292783ddbc3b23dee6ab..b1d2916b7acbfec698606f684dc1dff97352ad0f 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -59,7 +59,9 @@ For a full list of reference architectures, see - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. -6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large + repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to + [Large repositories](index.md#large-repositories) for more information. NOTE: @@ -150,66 +152,7 @@ monitor .[#7FFFD4,norank]u--> elb ## Requirements -Before starting, you should take note of the following requirements / guidance for this reference architecture. - -### Supported CPUs - -This reference architecture was built and tested on Google Cloud Platform (GCP) using the -[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). - -Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported. - -NOTE: -Any "burstable" instance types are not recommended due to inconsistent performance. - -### Supported infrastructure - -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, -or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. -However, this does not constitute a guarantee for every potential permutation. - -See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - -### Additional workloads - -The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with -good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, -such as security software, you may still need to adjust the specs accordingly to compensate. - -This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). - -As a general rule, it's recommended to have robust monitoring in place to measure the impact of -any additional workloads to inform any changes needed to be made. - -### Large repositories - -The Reference Architectures were tested with repositories of varying sizes that follow best practices. - -However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance -of Git and in turn the environment itself if best practices aren't being followed such as not storing -binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging -when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) -taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and -CI pipelines alike. - -As such, large repositories come with notable cost and typically will require more resources to handle, -significantly so in some cases. It's therefore **strongly** recommended then to review large repositories -to ensure they maintain good health and reduce their size wherever possible. - -NOTE: -If best practices aren't followed and large repositories are present on the environment, -increased Gitaly specs may be required to ensure stable performance. - -Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) -for more information and guidance. - -### Praefect PostgreSQL - -It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and -that to achieve full High Availability a third-party PostgreSQL database solution will be required. -We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). +Before starting, see the [requirements](index.md#requirements) for reference architectures. ## Setup components @@ -1178,7 +1121,7 @@ NOTE: Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos not following these practices can significantly impact Gitaly performance and requirements. -Refer to the [Large Repositories](#large-repositories) for more info. +Refer to [Large repositories](index.md#large-repositories) for more information. The recommended cluster setup includes the following components: @@ -1485,14 +1428,14 @@ requirements that are dependent on data and load. NOTE: Increased specs for Gitaly nodes may be required in some circumstances such as -significantly large repositories or if any [additional workloads](#additional-workloads), +significantly large repositories or if any [additional workloads](index.md#additional-workloads), such as [server hooks](../server_hooks.md), have been added. NOTE: Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos not following these practices can significantly impact Gitaly performance and requirements. -Refer to the [Large Repositories](#large-repositories) for more info. +Refer to the [Large repositories](index.md#large-repositories) for more information. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -2334,7 +2277,9 @@ services where applicable): - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. -6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large + repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to + [Large repositories](index.md#large-repositories) for more information. NOTE: diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index acd5ff92da610c64036808f8b7f307cc27c89329..fa9da08fad1bc58645b0a85e7f281ae305b1e07c 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -50,7 +50,9 @@ full list of reference architectures, see - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. -6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large + repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to + [Large repositories](index.md#large-repositories) for more information. NOTE: @@ -144,66 +146,7 @@ monitor .[#7FFFD4,norank]u--> elb ## Requirements -Before starting, you should take note of the following requirements / guidance for this reference architecture. - -### Supported CPUs - -This reference architecture was built and tested on Google Cloud Platform (GCP) using the -[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). - -Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported. - -NOTE: -Any "burstable" instance types are not recommended due to inconsistent performance. - -### Supported infrastructure - -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, -or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. -However, this does not constitute a guarantee for every potential permutation. - -See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - -### Additional workloads - -The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with -good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, -such as security software, you may still need to adjust the specs accordingly to compensate. - -This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). - -As a general rule, it's recommended to have robust monitoring in place to measure the impact of -any additional workloads to inform any changes needed to be made. - -### Large repositories - -The Reference Architectures were tested with repositories of varying sizes that follow best practices. - -However, large repositories or monorepos (Several gigabytes or more) can **significantly** impact the performance -of Git and in turn the environment itself if best practices aren't being followed such as not storing -binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging -when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) -taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and -CI pipelines alike. - -As such, large repositories come with notable cost and typically will require more resources to handle, -significantly so in some cases. It's therefore **strongly** recommended then to review large repositories -to ensure they maintain good health and reduce their size wherever possible. - -NOTE: -If best practices aren't followed and large repositories are present on the environment, -increased Gitaly specs may be required to ensure stable performance. - -Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) -for more information and guidance. - -### Praefect PostgreSQL - -It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and -that to achieve full High Availability a third-party PostgreSQL database solution will be required. -We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). +Before starting, see the [requirements](index.md#requirements) for reference architectures. ## Setup components @@ -1236,7 +1179,7 @@ NOTE: Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos not following these practices can significantly impact Gitaly performance and requirements. -Refer to the [Large Repositories](#large-repositories) for more info. +Refer to [Large repositories](index.md#large-repositories) for more information. The recommended cluster setup includes the following components: @@ -1546,14 +1489,14 @@ requirements that are dependent on data and load. NOTE: Increased specs for Gitaly nodes may be required in some circumstances such as -significantly large repositories or if any [additional workloads](#additional-workloads), +significantly large repositories or if any [additional workloads](index.md#additional-workloads), such as [server hooks](../server_hooks.md), have been added. NOTE: Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos not following these practices can significantly impact Gitaly performance and requirements. -Refer to the [Large Repositories](#large-repositories) for more info. +Refer to [Large repositories](index.md#large-repositories) for more information. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -2363,7 +2306,9 @@ services where applicable): - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. -6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large + repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to + [Large repositories](index.md#large-repositories) for more information. NOTE: diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 2ddfb662be7bb93feb63a18bdf00b76b3c94e6bc..8684fe98b1dc34e75f1025d456375f27bea77132 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -56,7 +56,9 @@ costly-to-operate environment by using the - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. -6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large + repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to + [Large repositories](index.md#large-repositories) for more information. NOTE: @@ -147,66 +149,7 @@ monitor .[#7FFFD4,norank]u--> elb ## Requirements -Before starting, you should take note of the following requirements / guidance for this reference architecture. - -### Supported CPUs - -This reference architecture was built and tested on Google Cloud Platform (GCP) using the -[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). - -Newer, similarly sized CPUs are supported and may have improved performance as a result. For Omnibus environments, ARM-based equivalents are also supported. - -NOTE: -Any "burstable" instance types are not recommended due to inconsistent performance. - -### Supported infrastructure - -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, -or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. -However, this does not constitute a guarantee for every potential permutation. - -See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - -### Additional workloads - -The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with -good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, -such as security software, you may still need to adjust the specs accordingly to compensate. - -This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). - -As a general rule, it's recommended to have robust monitoring in place to measure the impact of -any additional workloads to inform any changes needed to be made. - -### Large repositories - -The Reference Architectures were tested with repositories of varying sizes that follow best practices. - -However, large repositories or monorepos (Several gigabytes or more) can **significantly** impact the performance -of Git and in turn the environment itself if best practices aren't being followed such as not storing -binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging -when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) -taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and -CI pipelines alike. - -As such, large repositories come with notable cost and typically will require more resources to handle, -significantly so in some cases. It's therefore **strongly** recommended then to review large repositories -to ensure they maintain good health and reduce their size wherever possible. - -NOTE: -If best practices aren't followed and large repositories are present on the environment, -increased Gitaly specs may be required to ensure stable performance. - -Refer the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) -for more information and guidance. - -### Praefect PostgreSQL - -It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and -that to achieve full High Availability a third-party PostgreSQL database solution is required. -We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). +Before starting, see the [requirements](index.md#requirements) for reference architectures. ## Setup components @@ -1174,7 +1117,7 @@ NOTE: Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos not following these practices can significantly impact Gitaly performance and requirements. -Refer to the [Large Repositories](#large-repositories) for more info. +Refer to [Large repositories](index.md#large-repositories) for more information. The recommended cluster setup includes the following components: @@ -1482,14 +1425,14 @@ requirements that are dependent on data and load. NOTE: Increased specs for Gitaly nodes may be required in some circumstances such as -significantly large repositories or if any [additional workloads](#additional-workloads), +significantly large repositories or if any [additional workloads](index.md#additional-workloads), such as [server hooks](../server_hooks.md), have been added. NOTE: Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos not following these practices can significantly impact Gitaly performance and requirements. -Refer to the [Large Repositories](#large-repositories) for more info. +Refer to [Large repositories](index.md#large-repositories) for more information. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -2309,7 +2252,9 @@ services where applicable): - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. -6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large + repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to + [Large repositories](index.md#large-repositories) for more information. NOTE: diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index ae638ce08dc27cb2582132c6a694834864401a05..cb7e46d20ab8fb8392b7c72af034f0ec20fbb3f9 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -122,6 +122,88 @@ classDef default fill:#FCA326 linkStyle default fill:none,stroke:#7759C2 ``` +## Requirements + +Before implementing a reference architecture, refer to the following requirements and guidance. + +### Supported CPUs + +These reference architectures were built and tested on Google Cloud Platform (GCP) using the +[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) +CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). + +Newer, similarly-sized CPUs are supported and may have improved performance as a result. For Omnibus GitLab environments, +ARM-based equivalents are also supported. + +NOTE: +Any "burstable" instance types are not recommended due to inconsistent performance. + +### Supported infrastructure + +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and +their services, or self managed (ESXi) that meet both: + +- The specifications detailed in each reference architecture. +- Any requirements in this section. + +However, this does not constitute a guarantee for every potential permutation. + +See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. + +### Additional workloads + +These reference architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab +setups with good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the +nodes, such as security software, you may still need to adjust the specs accordingly to compensate. + +This also applies for some GitLab features where it's possible to run custom scripts, for example +[server hooks](../server_hooks.md). + +As a general rule, you should have robust monitoring in place to measure the impact of any additional workloads to +inform any changes needed to be made. + +### No swap + +Swap is not recommended in the reference architectures. It's a failsafe that impacts performance greatly. The +reference architectures are designed to have memory headroom to avoid needing swap. + +### Large repositories + +The relevant reference architectures were tested with repositories of varying sizes that follow best practices. + +However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance +of Git and in turn the environment itself if best practices aren't being followed such as not storing binary or blob +files in LFS. + +Repositories are at the core of any environment and the consequences can be wide-ranging when they are not optimized. +Some examples of this impact include: + +- [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) taking longer and consuming high CPU + and memory resources. +- Git checkouts taking longer that affect both users and CI/CD pipelines alike. + +As such, large repositories come with notable cost and typically require more resources to handle, (significantly more +in some cases). You should review large repositories to ensure they maintain good health and reduce their size wherever +possible. + +NOTE: +If best practices aren't followed and large repositories are present on the environment, increased Gitaly specs may be +required to ensure stable performance. + +Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) +for more information and guidance. + +### Praefect PostgreSQL + +[Praefect requires its own database server](../gitaly/praefect.md#postgresql) and +that to achieve full High Availability, a third-party PostgreSQL database solution is required. + +We hope to offer a built in solutions for these restrictions in the future but, in the meantime, a non HA PostgreSQL server +can be set up using Omnibus GitLab, the specifications reflect. Refer to the following issues for more information: + +- [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). +- [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). + ## Recommended cloud providers and services NOTE: