How to optimize CloudLinux OS limits

Editor's Notes

• #3: LVE is a kernel level technology developed by the CloudLinux team. The technology has common roots with container based virtualization and uses cgroups in its latest incarnation. It is lightweight and transparent. The goal of LVE is to make sure that no single web site can bring down your web server as today, a single site can consume all CPU, IO, Memory resources or Apache processes. LVE prevents that. It is done via collaboration of Apache module, PAM module and kernel.   mod_hostinglimits is Apache module that:   •detects VirtualHost from which the request came; •detects if it was meant for CGI or PHP script; •puts Apache process used to serve that request into LVE for the user determined via SuexecUserGroup directive for that virtual host; •lets Apache to serve the request; •removes Apache process from user's LVE. The kernel makes sure that all LVEs get fair share of the server's resources, and that no customer can use more then the limits set for that customer. Today we can limit CPU, Memory (virtual and physical), IO, number of processes as well as the number of entry processes (concurrent connections to apache).
• #4: LVE limits do not depends on the power of your server they only depend on how fast you want your hosting accounts to be. Default limits are suitable for 99 percent of shared hosting servers. With more powerful hardware you can host more clients per server. On the right side you can see common limits for low end customers and high end customers, where main difference is increased twice CPUspeed, physical memory limits and entry processes limit.
• #5: As of now with CloudLinux features you can limit PHP and CGI scripts, processes launched over ssh session, cron jobs, email pipes, mysql queries (using MySQL governor). Static content is not limited by default however this is controlled with AllowHandler in modhostinglimits config file, if changed to wildcard character we can limit all requests.
• #6: Default limits are followig: CPUSpeed 100 percent, physical memory 256Mb , virtual memory 0, input-output 1Mb per second, IOPS one thousand twenty four, Number of processes 100 and entry processes twenty. We strongly recommend disabling VMEM limits as physical memory is a much better and accurate way to limit memory for shared hosting.
• #7: The very first purpose of CloudLinux is to limit resources that could be taken by websites. Let compare LVE with apache Rlimits. CPU with RLIMIT you set it per apache process. That is an amount of seconds each process will execute. After that process will be killed. It does not limit amount of CPU that can be used at the same time by account. LVE CPU is for whole user account. Is the total amount of CPU (in fractions of the core) all processes within account can use at the same time. If processes try to use more, they will be throttled. Memory with RLIMIT is limiting memory consumption per process. If you have one hundred processes they can take one hundred multiplied by limit. LVE Memory limit is the total memory consumption for all processes within accounts. Total memory that account can consumed doesn’t depend on number of processes. InputOutput and IOPS limits are not possible with apache Rlimits, input-output limit is amount of data per second that can be read/written by all processes in account. While IOPS is number of input-output operations allowed per second. Number of processes is same with RLIMIT and LVE – that is a maximum number of processes allowed per account. However LVE counts all processes, not only launched by apache. Entry processes limit is not available in apache, while with LVE it means maximum number of apache connections for account.
• #8: CPU SPEED limit allows to set CPU limit in terms of % of a single core, or as a fixed number of Hz. Setting speed to 100% means LVE could use one full core. Setting it to one hundred fifteen means 1 and a half core. Speed in herz would automatically detect CPU speed of each core, and adjust the CPU scheduler to make sure user cannot go over that limit. For example on 1ghz CPU, setting of --speed=2ghz would mean 2 cores, while on 4ghz CPU same setting would mean 1/2 of a core. Once limit reached processes are interrupted so slow down. Hyper threading counts as a separate core.
• #9: In linux system one thread can occupy one core, if limit is set to less then one core this will cause automatic throttling of all requests. It’s quite dangerous to set speed limit less then one core as in this case system CPU usage will be higher due to context switching needed to fit the limit. You will notice %sys overhead in top output. More accounts with less then 1 core limit will cause higher load average. Half core means requests are two time slower but increasing .
• #10: Limits should be not more then a half of all cores. Much better if you keep one quarter of all cores as maximum speed limit for high-end accounts. If limits are higher then a half of all cores you need just two accounts to overload server.
• #11: IO limits restrict the data throughput for the customer. They are in KB/s. When limit is reached, the processes are throttled (put to sleep). This makes sure that processes within LVE cannot go over the limit,. Yet don't stop working, nor getting killed - they just work slower when the limit is reached. The IO limits will only affect DISK IO, and will have no effect on network. It also doesn't take into consideration any disk cache accesses. So, even if file is loaded from disk cache 1000 times -- it will not be counted towards IO limits.
• #12: SSD provides better throughput, if HDD average speed is 30-100 MB/s then SSH could give 80-500 MB/s . Limits are still set per account. For Hard Disk Drives you may leave default 1MB/second. While for Solid State Drive 4MB/second could be used. For high-end accounts you may want to use 5MB/s with HDD and 10MB/s with SSD.
• #13: IOPS limits restrict the total number operations per second. Disk operations are not only read and write, they include open, close, seek, dir.. etcetra. When the limit is reached the operations stops until current second expires. Default is set to one thousand and twenty four operations per second.
• #14: SSD provides significantly higher IOPS rate. With HDD normal is from 100 to 400 iops
• #15: Memory could be controlled by virtual and physical memory limits. Virtual memory limit corresponds to the amount of memory that processes can allocate within LVE. You can see individual process virtual memory usage by monitoring VIRT column in top output for the process. When process tries to allocate more memory then allowed with vmem limit kernel will not allow doing this and in most cases will cause process to fail. Physical memory limit corresponds to the amount of memory actually used by end customer's processes. You can see individual process physical memory usage by monitoring RES column in top output for the process. Because similar processes (like PHP) share a lot of their memory, physical memory usage is often much lower then virtual memory usage.
• #16: Physical memory default limit is set to 256MB. You may check physical memory usage in RSS field in ‘ps’ output or in RES column in ‘top’ output. Additionally physical memory includes shared memory used by the customer, as well as disk cache. When LVE goes over physical memory limit, CloudLinux will first free up memory used for disk cache, and if that is not enough, it will kill some of the processes within that LVE. This usually cause web server to show five-zero-three error page. Physical memory limit is much better way to limit memory for shared hosting. We strongly recommend setting VMEM to zero and use only Physical memory limit.
• #17: NPROC controls the total number of processes within LVE. Once limit reached no new process can be created, until another dies. Default limit is one hundred. The limit protects against fork bombs and similar attacks. Most users will never hit this limit under normal circumstances. For high-end users you may set it two hundred, however it is safe to increase even to one thousand.
• #18: Entry processes limit control the number of entries into LVE. It is also know as 'Apache concurrent connections' limit as the process enters into LVE when there is a new HTTP request for CGI/PHP. Each time a process 'enters' into LVE, we increment the counter. Each time process exits LVE, we decrement the counter. We don't count processes that are created inside LVE itself. Default limit is twenty. Once limit reached no new processes could enter LVE and will cause web server to show error 508 page (Resource Limit Reached). The limit was created to prevent DoS attacks against web server when attacker tries to reach MaxClients created large amount of slow requests. If MaxClients reached apache will not respond to new connections and from outside it appears to be down. The issue is worsened by CPU limits as once site starts to get slow due to CPU limit – it will respond to requests slower and slower, causing more and more connections to be tied up. With entry processes limit such attack will fail as one site will reach his EP limit while other will keep working.