Portal Home > Knowledgebase > Articles Database > How to reduce OpenVZ memory usage by >50%
How to reduce OpenVZ memory usage by >50%
| Posted by cieslakd, 04-18-2009, 02:29 PM |
| Few months ago I bought new small VPS box (OpenVZ, 128 MB RAM) in order to place there a new monitoring node of my site monitoring system. Such small amount of RAM is a challenge for operating system optimisation techniques (OpenVZ doesnt have swap as Xen does).
First of all I discovered that apache2-mpm-worker (Apache implementation that uses threads) consumes more memory (100MB) than the classic version that use separate processes (20MB). I had to switch to apache2-mpm-prefork version then.
Next unpleasant suprise: small Python app eats 100MB of virtual memory! I checked that virtual (not resident) memory is taken into account by VPS. I applied some tools to locate memory bottleneck, but without success. Next I added logs with current memory usage to track call that causes big memory consumption. I tracked the following line:
is guilty for high memory increase. After few minutes of googling I located problem: default stack size for a thread. Details:This line creates few threads to handle concurrent callsStack size is counted towards virtual memoryDefault stack size is very high on Linux (8MB)Every thread uses separate stack=> multi threaded application will use at least number_of_threads * 8MB virtual memory!
First solution: use limits.conf file. I altered /etc/security/limits.conf file and changed default stack size. But I couldnt make this change to alter Python scripts called from Apache (any suggestions why?).
Second (working) solution: lower default stack size using ulimit. For processes launched from Apache I altered /etc/init.d/apache2 script and added:
Now every thread (in apache / Python application) will use only 128 kB of virtual memory (I lowered VSZ from 70 MB to 17 MB this way). Now I have additional space to enlarge MySQL buffers to make DB operations faster.
Theres even better place to inject ulimit system-wide: you can insert this call in:
script. Then ulimit will be applied to all daemons (as Apache) and all login sessions. I reduced virtual memory usage by 50% this way.
Note: you may increase stack size on stack overflow errors. In my opinion 256 kb is safe option for most systems, you may increase if in doubt. Still memory savings are biig :-)
|
| Posted by RBBOT, 04-18-2009, 04:11 PM |
| The mpm version of apache may use more memory when idle, but when under heavy load the prefork version will fork more threads increasing its memory usage above that of the mpm. If you want to reduce it further consider lighttpd or nginx.
|
| Posted by cieslakd, 04-18-2009, 04:24 PM |
| Both (lighttpd or nginx) are multithreaded, so the problem (big stack per thread) remains. If you decrease stack size any solution will have lower memory requirements.
|
| Posted by mwatkins, 04-18-2009, 09:12 PM |
| Another mem space saving area to look for is rsyslogd - if you are running Linux and by default it is using rsyslogd, it, at least on Debian 5, is a RAM hog. I reduced it's stack size considerably. About 7 weeks ago I posted something on this, perhaps that's gone now. Short and sweet: ulimit -s 128 in the init script for rsyslog.
On multi-threading... do you have other choices?
Part of our business involves hosting a lot of Python apps - custom apps for clients. Each runs as the client user; we front end them in most cases with lighttpd using either proxy (not often) or SCGI (more often). We aren't opposed to native python web servers being exposed to the internet either but most have some static content that lighttpd is good at dealing with using few cycles or RAM.
One thing that is different between our case and yours is that none of our Python apps are multithreaded. Lighttpd and Apache might be, but the Python app is always external in our case (no mod_python here, nor mod_wsgi but that is for other reasons). The app framework uses a forking server model which is simple and quite efficient.
I can pump through 624 serial requests a second - that's a single client. Using "ab" for a quick and dirty benchmark (run on an OpenVS VPS, a half decent 4 core machine):
1 -> 624 req/s
2 -> 1220
3 -> 1492
4 -> 1556
20 -> 1431
50 -> 1426
I let the app turn up a default max 5 worker processes; generally I find that most apps I deploy for clients between 2 and 3 are enough. Here's that series run with maximum children of 2:
1 -> 658 req/s
2 -> 1598
3 -> 1277
4 -> 1276
20 -> 1191
50 -> 967
You can see it still does the job (try that with PHP). No failures, just degraded (but perhaps perfectly acceptable) performance.
This simple app (a very simple wiki) consumes ~ 10MB VSZ, 8MB RSZ, 2.5MB SZ. I have some production apps which trend around 25MB VSZ per child and of course there are exceptions out side of this.
What I like about the multi process model is I don't have to concern myself about threads, I still get performance and scalability... while retaining simplicity. If a child dies, another replaces it.
|
| Posted by wump, 10-08-2010, 06:53 AM |
| This was really useful for lowering my enormous Python WSGI process from 300+ MB to 100+ MB
Thanks!
|
Add to Favourites 
Print this Article
Also Read
