Drupal scaling and performance tuning - Part 3
08 Dec 2011Running a Developer Portal is not an easy task as I thought.
When it comes to WSO2 Oxygentank It is more critical as WSO2 community is
heavily depend on the portal.
To be frankly if the site went offline for 1 minute, I used to get more than 20 email / IM chats or
calls. Most of the solutions described in the web really didn't work for us. So we had to find our own way
to stabilize the system. After spending so much of time configuring and tuning we ended up with a scalable and
stable solution. In part 1 and 2 I have
described methods which we used.
During peak hours Our portal started to send 502 Bad gateway message from Nginx. The reason
was due to mysql high-load in master 1 server, Nginx didn't receive the response with in
defined time.
Solution was we configured Nginx with backup nodes setting instead of setting up 4 nodes with weight
balancing to handle the load.
Sample nginx.conf
upstream wso2.org {
server node1server:80 weight=5 fail_timeout=20s;
server node2server.org:80 weight=5 fail_timeout=20s;
server node3server:80 backup;
server node4server:80 backup;
}
with this setup
- Nginx always send traffic to node1 and node2 servers
- Primary data source is master1 mysql instance.
- Secondary data source is master2 MySQL instance
- Nginx send traffic to node3 and node4 when node1 and node2 can not response
- Primary data source is master2 MySQL Instance
- Secondary data source is master1 MySQL instance
with that setting we were able to keep the site with 0% downtime.
(Overview of WSO2 Oxygentank)
Nginx always looked after load balancing when http/s traffic get high and with the backup node
method we were able to keep our MySQL instance up and running without getting meltdown.