Drupal scaling and performance tuning - Part 3

Running a Developer Portal is not an easy task as I thought. When it comes to WSO2 Oxygentank It is more critical as WSO2 community is heavily depend on the portal.

To be frankly if the site went offline for 1 minute, I used to get more than 20 email /  IM chats or calls. Most of the solutions described in the web really didn't work for us. So we had to find our own way to stabilize the system. After spending so much of time configuring and tuning we ended up with a scalable and stable solution. In part 1 and 2 I have described methods which we used.

During peak hours Our portal started to send 502 Bad gateway message from Nginx. The reason was due to mysql high-load in master 1 server, Nginx didn't receive the response with in defined time. 

Solution was we configured Nginx with backup nodes setting instead of setting up 4 nodes with weight balancing to handle the load.

Sample nginx.conf

upstream wso2.org {
    server node1server:80 weight=5 fail_timeout=20s;
    server node2server.org:80 weight=5 fail_timeout=20s;
    server node3server:80 backup;
    server node4server:80 backup;
  }

with this setup 
  1. Nginx always send traffic to node1 and node2 servers
  2. Primary data source is master1 mysql instance.
  3. Secondary data source is master2 MySQL instance
When node1 or node2 couldn't response to Nginx with in 20 seconds we route the traffic to backup nodes
  1. Nginx send traffic to node3 and node4 when node1 and node2 can not response 
  2. Primary data source is master2 MySQL Instance
  3. Secondary data source is master1 MySQL instance
with that setting we were able to keep the site with 0% downtime.


(Overview of WSO2 Oxygentank)

Nginx always looked after load balancing when http/s traffic get high and with the backup node method we were able to keep our MySQL instance up and running without getting meltdown.   
comments powered by Disqus