Monday, February 11, 2013

Setting up nGinX with a GSLB/Reverse Proxy on AWS


By no means is this suppose to be a definitive guide to setting up nGinX on AWS as a Global Server Load Balancer (GSLB). Its goal is to serve as a reminder of what I have done to satisfy my requirements. That said, please feel free to comment and/or correct any part, as I am an nGinX novice.


Provide a means to route global traffic accurately based on the source IP of the client. This means nGinX must be able to determine anything from the source IP's country code to the organizations code the IP is originating from. 
Since nGinX does not work at the dns level (aka over port 53) - the magic that happens behind the scenes where traffic is routed and the users is unaware can not take place. (I will go into this in my next blog entry when I discuss setting up Stingray Traffic Manager [STM] as a Global Load Balancer [GLB] on AWS/EC2) 
nGinX must route all requests individually. 

Step by Step

1) Get the source code

    Create a source directory 
    > sudo mkdir /etc/nginx/src

    Go to the newly created source directory
    > cd /etc/nginx/src

    Get the source code
    > wget

    Expand the compressed file
    > gunzip nginx-1.3.10.tar.gz
    > tar xvf nginx-1.3.10.tar

2) Get the global IP data sets

Maxmind provides a number of GeoIP databases at different granularities. Several are supplied at no cost. You can use yum to install a no cost lite GeoIP database. 

  > yum install GeoIP-devel

          The GeoIP database is placed:  /usr/share/GeoIP/GeoIP.dat

If the yum version of the database is outdated, it is possible to get the latest from the maxmind site with the following steps:
    > mv /usr/share/GeoIP/GeoIP.dat /usr/share/GeoIP/GeoIP.dat_bak
    > cd /usr/share/GeoIP/
    > wget
    > gunzip GeoIP.dat.gz

3) Compile

If there is an initial installation of nGinX, the newly compiled version can be based from it. Its also a good opportunity to change the directory structure around so that nGinX is centralized more. I personally find it annoying to hunt up and down the file structure in order to look at logs then changes configs, etc. However, I'm sure this is done for a reason so for a production deployment, I'd suggest leaving it how it is.

Lets start with pulling the current configs from the installed version - manually formatted:

    > nginx -V
      nginx version: nginx/1.3.9
      built by gcc 4.6.2 20111027 (Red Hat 4.6.2-2) (GCC)
      TLS SNI support enabled
      configure arguments: 
        --with-cc-opt='-O2 -g'

From the source directory the was created earlier, run the following command to get the make ready. Notice the alterations to path variables and new modules. These next few commands can take some time to execute. Also, if you encounter errors - please see the Troubleshooting section below to help resolve the issue. 

    > ./configure --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/etc/nginx/logs/error.log --http-log-path=/etc/nginx/logs/access.log --pid-path=/var/run/ --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/etc/nginx/cache/client_temp --http-proxy-temp-path=/etc/nginx/cache/proxy_temp --http-fastcgi-temp-path=/etc/nginx/cache/fastcgi_temp --http-uwsgi-temp-path=/etc/nginx/cache/uwsgi_temp --http-scgi-temp-path=/etc/nginx/cache/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-cc-opt='-O2 -g' --with-http_geoip_module --with-debug
    > make
    > make install
        Done!  ~~> This part!

4) Start & Confirm

Since nGinX is hosts everything on port 80 and has a default homepage - it can be tested by simply starting and pointing to the webserver hosting nGinX.
    > service nginx start

Open a browser and navigate to the nginx server. If you see the homepage, we are good to go. 

5) Add GSLB/Reverse Proxy

First and foremost, you will need to define at least 2 endpoints to distribute requests to and a-record (literally) for the primary domain. Consider the following: --> A-Record --> nginx.ip.address --> CNAME --> --> CNAME -->

Add the following to the nGinX config. It will setup the geoIp database, geo targeting as per country codes [NA-North America & AS-Asia] and a reverse proxy. 

    geoip_country /usr/share/GeoIP/GeoIP.dat;

    if($geoip_city_continent_code=NA){proxy_pass http://ad-1;}
    if($geoip_city_continent_code=AS){proxy_pass http://ad-2;}

    upstream  ad-1 {  server; }
    upstream  ad-2 {  server; }

Yes - that's it!

6) Test

There are other configurations you can setup to make it easier to test. However, for this config, you will need access to a browser in North America and in Asia.

Be sure to clear your browser cache before testing.  


This is an assortment of issues and thier solutions that I ran into while trying to build nGinX. 

Find nginx without all the noise
A lot of null data is printed making it hard to read the page
FIX: > find / -name nginx 2>/dev/null

WinSCP - make it work with sudo
The files you are trying to touch are owned by root and WinSCP won't let you do anything.
FIX: todo

Missing gcc compiler
checking for OS + Linux 3.2.34-55.46.amzn1.x86_64 x86_64checking for C compiler ... not found./configure: error: C compiler gcc is not found
FIX: > yum install gcc

Missing PCRE
./configure: error: the HTTP rewrite module requires the PCRE library.You can either disable the module by using --without-http_rewrite_moduleoption, or install the PCRE library into the system, or build the PCRE librarystatically from the source with nginx by using --with-pcre= option.

Missing make
make is missing---
FIX: > yum install make 

Missing g++
compile error 
FIX: > yum install gcc-c++

PCRE not making after g++ is installed
files have been corrupted after the make and must be cleaned up
FIX: from the /usr/local/src/pcre-8.10 dir 
    > make distclean    

    > ./configure    
    > make    
    > make install

nGinX ./configure: error: the HTTP gzip module requires the zlib library
need to install zlib 
FIX: find out what is avail
    > yum search zlib
    > yum install zlib-devel.x86_64

nGinX ./configure: error: SSL modules require the OpenSSL library
FIX: > yum install openssl-devel.x86_64

PHP Is missing
Fix: > yum install php-fpm.x86_64

Install Aptitude
FIX: Download and Install
      > wget
   > rpm -Uvh rpmforge-release*rpm

How to STOP nGinX (3 options)
      > net service nginx stop
   > /usr/sbin/nginx -s stop
   > kill -QUIT $( cat /usr/local/nginx/logs/ )

No comments:

Post a Comment