I have wanting to install nginx to run sarnworld forum for a very long time now. Previously, there were other sites that were on the same machine so finally the past summer I moved sarnworld to its own machine. Now I finally have done it although with a few bumps along the way which I know alot of members probably were irritated by . I say that because really I don’t need to install anything. The default setting is fine (with apache) because I have alot of resources to throw at the site but as a geek you want to just try to milk more out of your system so I went ahead and did this. Some of the roadblocks that I ran into were because SW was now running on invision forum version 3 which was a bit more picky and had some things which took some work to get it to work on nginx.

These are the road blocks I ran into in order

1.) forum wouldn’t work due to me not including the reflection option in the php compile
2.) nginx had permission problem with the graphics on the forum and the base domain couldn’t load up.
3.) nginx had an issue with the new FURL which uses htaccess so nginx config had to be modified
4.) The wiki also needed a bit of modification on nginx config to work
5.) Ran into a problem in compiling the modules
6.) nginx had 502 errors after loading up the caching modules

First I referenced an old post I made before (SW video site link)

I did the php-fpm install. Edit the php-fpm.conf file and make sure to change the user and group.

Of course on the php compile you should do it your way just make sure you include –enable-fpm and –enable-reflection

./configure --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib --enable-mbstring --disable-pdo --disable-debug --enable-pic --enable-inline-optimization --with-xml --enable-sockets --enable-sysvsem --enable-sysvshm --enable-pcntl --enable-mbregex --with-mhash --enable-xslt --enable-memcache --enable-zip --with-pcre-regex --with-mysql --with-mysql-sock --with-gd --with-jpeg-dir=/usr/lib --enable-gd-native-ttf --without-sqlite --enable-reflection

I then compile nginx in the default folder and modules

Along the way I referenced This Site

But when I ran the pecl command I got this error

ran into the "pecl.php.net is using a unsupported protocal - This should never happen." error

To fix this error I found this link

Found the 2 channels and deleted them

/usr/local/lib/php/.channels/
/usr/share/pear/.channels/

Then I ran

pear update-channels

Now I go onto to install the 3 modules on the other link.

/usr/local/bin/pecl install memcache
 
Build process completed successfully
Installing '/usr/local/lib/php/extensions/no-debug-non-zts-20060613/memcache.so'
install ok: channel://pecl.php.net/memcache-2.2.5
configuration option "php_ini" is not set to php.ini location
You should add "extension=memcache.so" to php.ini

/usr/local/bin/pecl install apc
 
Build process completed successfully
Installing '/usr/local/lib/php/extensions/no-debug-non-zts-20060613/apc.so'
install ok: channel://pecl.php.net/APC-3.0.19
configuration option "php_ini" is not set to php.ini location
You should add "extension=apc.so" to php.ini

I ran into an error on the 3rd module.

/usr/local/bin/pecl install syck-beta
 
checking for syck files in default path... not found
configure: error: Please reinstall the syck distribution
ERROR: `/tmp/pear/temp/syck/configure' failed

Found the solution on this link.

I followed the instructions.

http://rubyforge.org/frs/?group_id=224&release_id=2202

wget http://rubyforge.org/frs/download.php/4492/syck-0.55.tar.gz

untar, ./configure, make, make install.

ran the command again

/usr/local/bin/pecl install syck-beta 
 
configuration option "php_ini" is not set to php.ini location
You should add "extension=syck.so" to php.ini

I then edited php.ini and added all the modules

Now I discovered Nginx had permission issue (403 error) with loading up the graphics on the forum. I posted here and here.

I wished I saw the latter post which gave me the correct because it took me a bit of time to figure it out. It was basically a counter intutitive solution. Initially I thought it was php-fpm which was the issue but it was actually nginx. Basically it was the user line that I had to add.

I also had to add extra stuff to get mediawiki to work with nginx. I used this link as a reference.

Here is my nginx config

Make sure you include

user usertim usergroup;

if the files in the domains are owned by the user and group.

On the top of the nginx.conf file

domains part of the config. Note that I had to include both /discuzz and discuzz/ just in case some people include a / at the end. This setting is with FURL enabled with .htaccess in IPB3.

$INFO['use_friendly_urls'] = '1';

 
server {
  listen 80;
  server_name  www.sarnworld.com;
  root /blah/blah;
  location / {
    index index.php index.html index.htm;
    }
 
# This is your php-fpm section.  Replace it accordingly (For example if you decide to run it on a different port).
  location ~ \.php$ {
    fastcgi_pass   127.0.0.1:9000;
    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include        fastcgi_params;
 
 }
 
  location /discuzz/ {
    index index.php index.htm index.html;
    try_files $uri $uri/ /discuzz/index.php?q=$uri;
   }
 
location /discuzz {
    index index.php index.htm index.html;
    try_files $uri $uri/ /discuzz/index.php?q=$uri;
   }
 
# This is for nginx to deny access to .htaccess file.
  location ~ /\.ht {
            deny  all;
          }
     }
 
server {
  listen 80;
  server_name wiki.sarnworld.com;
  root /blah/blah/blah;
  location / {
    index index.php index.html index.htm;
  }
 
# This is your php-fpm section.  Replace it accordingly (For example if you decide to run it on a different port).
  location ~ \.php$ {
    fastcgi_pass   127.0.0.1:9000;
    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
   include        fastcgi_params;
  }
 
#without this mediawiki wouldn't work
 location ~  ^/index.php/(.+) {
           rewrite ^/index.php/(.+) /index.php?title=$1 last;
       }
 
# This is for nginx to deny access to .htaccess file.
  location ~ /\.ht {
            deny  all;
          }
 
     }

I ran into another issue where if i load the APC extension in php.ini, nginx gives a 502 error and everything dies.

I found this thread but it didn’t give me any clues.

Finally I found this thread which let me know that Zend optimizer was conflicting with APC.  Great clue.  I needed ioncube loaded for some encrypted mod I was using on the forum but I didn’t need zend optimizer so I edited php.ini and commented all lines with zend_extension_ except for the one that loads ioncube.  Restarted php-fpm with apc extension loaded and the site wasn’t throwing the 502 errors anymore.

WOOHOO

Now to install memcache and get it to work you need a dependency (libevent). Found link for a reference.

installed memcache according to that and ran using this command. 

/usr/local/bin/memcached -u root -d -l 127.0.0.1 -p 11000 -m 128

It runs on port 11000 and uses up to 128 megs of ram and localhost

Now I activate APC and Memcache on sarnworld

edited conf_global.php file in IPB to enable the 2 caching softwares. I had to send a ticket in IPB customer service to ask how to do it. It turned out to be pretty simple as the support is built in.

For APC add in.

$INFO['use_apc'] = '1';

for Memcache

$INFO['use_memcache'] = '1';
$INFO['memcache_server_1'] = 'localhost';
$INFO['memcache_port_1'] = '11000';

According to the IPB tech if you have more the 1 memcache server, just duplicate the last 2 lines, increasing the server and port numbers.. server_2, server_3, etc…

Now to get into APC settings in detail I found this link for a reference.

edit php.ini and I added these entries.

[apc]
 
apc.enabled = 1
apc.shm_segments = 1
apc.shm_size = 30
apc.optimization = 0
apc.ttl = 7200
apc.user_ttl = 7200
apc.num_files_hint = 2000
apc.mmap_file_mask = /tmp/apc.XXXXXX

Then make a symlink to the apc.php to a web accessible location to see the statistics.

ln -s /usr/local/lib/php/apc.php /blahblah/apc.php

It seems that I don’t really need memcache so I may just close it off in the future. Most recommendations seem to be just to enable APC and it would be good enough but for now why the heck not. Just run it all since I have all the resources. I want to improve the speed of the forum

On a side note I found this memcache link for mediawiki but I will work on that later

So I’ve been attempting to upgrade the forum (currently 2.3.5) to 3.0.2 but I keep running into problems.

It all started off when I see this error “ALTER TABLE ibf_members CHANGE id member_id MEDIUMINT(8) NOT NULL auto_increment;
Duplicate entry ‘1′ for key 1″

After this my member_id table get deleted and it causes a chain reaction of errors that made it impossible for me to upgrade the forum.

After spending a few hours, I finally found the solution.  First I found an IPB thread that talked about something “funky” on the members table.

Well my brain finally noticed something strange on the members table.  Basically THE FIRST RECORD I had was a 0 with no username (just a blank), this was what was the problem.  I deleted that first record and everything started working.  Incredible that I didn’t see this from the beginning.

DOH!

Why Nginx?

Its hard to be a server/linux/software geek and not know about Nginx. I’ve been meaning to play with Nginx for a while. Thought about installing it on SW VPS but I just never found the time to do it. I did attempt to install nginx with RT 3.8 and that was a total failure on my part and mainly because I couldn’t get RT’s quirky perl to work. I will give that another try since I understand Nginx much better this time around.

Before I start I like to thank Boris from WHT, Nathan of SM Script, and #nginx on freenode IRC. and yawn.it

Although I mention using SM script, you can use this tutorial for pretty any scripts out there that uses php and .htaccess

A large part of this post is compromised of this tutorial I found on the web on nginx. Really well written and modified a little by me for CentOS5.

First, lets start with some packages we needed to compile nginx with no errors. Make sure all the development packages are installed. And of course a compiler (gcc)!

yum install ffmpeg-devel libxml2-devel libXpm-devel libjpeg-devel libpng-devel mysql-devel libtool make patch gcc

Next, we need to compile a custom version of php 5.2.6 patched and integrated with php-fpm and ffmpeg-php (for SM script). CentOS default php binary is located in “/usr/bin”, the compile of the source code defaults in “/usr/local/bin”. The nice thing about this whole process is the fact that you can have both php binaries installed and depending on what you want to turn on (apache or nginx), you can just activate it accordingly.

Download php 5.2.6 and php-fpm. Patch php.

wget http://us.php.net/get/php-5.2.6.tar.gz/from/this/mirror
tar -xvvzf php-5.2.6.tar.gz
wget http://php-fpm.anight.org/downloads/head/php-5.2.6-fpm-0.5.9.diff.gz
gzip -cd php-5.2.6-fpm-0.5.9.diff.gz | patch -d php-5.2.6 -p1

Download ffmpeg-php and build it right into the binary (I prefer it this way).

wget http://downloads.sourceforge.net/ffmpeg-php/ffmpeg-php-0.6.0.tbz2?modtime=1224044751&big_mirror=0
bunzip2 -c ffmpeg-php-0.6.0.tbz2 | tar xvf -
mv ffmpeg-php-0.6.0 php-5.2.6/ext/ffmpeg
cd php-5.2.6
autoconf
./configure --enable-fastcgi --enable-fpm --with-zlib --enable-mbstring --with-mysql --with-mysql-sock --with-gd --with-jpeg-dir=/usr/lib --enable-gd-native-ttf --without-sqlite --disable-pdo --disable-reflection --with-ffmpeg=yes
make all install
strip /usr/local/bin/php-cgi

Start configuring php-fpm.conf
On a side note I copied the sample php.ini from the source folder. This was a mistake as the sample didn’t have short_open_tag enabled. This was to give me a big headache for a while. Best thing is to copy your default CentOS php.ini which also already have ioncube loaded. The default location for your new php binary will be loaded from /usr/local/lib

cp /etc/php.ini /usr/local/lib
vi /usr/local/etc/php-fpm.conf

Find these two lines. Remove the Comments (arrows) and replace the “nobody” with the user and group you would like php-fpm to run as.

 <!--  <value name="user">nobody</value>         -->
<!--  <value name="group">nobody</value>      -->

If you don’t do this, starting php-fpm will show this error.

fpm_unix_conf_wp(), line 124: please specify user and group other than root, pool ‘default’

Next we will compile Nginx (There is an easy way to install using RPM but I’m not sure what options and where it gets put. Link here)

Although the default modules that are activated with nginx is plenty enough, I wanted to enabled two extra modules. One is http_stub_status_module and flv streaming. As of version 2.1 SM Script does not support FLV seeking but the new update for the flash player should give that ability so might as well get ready by activating the module.

You will need pcre library to compile nginx.

wget http://sysoev.ru/nginx/nginx-0.6.32.tar.gz
tar -xvvzf nginx-0.6.32.tar.gz
wget http://downloads.sourceforge.net/pcre/pcre-7.8.tar.gz?modtime=1220617433&big_mirror=0
tar -xvvzf pcre-7.8.tar.gz
cd nginx-0.6.32
./configure --with-http_stub_status_module --with-http_flv_module --with-pcre=/path/to/pcre-7.8/
make
make install

Edit Nginx config file

vi /usr/local/nginx/conf/nginx.conf

Use the following nginx config. This is a basic config that will run nginx with SM Script. (NOTE: Config isn’t optimized for best performance with script). Replace “root /var/www/html” with your path for SM script. Click here to get all the converted rewrites for nginx.

worker_processes  1;
events {
    worker_connections  1024;
}
 
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    # the 300m represent 300megs.  This is the max upload.  Change it if you need.
    client_max_body_size 300m;
 
server {
  listen 80;
# replace domainname.com with your domain name.  I discover you need to do this for encoded files.
  server_name  domainname.com;
# Replace /var/www/html with your path to SM script.
  root /var/www/html;
  location / {
    index index.php index.html;
  }
 
# This is your php-fpm section.  Replace it accordingly (For example if you decide to run it on a different port).
  location ~ \.php$ {
    fastcgi_pass   127.0.0.1:9000;
    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include        fastcgi_params;
  }
 
# This enable flv streaming allowing fast forwarding.  Next version of SM should support it.
location ~ \.php {
            flv;
        }
 
# This is for nginx to deny access to .htaccess file on your SM script root.
# Leave it there so you can easily turn on apache if the need arise.
  location ~ /\.ht {
            deny  all;
        }
 
# Copy all the converted SM .htaccess rules here.  Yes all of it needs to go in there.
rewrite ^/signup/$ /index.php?view=signup last;
....
....
....
 
       }
 
}

On a side note: Here is how the apache rewrite is converted to nginx
Apache: RewriteRule ^signup/$ index.php?view=signup [L]
Nginx: rewrite ^/signup/$ /index.php?view=signup last;

Replace it with all lines following this rule and it will work.

start nginx and php-fpm

php-fpm start
/usr/local/nginx/sbin/nginx

Since I can’t find any init script I do the lazy route of starting up the services by putting the commands in rc.local These commands will automatically run next time the VPS is restarted.

edit the file and put the 2 start commands in the file.

vi /etc/rc.local

php-fpm start
/usr/local/nginx/sbin/nginx

Now if you decide to stop nginx and start apache for whatever reason all you have to do is.

php-fpm stop
kill `cat /usr/local/nginx/logs/nginx.pid`
/etc/init.d/httpd start

Some Notes:

You may see this error when you browse the site after successfully starting nginx.
“No input file specified”
This error pops up if your fastcgi_param isn’t set correctly.

Part 2 is done!

Continue onto Part 3 which will involve setting up a Remote Backup and MySQL replication.

I installed OpenVZ on the new video server and then created a 850 gig CentOS5 VPS.

So why did I decide to go with the VPS route when I could just use the dedicated server itself? Here are the advantages I see when you running as an OpenVZ VPS in this case.

• Might have some issues when you ask techs to format your server in a specific way for Xen.  This is especially true if you are planning to use hypervm.  And also this might cost money.

• Overhead isn’t so bad with virtualization; especially with OpenVZ.

• I could run another independent VPS for some tiny experiment if I wanted to.

• If I screw up on something, it is painless to recreate the VPS and start fresh. Even if you have free OS reloads, it is still a hassle.

• On the worse case scenario where the VPS is compromised badly, you can quickly restore everything.  For example you can move your hundreds of gigs of videos to the main platform easily with a command and then destroy and recreate your VPS.  On Xen this would be impossible if your VPS space is almost full.

So lets continue on with the rest of the basic settings.

Installed Jason’s ever awesome repo. Get the near latest versions of the softwares.

Did the good ole basic commands to get things rolling.

yum -y update
yum install php php-mysql php-gd mysql mysql-server httpd vsftpd

Setup some really anal firewall rules.

Installed the SM script.  Make sure you enable .htaccess in httpd.conf

Started on the ffmpeg, mencoder and flv2 for the script

I just have to make a point that I used to compile all these by hand. Sure it was geeky but I have to say, using rpmforge really saves time. Sure I don’t have the latest versions but its SOOO easy!

Here’s a MUCH EASIER way.

Installed the rpmforge repo rpm

wget http://dag.wieers.com/rpm/packages/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.i386.rpm
rpm -ivh rpmforge-release-0.3.6-1.el5.rf.i386.rpm
yum install ffmpeg mencoder flvtool2

Patiently wait for the 73 packages to download.

Continue to Part 2 (installing Nginx webserver and other things).

So I rented a server for our new video server. My plan was to just prepare and to release it after at least a week or two.

BUT after a few days, I was checking the bandwidth meter and I saw my bandwidth going through the roof (100gig+). I went and downloaded the bw monitor and saw that I was at that moment pumping out about 27 mbits. I ran top and saw a few really bad things. First I saw utorrent running; second I saw a vnc server running; third I saw the user that was running and it was a shell user that I didn’t create.

Initially I was shocked because it made no sense other then the fact that I got rooted. Basically I got owned and this was definitely something that never happened. I’ve had situations where I got either SQL injected or have a run away script due to carelessness but NEVER one where I had practically zero services running other then SSH AND I had a rogue shell user running services that only a root user could run.

First thing I did was to kill the rogue services that were running. Then I set some very anal firewall rules. Then I started investigating. Things got interesting real quick.

After spending quite sometime into it, I was now pretty sure my server was compromised before I even got access to it.  At least before I even SSH in.

Here are some excerpts from the history and secure logs that shows what happened with some of my comments.

First we start off with the History log.

1  yum install yum-fastestmirror -y
2  yum update -y   ---- (Technician runs yum update)
3  cd /etc/
4  cd sysconfig
5  ls
6  cd network-scripts/
7  ls
8  cp ifcfg-eth0 ifcfg-eth0:0    ---- (Technician gives me another IP)
9  vi ifcfg-eth0:0
10  reboot   ----  (Disconnected)
11  setup    ----  (The same person or another person? comes in)
12  passwd    ----  (root pass word changed? is this the root pass I was emailed?)
13  uname -a
14  cd /home
15  ls
16  top
17  df -h
18  tp
19  top
20  df -h
21  uptime
22  ls
23  mkdir /var/spool/sql  ---- (same user now creating the hacked account)
24  cd /var/spool/sql   ----  (this is the home folder of the account that was using my server)
25  adduser sqlserver   ----  (person adds in the user)
26  passwd sqlserver    ----  (made a password)
27  nano /etc/passwd    ----  (changed his login path; promoted himself to root UID:0)
28  cd /var/log    ----  (to the logs)
29  nano secure   ----  (either edited or changed some stuff)
30  id sqlserver
31  exit      ----  (he leaves)
32  ls
33  exit
34  ls       ----  (This is me connected)
35  ifconfig
36  ifconfig
37  cat /proc/cpuinfo    ----  (me making sure its quadcore)
38  free -m
39  top -c
40  df -h          ----  (me checking a bunch of stuff)
41  lvdisplay
42  uname -a
43  w
44  passwd     ----  (me changing the root pass)
45  ls
46  w
47  ls
48  vi /etc/selinux/config    ----   (me making sure selinux was disabled)
49  pwd
50  wget http://download.lxlabs.com/download/hypervm/production/hypervm-install-slave.sh   ----    (me installing openVZ)

Now here are the entries from the Secure log in /var/log

 
Oct  2 19:23:58 server700 login: pam_unix(login:session): session opened for user root by LOGIN(uid=0)
Oct  2 19:23:58 server700 login: ROOT LOGIN ON tty1

This is the first entry in the log (console login)  Probably the technician running yum.

Oct  2 19:32:00 server700 sshd[2565]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Oct  2 19:34:13 server700 login: pam_unix(login:session): session opened for user root by LOGIN(uid=0)
Oct  2 19:34:13 server700 login: ROOT LOGIN ON tty1

Here is the 2nd time a console login was done and I’m assuming this login was the one that added the sqlserver user which was running utorrent on my box and sucking up my BW.

Oct  2 20:24:48 server700 sshd[3081]: pam_unix(sshd:session): session closed for user root
Oct  2 21:11:41 server700 sshd[3941]: Accepted password for root from 100.100.100.100 port 63266 ssh2
Oct  2 21:11:41 server700 sshd[3941]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct  2 21:11:49 server700 passwd: pam_unix(passwd:chauthtok): password changed for root

Here is ME logging in and changing my root password.

Oct  2 21:19:39 server700 sshd[2793]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct  2 21:20:17 server700 groupadd[2867]: new group: name=lxlabs, GID=501
Oct  2 21:20:17 server700 useradd[2871]: new user: name=lxlabs, UID=500, GID=501, home=/home/lxlabs, shell=/sbin/nologin

Here is me logging back in and installing hypervm on my box.

Oct  2 21:36:04 server700 sshd[6628]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Oct  2 21:47:06 server700 sshd[8321]: Accepted password for sqlserver from 200.200.200.200 port 10170 ssh2
Oct  2 21:47:06 server700 sshd[8321]: pam_unix(sshd:session): session opened for user sqlserver by (uid=0)

You can see that this account logs in with ROOT Access.  And if you remember in the history above, this person probably changed UID to 0 (by editing /etc/passwd)).

This shows that whoever compromised the server did it from the console. Whether it is from someone inside or someone who gained kvm/ip access (which is pretty scary) is something I don’t know for sure.

Surprise that the person left alot of footprints behind but I’m glad it happened this way so I was able to at least proved my case.

I send the information to the server’s company and complained about the incident. Thankfully they decided to compensate me which ended up being about a month of free hosting and of course a free OS reload.

Good thing that the site was in the developmental stage and wasn’t even ready for prime time. So it wasn’t a terrible loss other then me wasting time putting a few hours into it. Sure it was easy but I would have to do alot of those recompiles again.

So I decided to install some stuff to make the forum runs faster.

First was the SpeedUp mod.

This was really easy and you instantly notice the speed increases on the forum.

Now the next thing was to install Xcache.  On the older server, I’ve always had Eaccelerator  and it worked fine.  For some reason I never thought about installing one of these on the new server.  Probably because it was a quadcore and I just didn’t feel like I needed to do it which is obviously the wrong attitude.

Looking for some basic instructions I found two.

http://www.php.ph/2008/08/10/centos-5-installing-xcache-from-source/

http://www.jasonlitka.com/2006/12/20/php-caching-and-acceleration-with-xcache/

Initially I did the compile myself and everything looked good til I restarted apache.  Apache wasn’t starting up at all.  I tried it again and it still didn’t work.  Then I got lazy and decided to use the ever excellent Jason’s repo (Jason keep up the good work).

Ran yum install php-xcache

Restarted and promptly killed apache as it can’t start.  Just not getting any love tonight!

As I suspected this was due to lxadmin and since my work had support with them, I promptly got on live support.

The tech instantly knew my problem and told me to run

/script/upcp

after the yum install.  Mentioned to me that xcache was unstable.

Well after the command xcache.ini was gone from /etc/php.d/

Apparently, it got moved inside /etc/php.ini

I wasn’t sure where we were going with this so I asked the tech guy about it.

He told me to log into the LXADMIN panel and look at the php config.  Well turns out now xcache was now an option that I can enable.  So I checked the option then restarted apache and checked with phpinfo().

Well Xcache is finally listed in there.  The forum didn’t feel faster though; not like with the SpeedUP mod but I will be monitoring the load and see if it decreases as the load actually climbs a little bit during some busy time (load up to 4 or 5).

lxadmin panel is coming along nicely and I really like the support.  Especially love the way they patch things super quick.

Still prefer good ole shell though