Install Varnish, APC and Memcache with Drupal and Pressflow 7

[html format="ckeditor" different_values="0"]

In this tutorial I have used Centos 5.4 , Apache, PHP 5.3 and Mysql 5.5 DB Server.

This Installation will work as shown in the following diagram.

Install Plain Centos/RHEL on a Server. We need additional Repositories to install required packages and its dependencies.

First I will install EPEL and remi repo. These Repo will be used to install PHP 5.3 and MySQL 5.5.

rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-5.rpm

Installation.

1. Apache web server.

# yum install httpd -y

2. PHP and APC.

We will be installing PHP 5.3 and packages required to run Drupal with APC.

# yum install php53 –enablerepo=remi

# yum install php-gd php-pdo php-xml php-mbstring php-pear php-ldap php-pecl-apc php-devel php-mcrypt php-mysql php-pecl-memcache --enablerepo=remi

3. MySQL

# yum install mysql mysqlserver –enablerepo=remi

4. Varnish

# rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish-rel…

# yum install varnish

5. Memcached

# rpm -Uhv http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3…

# yum install memcached libmemcached libmemcached-devel

This will install Apache web server , Mysql server, Varnish, APC, Memcache along with all dependecies.

Download Drupal/Pressflow and keep it in /var/www/html.

Configurtion.

1. Apache

Start Server with the following command

# server httpd start

Enable Apahce to start on boot.

# chkconfig httpd on

we will configure Apache to run on port 8080 instead of default Port 80 as port 80 will be used by varnish.

# vim /etc/httpd/conf/httpd.conf

search for Listen 80 and replace it with Listen 8080. Save and exit from file.

Restart Apache Web server.

# service httpd restart.

To verify Apache web server is working open browser and give following url.

http://localhost:8080

This will open Apache Default page.

2. MySQL.

Start Mysql Server.

#service mysqld start.

Configure Mysql to start on boot.

#chkconfig mysqld on

Set Password for admin

#mysqladmin -uroot password “password”

Create DB requried by Drupal

#mysqladmin -uroot -ppassword create drupaldb.

3 Varnish

Start Varnish Server.

#service varnish start.

Configure Varnish to start on boot.

#chkconfig varnish on.

Create Varnish rule file

vim /etc/varnish/pressflow.vcl

# This is a basic VCL configuration file for varnish.  See the vcl(7)
# man page for details on VCL syntax and semantics.
#
 
# TODO: Update internal subnet ACL and security.
 
# Define the internal network subnet.
# These are used below to allow internal access to certain files while not
# allowing access from the public internet.
# acl internal {
#  "192.10.0.0"/24;
# }
 
# Default backend definition.  Set this to point to your content
# server.
#
backend default {
  .host = "127.0.0.1"; (IP Address of Apache Server)
  .port = "80";
}
 
# Respond to incoming requests.
sub vcl_recv {
  # Use anonymous, cached pages if all backends are down.
  if (!req.backend.healthy) {
    unset req.http.Cookie;
  }
 
  # Allow the backend to serve up stale content if it is responding slowly.
  set req.grace = 6h;
 
  # Do not cache these paths.
  if (req.url ~ "^/status\.php$" ||
      req.url ~ "^/update\.php$" ||
      req.url ~ "^/admin$" ||
      req.url ~ "^/admin/.*$" ||
      req.url ~ "^/flag/.*$" ||
      req.url ~ "^.*/ajax/.*$" ||
      req.url ~ "^.*/ahah/.*$") {
       return (pass);
  }
if (req.http.Accept-Encoding) {
  if (req.http.Accept-Encoding ~ "gzip") {
    # If the browser supports it, we'll use gzip.
    set req.http.Accept-Encoding = "gzip";
  }
  else if (req.http.Accept-Encoding ~ "deflate") {
    # Next, try deflate if it is supported.
    set req.http.Accept-Encoding = "deflate";
  }
  else {
    # Unknown algorithm. Remove it and send unencoded.
    unset req.http.Accept-Encoding;
  }
}
  # Do not allow outside access to cron.php or install.php.
  #if (req.url ~ "^/(cron|install)\.php$" && !client.ip ~ internal) {
    # Have Varnish throw the error directly.
  #  error 404 "Page not found.";
    # Use a custom error page that you've defined in Drupal at the path "404".
    # set req.url = "/404";
  #}
 
  # Always cache the following file types for all users. This list of extensions
  # appears twice, once here and again in vcl_fetch so make sure you edit both
  # and keep them equal.
  if (req.url ~ "(?i)\.(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)(\?.*)?$") {
#if (req.url ~ "^/themes/" && req.url ~ "\.(css|js|png|jpg|gif|jp(e?)g)") {
#    unset req.http.Cookie;
    return (lookup);
  }
#return (lookup); 
# Remove all cookies that Drupal doesn't need to know about. We explicitly
  # list the ones that Drupal does need, the SESS and NO_CACHE. If, after
  # running this code we find that either of these two cookies remains, we
  # will pass as the page cannot be cached.
  if (req.http.Cookie) {
    # 1. Append a semi-colon to the front of the cookie string.
    # 2. Remove all spaces that appear after semi-colons.
    # 3. Match the cookies we want to keep, adding the space we removed
    #    previously back. (\1) is first matching group in the regsuball.
    # 4. Remove all other cookies, identifying them by the fact that they have
    #    no space after the preceding semi-colon.
    # 5. Remove all spaces and semi-colons from the beginning and end of the
    #    cookie string.
    set req.http.Cookie = ";" + req.http.Cookie;
    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");  
    set req.http.Cookie = regsuball(req.http.Cookie, ";(SESS[a-z0-9]+|SSESS[a-z0-9]+|NO_CACHE)=", "; \1=");
    set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ] [^;]*", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
 
    if (req.http.Cookie == "") {
      # If there are no remaining cookies, remove the cookie header. If there
      # aren't any cookie headers, Varnish's default behavior will be to cache
      # the page.
      unset req.http.Cookie;
    }
   if (req.http.Cookie ~ "(VARNISH|DRUPAL_UID|LOGGED_IN)") {
     return (lookup);
   }
    else {
      # If there is any cookies left (a session or NO_CACHE cookie), do not
      # cache the page. Pass it on to Apache directly.
      return (pass);
#      return (lookup);
    }
  }
}
 
# Set a header to track a cache HIT/MISS.
sub vcl_deliver {
  if (obj.hits > 0) {
    set resp.http.X-Varnish-Cache = "HIT";
  }
  else {
    set resp.http.X-Varnish-Cache = "MISS";
  }
}
 
# Code determining what to do when serving items from the Apache servers.
# beresp == Back-end response from the web server.
sub vcl_fetch {
  # We need this to cache 404s, 301s, 500s. Otherwise, depending on backend but
  # definitely in Drupal's case these responses are not cacheable by default.
  if (beresp.status == 404 || beresp.status == 301 || beresp.status == 500) {
    set beresp.ttl = 10m;
  }
 
  # Don't allow static files to set cookies.
  # (?i) denotes case insensitive in PCRE (perl compatible regular expressions).
  # This list of extensions appears twice, once here and again in vcl_recv so
  # make sure you edit both and keep them equal.
  if (req.url ~ "(?i)\.(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)(\?.*)?$") {
    unset beresp.http.set-cookie;
#    return (lookup);
  }
 set beresp.ttl = 1800s;
  # Allow items to be stale if needed.
  set beresp.grace = 6h;
}
 
# In the event of an error, show friendlier messages.
sub vcl_error {
  # Redirect to some other URL in the case of a homepage failure.
  #if (req.url ~ "^/?$") {
  #  set obj.status = 302;
  #  set obj.http.Location = "______________";
  #}
 
  # Otherwise redirect to the homepage, which will likely be in the cache.
  set obj.http.Content-Type = "text/html; charset=utf-8";
  synthetic {"
<html>
<head>
  <title>Page Unavailable</title>
  <style>
    body { background: #303030; text-align: center; color: white; }
    #page { border: 1px solid #CCC; width: 500px; margin: 100px auto 0; padding: 30px; background: #323232; }
    a, a:link, a:visited { color: #CCC; }
    .error { color: #222; }
  </style>
</head>
<body onload="setTimeout(function() { window.location = '/' }, 5000)">
  <div id="page">
    <h1 class="title">Page Unavailable</h1>
    <p>The page you requested is temporarily unavailable.</p>
    <p>We're redirecting you to the <a href="/">homepage</a> in 5 seconds.</p>
    <div class="error">(Error "} + obj.status + " " + obj.response + {")</div>
  </div>
</body>
</html>
"};
  return (deliver);
}

Install Varnish Module for Drupal. Download it from http://drupal.org/project/varnish

Login in Drupal as Admin.

Click on Modules --> Install New Module --> Browse

select the downloaded Module

Press Install to Install this Module.

Progress Bar indicating Modules Instalaltions confirms when Module is installed

On Same page Click on Enable Newly added Module.

Select Varnish and click save configuration to enable Module.

Configure VarnishModule.

Select Configuration --> varnish . Fill the form with required details. Varnish control Key is in  /etc/varnish/secret file of varnish server.

Edit /etc/sysconfig/varnish to configure varnish

DAEMON_OPTS="-a :80\ #(varnish server will run on this port)
                -T localhost:6082 \ #(varnish mgmt port)
                                 -f /etc/varnish/drupal.vcl \ #(varnish rule file)
            -u varnish -g varnish \ #(varnish user and group )
            -s file,/var/lib/varnish/varnish_storage.bin,1G" #(varnish bin file path)
 Add Following lines in settings.php
################### Varnish and Cache Settings ###########
$conf['cache_class_cache_page'] = 'VarnishCache';
//Cache full pages to Varnish
$conf['cache_backends'] [] = 'sites/all/modules/contrib/varnish/varnish.cache.inc';
$conf['page_cache_invoke_hooks'] = False;
$conf['reverse_proxy'] = True;
$conf['cache'] = 1;
$conf['cache_lifetime'] = 1;
$conf['page_cache_maximum_age'] = 10800;
$conf['reverse_proxy_header'] = 'HTTP_X_FORWARDED_FOR';
$conf['reverse_proxy_addresses'] = array('192.168.170.128');
$conf['omit_vary_cookie'] = True;

4. APC

APC speeds up the processing of PHP by storing information used by PHP between processes. This shared memory prevents PHP from having to re-do a lot of repetitive work. APC module is already installed on our server . Check with following command.

#php -m|grep apc

If APC is installed it will return following text. Which means APC Module is installed.

apc

APC drupal Module is also required which can be downloaded from http://drupal.org/project/apc

Follow these steps to install Module in drupal.

Click on Modules --> Install New Module --> Browse

select the downloaded Module

Press Install to Install this Module.

Progress Bar indicating Modules Instalaltions confirms when Module is installed

Add following lines in settings.php

##################### APC Settings ##############################

$conf['cache_backends'] [] = 'sites/all/modules/apc/drupal_apc_cache.inc';
$conf['cache_class_cache_bootstrap'] = 'DrupalAPCCache';
$conf['cache_class_cache'] = 'DrupalAPCCache';

5. Memcache

Memcache allows Drupal to move many of its cache operations into fast memory and take the strain off of MySQL.  Memcache drupal Module is also required which can be downloaded from                 http://drupal.org/project/memcache

Follow these steps to install Module in drupal.

Click on Modules --> Install New Module --> Browse

select the downloaded Module

Press Install to Install this Module.

Progress Bar indicating Modules Instalaltions confirms when Module is installed

On Same page Click on Enable Newly added Module.

Select Memcache and click save configuration to enable Module.

By Default Memcache run on single port i.e 11211. It is adviseable to run Memcache on Multiple Port to enhance performace.

Replace /etc/init.d/memcached with the following code:

#! /bin/sh

# description: The memcached daemon is a network memory cache service.

# processname: memcached
# config: /etc/sysconfig/memcached
# pidfile: /var/run/memcached/memcached.pid
# Standard LSB functions
#. /lib/lsb/init-functions
# Source function library.
. /etc/init.d/functions

USER=memcached
OPTIONS=""

array[0]='-m 32 -p 11211'
array[1]='-m 16 -p 11212'
array[2]='-m 16 -p 11213'
array[3]='-m 16 -p 11214'
array[4]='-m 16 -p 11215'
array[5]='-m 32 -p 11216'
array[6]='-m 32 -p 11217'
array[7]='-m 16 -p 11218'
array[8]='-m 32 -p 11219'
array[9]='-m 16 -p 11220'

# Check that networking is up.
. /etc/sysconfig/network

if [ "$NETWORKING" = "no" ]
    then
    exit 0
fi

RETVAL=0
prog="memcached"

start () {

        #echo -n $"Starting $prog: "
        # insure that /var/run/memcached has proper permissions

      if [ "`stat -c %U /var/run/memcached`" != "$USER" ]; then
        chown $USER /var/run/memcached
      fi

    for i in "${array[@]}"
    do
        echo -n $"Starting $prog: $i"
        daemon memcached -d -u $USER $i -P /var/run/memcached/memcached.pid $OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch /var/lock/subsys/memcached
done
}

stop () {
    echo -n $"Stopping $prog: "
    killproc -p /var/run/memcached/memcached.pid /usr/bin/memcached
    killall /usr/bin/memcached
    RETVAL=$?
    echo
    if [ $RETVAL -eq 0 ] ; then
        rm -f /var/lock/subsys/memcached
        rm -f /var/run/memcached.pid
    fi
}

restart () {
            stop
            start
            }

# See how we were called.

case "$1" in
start)
    start
    ;;
stop)
    stop
    ;;
status)
    status memcached
    ;;

restart|reload|force-reload)
    restart
    ;;

condrestart)
    [ -f /var/lock/subsys/memcached ] && restart || :
    ;;
*)
echo $"Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
exit 1
esac

exit $?

Add following lines in settings.php

######################## Memcache settings ##############

include_once('./includes/cache.inc');
include_once('./sites/all/modules/contrib/memcache/memcache.inc');
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['memcache_servers'] = array(
'127.0.0.1:11211' => 'default',
'127.0.0.1:11212' => 'block',
'127.0.0.1:11213' => 'content',
'127.0.0.1:11214' => 'filter',
'127.0.0.1:11215' => 'form',
'127.0.0.1:11216' => 'menu',
'127.0.0.1:11217' => 'page',
'127.0.0.1:11218' => 'update',
'127.0.0.1:11219' => 'views',
'127.0.0.1:11220' => 'field',
);

$conf['memcache_bins'] = array(
'cache' => 'default',
'cache_block' => 'block',
'cache_content' => 'content',
'cache_filter' => 'filter',
'cache_form' => 'form',
'cache_menu' => 'menu',
'cache_page' => 'page',
'cache_update' => 'update',
'cache_views' => 'views',
'cache_field' => 'field',
'cache_views_data' => 'views',
);

$conf['cache_backends'] [] = 'sites/all/modules/contrib/memcache/memcache.inc';
$conf += array(
'memcache_extension' => 'Memcache',
'show_memcache_statistics' => 0,
'memcache_persistent' => TRUE,
'memcache_stampede_protection' => TRUE,
'memcache_stampede_semaphore' => 15,
'memcache_stampede_wait_time' => 5,
'memcache_stampede_wait_limit' => 3,
'memcache_key_prefix' => basename(realpath(conf_path())),
);

Save and exit file. Restart Memcached

# service memcached restart

Varnish, APC and Memache are installed.

Restart following services.

#service httpd restart

#service varnish restart

#service memcached restart

[/html]