Jan 3, 2012

Load Balancing and Clustering using mod_proxy_ajp on Apache HTTP Server 2.2.21 and Tomcat 7.0.23


I’ve been struggling to make load balancing and clustering working using these combination:

  • Apache HTTP Server 2.2.21 (httpd-2.2.21-win32-x86-openssl-0.9.8r) using mod_proxy_ajp with sticky session enabled.
  • Apache Tomcat 7.0.23 (apache-tomcat-7.0.23-windows-x64)
  • JDK 7 update 2 (jdk-7u2-windows-x64)
  • Windows 7 64 Bit
  • Spring 3.1

Some links that I’ve read:

  • www.mulesoft.com/tomcat-clustering
  • blogs.agilefaqs.com/2009/11/09/setting-up-tomcat-cluster-for-session-replication/
  • www.mulesoft.com/tomcat-context
  • tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html

This is my configuration:


# Required Modules
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule status_module modules/mod_status.so# Reverse Proxy
<Proxy balancer://mybalancer>
    BalancerMember ajp://localhost:8301 route=s1
    BalancerMember ajp://localhost:8302 route=s2
    BalancerMember ajp://localhost:8303 route=s3
ProxyPass / balancer://mybalancer/ stickysession=JSESSIONID|jsessionid# Forward Proxy
ProxyRequests Off<Proxy *>
    Order deny,allow
    Deny from none
    Allow from localhost
</Proxy># Balancer-manager, for monitoring
<Location /balancer-manager>
    SetHandler balancer-manager    Order deny,allow
    Deny from none
    Allow from localhost

server.xml for each tomcat (difference only in port number)

<Server port="8001" shutdown="SHUTDOWN">
    <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
    <Listener className="org.apache.catalina.core.JasperListener" />
    <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
    <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
    <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />    <GlobalNamingResources>
        <Resource name="UserDatabase" auth="Container"
            type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved"
            factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" />
    </GlobalNamingResources>    <Service name="Catalina">
        <Connector port="8101" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8201" />        <Connector port="8301" protocol="AJP/1.3" redirectPort="8201" />        <Engine name="Catalina" defaultHost="localhost" jvmRoute="s1">            <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
                <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false" 
                    notifyListenersOnReplication="true" />
                <Channel className="org.apache.catalina.tribes.group.GroupChannel">
                    <Membership className="org.apache.catalina.tribes.membership.McastService"
                        address="" port="45564" frequency="500" dropTime="3000" />
                    <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
                        <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" />
                    <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                        address="auto" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6" />
                    <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector" />
                    <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor" />
                <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="" />
                <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve" />
                <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener" />
                <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener" />
            </Cluster>             <Realm className="org.apache.catalina.realm.LockOutRealm">
                <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase" />
            <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">
                <Valve className="org.apache.catalina.valves.AccessLogValve"
                    directory="logs" prefix="localhost_access_log." suffix=".txt"
                    pattern="%h %l %u %t &quot;%r&quot; %s %b" />

port numbering (make sure no instances use the same port if deployed in the same server)

  • 80 -> Apache HTTP Server Port
  • 80xx -> Tomcat Server SHUTDOWN Port
  • 81xx -> Tomcat Connector Port (HTTP)
  • 82xx -> Tomcat SSL Redirect Port
  • 83xx -> Tomcat AJP Port
  • 40xx -> Tomcat tcp receive port for NioReceiver

Spring Controller

public class LoginController {
    public String show(@ModelAttribute("user") User user, HttpServletRequest request) {
        user.setUsername("YUSUF");        HttpSession session = request.getSession();
        Integer tambah = (Integer) session.getAttribute("tambah");
        if(tambah == null) tambah = new Integer(1);
        else tambah = new Integer(tambah.intValue() + 1);
        session.setAttribute("tambah", tambah);        return "login";


    <div class="mainFooter">
        Tambah = ${sessionScope.tambah}
        ID = ${pageContext.session.id}

So far the load balancing part is working, but the session replication is not.
Basically what I want is if I keep pressing refresh on the login page, the variable “tambah” will be incremented and kept in the session. And if the current tomcat node is down, the session will be replicated to the next tomcat node and the data is not gone. But this is what happening:

Login Screen:

Tambah = 39 
ID = C1D59C8CA5D10EB98C1DE08AC618204D.s1 

I take the tomcat1 down and keep tomcat2 and tomcat3 running, here is the Login Screen:

Tambah = 1

It seems the failover is not working, the session is not replicated, and the application created a new session. Can someone help point me to the right direction?


EDIT: I’ve solved the question, thanks to @Shane Madden, this is the changes I’ve made:

<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                        address="auto" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6" />

For each Tomcat, the tcp receiver port must be different (if running on the same server), for example use 4001, 4002, 4003, etc.


Thanks to @Shane Madden’s comments, I re-read the tomcat cluster docs in http://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html, especially in this part -> “If your Tomcat instances are running on the same machine, make sure the tcpListenPort attribute is unique for each instance, in most cases Tomcat is smart enough to resolve this on it’s own by autodetecting available ports in the range 4000-4100”.

The changes I’ve made is in each of the Tomcat instance’s server.xml, I make sure each port is different (for example 4001, 4002, 4003) :

<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"  
  address="auto" port="4001" autoBind="100" 
  selectorTimeout="5000" maxThreads="6" />

And voila! load-balancing and clustering is working (with the most basic config of course). I hope this post can help others on initial settings.

Related posts:

  1. Farm Deployer not working in Tomcat Cluster
  2. Ubuntu apache-tomcat webserver clustering/load balancing
  3. Apache load balancer limits with Tomcat over AJP
  4. HTTP 404 – /docs/appdev/ on fresh tomcat 7 install
  5. HTTP 404 – /docs/appdev/ on fresh netbeans tomcat 7 install

1 Comment

Leave a comment