Review

In the previous section, we've implemented load balancing using HAProxy and session sharing among our Tomcat instances. In this section, we will examine in-depth the HAProxy configuration file and setup its logging facilities.

Setting-up the Environment
- Download Tomcat
- Configure Tomcat
- Run Tomcat
- Download HAProxy
- Configure HAProxy
Load Balancing
- Default Setup
- Sharing Sessions
- Configure Tomcat to Share Sessions
- Retest Session Sharing
- Session Sharing Caveat
- Sharing Sessions
HAProxy Configuration
- Configuration File
- Logging

HAProxy Configuration

When it comes to HAProxy configuration, the best source of information is its online documentation at http://haproxy.1wt.eu/#docs. It's one massive text file of technical information though.

Configuration File

Not all information in that document applies to our configuration. Therefore, I have copied the relevant information only and pasted them as comments per line:

Take note of the following parts:

frontend http-in: We're telling HAProxy to listen to HTTP requests
default_backend servers: We declare a set of backend servers
stats uri /admin?stats: This is the URL to the stats page, relative to your hostname
stats realm Haproxy\ Statistics: This is the server name you see when you login to the stats page.
server tomcat1 127.0.0.1:8080 cookie JSESSIONID check inter 5000: Defines a server. In this case, a Tomcat server. Here we assigned the IP and port number.

HAProxy Logging

Logging is crucial in any serious application, and HAProxy has facilities to log its activities.
However, to setup one requires extra effort because to enable logging in HAProxy we need to know
Linux's logging facilities via the Syslog server and take into account the Syslog implementation in Ubuntu Lucid (10.04).

What is Syslog?

syslog is a utility for tracking and logging all manner of system messages from the merely informational to the extremely critical. Each system message sent to the syslog server has two descriptive labels associated with it that makes the message easier to handle. - Source: Quick HOWTO : Ch05 : Troubleshooting Linux with syslog

To enable logging, we need to:

Add a logging facility in haproxy.cfg
Add the logging facility to Syslog server

Add a logging facility in haproxy.cfg
Edit the haproxy.cfg file:

sudo gedit /etc/haproxy/haproxy.cfg

And declare the following:

We declared two logging facilities under the global section. Both facilities will send their log output to the Syslog server which is located at 127.0.0.1. The default port is 514. Each logger has its own unique name: local0 and local1.

Why are they named such? These are local facilities defined by the user to log specific deamons (see What is LOCAL0 through LOCAL7 ?).

Remember an optional level can be specified to a filter. Hence, local1 has an extra argument: notice. This means local1 will only capture logs with notice level as opposed to all, i.e. errors, debugs.

Reload haproxy by running the following command:

sudo haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

This command will not restart HAProxy. It will just reload the configuration file. This is good because you won't be killing active connections. If you get a missing file i.e /var/run/haproxy.pid or other errors, just kill the haproxy process and restart it:

kill -9 #####

where ##### is the process id

Add the logging facility to Syslog server
There are two solutions to achieve this.

Solution #1
a. Run

sudo gedit /etc/rsyslog.conf

And declare the following lines at the end:

b. Restart syslog server by running:

restart rsyslog

Solution #2
Instead of editing directly the rsyslog.conf, we can declare a separate configuration under /etc/rsyslog.d/ directory. If you inspect carefully the rsyslog.conf, you will see the following comments:

This setting will load all *.conf files under /etc/rsyslog.d/ directory.

a. Run

sudo gedit /etc/rsyslog.d/haproxy.conf

And declare the following lines at the end:

b. Restart syslog server by running:

restart rsyslog

Overflowing Logs

We've setup HAProxy logging. We can see the logs in /var/log/haproxy0a.log and /var/log/haproxy1a.log files. However, we also see them in /var/log/syslog.

This is bad because now we have redundant logs that just eats up space. You don't want that syslog to be polluted with HAProxy logs. That's the reason why we've setup a separate logging facility in the first place.

There are two ways to prevent this unwanted overflow:

Solution #1
1. Run

sudo gedit /etc/rsyslog.d/50-default.conf

And search for the following lines (right after the introductory comments):

And change them as follows:

This means local0 and local1 should not overflow to syslog.

b. Restart syslog server by running:

restart rsyslog

Solution #2
1. Run

sudo gedit /etc/rsyslog.conf

And find the following lines:

And change them as follows:

The addition of & ~ will prevent the logs designated to local0 from overflowing to other logging facilities.

Note: If you can't find those lines, maybe you've declared your configuration under /etc/rsyslog.d/haproxy.conf. If yes, follow the same steps.

b. Restart syslog server by running:

restart rsyslog

Rotate Logs

We've setup HAProxy logging. We've isolated the logs from overflowing to syslog. However, there's another problem. The HAProxy logs will soon pile-up and consume precious disk space. Gladly, Linux has a way to schedule and reuse the same lgo file and perform compression.

For more info of log rotation in Linux, please see Quick HOWTO : Ch05 : Troubleshooting Linux with syslog: Logrotate.

Again, there are two ways of handling this requirement:

Solution #1
a. Run

sudo gedit /etc/logrotate.d/haproxy

And add the following lines:

b. Restart syslog server by running:

restart rsyslog

Solution #2
Log rotation with rsyslog from the official rsyslog documentation. This is something I haven't tried yet but if you're willing to experiment, here's the link: http://www.rsyslog.com/doc/log_rotation_fix_size.html. This technique utilizes the output channels.

However, read the following notes:

Output Channels are a new concept first introduced in rsyslog 0.9.0. As of this writing, it is most likely that they will be replaced by something different in the future. So if you use them, be prepared to change you configuration file syntax when you upgrade to a later release.
- http://www.rsyslog.com/doc/rsyslog_conf_output.html

References

The following is a compendium of references that I found interesting to read further:

R: What is LOCAL0 through LOCAL7 ?
L: http://www.linuxquestions.org/questions/linux-security-4/what-is-local0-through-local7-310637/

R: Quick HOWTO : Ch05 : Troubleshooting Linux with syslog
L: http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch05_:_Troubleshooting_Linux_with_syslog

R: rsyslog official site
L: http://www.rsyslog.com/doc/rsyslog_conf.html

R: rsyslog.conf configuration file
L: http://www.rsyslog.com/doc/rsyslog_conf.html

R: UDP Syslog Input Module
L: http://www.rsyslog.com/doc/imudp.html

R: How to keep haproxy log messages out of /var/log/syslog
L: http://serverfault.com/questions/214312/how-to-keep-haproxy-log-messages-out-of-var-log-syslog

R: HAProxy Logging in Ubuntu Lucid
L: http://kevin.vanzonneveld.net/techblog/article/haproxy_logging/

Q: Install and configure haproxy, the software based loadbalancer in Ubuntu
A: http://linuxadminzone.com/install-and-configure-haproxy-the-software-based-loadbalancer-in-ubuntu/

Conclusion

That's it. We've completed our study of HAProxy and Tomcat clustering. We've learned how to setup, configure load balancing, and handle failover. We've also learned the important points when enabling session sharing. We've also studied HAProxy's configuration and logging facilities.

If you want to learn more about web development and integration with other technologies, feel free to read my other tutorials in the Tutorials section.

Share the joy: