Lynch Consulting Blog - Systems admin

Howto find files newer than a specific date using command line

Mon, 10 May 2010 23:19:00 --1000

While doing some server admin tasks the other day I needed to find all the files newer than a certain date. Using just the command line tools it was relatively simple but not obvious, so this is a not to self. The find utility has an option to find a file newer than another file. By creating and empty file with a specific creation date we can do the search:


touch timestamp -d 2010-01-01

To show all files newer than 2010-01-01 use:


find . -newer timestamp

Or to create a tar archive of them use xargs like so:


find . -newer timestamp | xargs tar -rf /root/filesnewerthan-2010-01-01.tar

Easy. Mark

proxy_http vs proxy_ajp benchmark

Thu, 18 Mar 2010 12:20:00 --1000

After I posted a previous blog entry about configuring railo & tomcat with apache and mod_proxy_http, Paul Kukiel and Gary Gilbert suggested that I should be using mod_proxy_ajp. This has been something I've been looking at, but haven't found a compelling reason for one over the other. Proxy AJP is claimed to be faster as it is a "Wire protocol" but I couldn't find any benchmarks around this. So I decided to do a very quick and dirty benchmark to satisfy my curiosity. This is not a scientific process, I just ran a simple railo testpage on the same machine with 50 threads of jmeter requests hitting it. First I enabled proxy_http and ran it four times, then enabled proxy_ajp and repeated. The config is below:


# Proxy HTTP config

	
	Order deny,allow
	Allow from all
	
	ProxyPassMatch ^/(.*\.cfm)$ http://testsite.railo:8080/$1
	ProxyPassReverse  /  http://testsite.railo:8080/


# Proxy AJP config

	
	Order deny,allow
	Allow from all
	
	ProxyPassMatch ^/(.*\.cfm)$ ajp://testsite.railo:8009/$1
	ProxyPassReverse  /  ajp://testsite.railo:8009/

Results:

Run	HTTP Requests/sec	AJP Requests/sec
Run 1	206.9	181.4
Run 2	203.9	143.6
Run 3	194.6	189.2
Run 4	204.6	191.4
Average	202.5	176.4

The results showed that the proxy_http module was faster - i.e. more requests per second could be pushed through. I'm putting this down to the fact that proxy_ajp has to convert the http request into it's binary format, while proxy_http really just has to pass it along. In different scenarios and network configurations the results may be different, but for now I'm going to stick with the http proxy. Proxy AJP has one other benefit, in that is passes along some extra flags such as whether the request is https or not, but for our purposes we don't need this. Cheers, Mark

Railo on Tomcat revisited - mod_proxy

Fri, 12 Mar 2010 11:01:00 --1000

Updated: Changed the linking between railo and tomcat to use shared.loader. I've been doing some more work on configuring railo to work flexibly in the numerous different environments we work in, and also making it simpler to set up. To that end I investigated the use of mod_proxy for linking it to apache instead of mod_jk. Advantages of this approach are:

Simple - communications are in plain http
Flexible - Load balancing can be easily added at the apache layer
Simple - No compiling mod_jk

Here are the basic install instructions for Railo/Tomcat/Apache on Ubuntu.

Download & Install Tomcat

Download tomcat and extract content:


tar xvzf apache-tomcat-6.0.26.tar.gz

Move Tomcat to a more appropriate place:


sudo mv apache-tomcat-6.0.26 /opt/tomcat

Download Railo

Download Railo custom version jars file Extract and move into Tomcat lib directory:


tar zxvf railo-3.1.2.001-jars.tar.gz
sudo mv railo-3.1.2.001-jars /opt/railo

Make Tomcat load the railo jars by editing catalina.properties to change the shared loader path:


shared.loader=/opt/railo/*.jar

Make Tomcat and Railo work together by modifying the web config file:


sudo nano -w /opt/tomcat/conf/web.xml

add the following inside the <web-app> element:



CFMLServlet
railo.loader.servlet.CFMLServlet
   
      configuration
      {web-root-directory}/WEB-INF/railo/
      Configuraton directory
   
   1


   CFMLServlet
   *.cfm


   CFMLServlet
   *.cfml


   CFMLServlet
   *.cfc

add the following inside <welcome-file-list> element:


index.cfm
index.cfml

Start up tomcat:


/opt/tomcat/bin/startup.sh

Once this is done you should be able to access the railo admin by going to the following URL:


http://127.0.0.1:8080/railo-context/admin/server.cfm

Back to Tomcat

To test our Railo installation, let's create a test site by adding a new virtual host in both Tomcat and Apache. We do this by modifying Tomcat server.xml file (/opt/tomcat/conf/server.xml )

Linking with Apache via Mod Proxy

Ensure the modules proxy and proxy_http are enabled. On Ubuntu this is done as follows:


sudo a2enmod proxy
sudo a2enmod proxy_http
sudo /etc/init.d/apache2 restart

Create vhost

Now we need to create a virtual host entry in Apache as well:



        DocumentRoot /vhosts/testsite.railo/www
        ServerName testsite.railo
        DirectoryIndex index.cfm
	#Proxy .cfm requests to railo
	
		
		Order deny,allow
		Allow from all
		
		ProxyPassMatch ^/(.*\.cfm)$ http://testsite.railo:8080/$1
		ProxyPassReverse  /  http://testsite.railo:8080/
	

	#Deny access to admin except for local/portforwarded clients
	
		Order deny,allow
		Deny from all
		Allow from 127.0.0.1

This tells apache to forward all requests for CFM files to the railo instance. Finally restart apache and railo and you should be good to go.


sudo /opt/tomcat/bin/shutdown.sh 
sudo /opt/tomcat/bin/startup.sh 
sudo /etc/init.d/apache2 restart

MySQL 5.1 logging changes - Log to DB and runtime config

Fri, 29 Jan 2010 08:15:00 --1000

While browsing around the MySQL site last night I discovered a number of nice new features of mysql 5.1 that relate to logging. These are:

Logging to DB instead of log files
Runtime configuration of logging.

Logging to DB instead of log files

Coming from a web development background rather than a sysadmin background I'm far more comfortable manipulating and analysing data using SQL. So to be able to log all the queries or just the slow queries for an application to the db during application development or load testing is a huge benefit. To enable logging to DB you can add the following to your my.cnf


log_output = TABLE

The logs will be written to the 'slow_log' and 'general_log' tables in the mysql database. Note - logging to tables has more overhead than logging to file, so would suggest using it primarily for development purposes. Full details of the options are on the mysql manual on log tables

Runtime configuration of logging.

This allows you to turn on and off logging without restarting MySQL - which just saves a little bit of time and makes it much nicer for debugging problems. To turn on the logging of all queries run:


SET GLOBAL general_log = 'ON';

And for just the slow query log:


SET GLOBAL slow_query_log = 'ON';

And to turn them both off use:


SET GLOBAL general_log = 'OFF';
SET GLOBAL slow_query_log = 'OFF';

If you also want to see queries not using indexes in the slow query log you can set the following variable:


SET GLOBAL log_queries_not_using_indexes = 'ON';

Hope it helps, Mark

Upcoming book review - Tomcat 6 Developer's Guide

Mon, 21 Dec 2009 21:12:00 --1000

I've just received a copy of the Tomcat 6 Developer's Guide from packt publishing to review. It's nice timing as I've been working with Tomcat 6 a bit lately and in the new year plan to move some of our production systems over to running Railo on top of Tomcat. After the extremely busy year Learnosity has had I'm looking forward to reading a few books over the break and coming back in the New Year with lots more ideas and technology to implement.

Apache Deflate Howto

Sun, 15 Nov 2009 19:15:00 --1000

For my own reference, settings to turn on apache mod_deflate. This sets turns it on for everything except gif,jpeg,png or mp3, as these are already well compressed.


        SetOutputFilter DEFLATE
	SetEnvIfNoCase Request_URI \
		\.(?:gif|jpe?g|png|mp3)$ no-gzip dont-vary

Cheers, Mark

Windows File Sharing (SMB/CIFS/Samba) over SSH

Sat, 14 Nov 2009 13:35:00 --1000

While working with a client recently setting up a Netgear VPN so he could securely access his internal file server. The VPN setup was straightforward but every time the VPN client connected to the VPN server the VPN server/firewall would crash - leaving no connectivity. In order to come up with a reliable solution to this we decided to use the SSH server we had available and tunnel the windows sharing across the local port forwards, much simpler and more reliable. Thanks to this article it was a breeze to set up. Steps are as follows:

Create loopback adapter on windows
Configure loopback adapter on windows
Reboot
Configure SSH connection
Test it all out

Create loopback adapter on windows

We'll give your computer an additional (fake) IP address, and we'll port forward to that address instead of the computer's real IP. Windows XP will continue to do file sharing on the real IP address. We'll assign it an IP of 10.0.0.1 (that's what we configured putty to use above.)

System->Control Panel->Add Hardware
Yes, Hardware is already connected
Add a new hardware device (at bottom of list)
Install the hardware that I manually select
Network adapters
Microsoft , Microsoft Loopback Adapter
(Go through the installation procedure.)

Configure loopback adapter on windows

Open your new fake ethernet adapter (Network Connections) , enter a made-up IP address (I suggest 10.0.0.1, which is a privately routable address that most folk don't use.)
Enable Client for Microsoft Networks.
Disable File and Printer Sharing for Microsoft Networks
Enable Interent Protocol (TCP/IP)
Click on properties for TCP/IP.
Enter your chosen IP address (10.0.0.1), subnet mask (255.255.255.0). You can leave gateway blank.
Under advanced->WINS, Enable LMHosts Lookup and Disable NetBIOS over TCP/IP

Reboot

In order to make it all work now it he appropriate time to reboot so windows initialises everything correctly.

Configure SSH connection

Download Putty
Enter IP address
Enter Auth Key (if using SSH keys)
Enter Port forwards for: (these connect the ports on you local machine to
- 10.0.0.1:137 to 127.0.0.1:137
- 10.0.0.1:138 to 127.0.0.1:138
- 10.0.0.1:139 to 127.0.0.1:139
- 10.0.0.1:445 to 127.0.0.1:445
Save the config.

Test it all out

Now to connect you need to do the following steps:

Open putty, load the settings and connect.
Open Exporer and type in: \\10.0.0.1\

You should now be connected to your remote windows system over a secure encrypted tunnel. Cheers, Mark

cp "No space left on device" problem - Solved

Sat, 14 Nov 2009 13:12:00 --1000

One of my backup scripts started reporting errors recently about running out of space:


cp: cannot create regular file `filename in here': No space left on device

Running the command df was showing lots of free space. However, a quick google on cp "no space left on device" turned up the suggestion to try:


df -i

This showed up the problem straight away - I had run out of inodes. Once the problem is identified it is generally easy to resolve, so I figured out that there was a cron job that was running a wget task and not discarding the output, and so had saved 600,000+ files in their home directory. However, when I tried to delete them I encountered another problem:


# rm filepattern*
bash: /bin/rm: Argument list too long

There were so many files that I couldn't use standard delete commands. Another quick google turned up this gem: I ran the following command to check that I was going to delete the correct files:


find . | grep filepattern

And then added the command to actually do the delete:


find . | grep filepattern | xargs rm

One final thing was to fix up the cron job that was causing the problem. Adding the parameter --delete-after to wget kept the directory nice and clean. All fixed. One of the things this has reinforced for me is how important it is to have /home on a seperate partition. If this had not been the case then the problem would have taken longer to happen (due to larger filesystem) but would have been more destructive - as all systems and processes would not have been able to create new files.

Two quick server tips

Sat, 15 Aug 2009 08:12:00 --1000

Here are two quick tips that I've recently found for server admins:

Automatically fixing file system errors
Ignoring directories from updatedb

Automatically fixing file system errors

If you have a remote server, i.e. at a data center 10km or 1000km away this should prevent some panic when rebooting the server remotely. Ext2/3 will do a filesystem check after a certain number of reboots or time. Most of the time any errors are fixed automatically, but certain errors by default require the root shell and the administrator to fix them. I've seen a few of these happen, but I have always replied yes to the prompts as I don't know enough about file systems to fix it I said no. So, to prevent the need to rush to a data center and plug in a keyboard and mouse just to press the "Y" key there is and option to automatically assume yes. On Ubuntu in the file /etc/default/rcS you need to change the following:


FSCKFIX=no
to 
FSCKFIX=yes

Ignoring directories from updatedb

If you have a backup server you may not want it to update the locate db for all your backup files, as it can take a very long time. To tell locate to ignore a directory you need to add it to the PRUNEPATHS line in the /etc/updatedb.conf file like so:


PRUNE_BIND_MOUNTS="yes"
PRUNEPATHS="/tmp /var/spool /media /srv"
PRUNEFS="NFS nfs nfs4 afs binfmt_misc proc smbfs autofs iso9660 ncpfs coda devpts ftpfs devfs mfs shfs sysfs cifs lustre_lite tmpfs usbfs udf rpc_pipefs"

In the above code I have added the /srv entry to make it ignore all my backups which are held under the /srv directory. Note: These tips were tested on Ubuntu linux, other distributions will have similar functionality but the file locations may vary. Cheers, Mark

Installing Dell OpenManage Server Administrator on Ubuntu 32bit

Tue, 04 Aug 2009 11:27:00 --1000

I found this article on Installing Dell OpenManage Server Administrator on Ubuntu x64 on Keith's Code and am posting here with modifications for 32 bit for my own reference

Dell's OpenManage Server Administrator is a powerful tool for keeping track of your server's health and making sure everything is running as it should. Unfortunately, Dell only released packages for RedHat and SuSE, so installing the software on Ubuntu can be tricky, at best.

There is some information on the Internet about getting the package to install on exotic distributions, but I never found anything definitive. This tutorial will guide you through installing the software and getting the web access up and running.

My setup is: Dell 2950 with 1 Intel 2.2GHz Processor, 2GB RAM, PERC 5/i SCSI Raid Controller with 2 73GB 15K RPM SAS disks running Ubuntu 8.04.2 LTS 32-bit.

1. Install the kernel modules

The first step is installing the needed kernel modules to support IPMI:

sudo modprobe ipmi_msghandler sudo modprobe ipmi_devintf sudo modprobe ipmi_si

Next,you will want to add those modules to the end of the /etc/modules file. This will allow them to be loaded at boot time.

2. Setup Apt to get the OMSA package

As stated before, Dell only released the OpenManage Server Administrator for RedHat and SuSE. However, someone was kind enough to create an installable Debian based package for us. In order to get access to that package, you will need to add the following line to /etc/apt/sources.list:

deb ftp://ftp.sara.nl/pub/sara-omsa dell sara

Then, you will need to install the Pulic Key for that repository. To do that, create a temporary directory in your home folder and run the following commands from that folder:
wget http://ftp.sara.nl/debian_sara.asc sudo apt-key add debian_sara.asc

3. Install the needed packages

Now it's time to install all the packages that will be needed:

sudo apt-get update sudo apt-get install snmp snmpd sudo apt-get install openipmi sudo apt-get install ipmitool sudo apt-get install dellomsa

4. Start SNMP

The following commands will enable SNMP and get it going:
sudo /etc/init.d/snmpd restart sudo /etc/init.d/dataeng start If you get and error like: Starting dsm_sa_datamgr32d: /opt/dell/srvadmin/dataeng/bin/dsm_sa_datamgr32d: error while loading shared libraries: libdcsmil32.so.5: cannot open shared object file: No such file or directory Starting dsm_sa_eventmgr32d: /opt/dell/srvadmin/dataeng/bin/dsm_sa_eventmgr32d: error while loading shared libraries: libdcsupt32.so.5: cannot open shared object file: No such file or directory You may need to reconfigure the dynamic linker so that it knows the new libraries are.
sudo ldconfig

5. Start the web services

Start the Dell OMSA web interface:
sudo /etc/init.d/dsm_om_connsvc start

6. Add a user to access the OMSA site

If your system is setup properly, you shouldn't have a password for the root user. To access the OpenManage client, you need to have root access. There is a way to do this without giving the root user a password. I created a user called dellroot that will be able to access the OMSA client:
sudo useradd dellroot sudo usermod -g root dellroot sudo passwd dellroot

Enter a secure password for the user when prompted.

7. Check it out

Now you should be able to access the OpenManage web interface by going to https://<servername>:1311 (e.g., https://www.example.com:1311)

Make sure you are using https and you add the port number of 1311 to the end of the URL. You can login using the dellroot user you just created.

If you are using firefox, you will have to add a security exception for the self-signed certificate the server is using.

Thanks Keith for the helpful reference. Cheers, Mark

SSH SOCK Proxying and preventing it

Wed, 01 Jul 2009 13:46:00 --1000

I've been using SSH portforwards extensively for getting access to remote networks securely for years, but today I came across another very useful thing you can do with SSH. That is use it as a SOCKS proxy. This is useful if you are on a corporate network that has specific restrictions on certain sites that you need to access. You can use the following command so set it up:


ssh -D 8080 your_ssh_server

This will create a socks proxy on port 8080 and if you configure your browser proxy settings to use a socks proxy on localhost port 8080 - see picture. Once this is done all your browser traffic will go through the proxy network.

Preventing Port Forwards on your SSH server

However, you may not want everyone who has a login to your ssh server from being able to do this, so you can conditionally limit access to who can do portforward with the following in you sshd_config file:


AllowTcpForwarding no
Match User someuser
         AllowTcpForwarding yes


AllowTcpForwarding no
Match Group somegroup
         AllowTcpForwarding yes

I found that little gem here on nabble.

HOWTO find files that are not world readable

Mon, 18 May 2009 21:47:00 --1000

I ran across some permissions errors on content that some users uploaded to a website. The files uploaded were not world readable so were coming up as forbidden: This simple find command shows the offending files:


/usr/bin/find . -type f ! -perm -004

A quickie but a goodie. Cheers, Mark

ColdFusion per vhosts mappings

Wed, 06 May 2009 13:32:00 --1000

One of the features of CF8 that I was most interested in and most let down on was per application mappings. The reason I was so keen was so that we could keep core library application files out of the webroot for additional security, but still have different versions of the same codebase. I.e. some sites running farcry3 some running farcry5 for example. This didn't transpire and so now we use the following technique on apache to map the folder in and then lock it down with apache. In the vhost section add the following:


Alias /farcry /path/to/non-webfolder/farcry

    Order Deny,Allow
     Deny from all

Note: it is important that there is no /farcry mapping defined in the CF Administrator or it will override this setting. What the above does, is to map in the non-web accessible folder, but then locking it down so it cannot be browsed from the web. This seems pointless but CF can still see the files and correctly resolves the paths even though they are secure from the outside world. This then allows us to version our sites independently - i.e. if we want to upgrade from farcry 5 to 5.1 we can do it one site at a time, instead of having to test them all at once. Note - Railo already has per vhost mappings which solve the same problem as this. But if you are on CF7 or CF8 then this works a treat. Cheers, Mark

Installing Railo on Tomcat via Apache on Leopard - Step by Step

Fri, 27 Mar 2009 14:45:00 --1000

After a bit of fiddling and head scratching I managed to install Railo on Tomcat via Apache on my laptop running on OS X 10.5.6.

Tomcat

Firstly, download Tomcat 6 extract content:


tar xvzf apache-tomcat-6.0.18.tar.gz

Move Tomcat to a more secure place:


sudo mv apache-tomcat-6.0.18 /usr/local/tomcat

Railo

Download Railo custom version extract and move into Tomcat lib directory:


tar zxvf railo-3.0.2.001-jars.tar.gz
sudo mv railo-3.0.2.001-jars/* /usr/local/tomcat/lib

Make Tomcat and Railo work together by modifying the web config file:


sudo nano /usr/local/tomcat/conf/web.xml

add the following inside the <web-app> element:



CFMLServlet
railo.loader.servlet.CFMLServlet
   
      configuration
      {web-root-directory}/WEB-INF/railo/
      Configuraton directory
   
   1


   CFMLServlet
   *.cfm


   CFMLServlet
   *.cfml


   CFMLServlet
   *.cfc

add the following inside <welcome-file-list> element:


index.cfm
index.cfml

Apache

Leopard OS comes with Apache 2, so you don't have to worry about installing it. However, you need to download The Apache Tomcat Connector source code. Next you need to compile the source so that the resulting binary file is compatible with your Intel Mac architecture. I got this from Eric Rank's blog cd into tomcat source:


cd tomcat-connectors-1.2.27-src/native

Edit the apache-2.0/Makefile.apxs.in file. Replace


mod_jk.la:
 $(APXS)  -c -o $@ -Wc,"${APXSCFLAGS} ${JK_INCL}" "${JAVA_INCL}" "${APXSLDFLAGS}" mod_jk.c ${APACHE_OBJECTS}

with:


mod_jk.la:
 $(APXS) -c -o $@ -Wc,"${APXSCFLAGS} -arch x86_64 ${JK_INCL}" "${JAVA_INCL}" "${APXSLDFLAGS} -arch x86_64 " mod_jk.c ${APACHE_OBJECTS}

configure the build files:


./configure --with-apxs=/usr/sbin/apxs

now go into apache-2.0 directory and build:


cd apache-2.0
make -f Makefile.apxs

Finally install


sudo make install

Now specify the connection between Apache and Tomcat. To do this you need to create workers.properties file. I created mine in /etc/apache2.


sudo nano /etc/apache2/workers.properties

Paste the following:


worker.list=default

worker.default.port=8009
worker.default.host=localhost
worker.default.type=ajp13
worker.default.lbfactor=1

Now we need to modify the Apache httpd.conf file:


sudo nano /etc/apache2/httpd.conf

Enable The Apache Tomcat Connector:


LoadModule jk_module    libexec/apache2/mod_jk.so

Underneath that tell Apache where your workers.properties file is located and add some logging info (could be useful):


# Mod_jk settings                                            
JkWorkersFile /etc/apache2/workers.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel debug
DirectoryIndex index.html index.htm index.cfm index.cfml

Back to Tomcat

To test our Railo installation, let's create a test site by adding a new virtual host in both Tomcat and Apache. We do this by modifying Tomcat server.xml file (/usr/local/tomcat/conf/server.xml )

Apache

Now we need to create a virtual host entry in Apache as well:



        JkMount /*.cfm default
        ServerName railo.local
        DirectoryIndex index.cfm


JkMount /*.cfm default

Tells mod_jk to use the connector specified in your workers.properties file when it encounters a .cfm extension.

Important

Notice that in my Apache VirtualHost entry there is no DocumentRoot. I originally had it in there and it was breaking my Apache-Tomcat connection. It was driving me mad. It's probably because document root is already specified in /usr/local/tomcat/conf/server.xml. One last thing, to start your Tomcat server type in this command:


/usr/local/tomcat/bin/startup.sh

shut down


/usr/local/tomcat/bin/shutdown.sh

My assumption is that the above steps would be very similar on other operating systems as long as you use the correct file paths. Good luck :-) Marko

HOWTO generate diff files from subversion

Tue, 03 Mar 2009 11:46:00 --1000

Here's a nice and quick way to generate a diff file from subversion and store it on your local machine.


svn diff filepath/myFile.ext > saveDirPath/myFile.ext.diff

Where filePath is the local path to your working directory in which your file is located. And saveDirPath is the path where you want to save your diff file. Diff files can be very useful when contributing to open source projects. You can also get a list of files changed between revisions. Marko