Lynch Consulting Blog - HOWTO

Howto find files newer than a specific date using command line

Mon, 10 May 2010 23:19:00 --1000

While doing some server admin tasks the other day I needed to find all the files newer than a certain date. Using just the command line tools it was relatively simple but not obvious, so this is a not to self. The find utility has an option to find a file newer than another file. By creating and empty file with a specific creation date we can do the search:


touch timestamp -d 2010-01-01

To show all files newer than 2010-01-01 use:


find . -newer timestamp

Or to create a tar archive of them use xargs like so:


find . -newer timestamp | xargs tar -rf /root/filesnewerthan-2010-01-01.tar

Easy. Mark

Flex 3 to Flex 4 Migration Howto

Mon, 22 Mar 2010 09:50:00 --1000

I recently completed migrating an application from Flex3 to Flex4, and for my own reference, here's how I did it. Note - this was done with flexsdk-4.0.0.13875, which was the most recent stable build at the time.

Phase 1: Get Flex3 code compiling with Flex4

Fix up stylesheets:

If you have a current Flex3 project and you are migrating to Flex4 you need to add the following lines at the top of your stylesheet.


@namespace "library://ns.adobe.com/flex/mx";
@namespace s "library://ns.adobe.com/flex/spark";

This makes the default non-namespaced items in your stylesheet refer to the MX components. eg:


Button{fontSize:18;}

And to style up spark buttons you simply use:


s|Button{fontSize:18;}

To statically link or not?

Flex 4 defaults to dynamically linking runtime shared libraries. This produces smaller files, but if you are making desktop type applications or developing code while not connected to the internet you'll probably want to statically link the files. Add this to your compiler option:


-static-link-runtime-shared-libraries=true

You now should have a happily compiling app, however, to use all the new goodness of flex 4 there are still some more steps to be done.

Phase 2: Using all of flex 4

Update namespace to 2009

To update the namespace references for flex 4 I did the following search and replace:


Replace:
xmlns:mx="http://www.adobe.com/2006/mxml"
With:
xmlns:fx="http://ns.adobe.com/mxml/2009" xmlns:mx="library://ns.adobe.com/flex/mx" xmlns:s="library://ns.adobe.com/flex/spark"

Replace mx with fx namespace


  and 
becomes:
  and

Similarly for other tags like this:


  to 
  to

Add Declarations for non visual components

Wrap tag around non visual compoents. The compiler will complain about all these errors so just follow through and wrap them in

tags until all the errors go away.

Migrate States

Finally, at least for the project I migrated, I needed to convert all the state tags to the new format of inline attributes. This shows up as the following error:


Error: State overrides may no longer be explicitly declared. The legacy states syntax has been deprecated.

So if you had something like this before:


 
  
   
            
... snip ...

It would now look something like this:


 
... snip ...

Migrate StyleManager References

Stylemanager is not called as a global class any more and you need to use fully qualified references so:


thisImage.source = StyleManager.getStyleDeclaration("Image").getStyle("brokenImageSkin");

becomes:


thisImage.source = styleManager.getStyleDeclaration("mx.controls.Image").getStyle("brokenImageSkin");

When these have all been converted, your application should compile. Then you can start leveraging the new functionality of flex4 and begin migrating your components to spark if necessary.

FFMpeg conversion - wmv to flv HOWTO

Fri, 12 Mar 2010 15:34:00 --1000

More of a quick note to self about converting video formats: To convert wmv to flash video:


ffmpeg -i input.wmv -ar 44100 -qmax 8 out.flv

This does the following:

-i input.wmv - Load input file
-ar 44100 - Resample audio to 44.1kHz.
-qmax 8 magic voodoo about quality. Gave better results than not using it.
out.flv save it as an flv.

Easy.

Split first name and last name with Openoffice/Excel

Tue, 23 Feb 2010 21:21:00 --1000

I've been dealing with a lot of spreadsheets of usernames recently and sometimes you get firstname and lastname in seperate columns, and sometimes in the same column. If you get them in the same column but need them in two columns here are two little formulae to do it:


FullName | FirstName   | Lastname    | 
Mark Lynch | =MID(A2,1,FIND(" ",A2,1)) | =MID(A2,FIND(" ",A2,1)+1,100) |

Which will look like:


FullName   | FirstName  | Lastname | 
Mark Lynch | Mark       | Lynch    |

This basically seperates the string on the space between the names and puts it into each column Cheers, Mark

Jmeter over SSH Socks proxy

Thu, 07 Jan 2010 10:25:00 --1000

I've been doing some testing recently where I need to connect via SSH server to a remote network to run some load testing. To do this I used a SSH sock proxy like I have previously blogged about. So I fired this up so that I could review the site I wanted to look at. It worked a charm through firefox but there is no where to set up the proxy in jmeter. To make it work you need to let the JVM know what proxy to use like so:


java -DsocksProxyHost=localhost -DsocksProxyPort=8080 -jar ApacheJMeter.jar

No jmeter will use the socks proxy on port 8080 on my local machine. Nice.

Windows File Sharing (SMB/CIFS/Samba) over SSH

Sat, 14 Nov 2009 13:35:00 --1000

While working with a client recently setting up a Netgear VPN so he could securely access his internal file server. The VPN setup was straightforward but every time the VPN client connected to the VPN server the VPN server/firewall would crash - leaving no connectivity. In order to come up with a reliable solution to this we decided to use the SSH server we had available and tunnel the windows sharing across the local port forwards, much simpler and more reliable. Thanks to this article it was a breeze to set up. Steps are as follows:

Create loopback adapter on windows
Configure loopback adapter on windows
Reboot
Configure SSH connection
Test it all out

Create loopback adapter on windows

We'll give your computer an additional (fake) IP address, and we'll port forward to that address instead of the computer's real IP. Windows XP will continue to do file sharing on the real IP address. We'll assign it an IP of 10.0.0.1 (that's what we configured putty to use above.)

System->Control Panel->Add Hardware
Yes, Hardware is already connected
Add a new hardware device (at bottom of list)
Install the hardware that I manually select
Network adapters
Microsoft , Microsoft Loopback Adapter
(Go through the installation procedure.)

Configure loopback adapter on windows

Open your new fake ethernet adapter (Network Connections) , enter a made-up IP address (I suggest 10.0.0.1, which is a privately routable address that most folk don't use.)
Enable Client for Microsoft Networks.
Disable File and Printer Sharing for Microsoft Networks
Enable Interent Protocol (TCP/IP)
Click on properties for TCP/IP.
Enter your chosen IP address (10.0.0.1), subnet mask (255.255.255.0). You can leave gateway blank.
Under advanced->WINS, Enable LMHosts Lookup and Disable NetBIOS over TCP/IP

Reboot

In order to make it all work now it he appropriate time to reboot so windows initialises everything correctly.

Configure SSH connection

Download Putty
Enter IP address
Enter Auth Key (if using SSH keys)
Enter Port forwards for: (these connect the ports on you local machine to
- 10.0.0.1:137 to 127.0.0.1:137
- 10.0.0.1:138 to 127.0.0.1:138
- 10.0.0.1:139 to 127.0.0.1:139
- 10.0.0.1:445 to 127.0.0.1:445
Save the config.

Test it all out

Now to connect you need to do the following steps:

Open putty, load the settings and connect.
Open Exporer and type in: \\10.0.0.1\

You should now be connected to your remote windows system over a secure encrypted tunnel. Cheers, Mark

Two quick server tips

Sat, 15 Aug 2009 08:12:00 --1000

Here are two quick tips that I've recently found for server admins:

Automatically fixing file system errors
Ignoring directories from updatedb

Automatically fixing file system errors

If you have a remote server, i.e. at a data center 10km or 1000km away this should prevent some panic when rebooting the server remotely. Ext2/3 will do a filesystem check after a certain number of reboots or time. Most of the time any errors are fixed automatically, but certain errors by default require the root shell and the administrator to fix them. I've seen a few of these happen, but I have always replied yes to the prompts as I don't know enough about file systems to fix it I said no. So, to prevent the need to rush to a data center and plug in a keyboard and mouse just to press the "Y" key there is and option to automatically assume yes. On Ubuntu in the file /etc/default/rcS you need to change the following:


FSCKFIX=no
to 
FSCKFIX=yes

Ignoring directories from updatedb

If you have a backup server you may not want it to update the locate db for all your backup files, as it can take a very long time. To tell locate to ignore a directory you need to add it to the PRUNEPATHS line in the /etc/updatedb.conf file like so:


PRUNE_BIND_MOUNTS="yes"
PRUNEPATHS="/tmp /var/spool /media /srv"
PRUNEFS="NFS nfs nfs4 afs binfmt_misc proc smbfs autofs iso9660 ncpfs coda devpts ftpfs devfs mfs shfs sysfs cifs lustre_lite tmpfs usbfs udf rpc_pipefs"

In the above code I have added the /srv entry to make it ignore all my backups which are held under the /srv directory. Note: These tips were tested on Ubuntu linux, other distributions will have similar functionality but the file locations may vary. Cheers, Mark

Installing Railo on Tomcat via Apache on Leopard - Step by Step

Fri, 27 Mar 2009 14:45:00 --1000

After a bit of fiddling and head scratching I managed to install Railo on Tomcat via Apache on my laptop running on OS X 10.5.6.

Tomcat

Firstly, download Tomcat 6 extract content:


tar xvzf apache-tomcat-6.0.18.tar.gz

Move Tomcat to a more secure place:


sudo mv apache-tomcat-6.0.18 /usr/local/tomcat

Railo

Download Railo custom version extract and move into Tomcat lib directory:


tar zxvf railo-3.0.2.001-jars.tar.gz
sudo mv railo-3.0.2.001-jars/* /usr/local/tomcat/lib

Make Tomcat and Railo work together by modifying the web config file:


sudo nano /usr/local/tomcat/conf/web.xml

add the following inside the <web-app> element:



CFMLServlet
railo.loader.servlet.CFMLServlet
   
      configuration
      {web-root-directory}/WEB-INF/railo/
      Configuraton directory
   
   1


   CFMLServlet
   *.cfm


   CFMLServlet
   *.cfml


   CFMLServlet
   *.cfc

add the following inside <welcome-file-list> element:


index.cfm
index.cfml

Apache

Leopard OS comes with Apache 2, so you don't have to worry about installing it. However, you need to download The Apache Tomcat Connector source code. Next you need to compile the source so that the resulting binary file is compatible with your Intel Mac architecture. I got this from Eric Rank's blog cd into tomcat source:


cd tomcat-connectors-1.2.27-src/native

Edit the apache-2.0/Makefile.apxs.in file. Replace


mod_jk.la:
 $(APXS)  -c -o $@ -Wc,"${APXSCFLAGS} ${JK_INCL}" "${JAVA_INCL}" "${APXSLDFLAGS}" mod_jk.c ${APACHE_OBJECTS}

with:


mod_jk.la:
 $(APXS) -c -o $@ -Wc,"${APXSCFLAGS} -arch x86_64 ${JK_INCL}" "${JAVA_INCL}" "${APXSLDFLAGS} -arch x86_64 " mod_jk.c ${APACHE_OBJECTS}

configure the build files:


./configure --with-apxs=/usr/sbin/apxs

now go into apache-2.0 directory and build:


cd apache-2.0
make -f Makefile.apxs

Finally install


sudo make install

Now specify the connection between Apache and Tomcat. To do this you need to create workers.properties file. I created mine in /etc/apache2.


sudo nano /etc/apache2/workers.properties

Paste the following:


worker.list=default

worker.default.port=8009
worker.default.host=localhost
worker.default.type=ajp13
worker.default.lbfactor=1

Now we need to modify the Apache httpd.conf file:


sudo nano /etc/apache2/httpd.conf

Enable The Apache Tomcat Connector:


LoadModule jk_module    libexec/apache2/mod_jk.so

Underneath that tell Apache where your workers.properties file is located and add some logging info (could be useful):


# Mod_jk settings                                            
JkWorkersFile /etc/apache2/workers.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel debug
DirectoryIndex index.html index.htm index.cfm index.cfml

Back to Tomcat

To test our Railo installation, let's create a test site by adding a new virtual host in both Tomcat and Apache. We do this by modifying Tomcat server.xml file (/usr/local/tomcat/conf/server.xml )

Apache

Now we need to create a virtual host entry in Apache as well:



        JkMount /*.cfm default
        ServerName railo.local
        DirectoryIndex index.cfm


JkMount /*.cfm default

Tells mod_jk to use the connector specified in your workers.properties file when it encounters a .cfm extension.

Important

Notice that in my Apache VirtualHost entry there is no DocumentRoot. I originally had it in there and it was breaking my Apache-Tomcat connection. It was driving me mad. It's probably because document root is already specified in /usr/local/tomcat/conf/server.xml. One last thing, to start your Tomcat server type in this command:


/usr/local/tomcat/bin/startup.sh

shut down


/usr/local/tomcat/bin/shutdown.sh

My assumption is that the above steps would be very similar on other operating systems as long as you use the correct file paths. Good luck :-) Marko

GZIP encoding in Adobe AIR/Flex - HOWTO

Fri, 06 Mar 2009 12:27:00 --1000

If you use HTTPService component in Flex or AIR to retrieve data from a server, you might want to consider gzipping the response. This can significantly reduce the packet size you get from the server, which results in a much better application performance. This means that response packet is compressed by Apache web server and your client will need to decompress it. Fortunately, both Flex and AIR 1.5.1 natively support GZIP compression. However there are a couple of things you need to watch out for. Firstly, you need to enable Apache Module mod_deflate in your virtual host. In your AIR application, you need to tell HTTPService component to accept gzip encoding and AIR will take care of the rest:



this.service.headers = {'Accept-Encoding':'gzip'};

If you are in a Flex project, specifying gzip encoding will throw the following error:


ArgumentError: Error #2096: The HTTP request header Accept-Encoding cannot be set via ActionScript.

This is because Flex apps run in a browser and the browser is responsible for determining encoding type and decompression (if required). Therefore, you cannot specify encoding type in Flex apps. Our problem is that we use one core library for AIR and Flex applications. So the issue above can be avoided by conditionally specifying encoding type depending on the framework used. This can be done in 2 lines of code:


if(Capabilities.playerType == "Desktop")
   this.service.headers = {'Accept-Encoding':'gzip'};

Note: this is not the complete solution - check out this entry for the rest of it :-)

Cheers Marko

HOWTO generate diff files from subversion

Tue, 03 Mar 2009 11:46:00 --1000

Here's a nice and quick way to generate a diff file from subversion and store it on your local machine.


svn diff filepath/myFile.ext > saveDirPath/myFile.ext.diff

Where filePath is the local path to your working directory in which your file is located. And saveDirPath is the path where you want to save your diff file. Diff files can be very useful when contributing to open source projects. You can also get a list of files changed between revisions. Marko

Non-case sensitive filesystem on Linux - HOWTO

Mon, 02 Feb 2009 20:58:00 --1000

I was setting up a ColdFusion web application on my dev machine and ran into lots of errors because of the case sensitivity of Linux (which i was using) versus the non-case sensitivity of windows (where the application was developed). I had a quick look to see if I could fix it up but quickly figured out that there were too many occurrences and so it was likely not worth fixing up at this point. To that end I set up a loopback vfat filesystem so that I could have a directory on my machine mounted as non-case sensitive. Note: this is not recommended for production systems - but is a handy fix for a temporary development problem.

Create a virtual disk


dd if=/dev/zero of=virtual.dsk bs=1048576 count=150

Format it


mkfs.vfat virtual.dsk

Mount it


sudo mkdir /mnt/vfat
sudo mount virtual.dsk /mnt/vfat -t vfat -o loop,owner,group,umask=000

You can set this up to mount every time by putting the following line in your /etc/fstab


/path/to/virtual.dsk /mnt/vfat  vfat  loop,owner,group,umask=000  0 0

Note - the same trick works on OSX in reverse. I.e. if you are developing on a mac, but deploying to a linux production environment you can create a virtual disk with a case-sensitive filename, which will mean that any case sensitive issues get picked up on your dev machine, and not on your production/staging machines. Cheers, Mark

SVN - Get list of files changed between revisions

Fri, 16 Jan 2009 13:18:00 --1000

For my own reference, here is how to get the list of files that have changed between any two subversion revisions.


svn diff --summarize -r5:10 http://svn.example.com/trunk

This will give something like:


A      http://svn.example.com/trunk/file1.txt
A      http://svn.example.com/trunk/file2.txt
M      http://svn.example.com/trunk/file3.txt

If you want to get the revisions from a revision to the latest then you can change the 10 to HEAD - eg:


svn diff --summarize -r5:HEAD http://svn.example.com/trunk

Cheers, Mark

Load Testing Asterisk Applications - HOWTO

Tue, 06 Jan 2009 14:00:00 --1000

I've been working on Load Testing some of our asterisk applications and thought I'd document the process here so I can find it when I need to do it again. There is a bit of setup to be done but when you get the whole picture is it's actually very easy. To do it you need at least 2 asterisk boxes - the one you are load testing (lets call it 'stressbox' for this example) and a another one to drive load to it which we'll call 'driver'. We need to do the following steps:

Create an sip account on stressbox for driver to connect to
Register the account from the driver machine
Create a dial plan which simulates a call
Create a call file for the call
Start you system monitoring
Create lots of call files

So here is the step by step:

Create an sip account on stressbox for driver to connect to

In the sip.conf file on stressbox (the one we are testing) add the following:


[loadtest]
type=friend
username=loadtest                                     
secret=1111
context=test
host=dynamic

Do a 'sip reload' in the asterisk console to register this change.

Register the account from the driver machine

Then to register the account from the driver machine add the following to sip.conf on driver:


register => loadtest:1111@xxx.xxx.xxx.xxx

Where xxx.xxx.xxx.xxx is the ip address of the stressbox server. Your driver machine should now be able to make calls to the stressbox machine and you should see a line that looks like this if you were watching the console:


-- Registered SIP 'loadtest' at yyy.yyy.yyy.yyy port 9350 expires 120

Create a dial plan which simulates a call

Then on the driver machine we need to create a dial plan which simulates the call we want to do and place it in extensions.conf. Here's an example dial plan:


[dialplan-loadtest1]
exten => s,1,Answer
;wait for welcome to play
exten => s,n,Wait(10)
;enter username
exten => s,n,SendDTMF(1234)
exten => s,n,Wait(4)
;enter password
exten => s,n,SendDTMF(0000)
;listen to menu
exten => s,n,Wait(10)
;select menu option 1
exten => s,n,SendDTMF(1)
exten => s,n,Wait(3)
;select menu option 2
exten => s,n,SendDTMF(2)
exten => s,n,Wait(4)

;Play some audio
exten => s,n,Playback(sampleaudio1)
;Hangup
exten => s,n,Hangup

This example does a login to a phone system and then selects some options and plays some audio and hangs up. The dial plan can be as simple or as complex as you like.

Create a call file for the call

Last config step is to create a call file which will tell asterisk on driver to call asterisk on stressbox. Create a file test.call with the following:


Channel: SIP/6@xxx.xxx.xxx.xxx
MaxRetries: 1
RetryTime: 5
WaitTime: 10
Context: dialplan-loadtest1
Extension: s
Priority: 1

A bit of explanation of these lines: Channel: SIP/6@xxx.xxx.xxx.xxx - the 6 is the extension to dial on the stressbox server - change as necessary. The ip adress should be the stressbox one. Context: dialplan-loadtest1 - this is the dial plan that will be run for this call.

Start you system monitoring

If you are going to do some load testing then you probably want to monitor the load on the machine - it is recommended to record the load of all machines involved - i.e. stressbox and driver to ensure that driver is not the bottleneck in the load test. For this I use a vmstat on linux via a little script I wrote which makes it easy to collect stats from multiple machines It would be used as follows:


./multivmstat stressbox,driver 5

This will connect via ssh to each box and run vmstat on them every five seconds. This can be graphed later - I'll post the scripts for this shortly.

Create lots of call files

You are not ready to test you system. To do this you need to create copies the test.call file and move them into /var/spool/asterisk/outgoing on the driver machine. I use a little script to do this:


#!/bin/sh
#usage: makecalls callfile count delaybetween

callfile=$1
count=$2
delaybetween=$3

callnumber=0
asteriskcalldir=/var/spool/asterisk/outgoing/
asteriskuser=asterisk

echo "Using $callfile for $count iterations with a $delaybetween second delay"

while [ $callnumber -lt $count ];
do
	callnumber=`expr $callnumber + 1`
	echo "Call Number: $callnumber"
	cp $callfile $callfile.tmp
	chown $asteriskuser:$asteriskuser $callfile.tmp
	mv $callfile.tmp $asteriskcalldir
	sleep $delaybetween
done

This is used as follows:


sudo ./makecalls test.call 5 2

This will make 5 calls with a 2 second delay between the start of each call. Note: It needs to be run as sudo so it can change the ownership correctly. Credits: Thanks to Stuart Elvish for his help with getting this working so nicely. Happy load testing. Cheers, Mark

Ubuntu Server upgrade howto

Sun, 16 Nov 2008 19:53:00 --1000

One of the nice things about Ubuntu is how easy it makes it to upgrade between versions - typically start up update-manager and it will prompt you if there is a new version available and handle all the details of the upgrade for you. I've done numerous upgrades this way and never had any issues. However, I have a number of ubuntu servers which I manage and previously I have upgraded them by changing the /etc/apt/sources.list and then running "sudo apt-get dist-upgrade" Today I discovered that there is an automated script to handle it - which I presume is what powers the gui version behind the scenes. You need to make sure you have update-manager-core installed:


sudo apt-get install update-manager-core

Then you simply run do-release-upgrade


sudo do-release-upgrade

I've just upgraded two different servers from Feisty as it has recently stopped being supported. I upgraded them to Gutsy and then to Hardy which is a Long Term Supported version. The whole process took about an hour of which it took about 5 minutes of my attention to answer a few simple questions. Who said Linux is hard :-) Cheers, Mark

Howto refresh /dev/disk/by-uuid on Ubuntu

Sat, 16 Aug 2008 09:08:00 --1000

I was recently setting up a Ubuntu server and was partitioning it after it had been installed. There was lots of free space on the drives as the root partition was only using a small portion of the disk. After running fdisk to partition it and mkfs.ext3 on the partitions to format them I couldn't see them in /dev/disk/by-uuid. A quick google presented the solution:


sudo udevtrigger

According to the man page this makes udev request the kernel device uevents, which in essence makes it read the disk info again and show it all up so you can mount it happily. Cheers, Mark