Lynch Consulting Blog - Linux

Lego Digital Designer on Ubuntu

Sat, 28 Aug 2010 17:48:00 --1000

Having seen the Lego Digital Designer I wanted to see if I could get it to run on Ubuntu, as there is a distinct lack of windows machines at home for my son to use. It was all relatively painless and it's quite amazing how far Wine has come.

Installing Lego Digital Designer

Go to the Lego Digital Designer site and download the Windows version. I got LDD 3.1.3 which was the latest version available at the time. Now - to make it all work you need to install Wine and I used the latest version from the wine PPA team. Add the wine ppa:


sudo add-apt-repository ppa:ubuntu-wine/ppa
[sudo] password for markl:

You'll then see:


Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv 883E8688397576B6C509DF495A9A06AEF9CB8DB0
gpg: requesting key F9CB8DB0 from hkp server keyserver.ubuntu.com
gpg: key F9CB8DB0: public key "Launchpad PPA for Ubuntu Wine Team" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Now to update the list of available software:


sudo apt-get update

And then install wine and winetricks


sudo apt-get install wine winetricks

Wine lets you run Windows software on other operating systems, in this case Ubuntu. Wine tricks provides a simple way to install native versions of some of the libraries that are not 100% compatible - by looking at the Wine App DB I discovered that it had problems with missing font's (Tahoma) and scrolling which was fixed by installing Quartz.


winetricks tahoma quartz

Then install double click on the install file for LDD and it should be up an running in no time. If you get a warning about it not being Executable you may need to right click on the application and select Properties - and then from the Permissions tab select "Allow executing file as program" Son is now very happy with Lego Digital Designer :-)

Howto find files newer than a specific date using command line

Mon, 10 May 2010 23:19:00 --1000

While doing some server admin tasks the other day I needed to find all the files newer than a certain date. Using just the command line tools it was relatively simple but not obvious, so this is a not to self. The find utility has an option to find a file newer than another file. By creating and empty file with a specific creation date we can do the search:


touch timestamp -d 2010-01-01

To show all files newer than 2010-01-01 use:


find . -newer timestamp

Or to create a tar archive of them use xargs like so:


find . -newer timestamp | xargs tar -rf /root/filesnewerthan-2010-01-01.tar

Easy. Mark

MySQL 5.1 logging changes - Log to DB and runtime config

Fri, 29 Jan 2010 08:15:00 --1000

While browsing around the MySQL site last night I discovered a number of nice new features of mysql 5.1 that relate to logging. These are:

Logging to DB instead of log files
Runtime configuration of logging.

Logging to DB instead of log files

Coming from a web development background rather than a sysadmin background I'm far more comfortable manipulating and analysing data using SQL. So to be able to log all the queries or just the slow queries for an application to the db during application development or load testing is a huge benefit. To enable logging to DB you can add the following to your my.cnf


log_output = TABLE

The logs will be written to the 'slow_log' and 'general_log' tables in the mysql database. Note - logging to tables has more overhead than logging to file, so would suggest using it primarily for development purposes. Full details of the options are on the mysql manual on log tables

Runtime configuration of logging.

This allows you to turn on and off logging without restarting MySQL - which just saves a little bit of time and makes it much nicer for debugging problems. To turn on the logging of all queries run:


SET GLOBAL general_log = 'ON';

And for just the slow query log:


SET GLOBAL slow_query_log = 'ON';

And to turn them both off use:


SET GLOBAL general_log = 'OFF';
SET GLOBAL slow_query_log = 'OFF';

If you also want to see queries not using indexes in the slow query log you can set the following variable:


SET GLOBAL log_queries_not_using_indexes = 'ON';

Hope it helps, Mark

Jmeter over SSH Socks proxy

Thu, 07 Jan 2010 10:25:00 --1000

I've been doing some testing recently where I need to connect via SSH server to a remote network to run some load testing. To do this I used a SSH sock proxy like I have previously blogged about. So I fired this up so that I could review the site I wanted to look at. It worked a charm through firefox but there is no where to set up the proxy in jmeter. To make it work you need to let the JVM know what proxy to use like so:


java -DsocksProxyHost=localhost -DsocksProxyPort=8080 -jar ApacheJMeter.jar

No jmeter will use the socks proxy on port 8080 on my local machine. Nice.

Learnosity are looking for a Web Ninja at a Mid to Senior level

Fri, 04 Dec 2009 16:03:00 --1000

4 December 2009, Learnosity are looking for a Web Ninja at a Mid to Senior level.

About Learnosity

Learnosity develop cutting edge tools for teachers and educators. Our flagship product Learnosity Voice uses the telephone to enable language students and teachers to interact on a one to one level. Our service:

Makes it practical for students to practice Oral and Aural skills
Is efficient and effective for teachers, as they can listen to each student individually at a time to suit them
Can be used for homework assignments or “High Stakes Assessments”

We are continuing to grow our core development team and we need another great developer to help us keep up with demand.

We need someone who can:

Use Javascript or Actionscript to create great user interfaces
Develop highly scalable web applications
Cut code with the best in the world

You will also need to be:

energetic with a butt kicking attitude
ready to create cutting edge web 2.0 apps
keen to continue learning new technologies
able to have a conversation with non technical people

You'll need:

3 or more years of programming experience
Expert in at least on Client side language (Actionscript or AJAX)
Expert in at least one Server side language (eg PHP, Java, ColdFusion, etc)
Understanding of Object Oriented design
Understanding of XHTML and CSS

It would be good if you have:

A degree in Computer Science, Engineering or similar.
been working with open source tools
been playing around with iPhone/Android applications
experience with some of Linux/VOIP/SIP/Asterisk/Jabber/XMPP

This is a full time role and you will be working in a casual workplace with flexible hours in the Sydney CBD. If this sounds like the job for you, email a covering letter explaining why you'll be great and your resume to mark@learnosity.com - no agencies please.

Normalising audio with sox

Wed, 25 Nov 2009 22:48:00 --1000

Update - as of sox 14.3.0 it is much simpler than described below - simply do: sox --norm before.wav after.wav Seem to be on a roll with Sox tonight. Also figured out how to normalise audio files nicely. The first thing to do with sox is to get it to calculate the max volume adjustment possible:


sox before.wav -n stat -v

This will return a number like: 4.234 You can then call sox again using this number:


sox -v 4.234 before.wav after.wav

And if you're feeling particularly good you can put the whole command on a single line as follows:


sox -v `sox before.wav -n stat -v 2>&1` before.wav after.wav

Easy when you know how.

Removing silence from audio using sox

Wed, 25 Nov 2009 22:11:00 --1000

I've had a look at this a few times before and each time I've gotten a little bit further and then gotten frustrated and failed. However, this time I finally managed to get past all the hurdles and jump through all the hoops to make this work. Scenario is a follows. Audio file is comes in in alaw or ulaw format and we want to trim any silence before and after it. First step is to convert it from alaw to pcm encoded wav file as SOX has issues filtering silence in alaw files


sox -t alaw in.alaw -c1 -2 -r8000 -e signed-integer temp.wav

Then we do the trimming in one fell swoop.


sox temp.wav out.wav silence 1 0.1 0.1% reverse silence 1 0.1 0.1% reverse

A huge thanks to this article for giving such a thorough overview of the how silence detection works in sox. Too easy, when you know how :-)

Apache Deflate Howto

Sun, 15 Nov 2009 19:15:00 --1000

For my own reference, settings to turn on apache mod_deflate. This sets turns it on for everything except gif,jpeg,png or mp3, as these are already well compressed.


        SetOutputFilter DEFLATE
	SetEnvIfNoCase Request_URI \
		\.(?:gif|jpe?g|png|mp3)$ no-gzip dont-vary

Cheers, Mark

Windows File Sharing (SMB/CIFS/Samba) over SSH

Sat, 14 Nov 2009 13:35:00 --1000

While working with a client recently setting up a Netgear VPN so he could securely access his internal file server. The VPN setup was straightforward but every time the VPN client connected to the VPN server the VPN server/firewall would crash - leaving no connectivity. In order to come up with a reliable solution to this we decided to use the SSH server we had available and tunnel the windows sharing across the local port forwards, much simpler and more reliable. Thanks to this article it was a breeze to set up. Steps are as follows:

Create loopback adapter on windows
Configure loopback adapter on windows
Reboot
Configure SSH connection
Test it all out

Create loopback adapter on windows

We'll give your computer an additional (fake) IP address, and we'll port forward to that address instead of the computer's real IP. Windows XP will continue to do file sharing on the real IP address. We'll assign it an IP of 10.0.0.1 (that's what we configured putty to use above.)

System->Control Panel->Add Hardware
Yes, Hardware is already connected
Add a new hardware device (at bottom of list)
Install the hardware that I manually select
Network adapters
Microsoft , Microsoft Loopback Adapter
(Go through the installation procedure.)

Configure loopback adapter on windows

Open your new fake ethernet adapter (Network Connections) , enter a made-up IP address (I suggest 10.0.0.1, which is a privately routable address that most folk don't use.)
Enable Client for Microsoft Networks.
Disable File and Printer Sharing for Microsoft Networks
Enable Interent Protocol (TCP/IP)
Click on properties for TCP/IP.
Enter your chosen IP address (10.0.0.1), subnet mask (255.255.255.0). You can leave gateway blank.
Under advanced->WINS, Enable LMHosts Lookup and Disable NetBIOS over TCP/IP

Reboot

In order to make it all work now it he appropriate time to reboot so windows initialises everything correctly.

Configure SSH connection

Download Putty
Enter IP address
Enter Auth Key (if using SSH keys)
Enter Port forwards for: (these connect the ports on you local machine to
- 10.0.0.1:137 to 127.0.0.1:137
- 10.0.0.1:138 to 127.0.0.1:138
- 10.0.0.1:139 to 127.0.0.1:139
- 10.0.0.1:445 to 127.0.0.1:445
Save the config.

Test it all out

Now to connect you need to do the following steps:

Open putty, load the settings and connect.
Open Exporer and type in: \\10.0.0.1\

You should now be connected to your remote windows system over a secure encrypted tunnel. Cheers, Mark

cp "No space left on device" problem - Solved

Sat, 14 Nov 2009 13:12:00 --1000

One of my backup scripts started reporting errors recently about running out of space:


cp: cannot create regular file `filename in here': No space left on device

Running the command df was showing lots of free space. However, a quick google on cp "no space left on device" turned up the suggestion to try:


df -i

This showed up the problem straight away - I had run out of inodes. Once the problem is identified it is generally easy to resolve, so I figured out that there was a cron job that was running a wget task and not discarding the output, and so had saved 600,000+ files in their home directory. However, when I tried to delete them I encountered another problem:


# rm filepattern*
bash: /bin/rm: Argument list too long

There were so many files that I couldn't use standard delete commands. Another quick google turned up this gem: I ran the following command to check that I was going to delete the correct files:


find . | grep filepattern

And then added the command to actually do the delete:


find . | grep filepattern | xargs rm

One final thing was to fix up the cron job that was causing the problem. Adding the parameter --delete-after to wget kept the directory nice and clean. All fixed. One of the things this has reinforced for me is how important it is to have /home on a seperate partition. If this had not been the case then the problem would have taken longer to happen (due to larger filesystem) but would have been more destructive - as all systems and processes would not have been able to create new files.

Chinese characters not working cfdocument for PDF exports - fixed

Mon, 26 Oct 2009 16:38:00 --1000

I've just spent quite a few hours grappling with an annoying PDF export issue where the font's were not displaying correctly. This is on a CFMX 7 (7.0.2) on Ubuntu Linux system. All the DB storage and application was using UTF8 throughout, but the PDF exports were showing nothing when they should have been showing Chinese characters. To fix this up you need to do the following:

1. Need fonts installed on system:


sudo apt-get install ttf-arphic-bsmi00lp ttf-arphic-gbsn00lp

2. Need fonts installed in CFAdmin

Go to CFAdmin and Font Management
Select directory /usr/share/fonts/truetype/arphic
Select "Add" and you should see 2 new fonts added.

3: CFdocument Hotfix for CF7.0.2

The CFDocument hotfix for 7.0.2 provides some essential fixes and makes this work - Follow the instructions to install: http://kb2.adobe.com/cps/402/kb402093.html

Configure cffont.properties file

The final step is to set the cffont.properties file so that it will check additional font sets for any characters it doesn't know about - i.e. Chinese characters. Each line in the file is like a CSS font-family rule, i.e. it will start at the left and look for the correct font to render the text. I appended the two extra font names to ensure they are checked, so the file looks like this:


defaultbasefont=
dialog=Arial, Helvetica, AR PL SungtiL GB,AR PL Mingti2L Big5
dialog.bold=Arial Bold, Helvetica-Bold, AR PL SungtiL GB,AR PL Mingti2L Big5
dialog.italic=Arial Italic, Helvetica-Oblique, AR PL SungtiL GB, AR PL Mingti2L Big5
dialog.bolditalic=Arial Bold Italic, Helvetica-BoldOblique, AR PL SungtiL GB, AR PL Mingti2L Big5
dialoginput=Courier New, Courier, AR PL SungtiL GB, AR PL Mingti2L Big5
dialoginput.bold=Courier New Bold, Courier-Bold, AR PL SungtiL GB, AR PL Mingti2L Big5
dialoginput.italic=Courier New Italic, Courier-Oblique, AR PL SungtiL GB, AR PL Mingti2L Big5
dialoginput.bolditalic=Courier New Bold Italic, Courier-BoldOblique, AR PL SungtiL GB, AR PL Mingti2L Big5
serif=Times New Roman, Times-Roman, AR PL SungtiL GB, AR PL Mingti2L Big5
serif.bold=Times New Roman Bold, Times-Bold, AR PL SungtiL GB, AR PL Mingti2L Big5
serif.italic=Times New Roman Italic, Times-Italic, AR PL SungtiL GB, AR PL Mingti2L Big5
serif.bolditalic=Times New Roman Bold Italic, Times-BoldItalic, AR PL SungtiL GB, AR PL Mingti2L Big5
sansserif=Arial, Helvetica, AR PL SungtiL GB, AR PL Mingti2L Big5
sansserif.bold=Arial Bold, Helvetica-Bold, AR PL SungtiL GB, AR PL Mingti2L Big5
sansserif.italic=Arial Italic, Helvetica-Oblique, AR PL SungtiL GB, AR PL Mingti2L Big5
sansserif.bolditalic=Arial Bold Italic, Helvetica-BoldOblique, AR PL SungtiL GB, AR PL Mingti2L Big5
monospaced=Courier New, Courier, AR PL SungtiL GB, AR PL Mingti2L Big5
monospaced.bold=Courier New Bold, Courier-Bold, AR PL SungtiL GB, AR PL Mingti2L Big5
monospaced.italic=Courier New Italic, Courier-Oblique, AR PL SungtiL GB, AR PL Mingti2L Big5
monospaced.bolditalic=Courier New Bold Italic, Courier-BoldOblique, AR PL SungtiL GB, AR PL Mingti2L Big5

The cffont.properties file is located in /opt/jrun4/servers/{instance name}/cfusion.ear/cfusion.war/WEB-INF/cfusion/lib on the jrun multiserver version.

5. Restart CF

Once you restart CF you documents should be coming out with their Chinese fonts intact. I'm sure there are additional fonts to add to get it working natively with all the other asian languages, so I'll add to this as and when I need/discover them. Please feel free to post a comment if you find any more font's that should be added to the list.

Two quick server tips

Sat, 15 Aug 2009 08:12:00 --1000

Here are two quick tips that I've recently found for server admins:

Automatically fixing file system errors
Ignoring directories from updatedb

Automatically fixing file system errors

If you have a remote server, i.e. at a data center 10km or 1000km away this should prevent some panic when rebooting the server remotely. Ext2/3 will do a filesystem check after a certain number of reboots or time. Most of the time any errors are fixed automatically, but certain errors by default require the root shell and the administrator to fix them. I've seen a few of these happen, but I have always replied yes to the prompts as I don't know enough about file systems to fix it I said no. So, to prevent the need to rush to a data center and plug in a keyboard and mouse just to press the "Y" key there is and option to automatically assume yes. On Ubuntu in the file /etc/default/rcS you need to change the following:


FSCKFIX=no
to 
FSCKFIX=yes

Ignoring directories from updatedb

If you have a backup server you may not want it to update the locate db for all your backup files, as it can take a very long time. To tell locate to ignore a directory you need to add it to the PRUNEPATHS line in the /etc/updatedb.conf file like so:


PRUNE_BIND_MOUNTS="yes"
PRUNEPATHS="/tmp /var/spool /media /srv"
PRUNEFS="NFS nfs nfs4 afs binfmt_misc proc smbfs autofs iso9660 ncpfs coda devpts ftpfs devfs mfs shfs sysfs cifs lustre_lite tmpfs usbfs udf rpc_pipefs"

In the above code I have added the /srv entry to make it ignore all my backups which are held under the /srv directory. Note: These tips were tested on Ubuntu linux, other distributions will have similar functionality but the file locations may vary. Cheers, Mark

HOWTO find files that are not world readable

Mon, 18 May 2009 21:47:00 --1000

I ran across some permissions errors on content that some users uploaded to a website. The files uploaded were not world readable so were coming up as forbidden: This simple find command shows the offending files:


/usr/bin/find . -type f ! -perm -004

A quickie but a goodie. Cheers, Mark

Non-case sensitive filesystem on Linux - HOWTO

Mon, 02 Feb 2009 20:58:00 --1000

I was setting up a ColdFusion web application on my dev machine and ran into lots of errors because of the case sensitivity of Linux (which i was using) versus the non-case sensitivity of windows (where the application was developed). I had a quick look to see if I could fix it up but quickly figured out that there were too many occurrences and so it was likely not worth fixing up at this point. To that end I set up a loopback vfat filesystem so that I could have a directory on my machine mounted as non-case sensitive. Note: this is not recommended for production systems - but is a handy fix for a temporary development problem.

Create a virtual disk


dd if=/dev/zero of=virtual.dsk bs=1048576 count=150

Format it


mkfs.vfat virtual.dsk

Mount it


sudo mkdir /mnt/vfat
sudo mount virtual.dsk /mnt/vfat -t vfat -o loop,owner,group,umask=000

You can set this up to mount every time by putting the following line in your /etc/fstab


/path/to/virtual.dsk /mnt/vfat  vfat  loop,owner,group,umask=000  0 0

Note - the same trick works on OSX in reverse. I.e. if you are developing on a mac, but deploying to a linux production environment you can create a virtual disk with a case-sensitive filename, which will mean that any case sensitive issues get picked up on your dev machine, and not on your production/staging machines. Cheers, Mark

Formatting XML for easier reading with Tidy

Wed, 28 Jan 2009 07:49:00 --1000

I've been working on performance tuning a clients application which makes heavy use of XML and in order to get my head around the structure of the XML I wanted to have a browse through the XML data. So I did a Charles capture of the data to have a look. Unfortunately the data was formatted in a single line with no line breaks and as it was just under 1MB it upset all my usual editors (Eclipse & Gedit) and made them become completely unresponsive. After a bit of hunting I finally found the right tool for the job - HTML Tidy. With the following command I was able to get the XML formatted nicely which made my editors play nice and made it a lot easier to get an understanding of the data in the XML packet.


tidy -xml -i input.xml > formatted.xml

This takes the input.xml and indents it nicely and saves it as formatted.xml. Cheers, Mark