Lynch Consulting Blog - Java

Upcoming book review - Tomcat 6 Developer's Guide

Mon, 21 Dec 2009 21:12:00 --1000

I've just received a copy of the Tomcat 6 Developer's Guide from packt publishing to review. It's nice timing as I've been working with Tomcat 6 a bit lately and in the new year plan to move some of our production systems over to running Railo on top of Tomcat. After the extremely busy year Learnosity has had I'm looking forward to reading a few books over the break and coming back in the New Year with lots more ideas and technology to implement.

"Show Full Columns" problem with CFMX and MySQL solved

Sun, 01 Jun 2008 21:01:00 --1000

We've been performing some load testing on a new website we've developed and our helpful sysadmin noticed lots of queries happening on the DB that looked like this:


SHOW FULL COLUMNS FROM `dbname`.`tablename`

We weren't intentionally doing these queries but they were coming from somewhere. A bit of detective work via google found a very enlightening article about this problem. A quick read of this explained:

It turns out ColdFusion was asking Connector/J for the metadata on every field, which in turn triggered a SHOW FULL COLUMNS query for every varchar and text column returned.

It also went on to say that is had been fixed as of Connector/J 5.07. On reading the release notes it mentions that the

Driver detects when it is running in a ColdFusion MX server (tested with version 7), and uses the configuration bundle coldFusion, which sets useDynamicCharsetInfo to false (see previous entry), and sets useLocalSessionState and autoReconnect to true.

However, from my testing it wasn't doing it as we were still seeing the problem. However, we are running the MultiServer version of ColdFusion which sits atop Jrun4. So I added the parameter directly to the querystring:


&useDynamicCharsetInfo=false

So it now looks like:


jdbc:mysql://127.0.0.1:3306/mydbname?allowMultiQueries=true&useDynamicCharsetInfo=false

This has stopped the problem, and given a significant performance gain. On one of the sites I was load testing it gave a 12% throughput increase and on another one which had less queries it gave a 4% increase. Hope it helps. Cheers, Mark

Job Vacancy Sydney AU - Graduate or Junior OO Programmer

Fri, 09 May 2008 00:17:00 --1000

We're looking for a Computer Science Graduate or similar with strong Object Oriented programming skills and the ability to grasp cutting edge technologies quickly. If you are looking to work with the latest technogies and are a motivated self starter with a positive "can do" attitude this is the job for you. The ability to take problems and deliver solutions is a must. There will be significant on the job learning as we are always pushing the boundaries and using the latest and greatest technologies. We are currently working on projects using Actionscript/Flex, ColdFusion, PHP, and Javascript, and are leveraging technologies such as VOIP, SMS, Instant Messenger (Jabber XMPP) to deliver cutting edge solutions. Suitable applicants will have a Bachelors Degree in Computer Science or Engineering, or similar tertiary qualification. Remuneration commensurate with experience.

Responsibilities would include:

Developing new functionality for Asterisk and VOIP applications
Creating Rich Internet Applications with Adobe Flex
Creation of Desktop applications with Adobe AIR
Design and development of new Web applications

Must haves:

A robust foundation in Object Oriented programming
Demonstrated experience in at least one OO language
Motivation to learn and push the boundaries
Understanding of XHTML and CSS

Preferable:

Experience with PHP, ColdFusion, Farcry CMS, XML, Web Standards etc
Competency using linux-based tools (SSH, bash, etc)
Familiar with using source control tools (Subversion, etc)
Familiarity with Actionscript or Flash

This role may be full time, part-time or on a contract basis depending on the candidates skills and experience. You will be working in a casual workplace with flexible hours in the centre of Sydney. If this sounds like the job for you email your resume to mark@lynchconsulting.com.au if interested - no agencies please.

Getting Flex 4.0 up and running on Ubuntu

Thu, 08 May 2008 20:41:00 --1000

I came across a blog entry today from Mike Morearty about the flex 4 source tree. It also mentioned the swfdump tool which looked interesting I downloaded it and went to the bin directory to try swddump but it didn't work straight away complaining about a missing jar file. I thought this would mean a lot of pain to get it all compiling but I was very surprised. Here were the steps to get it working on Ubuntu 8.04 Hardy Heron.

Step 1

Get the code from the repository svn co http://opensource.adobe.com/svn/opensource/flex/sdk/trunk/ flex4 This will check out the code to a directory called flex4 under your current directory. I'm assuming you have Subversion installed but if you don't you'll need to run this first.


sudo apt-get install subversion

Get the tools to compile flex


sudo apt-get install ant ant-optional sun-java6-jdk

That wasn't hard now was it?

Compile

Go into the correct directory and call "ant" which will use the build.xml to build it all.


cd flex4
ant

Some couple of minutes later you should get the following message


BUILD SUCCESSFUL
Total time: 2 minutes 41 seconds

Now you have a build of Flex4. Next step is to figure out how to configure Flex Builder to use the new compiler. NOTE: This is not a finished version of Flex4 yet. Just the work in progress. Cheers, Mark

MD5 Checksum for Images (Binaries) in CF

Mon, 14 Jan 2008 21:00:00 --1000

I've been working on a piece of an application which is needs to verify that the images received have not been modified or tampered by using MD5 hashes. Note: If you want to skip my ramble and get to the solution scroll to the bottom for the example code and download. I thought it would be nice and easy to do it with the CF Hash function and on first attempt it appeared to be:



Hash: #hash(myTextFile)#

Gives:


Hash: 2A3643821420D5349665D2231842FD89

However, the hashes get created on by CF but get verified by an Flex application so I was using the as3corelib's MD5 function and couldn't get the hashes to match. So I decided to try another MD5 implementation to see which of my functions was wrong - unfortunately it turned out that both of them were wrong but for two different reasons. I'll cover the AS3 problem in a different post. The output from md5sum was as follows:


$ md5sum firework.jpg 
2346b6ab017de31688ee35949612db07  firework.jpg

This didn't match what I had generated. A bit more head scratching and I realised that the problem was that I was trying to do an md5 hash on a binary file but the CF Hash() only takes a "string". After loading the file with "readbinary" the hash function didn't work (no surprise) and a bit of digging around led me to use some Java functionality to handle the generation of the Hash.

This leaves us with a ByteArray digest of the file - which we need to convert to a familiar hex encoded MD5 Hash.



	
	
	
		
	
	

Binary Hash: #checkSumHex#

This gives the following checksum:


Binary Hash: 2346b6ab017de31688ee35949612db07

Bingo - all working nicely now. I've packaged it all up to make my life easier I've packaged it up into a nice CFC - Crypto.cfc.

Solution and Download

Here's the example code and a running demo of it. The Crypto CFC is available from the subversion repository. Here is the code to use it:





MD5 Sum calculated from hashBinary()
Hash Binary: #md5#

Hope it helps. Cheers, Mark

Complex CF Webservices problem - solved

Mon, 12 Nov 2007 23:13:00 --1000

Couple of days ago I ran into ColdFusion Webservices problem using cfinvoke and complex arguments. The Java Webservice I was talking to expected 3 arguments - 2 strings and 1 array of structures. So far, I only had to pass String arguments to webservices, which worked fine. For some reason, and correct me if I'm wrong, ColdFusion doesn't seem to translate complex variables into correct soap-equivalent format when sending the soap packet out. I got the following error - "Web service operation 'myMethod()' with parameters {args} could not be found". I was definitely calling the right method, but CF Array I was sending was not recognised by the Webservice. I was pretty determined to get this working and decided to build the SOAP envelope manually and send it to the webservice using a CFHTTP POST. This actually worked very nicely and this is how I did it: [More]

JMeter load testing - Regex problem solved

Tue, 09 Oct 2007 08:19:00 --1000

I've been having fun playing with JMeter to do some load testing on some of our applications recently and I came across this problem which had me stumped. Eventually my friendly sysadmins put me out of my misery. So to prevent other people from the same issues - here is the problem and the solution. JMeter is a great load testing application but it can be a bit tricky to when you are learning it. For the particular testing I needed to get it to do the following:

Login in as a user
Retrieve the valid userid
Create a new test for the user
Get the testid back
Load the test details
Submit the test

I pretty quickly got everything working with some regex's to extract the userid and testid as required and my test case looked like the picture above. Everything was working except the final submission of the test, which was failing as the regex variable was always using the default value, despite the fact that is was working for the previous step. It turns out that the regex needed to be nested inside the request it was referring to or it would be called after every request, which in most cases would return the default value as the regex wouldn't match. Once I put the regex's inside the requests the were not constantly being reset and everything worked a treat. Hope it helps. Mark

Cross platform Database IDE

Sat, 15 Sep 2007 05:29:00 --1000

Ever since I've moved to using Linux full time I've been looking for a solid Database IDE. I've tried and tested quite a few.

Aqua Data Studio

This is my current DB IDE of choice and has some significant features in the new version 6.5 that means I'll be staying with it for a while, particularly as it runs on Linux, Macs and Windows identically.

Schema Comparison - very handy for keeping development and production db's schemas in sync.
Charts & Pivot Tables - makes it easy to create adhoc graphs withouth having to resort to Excel/OpenOffice Calc
Morph to Delimited List - A simple but powerful function that saves a lot of time. Allows you to take the results of a sql query and convert to comma seperated list. I've done dozens of adhoc CF scripts to handle this before :-)
Subversion Support - As we use subversion for our code it's handy to have it built in.

MySQL GUI tools (Query Browser & Admin)

These have some very nice functionality but have always found them to be pretty unstable - they just crash a lot with no explanation.

Navicat

This is pretty good option, but the Linux version is the poor cousin to the Windows & Mac versions.

Love to hear any comments feedback or suggestions.

howto document your code with ASDoc

Thu, 23 Aug 2007 20:52:00 --1000

Last night I was trying to document our AS3 code, using ASDoc, for a project that we have been working on. For those of you who don't know, ASDoc is a command line utility which allows you to generate code documentation in HTML format. A good example of what ASDoc output looks like would be, Flex 2 Language Reference. Documenting code wasn't as easy as I thought it would be, so I thought I'd share this with you and hopefully it will save someone some time. I use an Intel Mac and this is what you need to do to get your basic code documentation generated successfully:

1. In your terminal window go to your Flex SDK bin directory. Typically this is located under /Applications/Adobe Flex Builder 2/Flex SDK 2/bin

2. From "bin" directory run this command:

./asdoc -doc-sources=/Users/marko/Documents/workspace/yourFlexProject/  -source-path=/Users/marko/Documents/workspace/yourFlexProject/

If you have any external Libraries in your Flex project you will need to include them in your command: -external-library-path=/path_to_your_swc/myLib.swc The key thing to remember for Mac users is that you need to run the command from Flex SDK bin directory and use -source-path option to tell the compiler where the source is located. -doc-sources will tell the compiler to generate output for the current directory and all subdirectories recursively. If you run this utility from any other directory other than "bin", you will get NoClassDefFoundError. I haven't tried running ASDoc in Windows yet, but it should be very similar to running it on a Mac. And since ASDoc is a command line java utility, you would think it would run on Linux too. Well, for all you Linux users out there, this is the message I got from the compiler: Error: Could not create toplevel.xml: Unknown OS for ASDoc. ASDoc will only run on Windows and Mac OS X. That last sentence is pretty self explanatory :) ASDoc is nicely documented here. Cheers
Marko

Install SSL certificate in ColdFusion

Wed, 20 Jun 2007 03:18:00 --1000

If you are using self signed ssl certs for web-services you may run across the following error:


 javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

If you browse to the URL using Firefox you will likely get a certificate warning and a prompt which allows you to continue anyway despite the warning. With CF you can't do this so you have to add the certificate to the list of "known" certs using the java keytool.

Instructions:

First - save the certificate to your desktop using Internet Explorer or the Cert Viewer Plus plugin for firefox. Then open a command prompt and go to cfusion jre/bin directory


$ cd /opt/jrun/jre/bin

From here you should have see access to the keytool application which is how you modify the security settings. You can show all certs:


./keytool -list -storepass changeit -noprompt -keystore ../lib/security/cacerts

or Import a new cert:


./keytool -import -storepass changeit -noprompt -alias mynewcert
-keystore ../lib/security/cacerts -trustcacerts -file mynewcert.cer

Explanation of the options:

alias - this is a friendly name for the cert that you can specify
storepass is the password for the security store. Unless you have already changed it it will be 'changeit'
file - this is the certificate file you have downloaded in your browser.

If it reports the following error:


keytool error: java.io.FileNotFoundException: ../lib/security/cacerts
(Permission denied)

Check that you have write permissions to the cacerts file. Note: If you are a Mac user you will find that ColdFusion may not have JRE directory installed under JRun. This is because CF uses OS X Java install, so you will need to run your keytool command from the following directory:


/Library/Java/Home/bin

To find out the location of your Java Home, log in CF Administrator and click on Settings Summary. Hope it helps and thanks to the other people who have posted about this. Sources:

Using Keytool to Import SSL Certificates into Sun JDK

Useful package for Ubuntu - ubuntu-restricted-extras

Tue, 12 Jun 2007 05:34:00 --1000

I just happened across this package today - and while I had pretty much everything already installed it is a handy way to add some of the essential packages which cannot be distributed in the main Ubuntu distribution due to licensing issues. It installs the following:

Flash Player 9
Sun Java 6.0 runtime and browser plugin
Microsoft Core Fonts - Arial, Verdana etc
GStreamer Video Codecs

Installing is as simple as:


sudo apt-get install ubuntu-restricted-extras

or search for it in Synaptic. Cheers, Mark

Memcached configuration options

Tue, 08 May 2007 22:57:00 --1000

From the previous discussion with Michael Long I realised that the memcached solution doesn't force you to only have a single centralised cache store. What it does is allow you to configure you caching in a more flexible manner (at the cost of adding another layer in). For example, you could configure you caching in a number different ways: Note that these are theoretical and I haven't yet tested these configurations, keen to hear if anyone else has/or does though.

Stable Performance

1 memcached instance per server
2 or 3 CF intances per server

Pros:

Ability to rolling restart CF instances without clearing cache
Ability to clear entire cache (or parts of it) for the server via cf template
More efficient memory usage than storing 1 copy per instance
Each instance can share work of other instances i.e. cache is generated by one server instead of each server

Cons:

Additional memcached layer

Maximum Performance

2+ memcached instances shared across cluster
2 or 3 CF intances per server

Pros:

Ability to rolling restart CF instances without clearing cache
Ability to clear entire cache for cluster (or parts of it) for the server via cf template
Very efficient memory usage across entire cluster
Each instance can share work of other instances across cluster i.e. cache is generated by one server instead of each server

Cons:

Performance degration of particular pages if memcached server holding specific cache is down.

CF Emulation

1 memcached instance per cf instance
1 CF intances per memcache

Pros:

Each CF instance has dedicated cache
CF memory is only used for requests not caching

Cons:

Dependancy on memcached

One of the reasons that I'm keen to move the cache data out of the CF process is that I've found CF to become unstable when it's using all it's memory and this is an elegant way of removing some of the memory usage. Also the JVM seems to have limits on how much ram it can use. This way it leaves all the memory in CF for processing requests.

Faster alternative to CFWDDX - java serialize

Tue, 08 May 2007 09:24:00 --1000

As part of looking at implementing the memcached client for CF I've been looking at using the native Java serialize methods instead of the cfwddx serialization that I put in first time to get it to work. With a bit of help from Max Nyman I've created some tags to handle java serialization in a similar manner to cfwddx. Some rough benchmarks on this using an array of numbers between 100 and 10,000 elements and repeating between 100 and 10000 times resulted in the following general observations: JavaSerialize on average took 60% of the execution time of cfwddx for a serialize and deserialize operation. In addition to this - the data length was typically 60% of the wddx length. This has significant benefits when it is used for caching in memory. It means you can fit more data in the same amount of ram or use less ram! The tag is available here To serialize from CF use the following:

To serialize to CF use the following:

Note that fromCF function returns a binary object - which would need to be converted to base64 if you want to transfer them via a form submission. Another Note - the customtag does add some overhead, so I would suggest putting the code directly into any high performance code (eg caching tags, like cfaccelerate and memcached) but for general use it is more convenient in the custom tag.

MemCached Client for CFMX - alpha version

Wed, 02 May 2007 07:10:00 --1000

I've been looking at using memcached in order to speed up a few sites that I've been working on.

What does it do?

Memcached is basically a large RAM cache that is distributed across a network and can be shared between many machines. Many CF applications use the application scope to cache information - including the wonderful CF_Accelerate tag. This works great for single servers (and instances) but once you need to scale beyond this you end up with each server having duplicate copies of the cached data. Memcache gives you a central place to store this info and allows you to move it out of the ColdFusion memory space which uses as much memory on the network as you have available. Plenty more info on the memcache site.

How do I use it?

Install memcached on a machine (using apt on ubuntu is the following)


sudo apt-get install memcached

Then run it (the -vv gives extra debugging info):


memcached -vv

Download the java api for memcached (for CF7 you need the 1.3.2 version unless you have upgraded your VM) Copy it into the lib folder for CF and restart CF eg:


sudo cp java_memcached-release_1.3.2.jar /opt/jrun/lib

Get the CFC:

Download the MemCached.cfc from SVN.
It can also view in the friendlier trac

Then try the following test code:



	oMemCached = createObject("component","MemCached").getPoolInstance();
	oMemCached.shutdown();
	
	//List of server:ports to use

	serverList = '127.0.0.1:11211';
	
	//Create a new memcached object - this could be stored in application
	
	oMemCached = createObject("component","MemCached").init(serverList=serverList);
		
	//Create some data to store
	
	aTemp = ArrayNew(1);
	aTemp[1] = '34343';
	aTemp[2] = '134343';
	aTemp[3] = '234343';
	aTemp[4] = '334343';
	aTemp[5] = '434343';

	//Save to keys (one add and one set)

	oMemCached.add('key',aTemp);
	oMemCached.set('key2',aTemp);

The cfc is not production ready - but is working in some basic tests. One area that is not working (as far as I can tell) is the failover of servers. Have a play, and all comments welcome.

OO Part 1: From code libraries to instances

Mon, 15 Jan 2007 05:49:00 --1000

Code libraries

When people first start using CFC's they often use them as a handy way to group related functions together. For example, if you could put all the functions that deal with users into the same cfc and you may have something like the following pseudocode:


User.cfc
 function getUserName(id)
 function getFirstName(id)
 function getLastName(id)
 function authenticate(username,password)
 function getAllUsers()
 function getUserCount()
 function createUser(username,password,firstname,lastname)
 function updateUser(id,username,password,firstname,lastname)

While this is a useful and valid way to use CFC's it isn't using them to their full potential.

Dealing with a single user

When working in object oriented way we try to discern what the different objects in a system would be. In this case User would be an object - and the user would know certain things about itself (name, username, password, date created) and just as importantly it would not know certain things (eg: total number of users in system)


User.cfc
 var id  //We store the user id in the cfc

 function init(id)
 function getUserName() //id is not required as it is already in the instance
 function getFirstName() 
 function getLastName()
 function updateUser(username,password,firstname,lastname) //also no id required here

This "instance" of a user gives us access to all the properties that a user knows about but shields us from many of the functions that are not relevant. We now have a user that we can easily conceptualise and work with without having all of the other methods such as getAllUsers() or getUserCount() getting in our way. It also allows a team of programmers to have a set of guidelines as to where code should live. If it deals with a single user it will be in the user CFC

But how do we create a new user

You'll notice that the above code has no way of creating a user - this is because it doesn't make sense for a user to be able to create another user (we're not talking about giving birth here). So how do we create a user. We need a place to put the create user method that is outside of the User.cfc - and I have typically used something like UserManager.cfc. UserManager is also responsible for all functions that deal with more than one user.


UserManager.cfc
 function authenticate(username,password)  // We don't have a valid user at this point
 
 function getAllUsers()
 function getUserCount()
 function createUser(username,password,firstname,lastname)

This works really well in that there is now a logical place to put all of the code relating to Users and if all the people working on the application know where things should go.

Two CFC's for each object type?

However having two CFC's for each object type can get really messy, espcially when you are drawing Class diagrams. The java solution to this lies in the use of Static methods. Static methods (or Class methods) are functions that are available on the Class as opposed to the Instance - i.e. when we are dealing with the concept of Users as opposed to a specific user Steve Bennett who works on level 5. So now we are back to having the following:


User.cfc
 var id
// static methods

 function authenticate(username,password)
 function getAllUsers()
 function getUserCount()
 function createUser(username,password,firstname,lastname)
 function init(id)

//Instance methods

 function getUserName() //id is not required as it is already in the instance

 function getFirstName() 
 function getLastName()
 function updateUser(username,password,firstname,lastname) //also no id required here

But aren't we back where we started with everything jumbled together? Not really, we now have a obvious set methods that deal with a single instance and another set that deal with multiple users. In this example you only have half the number of methods to look through when you are dealing with a specific user (i.e. all the instance methods) and the same when you are dealing with multiple users.

Enforcing it in ColdFusion

However Coldfusion doesn't naturally handle the differentiation of Static methods from Instance methods - however with a bit of ingenuity (aka hackery) we can enforce it in our CFC's. I've got a working prototype of this and I'll post some more on it shortly.

Lynch Consulting Blog - Java

Upcoming book review - Tomcat 6 Developer's Guide

"Show Full Columns" problem with CFMX and MySQL solved

Job Vacancy Sydney AU - Graduate or Junior OO Programmer

Responsibilities would include:

Must haves:

Preferable:

Getting Flex 4.0 up and running on Ubuntu

Step 1

Get the tools to compile flex

Compile

MD5 Checksum for Images (Binaries) in CF

Solution and Download

MD5 Sum calculated from hashBinary()

Complex CF Webservices problem - solved

JMeter load testing - Regex problem solved

Cross platform Database IDE

howto document your code with ASDoc

Install SSL certificate in ColdFusion

Instructions:

Useful package for Ubuntu - ubuntu-restricted-extras

Memcached configuration options

Stable Performance

Maximum Performance

CF Emulation

Faster alternative to CFWDDX - java serialize

MemCached Client for CFMX - alpha version

What does it do?

How do I use it?

OO Part 1: From code libraries to instances

Code libraries

Dealing with a single user

But how do we create a new user

Two CFC's for each object type?

Enforcing it in ColdFusion

Further reading