CFMX - SOAP vs REST benchmarks
Currently the application consists of a AIR/Flex frontend and a CF backend which is using SOAP web services to communicate. I wanted to understand the level of overhead that the SOAP layer added to each coldfusion request and understand if it would be beneficial to move away from using SOAP to using a REST type direct XML format.
Test 1: SOAP CFC vs REST CFM
Initally I created two files which were a very simple hello world example to test each scenario.
soapservice.cfc:
<cffunction name="hello" access="remote" returntype="string">
<cfreturn 'hello world'>
</cffunction>
</cfcomponent>
and the CFM/REST version
xmlservice.cfm:
<cfcontent reset="true" type="text/xml">
<cfparam name="url.method" default="">
<cfswitch expression="#url.method#">
<cfcase value="hello">
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>Hello World</str></cfoutput>
</cfcase>
<cfdefaultcase>
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>Unknown Method</str></cfoutput>
</cfdefaultcase>
</cfswitch>
The results of the load tests were striking
The rest version of the code was 4 to 5 times faster, which meant 1000 second per minute instead 200 requests per second.
However, looking at the code I thought it was an unfair comparison as the SOAP one was using OO-style code which was more maintainable. I'm very much an OO person so I didn't want to have to live with horribly unmaintainable code just to get performance.
Test 2: SOAP CFC vs REST CFM with CFC
I created a new version of the CFM file which this time leveraged the soapservice.cfc which meant I could use the same code but bypass the seemingly very slow SOAP layer in Coldfusion.
xmlservicewithcfc.cfm:
<cfcontent reset="true" type="text/xml">
<cfparam name="url.method" default="">
<cfswitch expression="#url.method#">
<cfcase value="hello">
<cfset oSS = createObject("component","soapservice")>
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>#oSS.hello()#</str></cfoutput>
</cfcase>
<cfdefaultcase>
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>Unknown Method</str></cfoutput>
</cfdefaultcase>
</cfswitch>
So this version now uses a REST type request but leverages the CFC code for the functionality of "hello world".
This was not quite as good performance as the pure CFM code with only 3 to 4 times the performance of the SOAP version - but still not bad at all.
Test 3: SOAP CFC vs REST CFM with CFC with caching
One thing that is known to be reasonably costly in CF is the instantiation of CFC objects. A typical way to avoid this is to instantiate the objects once and then store them in the persistent application scope. This technique will work for lots of the code I am looking to optimise so I wanted to see how much impact it would have.
xmlservicewithcfccached.cfm:
<cfcontent reset="true" type="text/xml">
<cfapplication name="xmlservicewithcfcached" sessionmanagement="false" clientmanagement="false">
<cfparam name="url.method" default="">
<cfswitch expression="#url.method#">
<cfcase value="hello">
<cfif NOT structKeyExists(application,'oSS')>
<cfset application.oSS = createObject("component","soapservice")>
</cfif>
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>#application.oSS.hello()#</str></cfoutput>
</cfcase>
<cfdefaultcase>
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>Unknown Method</str></cfoutput>
</cfdefaultcase>
</cfswitch>
The performance of this version of the code was almost identical to the 1st version - i.e. 4 to 5 times faster than using the SOAP version but it more easily maintainable.
Is this a valid benchmark?
In a word no - in the strict sense of a reproducible benchmark. It was done on my development laptop as I don't have lots of spare machines lying around.
The specs of the machine are as follows:
- Intel Core 2 Duo T7200 @ 2GHz
- 3GB Ram
- Ubuntu 7.10 (gutsy) with standard 2.6.22-14-generic kernel
- Coldfusion 7.0.2 - Multiserver install - default config
- Apache JMeter 2.3.1 for load testing
Each thread in the tests ran 10,000 loops - so for example the tests where I ran 50 threads would be 10,000 x 50 = 500,000 requests.
The entire test was conducted over the course of one day and there were no reboots during the tests. Mostly the same programs were running but it was connected to the internet so minor variations in load and traffic could have affected the results.
What about CF8?
After I finished the testing on CF7 (which is what we currently have on our Production servers) I tried it out on the CF8 Install to see how it compared - the details are detailed below.
Show me the stats
So here are the pretty stats and charts that I created to understand all the data:
Graph 1:
Table 1 - figures for Graph 1:
Graph 2:
Table 2 - figures for Graph 2:
Need more stats?
I have also uploaded the spreadsheet that I created to generate these stats in the original OpenDocument format and also as an Excel file.
Conclusion
For me - I'll be definitely looking at using the REST methods further on any high traffic sites where I might previously have used SOAP.
What are we gaining
- Lots of performance - a 5 fold performance increase can't be ignored.
- Smaller data being transferred as no SOAP envelope overhead
- More control over data transferred
- Ability to use conventional debugging and caching techniques
What are we giving up
- Automatic handling of complex objects
- Lots of overhead
I'd love to hear your thoughts on this - what else am I giving up?
Cheers, Mark
