CFMX - SOAP vs REST benchmarks
Currently the application consists of a AIR/Flex frontend and a CF backend which is using SOAP web services to communicate. I wanted to understand the level of overhead that the SOAP layer added to each coldfusion request and understand if it would be beneficial to move away from using SOAP to using a REST type direct XML format.
Test 1: SOAP CFC vs REST CFM
Initally I created two files which were a very simple hello world example to test each scenario.
soapservice.cfc:
<cffunction name="hello" access="remote" returntype="string">
<cfreturn 'hello world'>
</cffunction>
</cfcomponent>
and the CFM/REST version
xmlservice.cfm:
<cfcontent reset="true" type="text/xml">
<cfparam name="url.method" default="">
<cfswitch expression="#url.method#">
<cfcase value="hello">
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>Hello World</str></cfoutput>
</cfcase>
<cfdefaultcase>
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>Unknown Method</str></cfoutput>
</cfdefaultcase>
</cfswitch>
The results of the load tests were striking
The rest version of the code was 4 to 5 times faster, which meant 1000 second per minute instead 200 requests per second.
However, looking at the code I thought it was an unfair comparison as the SOAP one was using OO-style code which was more maintainable. I'm very much an OO person so I didn't want to have to live with horribly unmaintainable code just to get performance.
Test 2: SOAP CFC vs REST CFM with CFC
I created a new version of the CFM file which this time leveraged the soapservice.cfc which meant I could use the same code but bypass the seemingly very slow SOAP layer in Coldfusion.
xmlservicewithcfc.cfm:
<cfcontent reset="true" type="text/xml">
<cfparam name="url.method" default="">
<cfswitch expression="#url.method#">
<cfcase value="hello">
<cfset oSS = createObject("component","soapservice")>
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>#oSS.hello()#</str></cfoutput>
</cfcase>
<cfdefaultcase>
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>Unknown Method</str></cfoutput>
</cfdefaultcase>
</cfswitch>
So this version now uses a REST type request but leverages the CFC code for the functionality of "hello world".
This was not quite as good performance as the pure CFM code with only 3 to 4 times the performance of the SOAP version - but still not bad at all.
Test 3: SOAP CFC vs REST CFM with CFC with caching
One thing that is known to be reasonably costly in CF is the instantiation of CFC objects. A typical way to avoid this is to instantiate the objects once and then store them in the persistent application scope. This technique will work for lots of the code I am looking to optimise so I wanted to see how much impact it would have.
xmlservicewithcfccached.cfm:
<cfcontent reset="true" type="text/xml">
<cfapplication name="xmlservicewithcfcached" sessionmanagement="false" clientmanagement="false">
<cfparam name="url.method" default="">
<cfswitch expression="#url.method#">
<cfcase value="hello">
<cfif NOT structKeyExists(application,'oSS')>
<cfset application.oSS = createObject("component","soapservice")>
</cfif>
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>#application.oSS.hello()#</str></cfoutput>
</cfcase>
<cfdefaultcase>
<cfoutput><?xml version="1.0" encoding="utf-8"?><str>Unknown Method</str></cfoutput>
</cfdefaultcase>
</cfswitch>
The performance of this version of the code was almost identical to the 1st version - i.e. 4 to 5 times faster than using the SOAP version but it more easily maintainable.
Is this a valid benchmark?
In a word no - in the strict sense of a reproducible benchmark. It was done on my development laptop as I don't have lots of spare machines lying around.
The specs of the machine are as follows:
- Intel Core 2 Duo T7200 @ 2GHz
- 3GB Ram
- Ubuntu 7.10 (gutsy) with standard 2.6.22-14-generic kernel
- Coldfusion 7.0.2 - Multiserver install - default config
- Apache JMeter 2.3.1 for load testing
Each thread in the tests ran 10,000 loops - so for example the tests where I ran 50 threads would be 10,000 x 50 = 500,000 requests.
The entire test was conducted over the course of one day and there were no reboots during the tests. Mostly the same programs were running but it was connected to the internet so minor variations in load and traffic could have affected the results.
What about CF8?
After I finished the testing on CF7 (which is what we currently have on our Production servers) I tried it out on the CF8 Install to see how it compared - the details are detailed below.
Show me the stats
So here are the pretty stats and charts that I created to understand all the data:
Graph 1:
Table 1 - figures for Graph 1:
Graph 2:
Table 2 - figures for Graph 2:
Need more stats?
I have also uploaded the spreadsheet that I created to generate these stats in the original OpenDocument format and also as an Excel file.
Conclusion
For me - I'll be definitely looking at using the REST methods further on any high traffic sites where I might previously have used SOAP.
What are we gaining
- Lots of performance - a 5 fold performance increase can't be ignored.
- Smaller data being transferred as no SOAP envelope overhead
- More control over data transferred
- Ability to use conventional debugging and caching techniques
What are we giving up
- Automatic handling of complex objects
- Lots of overhead
I'd love to hear your thoughts on this - what else am I giving up?
Cheers, Mark
Thanks for the comment - I did look at it briefly earlier in the testing but didn't spend much time benchmarking it. I've just run a couple of tests on it and it appears to be about 2-4 times faster than using SOAP which is still a good bit slower than the direct method. This is on CF7 which only supports returning a WDDX packet.
For the extra speed and control over the output that is returned I'm still favouring the direct cfm request which returns XML (or any other format you wish).
Cheers,
Mark
I'd really like to see the CF8 numbers when calling the CFC directly like: http://mydomain.com/myservice.cfc?method=foo.
I hear ya that it's 2-4x times faster on CF7, but that sa pretty big gap (200-400%), and considering everything is much faster on CF8... well... I'm just curious.
Anyway, thanks for the great info.
Thanks for doing this legwork for us - I've been in a few REST vs. SOAP wars recently, and documentation like this would be great to have on hand.
One question I've got is why you're using SOAP or REST as a backend to your Flex/AIR application instead of AMF/RemoteObject? My experience is that the binary format of AMF blows both out of the water when it comes to performance. Have you found something I've missed?
I did a couple of tests (not all of them) using CF8 with REST on the CFC with format json and format plain.
1 Thread
- CF7 CFC - SOAP - 181.78
- CF8 CFC - SOAP - 192.35
- CF8 CFC - JSON - 785.79
- CF8 CFC - Plain - 822.64
- CF8 CFM with CFC XML - 925.41
- CF8 with CFC XML cached - 1095.29
5 Threads
- CF7 CFC - SOAP - 238.83
- CF8 CFC - SOAP - 363.29
- CF8 CFC - JSON - 1259.57
- CF8 CFC - Plain - 1298.94
- CF8 CFM with CFC XML - 1501.05
- CF8 with CFC XML cached - 1646.58
In summary - the performance is most comparable with the CFM version that which instantiates the object each time - however the direct CFC methods are slightly slower. Even through there is more code going through the CFM is more efficient particularly if the instantiation is cached.
@Joe,
Re AMF - It's on my list of things to check, it's just a little trickier to load test as it's not plain text. I know it works very well on the network layer as the data is shorter, but what I'm really testing here is the actual CF processing efficiency of it. I'll try to do a capture of the binary format and run the load test on it. If I get to it I'll post the results.
I managed to get the AMF load testing working for the same code. I did a WireShark capture of the binary data and sent it to the server.
Unfortunately the results were rather disappointing.
Comparisons of CF8 SOAP vs CF8 AMF - 10000 iterations per thread:
- 1 Thread - SOAP: 192.35 - AMF : 511.74 - 2.66 times faster
- 2 Threads - SOAP: 307.3 - AMF : 469.83 - 1.53 times faster
- 3 Threads - SOAP: 338.09 - AMF : 437.41 - 1.29 times faster
- 4 Threads - SOAP: 376.83 - AMF : 350.27 - 0.93 times faster (i.e. slower)
- 5 Threads - SOAP: 363.29 - AMF : 221.95 - 0.61 times faster (i.e. slower)
It should also be noted that after I ran a 10 threads x 10,000 iterations (100,000) requests when it had completed the CF server continued to be under heavy load. This is very unstable behaviour and leads me to believe there is a memory leak or other similar bug in the AMF code.
These results are very worrying and lead to the following conclusions:
- AMF is faster than SOAP but not faster than REST. About half the performance of REST.
- There is a bug in the AMF code which makes it slow down over time
- AMF on CF does not handle multiple threads very well - notice how all the other charts go up initially as threads are added but the figures for AMF start higher but go down.
AMF is also significantly harder to debug and load test in comparison with REST type systems so for now I'm definitely recommending using REST.
BTW - If anyone wants to try to replicate these tests I'm happy to provide any or all files used.
Cheers,
Mark
One of the things advantages I often hear is how compact AMF is - which leads to a smaller data size and hence faster performance.
In order to do the AMF load testing I had to extract the AMF binary payload from the HTTP request and it turns out that for this simple method call the payload was 335 bytes. Which is not insignificant. I didn't look at the response packet as that was not what I was testing.
With the header and everything else the Avg Bytes per request were as follows:
- SOAP - 463 bytes
- AMF - 376 bytes
- REST - 62 bytes
Now this is a very simple example but nonetheless - AMF does have significant overhead a low levels and given the stability problems I've seen I'll be steering clear of it for now.
Cheers,
Mark
http://www.ximpleware.com/2.3/benchmark_2.3_parsin...
I've seen this a while ago - and while on the surface it seems to disagree with what I have measured I don't believe it does.
My results above focus solely on the server side, as for the project that we were interested in, the client side parsing was not a bottleneck at all. However, the generation of all the content on the server was.
Also - what I found was that while the AMF code appeared fast at low numbers of requests it plateaued early (i.e. it didn't scale nicely to large load), where as the simple generation of XML was easy/fast to create, and scaled very nicely as well as allowing standard HTML style caching to be utilised (though this is not included in the benchmarks).
Cheers,
Mark