Back in June I made an updated public beta of BenchTest available. Since then 45 valid complete test results were submitted by 19 different people. It’s just a beginning of the community testing, but enough to make some comparisons and share the results with you, so here they come.
I have also slightly updated the tool itself based on my findings, so it’s a great time to download the updated version, test your current FileMaker setup and compare it to the results submitted by others.
What do the results say?
You might think that 45 results is not so many, but when you consider that each result set contains 111 scores, along with other interesting factors, such as CPU and disk speed, it’s becoming a lot to make some reasonable sense of.
So I exported all the data into a big table and started adding conditional formatting, so that I could start looking for interesting patterns...
When latency matters
The first pattern I spotted was the dependency of write operations speed (replacing field values across many records) on latency. The table clearly showed that latency (the time needed for a single round trip of a request sent from FileMaker Pro to FileMaker Server and receiving the server’s response back) is the single most important factor affecting the speed of these small change commits.
What this means is that if your app relies on a lot of small changes being committed from the client to the server in a small amount of time, you had better do your best to minimize the latency. If you can’t then you have to use optimization techniques specifically addressing this, such as Todd Geist’s transactions or Vince Menanno’s local editing helper file.
When server speed matters
Another interesting pattern, although not so obvious, revealed some operations whose primary performance factor seems to be the server’s performance. These operations are performing find and gathering data for Bruce Robertson’s virtual list.
It was not so easy to differentiate between the impact of processing speed and disk speed as most of the fast machines tested had also fast hard drives and vice versa. What I found quite interesting though, besides the fact that Macs with the new M1 processor were among the fastest server machines tested so far, is that almost all Macs tested were faster than the fastest Linux instance tested.
I am curious what we’ll see in the results once someone tries some faster machines running Linux, including the M1 Macs…
What can you find out?
I am sure that with the help of some great data visualization tools, such as Tableau or Power BI, even more dependencies and patterns could be revealed, but I am not so good in this, so let me leave it up to you to find out what else you can discover. Here you can download the table in the format you prefer:
Download for Numbers Download for Excel
What’s new in BenchTest 2.0b6
As I mentioned earlier, I also updated the BenchTest tool based on some of my findings. I have modified the overall score calculation method, added buttons for exporting and importing test results, added collecting more server characteristics during each test, and more…
Weighed average
My first discovery was the fact that some of the scores can get insanely high or low, specifically for the first iteration of some tests. I assume this might be caused by things like the people running the tests not always discarding the client caches before running the tests.
So I calculated the variance of each test's scores across the collected results, and changed the overall score to use weighed average instead of plain average.
Now the overall score makes much better sense, although I may still have to adjust the weights of individual tests a bit more.
Exports, imports, and charts
As some of you pointed out, when testing multiple different servers, it may be handy to be able to have BenchTest installed on all of them first, and then somehow consolidate all the collected results to a single copy.
Of course, you could do that manually since you have the full access unlocked, but I thought making it easier could be a great feature, so I added buttons for exporting and importing the test results. I have also added a button to export the big table mentioned above, and another to let you select which test results appear in the comparison charts.
You can also click on the portal columns to change the sort order…
More environment info, better structured
When looking for patterns in the big table, I also discovered it might be useful to record some more factors the performance may depend on, such as operating system version, or CPU and disk speed of the server. So the new version of BenchTest collects this info and displays it for every test session in a slightly better structured form.
In the next update I am also planning to add network bandwidth measurement. Is there anything else you think I should collect? Don’t hesitate and let me know.
Get the new version
If you opened the previous version of BenchTest it has probably already told you a new version is available and you have probably already downloaded it. Otherwise you can download it here:
Download BenchTest Download Small Data File (166 MB) Download Ready-To-Use Data (2.2 GB)
The data files have not changed, so if you have already downloaded one with the previous version, you don’t have to download it again. Otherwise download the small one if downloading 2 gigabytes would take you 10 hours or more, otherwise it’s better to download the larger one with the test data already pre-generated.
Latest community results
Finally, since BenchTest can now export and import test results, why not compare all the results we received in your own copy of the tool?
Download Test Results for Import
Then go ahead, perform more tests, see if you can discover some interesting dependencies, and let me know when you do, so that I can share your findings with the rest of the community in my next article.
