During the January 2021 FileMaker Performance Lab meeting I shared the first beta version of BenchTest, my performance testing tool, with the meeting participants. I was working on it since then and today I am pleased to announce the first version ready for you to easily compare the performance of your own FileMaker Server or hosting with my reference setup.
BenchTest 2.0b3 is here
Starting from today, a new version of BenchTest, marked as beta 3, can be downloaded for free directly from this article. It comes with a pre-configured “standard” set of 111 performance tests, which take about 10 minutes when executed locally on a 9 years old MacBook Pro. I have already done some comparisons myself, so read along to find out how cloud hosting compares to Mac mini colocated in a data center or to the new Mac mini M1 on a local network, or how FileMaker Server 19.3 compares to FileMaker Server 19.2, especially on Apple silicon.
If you can’t wait to try it out yourself, start by watching this short video to learn how to use the tool to run your own test and submit it for comparison:
As soon as you’re ready, feel free to download the current beta version of BenchTest and the test data by clicking on the buttons below:
Download BenchTest Download Small Data File (166 MB) Download Ready-To-Use Data (2.2 GB)
If you download the small data file, you will have to generate the test data yourself, which can easily take over 10 hours. I decided to keep the data a separate download so that you don’t have to re-download it when BenchTest gets updated as the test data is most likely going to get updates much less often than the BenchTest tool itself.
Reference Test
I have been using the same primary test machine since the beginning of my performance testing, so I decided to use it as a reference setup for the standard test as well. Any new hardware you can get for an on-premise FileMaker Server should perform better than the 13 years old Mac Pro, unless you make some mistake in the configuration, such as not having enough RAM or free disk space.
Running the standard test over a local network took a bit over 28 minutes with the Mac Pro early 2008 as my FileMaker Server.
To have an additional reference for cloud hosting options as well, I have also performed the standard test set with a simulated WAN connection, configured using the Network Link Conditioner to have 100Mbps downstream and 10Mbps upstream bandwidth. Latency under 50 milliseconds is usually considered very good, so I set both downstream and upstream delay to 25 ms, resulting in the whole round trip taking 50 ms.
We’re probably lucky because the DragonCloud hosting with data center in Brno we use for some of our customers, had the latency ranging from 5 to 10 milliseconds during my test. But let’s see what values you get from your own measurements with your hosting providers…
Initial Tests for Comparison
For this article I initially performed the standard test on two setups we use for our customers. One was a 7 years old Core i5 Mac mini colocated in a data center in Prague, running FileMaker Server 19.1.2 on macOS 10.14 Mojave. The other one was an entry-level virtual server running Windows Server 2019, hosted by DragonCloud in their data center in Brno.
I also tried the new Mac mini with the M1 processor and I was using the 13-inch Core i5 MacBook Pro early 2015 with FileMaker Pro 19.2.1 as a client for all the tests.
The results did not reveal any huge surprise as you can find out in the charts comparing the results below, except for the just released FileMaker 19.3 for which I updated the article. So if you have already read it before 19.3 was released, it may be, for the first time, a good reason to read the same article twice… ;-)
Scoring Method
My current “standard test set” consists of 111 individual tests. The score for each test is calculated simply as reference time / measured time. In other words, the score simply tells you how many times your tested setup is faster than my reference setup.
I have used a simple average of all 111 score values for the overall score for now. Going forward (for the final release of BenchTest 2.0) I am planning to replace it with a weighted average by assigning a specific weight (importance) value to each of the 111 tests. I will perhaps also include some evaluation of the latency, CPU speed, disk reading speed and disk writing speed values. If you have any suggestions on how to calculate the overall score in the final version, please do let me know, I will appreciate your feedback.
First Results
I guess you’re curious how my first tests went, so here are finally the comparisons I promised. Let me start with the overall score. No big surprise here, Mac mini M1 on LAN scored the best. What did surprise me is how much the FileMaker Server 19.3 running natively on Apple silicon scored better than 19.2 running under Rosetta. A nice discovery is that the colocated Mac mini performed even slightly better than the old Mac Pro on local network.
For read-only operations it’s hard to get accurate results because anything beyond first test can be significantly affected by caching. So I have chosen the first Perform Find result from the first iteration of the test as the most relevant.
Note that these charts show score, not time, so wider bar means faster setup. I have also compared searching for the three-word query “a h m”, which seems to be one of the busier find queries at least for FileMaker.
Sorting records is where it started being interested, especially with the M1 Mac mini. This is, however, strongly dependent on caching and exact FileMaker Pro and FileMaker Server version combination since the way how FileMaker sorts records has changed a lot since the first release of FileMaker 19 last year, so don’t rely on this chart heavily and let me see if I can maintain consistent results when repeating the test multiple times on the same setup.
Modifying data is a usual bottleneck of FileMaker solutions where latency and bandwidth has higher impact than ever. The easiest way to quickly modify large set of records is the Replace Field Contents script step. Even though you could expect this to be off-loaded to the server automatically, it still does not seem to.
The takeaway seems to be that network performance is the primary factor for data manipulation performance. It’s also a reminder that FileMaker always transfers the whole record between the client and the server, so the more often you make changes the more you should be careful before adding too many fields to the same table.
Even more interesting is that when I replaced Replace Field Contents with a Set Field loop, then the old Mac Pro even outperformed the new M1 Mac mini. This was actually the biggest surprise from these tests for me.
When examining the results further, I discovered that there was a significant difference between network latency. Even though it was generally very low (mostly under 1 millisecond) it ranged from 0.3 ms to 0.5 ms on the Mac Pro, while latency connecting to the M1 Mac mini, even using the same cable and Ethernet switch, was between 0.7 ms and 1 ms. When looping through 1000 records the latency alone contributed to the total execution time of the test 1000 times.
The last test I am going to show compared here, is a simulation of the popular Virtual List technique. In this case I am using Perform Script On Server to gather data and a virtual list table with unstored calculations to render the data from a global variable.
As you can see above, Virtual List still remains a good way to overcome the limitations of poor network connection at the cost of server performance and some extra development effort. If you haven’t used the technique yet, make sure to check it out at ISO Productions FileMaker Magazine, FM Forums, FileMakerHacks, FileMaker Pro Gurus, Michael Rocharde, Soliant Consulting, SeedCode, or other FileMaker related forums and websites.
More Results
However interesting my results are, it’s getting tens and hundreds of your tests in the mix what will make this really valuable. So that’s why I am now making BenchTest available for everyone to download and try.
The most important value comes form the performance or efficiency of the user-app-software-hardware combo. You can affect efficiency of users by training the users and properly designing processes. App’s performance can be fine-tuned by using good development techniques and optimizing your code, which our FM Bench can help you with. BenchTest will hopefully help you to easily select the right environment to deploy your app to.
So don’t hesitate, download it, and submit your results.
As soon as we collect enough results and finalize the scoring method, we’re going to make all submitted results available online on our website. In the meantime, feel free to get in touch if you have any question or idea.
