Meet BenchTest 2, our free benchmarking tool for Claris FileMaker. You can use it to measure and compare performance of different techniques to do the same thing, different versions of FileMaker Pro and FileMaker Server, or different client, server and network configurations.
We created BenchTest for FileMaker DevCon 2013 to show how different ways to do the same thing compare to each other in terms of performance. We compared speed of aggregate functions with summaries, different ways to count related records, finds in indexed fields vs unindexed, and several more techniques developers use every day without even thinking if there might be a better way.
Then we used the tool internally to run over 200,000 tests within our FileMaker Performance Lab to find out how each new version of FileMaker compared to its predecessors, how fast FileMaker runs on Windows compared to Mac, or when it's good to use Perform Script on Server and when not. It has become our tool to find out what's the right way to do something when we have multiple choices.
Now it can be your tool as well because it's available for free right here. And here you can watch a short video to learn how to use it:
Before you start
Note that you have to download BenchTest in two pieces. One is the tool itself, the other one is the test data. That's because the test data is much larger download (over 2 gigabytes if you choose the ready-to-use version), but it's not going to be updated as often as the tool itself, so we don't want you to have to re-download it every time we release a minor update of BenchTest.
After you unzip the downloaded files, make sure to add the data file to the same folder as the other files. Then you can open the file named BenchTest.fmp12.
You can start exploring the 14 groups of tests right away, but in order to use automated testing or examine the test scripts, you want to also unlock full developer access. So just click on the green button that’s there for this purpose.
To find out how fast your whole environment is, including FileMaker Pro as a client, FileMaker Server, and the network connection between the two, simply host the four BenchTest files on your FileMaker Server and open the tool from there.
Important: If your server is exposed to public network, make sure to first modify the full access account credentials and disable or delete the other accounts.
Getting started with Standard Test
The easiest to use feature to start with is the Standard Test. It’s a set of tests, specifically selected to measure performance of common operations most FileMaker solutions rely on every day, from searches, sorts, through creating and modifying records, to deleting records. The standard test also includes some popular techniques, such as virtual list, and common issues, such as working with a file whose data structure can bring FileMaker to its knees.
The standard test is meant to provide a consistent test that anyone can run and compare its results with the others. That’s why it doesn’t just show the time it took to perform each job, but also calculates a performance score, i.e. how many times is your specific configuration faster than our reference configuration, currently consisting of 2015 MacBook Pro as a client, and 2010 Mac Pro as a server, running FileMaker 19.2v1.
Once you collect multiple test results from different configurations, you can also easily see how they compare to each other. You can display a comparison chart by clicking on the chart icon beneath the list of collected results.
The chart will show you all results compared by overall score, calculated as a weighed average of the score from all performed tests. Wider bar means faster configuration here, while green bars represent configurations faster than our reference setup and purple bars indicate the tested configuration performed worse than our reference setup.
You can also compare results on every single test performed. Simply click on the chart icon next to any of the performed tests.
This way you can easily see how different configuration parameters, such as FileMaker version, network latency or hard disk speed affect individual FileMaker features and techniques.
Finally, since every valid standard test result is now also published on the BenchTest homepage, you can easily compare your results with those collected by other users and see how fast environment you can get to run your FileMaker apps on.
Trying individual tests
If you’re in the middle of development and want to simply find out what’s a faster technique to do something, you can just click on one of the 14 numbered tabs and run individual tests by clicking on the provided buttons. On the tab 12, for instance, you can compare various different ways to collect and sort records to display in a virtual list.
You may be surprised how big difference there can be between sorting values in JSON structure, as tab-delimited values, and getting them already sorted in the first place.
By examining the actual test scripts (after you have unlocked full access), you can even learn some alternative techniques you didn’t know about. Or you can add your own alternative and compare it to the pre-defined ones.
Automating series of tests
Any performance testing starts to be serious when you can repeat the same test multiple times and start getting consistent results. That’s what we’ve used within our performance lab for the last few years, and you can now use it as well. Start by switching to the Auto tab and either duplicating one of the pre-defined test sets, or creating a new one by manually entering its name.
You can then define one ore more jobs, where for each job you specify how many test records to operate on, how many times (iterations) to repeat the test, whether it should be performed on server or using workers, and which of the test scripts to run. You can also enter a FileMaker calculation to define condition that has to be met to actually execute the job.
For Test 8 X5 (performing custom find query) it is even possible to specify various different queries to try, so you can find out how different find request affect the time needed to perform the query.
Once you have you test set ready, you can simply perform the whole sequence by clicking on the Run button. This way, you can repeat exactly the same set of tests on different configurations, different FileMaker versions or other conditions you want to compare.
If your test sequence takes a long time to execute, you may also want to get notified about its progress. BenchTest uses the PushOver service for this, so you can just paste in your PushOver user ID and select what events you wish to be notified about.
Meet us to share and discuss
Whether you are just curious how fast your FileMaker setup is compared to others, or you’re trying to select the fastest FileMaker hosting for your app, or you run your own set of tests to find out something new, it’s always great to discuss your discoveries with others, talk about what we can do to get the most out of the tools we have, or how to optimize our apps to please our customers and users.
That’s why we started hosting the online Performance Lab Meetings and the next one is just one day ahead…
So don’t hesitate to grab a seat and come to meet us to talk about BenchTest and FileMaker performance in general. And if you’re reading this too late to register for October 27, 2021, don’t worry, there will be other meetings in the future. Or you may contact us directly if you have anything to ask about.
