Our free benchmarking tool BenchTest has been updated to version 2.1. The update brings a new section dedicated to our Initiative '24, focusing on measuring performance of the FileMaker calculation engine, as well as several other small improvements to give you better insight into performance of various FileMaker versions and techniques to make good decisions for your development.

We have also reached over 900 collected BenchTest results, with over 700 submitted by the community. That's an amazing collective effort! Keep them coming and feel free to check all the published results at the BenchTest page. Use them to compare how different software and hardware configurations perform when running typical FileMaker tasks. Check which hosting option is likely to provide the best value for price. Make great decisions based on data, not guessing.

 

News for Initiative '24 - Calculation Engine Performance

The new tab in BenchTest introduces our Initiative '24, along with our current progress in getting community votes for Claris to make the FileMaker calculation engine significantly faster. If you have not heard about this initiative yet, make sure to check it out and learn why we think it's going to make life better for all FileMaker users.

 

But don't take my word for it. Just one extra click away, you'll find a list of pre-defined test calculations, which you can test in different version of FileMaker Pro to see how/whether the calculation engine's performance has changed.

The just released BenchTest 2.1 includes the exact calculations I set as reference for math and text operations when announcing the Initiative '24 last year, then 4 different calculations testing JSON functions, and 4 others testing two performance related bugs reported years ago but still not fixed as of today, one affecting the Position function and another one in FilterValues. But don't constrain yourself to just these few examples. Feel free to add your own test calculations, especially such that you use heavily in your real solutions, and please do send them to me to include in the next BenchTest update if you find them to reveal interesting or surprising results.

In the screenshot below, you can see FileMaker Pro 19.3 compared to FileMaker Pro 19.2 running on MacBook Pro M4 Max. Those are actually the only adjacent versions from 18 to 21 where I was able to measure a significant speed difference. For the case you do not recall, 19.3 was the first version compiled natively for Apple silicon, so this screenshot shows the speed of that compared to version 19.2 executing under Rosetta emulation...

 

Unlike the other performance tests in this tool, the concept of this tab is slightly different. Instead of running a specific test exactly defined number of times and measuring the time they take, here I took the opposite approach. Each test calculation is repeatedly executed in a loop for no more than 1 second, and I am counting how many times it gets evaluated. This is repeated 30 times and the best result is considered the most accurate.

 

The reason I am returning the best of the 30 collected results is that slower executions are more likely to be affected by the computer doing something else as well, so the fastest one is is actually the closest to the ideal of the processor not having anything else to do other than evaluating our calculation.

Sure, I could run it more times to get even more precise result but I thought 30 seconds is a good balance between waiting for the test to complete and getting a meaningful result.

 

More charting options

Previous versions of BenchTest provided comparison charts primarily for the standard test set, consisting of 111 performance tests. You could get a chart comparing the overall score, or one that compared single test score across different runs you executed on different setups. In this version, you can now visually compare also individual tests within a single run with each other.

To do that you first have to select one test as referential. Score of all remaining test is then calculated to show how many times faster they were.

Then you can use the chart icon next to each test to include or exclude the result from your comparison, and request the chart by clicking on the chart icon beneath the portal.

You can also customize the labels shown next to each result in the chart, for the case you want to take a screenshot and include the chart in a presentation or article. This makes you able to easily compare different techniques doing the same thing, for example.

 

Other improvements

Remaining improvements make using the tool easier and more efficient, one step at a time:

• Filter by test set lets you keep rsults from different scenarios together in one instance but still compare only relevant ones
• Iteration filter on individual test result chart lets you constrain the chart to show results only from specific iteration
• New variants of Test 3 let you test and compare the new loop flush options added in FileMaker 20.3
• Score of each test is now finally cached for better performance when navigating between different views
• Improved compatibility with iOS, iPadOS and WebDirect lets you test performance of those platforms as well
• The Toolbox Plug-In is no longer needed for precise measurements with FileMaker 19.6 or newer thanks to the new Get(CurrentTimeUTCMicroseconds) functiion
• Added validation for info submitted with results will hopefully help to make the published results a bit cleaner and more meaningful
• GetPreciseTimestamp custom function fixed to return consistent timestamps with and without Toolbox

 

The show must go on

I don't stop testing and challenging performance of every new version of FileMaker released. Initiative '24 continues until we reach the final goal, making FileMaker calculations at least 24 times faster compared to FileMaker 2023. I hope that BenchTest made it easy enough for you to take your part as well and I keep looking forward to receiving all the test results and discoveries from all over the community. So don't hesitate to run your own tests, submit your results, share your discoveries, and support our endeavor to make FileMaker faster for everyone.

Only 302 votes are missing at the time of this writing to make my idea #1 on the list for Claris to consider. Help me finally push it there...

Vote now, or convince someone else if you have already voted.