Another Performance Lab Meeting has been great and inspirational again! 27 (out of 52 registered) developers interested in FileMaker performance joined me online to discuss how FileMaker 19.5 compares to the previous versions, how fast it runs on the new Apple silicon Macs, and what's new in BenchTest, our tool for testing and comparing performance of different FileMaker versions and configurations.
If you missed the meeting or want to recall some specific moments, you can now watch the whole recording:
Now let me recap the most significant news I shared in the meeting.
New version of BenchTest
I received great feedback from developers who tried BenchTest 2.0, including one bug report. Specifically, when running the standard test on Windows, the testing got sometimes interrupted by an error in the 7th and/or 8th step. I figured out this was due to the Web Viewer not always being ready to perform JavaScript. So I have modified the way how tests involving JavaScript are performed and an updated version of BenchTest is now available for you to download and try.
There are some other minor improvements as well, but the testing and scoring remains unchanged, so results collected by the new version are comparable to those collected by the previous one.
Better display of results on our web
As results kept coming from the community, it became clear that we need to separate them into more categories. For example, many tests were performed in a copy of FileMaker Pro running on the same machine as the FileMaker Server, thus completly eliminating the impact of (even low) network latency. Obviously, these results appeared as the overall leaders. On the other end of the spectrum, tests performed when connecting over the WAN appeared very low in the list due to the high network latency. So we separated these results to their own categories to make the comparison and the leaders in each category much clearer.
How fast are Apple silicon Macs
Last year I tested the new Mac mini M1 with both FileMaker 19.2, which was still not native and needed Rosetta, and FileMaker 19.3, which finally utilized the power of the Apple silicon with a native code. This year in January I received the long-awaited MacBook Pro with the M1 Max processor, and just a week before my June lab tests we also got our first Mac Studio. When used as a server for a single client, Mac Studio (with M1 Max) outperformed the Mac mini M1 by nice 33% when running FileMaker Server 19.4 and by 57% when running FileMaker Server 19.5.
Yet more interesting, however, was testing 10 concurrent clients connecting to the Mac Studio. Even when testing the most server-centric operations, which was getting data for virtual list, the Mac Studio remained mostly bored, only utilizing 4-6 processor cores.
This indicates that Mac Studio, even with the M1 Max processor, is likely to easily handle around 50 busy concurrent clients without the hardware becoming the bottleneck.
Compared to the M1 Mac mini, the M1 Max Mac Studio was able to serve the 10 concurrent users almost twice as fast...
Concurrent users testing
To make my test results as close to real use as possible, I run real concurrent clients as VMWare virtual machines. Using the Mac Pro (late 2013) with 96 GB of RAM, up to around 10-12 concurrent clients can be tested without significant degradation of each client's performance.
To easily spin up multiple concurrent clients, I have created a pre-configured "master" virtual machine with FileMaker Pro set as a startup item and BenchTest automatically opening on launch. I am then using AppleScript to create and run specified number of "linked clones". If you would like to try the same thing, feel free to download my AppleScript scripts below:
How fast is FileMaker 19.5
Finally, the most awaited comparison was the recently released FileMaker 19.5 compared to FileMaker 19.4. The new version contains around 400 bug fixes and several performance improvements, at least according to the release notes for FileMaker Pro and FileMaker Server.
The good news is that the overall performance score confirmed around 10% improvement on all the configurations I tested.
In specific cases, however, FileMaker 19.4 performed better than 19.5, mostly when running the Replace Field Contents operation. This has already been confirmed as a known issue and is going to be addressed by the 19.5.2 update.
The most significant improvement was expected from the new ability to evaluate summaries on the server. Unfortunately, none of my test results reliably indicated this was actually happening. There was some difference when running a single VMWare client against Mac Studio, but my expectation was much higher...
Discussing this in the meeting, however, led me to the conclusion that my tests were most likely affected by previous test causing the summarized data to be already downloaded to the client's local cache, so it would be good to try it out again as an isolated test. And since I had planned to be a guest of Richard Carlton's live stream on the same topic just 2 days later, I didn't wait too long to perform these extra tests...
Update from Richard Carlton's live stream
On Thursday the 30th of June it was my honor to talk about FileMaker 19.5 peformance again as a guest of FMTraining TV. If you did not attend and are interested in the whole stream, you're more than welcome to watch the recording:
The highlight from the session is that FileMaker 19.5 really calculates summaries on server if given conditions are met. Two imporant conditions missing from the Claris support article on this topic are:
- Both FileMaker Pro and FileMaker Server need to be version 19.5 or later
- The summarized data must not be already fully downloded to the client's local cache
But the peformance improvement when this actually happens can be huge:
It even seems the impact of calculating summaries on server is the bigger the slower hardware is used for the server...
But the biggest impact of this improvement can be noticed on a slow network (especially with high latency values):
Of course the actual impact on your use case strongly depends on how often the conditions for calculating summaries on server will actually be met. But if you connect to your FileMaker Server over a slow network or large distance (such as when it's hosted in the cloud), and you only need the summaries but not the actual records being summarized, this single new feature can be a game changer for you.
Just beware one key observation I made during my testing. This only applies to summary fields, not to the aggregate functions.
While until 19.4 it was safe to say it does not really matter whether you use aggregate functions, such as Sum, over a relationship, or get a value from the remote summary field over the same relationship, as of version 19.5 this is no longer true. Unless/until Claris makes this feature available for aggregate functions as well, it will be much better to create summary fields in many cases.
I have to say this round of tests (and meetings) was really exciting and I am looking forward to seeing what comes in the next version of FileMaker. Until then, please stay tuned and don't hesitate to let me know if you have any questions or comments.
