CVE-2016-5384 Patch slows down fontconfig and the start up of any desktop applications

The CVE-2016-5384 patch shipped in Fontconfig-2.12.1 dramatically slows down fontconfig, fc-scan, fc-match, fc-cache and the startup of almost all desktop applications on my computer (openSUSE Tumbleweed).

$ time fc-match "sans"
OpenSans-Regular.ttf: "Open Sans" "Regular"

real    0m3.179s
user    0m3.132s
sys     0m0.032s

$ fc-match -V 
fontconfig version 2.12.1

and it only takes 1/60 of the time for fontconfig-2.12.0:

$ time fc-match "sans"
OpenSans-Regular.ttf: "Open Sans" "Regular"

real    0m0.055s
user    0m0.052s
sys     0m0.000s

$ fc-match -V
fontconfig version 2.12.0

At least one other person has experienced the similar change on Redhat’s bugzilla (https://bugzilla.redhat.com/show_bug.cgi?id=1350891#c18). Anyone else also seeing similar results? Is there a way to keep the CVE-2016-5384 patch but restore the speed of fontconfig?