IDA C++ plugin authors, who wish to link such plugins against Qt 5.x libraries.
One of our customers, Aliaksandr Trafimchuk, recently reported that whenever IDA was run with a plugin of his that links against the Qt libraries that we ship, IDA would crash at exit-time (at least on Windows.)
Aliaksandr already did most of the work of figuring out exactly what was causing the crash, and even had a work-around (more like a kludge, as he pointed out, really) for it, but he still wanted to let us know about it so we are aware of the problem & perhaps can communicate about it.
The crash is an access violation, in an area of memory that doesn’t seem to be mapped by any stack, heap, DLL code or data.
The stack reveals that the crash happens at QCoreApplication::~QCoreApplication()-time (i.e., at application exit), when the QFontCache is freeing/releasing its entries:
Qt5Core.dll!QT::QSettingsGroup::~QSettingsGroup() Qt5Gui.dll!QT::QMapNode::destroySubTree() Qt5Gui.dll!QT::QFontCache::clear() Qt5Gui.dll!QT::QFontCache::~QFontCache() [External Code] Qt5Gui.dll!QT::QThreadStorage::deleteData(void * x) Qt5Core.dll!QT::QThreadStorageData::set(void * p) Qt5Gui.dll!QT::QFont::cleanup() Qt5Gui.dll!QT::QGuiApplicationPrivate::~QGuiApplicationPrivate() [External Code] Qt5Core.dll!QT::QObject::~QObject() Qt5Core.dll!QT::QCoreApplication::~QCoreApplication() idaq.exe!013426c5() Unknown (...)
Our customer’s plugin uses a UI description file, that needs to be processed by Qt’s uic (UI-compiler). The generated code contains lines such as these:
label = new QLabel(TestDialog);
label->setObjectName(QStringLiteral("label"));
QFont font;
font.setFamily(QStringLiteral("Comic Sans MS"));
Note the use of QStringLiteral.
QStringLiteral typeThis is an optimization that came in Qt 5.x, and that causes actual QString instances to be laid out in the .rodata section of the program (together with a special refcount value that is -1, meaning “don’t touch this refcount”.)
Although at exit-time, this “static const” in-.rodata-QString instance wouldn’t be modified (because of the -1 refcount), simply reading it will cause a crash, since the section holding it has been removed from memory.
This is a known limitation/problem, too: https://bugreports.qt.io/browse/QTBUG-46880
This is where the problem lies: at exit-time, IDA will:
QCoreApplication goes out of scope, which will perform (among other things) the QFontCache cleanup. QFontCache still refers to literal QString data, in a section that is now gone (it was discarded at #1) In fact, Qt expects that any binary that uses Qt libraries should remain in memory, so that some optimizations (such as the QStringLiteral) will continue to work. That’s why, when Qt unloads some of its own plugins, it doesn’t really unload those from memory.
Although the Qt library maintainers consider that having such limitations on binaries that link against Qt is acceptable, I personally hope they try to keep those restrictions as minimal as possible.
In any case, concerning this QStringLiteral issue, we have a way out: at compilation-time, pass the compiler the following flag: -DQT_NO_UNICODE_LITERAL
This will turn the QStringLiteral() expression into a QString::fromUtf8(), which will allocate the memory on the heap and the plugin should work just fine.
Another possibility reported by an IDA user (but untested by us), is to add the following after the Qt headers #include directives:
#undef QStringLiteral #define QStringLiteral(_str) _str
With this method, the literal C-style string will be implicitly converted to a QString, using the default conversion rules.
The kludge (which is Windows-specific) consists of calling LoadLibrary(szMyPluginFilePath), thereby somewhat artificially incrementing the refcount of his plugin, which will cause it to remain in memory & thus the ~QFontCache cleanup will succeed.