Intended audience
IDA C++ plugin authors, who wish to link such plugins against Qt 5.x libraries.
The problem
One of our customers, Aliaksandr Trafimchuk, recently reported that whenever IDA was run with a plugin of his that links against the Qt libraries that we ship, IDA would crash at exit-time (at least on Windows.)
Aliaksandr already did most of the work of figuring out exactly what was causing the crash, and even had a work-around (more like a kludge, as he pointed out, really) for it, but he still wanted to let us know about it so we are aware of the problem & perhaps can communicate about it.
The crash is an access violation, in an area of memory that doesn’t seem to be mapped by any stack, heap, DLL code or data.
The stack reveals that the crash happens at QCoreApplication::~QCoreApplication()-time (i.e., at application exit), when the QFontCache is freeing/releasing its entries:
Qt5Core.dll!QT::QSettingsGroup::~QSettingsGroup() Qt5Gui.dll!QT::QMapNode::destroySubTree() Qt5Gui.dll!QT::QFontCache::clear() Qt5Gui.dll!QT::QFontCache::~QFontCache() [External Code] Qt5Gui.dll!QT::QThreadStorage::deleteData(void * x) Qt5Core.dll!QT::QThreadStorageData::set(void * p) Qt5Gui.dll!QT::QFont::cleanup() Qt5Gui.dll!QT::QGuiApplicationPrivate::~QGuiApplicationPrivate() [External Code] Qt5Core.dll!QT::QObject::~QObject() Qt5Core.dll!QT::QCoreApplication::~QCoreApplication() idaq.exe!013426c5() Unknown (...)
Why does that happen?
Our customer’s plugin uses a UI description file, that needs to be processed by Qt’s uic (UI-compiler). The generated code contains lines such as these:
label = new QLabel(TestDialog);
label->setObjectName(QStringLiteral("label"));
QFont font;
font.setFamily(QStringLiteral("Comic Sans MS"));
Note the use of QStringLiteral.
The QStringLiteral type
This is an optimization that came in Qt 5.x, and that causes actual QString instances to be laid out in the .rodata section of the program (together with a special refcount value that is -1, meaning “don’t touch this refcount”.)
Although at exit-time, this “static const” in-.rodata-QString instance wouldn’t be modified (because of the -1 refcount), simply reading it will cause a crash, since the section holding it has been removed from memory.
This is a known limitation/problem, too: https://bugreports.qt.io/browse/QTBUG-46880
The plugin lifecycle
This is where the problem lies: at exit-time, IDA will:
- unload plugins
- proceed until its
QCoreApplicationgoes out of scope, which will perform (among other things) theQFontCachecleanup. - alas, at that time, the
QFontCachestill refers to literalQStringdata, in a section that is now gone (it was discarded at #1)
In fact, Qt expects that any binary that uses Qt libraries should remain in memory, so that some optimizations (such as the QStringLiteral) will continue to work. That’s why, when Qt unloads some of its own plugins, it doesn’t really unload those from memory.
Although the Qt library maintainers consider that having such limitations on binaries that link against Qt is acceptable, I personally hope they try to keep those restrictions as minimal as possible.
The solution
In any case, concerning this QStringLiteral issue, we have a way out: at compilation-time, pass the compiler the following flag: -DQT_NO_UNICODE_LITERAL
This will turn the QStringLiteral() expression into a QString::fromUtf8(), which will allocate the memory on the heap and the plugin should work just fine.
Another possible solution
Another possibility reported by an IDA user (but untested by us), is to add the following after the Qt headers #include directives:
#undef QStringLiteral #define QStringLiteral(_str) _str
With this method, the literal C-style string will be implicitly converted to a QString, using the default conversion rules.
Footnotes
The kludge (which is Windows-specific) consists of calling LoadLibrary(szMyPluginFilePath), thereby somewhat artificially incrementing the refcount of his plugin, which will cause it to remain in memory & thus the ~QFontCache cleanup will succeed.
