BASE: The detection plugin is almost always loaded in memory

[ccawley2011]

If I understand the code in base/main.cpp correctly, the detection plugin is loaded when ScummVM starts, and then keeps it loaded in memory when a game is running so that the engine can access the detection entry and options data, and is only unloaded briefly when the engine is de-initialized, before being loaded again for use with the launcher. The detection plugin is 2.86 MB for the DS port, which means this limits the amount of memory available to engines.

[Die4Ever]

I tried to see if the plugin system could handle the unloading of the global data and string tables, but that doesn't seem to work

in groovie/detection.cpp I put

static byte garbage[1024*1024*1024];

and did a memset on it to make sure it actually touches that memory
then in base/main.cpp

#if defined(UNCACHED_PLUGINS) && defined(DYNAMIC_MODULES) && !defined(DETECTION_STATIC)
                        // Unload all MetaEngines not needed for the current engine, if we're using uncached plugins
                        // to save extra memory.
                        PluginManager::instance().unloadPluginsExcept(PLUGIN_TYPE_ENGINE_DETECTION, NULL);
                        PluginManager::instance().unloadAllPlugins();
                        warning("stop!");
                        while(1) {}
#endif

using top, the memory usage stays high

either the plugin system will need to be upgraded to find a way to delete the global data and string tables, or the detection data would have to be put into binary data files and read on demand

there could possibly be a way to automate the generation of the data files without changing all the code for them, idk how much we care about DS support and other devices with 4MB of RAM though

[Die4Ever]

also, in the AdvancedMetaEngineDetection constructor I did this

    for (const byte *descPtr = _gameDescriptors; 1; descPtr += _descItemSize) {
        total_size += _descItemSize;
        auto p = (const ADGameDescription *)descPtr;
        for (auto f : p->filesDescriptions) {
            if (f.fileName)
                total_size += strlen(f.fileName) + 1;
            if (f.md5)
                total_size += strlen(f.md5) + 1;
        }
        if(p->extra)
            total_size += strlen(p->extra)+1;
        if(p->gameId)
            total_size += strlen(p->gameId)+1;

        if (p->gameId == nullptr)
            break;
    }
    total_size += sizeof(AdvancedMetaEngineDetection);

total_size was 3,635,964 bytes, without the strlens it was still 2,962,304 bytes

[Die4Ever]

made a proof of concept "fix" ​https://github.com/scummvm/scummvm/pull/3381

[somaen]

Probably a bit too complex to solve for 2.9.0, but looking at the issues that we have with Wii/PSP regarding memory, it might be good to approach this.

[sev-]

This has been in fact solved in ​https://github.com/scummvm/scummvm/pull/5829

Closing