After playing the game a bit it's obvious that their caching and UI management in general is just horribly broken. But surely dumping everything at Mem/2 (especially every 10th of a second, the way aC's script tries) is way too aggressive...at that rate I might as well just install a card with half as much RAM.
I'd suggest opening the IS console and echoing TextureMem while you fiddle around and open/close various windows, to see what's actually going on. I expect it varies wildly depending on various game engine settings, as well what card/drivers you're using and how well they support the various texture blitting/compressing/scaling/etc features the engine might be using. Try to keep track of the largest jump you see...if you have 150 one minute and 50 the next, that tells you you're probably going to want at least 100 free at any given time.
If you use a script like aC's above, set the wait to something a bit more reasonable...at least a few seconds in between checks. Flushing 6 times while a slow-to-open window is still popping up isn't going to buy you anything, it's more likely to hurt. I'd even consider a sub-loop that waits a few seconds once it detects things getting low, to make sure it stays that way for a bit...the game could just be in the middle of one of it's own garbage collection routines and/or swapping things around to reduce fragmentation.
