The code engine is definitely not thread-safe, but it's only a few dynamically generated instructions, running in an interrupt handler. I doubt even another interrupt would likely interrupt it, let alone a thread switch. I would think the interrupt vectors would be above the threading system, and would be where context switching takes place.
In at least some cases, it seems to have to do with the amount of available memory. Para once explained that the code F0000319 00?? would change the amount of memory the game sees available to it, where 40=4MB and 80=8MB, and 78=just under 8MB - enough for nearly all 8MB games to run while leaving a bit of space for the code engine. I found that just using that code with OoT (instead of EE000000 0000) solves all the freezing issues, whether you have the Code Generator active or not. Before that, the rule was always to either use EE000000 0000 or enable the generator. I also seem to remember reading that the EE code type actually does exactly the same thing as that code, just setting the available memory to 4MB.
It's as if the crashing is related to the amount of available memory. Likely the standard libraries used by later games probe for and/or clear out as much memory as they can find, even if the game only uses 4MB (which would explain why OoT is affected even though it doesn't use the Expansion Pak). I can't imagine just what is being done differently though. There are also apparently some glitches/hacks for OoT that only work with the Expansion Pak present (otherwise it just crashes); that could just be because they're causing out-of-bounds array accesses that fall within the upper 4MB, that causes an exception if it's not there.
I haven't looked at OoT's code, but I suspect that the SDK provides the usual malloc/free, and will use as much memory as is available (probing to see how much there is and storing that amount at 80000318), while the game developers decided (likely for speed and to avoid fragmentation) to use a different allocator which assumes the game has 4MB. So normally the game will never touch the upper 4MB, though it might destroy its contents at startup (which normally are undefined anyway) in probing to see if it's there. The question then is why the crashes are so random.
I suspect the reason for that is that while the game won't touch that area, the SDK code will store various information there if it's available, and it's never come up in hacking simply because nobody's really played with that code. So in certain circumstances some important data gets stored there only to be overwritten by the code handler. In fact it could even be that the game does store things there if it's available, and just nobody's noticed because it's not very interesting data. The games are designed to have full control of the hardware and run alone, so even though the game is designed not to use more than 4MB at a time, that doesn't mean it will only ever use the first 4MB of RAM; just that it will never be using more than 4MB of the total amount at once.
Mario Kart uses a similar allocation system, so I wouldn't be surprised if OoT does as well. The game simply has a large static buffer, big enough to hold all the textures, vertices, etc of the most complex track, and has a pointer to the end of the buffer. When it copies data from ROM it copies it into that buffer and moves the pointer back to the beginning of the data. For a game like this where everything will be deallocated before loading the next level, it has several advantages. No fragmentation and no time required to deallocate. You just reset the pointer to the end of the buffer and overwrite whatever was there before, because it's no longer needed. In this case the amount of RAM actually available to the game is irrelevant, as it will just use this static buffer which is a fixed size. It also makes it trivial to measure the amount of free memory; just compare the pointer to the beginning address of the buffer. (Though Mario Kart doesn't bother with this and will just write over code if it runs out of memory; normally, that should never happen, so there's no point checking for it.)
(A bonus of this method for us is that all you have to do is find the code that sets the initial pointer, and change it to 80800000. Bam. The game now uses all of the upper 4MB for its resources, and you still have the old buffer to put whatever you want in, that should never be overwritten. If you knew how to detect the Expansion Pak it'd be trivial as well to do this only when it's available.
OoT, then, may well do similar with multiple buffers, and set one of them to the very end of the available memory space. So when it sees 8MB available it sets that to the very end of available RAM, and when the buffer gets full enough it ends up overwriting the code handler that's stored right near the end of RAM.
As for only freezing when GS Button codes are active, the most likely reason is the code handler simply being longer, due to the added check for the button in each code. Another possibility is caching weirdness, or ROM access conflict - the code must probe some address on the cartridge bus to test the button, and that could potentially conflict with the game trying to read something from ROM. That would especially explain why you hear blips and pops in the music when using the generator, more often in unstable areas, as the game is constantly streaming music from ROM.
is now live (but may take a few tries to load)
currently down; check out my PSP/DS/Game Boy/Windows/Linux homebrew, ROM hacks, and Gameshark codes!