[LLVMdev] Stack overflow in Legalize Op

[Duncan Sands]

Hi,

> This isn't running in a child process, but this is a win32 machine.
> 
> The problem I'm having is LegalizeOp is recursively calling itself 6
> times, followed by 3 calls to ExpandEXTRACT_VECTOR_ELT, with each
> LegalizeOp pushing 20KBof data onto the stack and Expand pushing 800B of
> data on the stack. So that is 9 function calls pushing ~140KB of data
> onto the stack. This set of 9 function calls repeats itself 25 times,
> pushing almost 3.5 MB of data onto the stack.
> 
>  
> 
> This is occurring with v16i32 data type being converted to v4i32. If any
> larger vector integer data type was used, the stack usage would probably
> be even more extreme.

try svn, or even llvm 2.5.  There type legalization has been rewritten
and in particular is no longer recursive.

Ciao,

Duncan.