Fix GC background thread start in OOM (#5840)

This change fixes a problem that happened when the `gc_heap::create_bgc_thread`
calls SetupUnstartedThread and it fails to allocate a Thread object and throws.
The GC doesn't expect that and so when the stack is unwound, the `gc_heap::gc_started`
stays set. Now we try to allocate memory for the Throwable for the exception, so we go
to GC heap and since there is not enough space, we end up calling
`gc_heap::try_allocate_more_space`, which calls `gc_heap::wait_for_gc_done`.
And that’s the end of it, since the `gc_started` is still set and we wait forever.

The fix is to catch exception from the SetupUnstartedThread. I have also fixed couple of
places where this method was being called and the exception was not expected.

As an additional cleanup, I have moved the thread creation in GC to the GCToEEInterface.

1 files changed, 0 insertions, 13 deletions

diff --git a/src/gc/gcpriv.h b/src/gc/gcpriv.h
index 9f6abe40d4..703059a636 100644
--- a/src/gc/gcpriv.h
+++ b/src/gc/gcpriv.h
@@ -2736,19 +2736,6 @@ protected:
     static
     uint32_t __stdcall bgc_thread_stub (void* arg);
 
-#ifdef FEATURE_REDHAWK
-    // Helper used to wrap the start routine of background GC threads so we can do things like initialize the
-    // Redhawk thread state which requires running in the new thread's context.
-    static uint32_t WINAPI rh_bgc_thread_stub(void * pContext);
-
-    // Context passed to the above.
-    struct rh_bgc_thread_ctx
-    {
-        PTHREAD_START_ROUTINE   m_pRealStartRoutine;
-        gc_heap *               m_pRealContext;
-    };
-#endif //FEATURE_REDHAWK
-
 #endif //BACKGROUND_GC
  
 public: