Fix Unix stack overflow detection

On Linux (and FreeBSD), we were not distinguishing a stack overflow detected
using a probe and general access violation. So instead of aborting the process,
we were sending it a managed access violation exception.
This change checks for the case when the access violation happens in the page
below the stack limit and aborts right in the sigsegv_handler if it does.
On OSX, we were detecting it, but there was a problem in it - it could only
detect it if the probed address was closer than 512kB to the current SP.
So e.g. an application failing due to allocating 1MB structure on stack didn't
pass this test.
The fix here was to remove the distance limit.

1 files changed, 15 insertions, 0 deletions

diff --git a/src/pal/src/exception/signal.cpp b/src/pal/src/exception/signal.cpp
index 69519d4a87..ef7b569f69 100644
--- a/src/pal/src/exception/signal.cpp
+++ b/src/pal/src/exception/signal.cpp
@@ -283,6 +283,21 @@ static void sigsegv_handler(int code, siginfo_t *siginfo, void *context)
         record.ExceptionAddress = GetNativeContextPC(ucontext);
         record.NumberParameters = 2;
 
+        if (record.ExceptionCode == EXCEPTION_ACCESS_VIOLATION)
+        {
+            // Check if the failed access has hit a stack guard page. In such case, it
+            // was a stack probe that detected that there is not enough stack left.
+            void* stackLimit = CPalThread::GetStackLimit();
+            void* stackGuard = (void*)((size_t)stackLimit - getpagesize());
+            if ((siginfo->si_addr >= stackGuard) && (siginfo->si_addr < stackLimit))
+            {
+                // The exception happened in the page right below the stack limit,
+                // so it is a stack overflow
+                write(STDERR_FILENO, StackOverflowMessage, sizeof(StackOverflowMessage) - 1);
+                abort();
+            }
+        }
+
         // TODO: First parameter says whether a read (0) or write (non-0) caused the
         // fault. We must disassemble the instruction at record.ExceptionAddress
         // to correctly fill in this value.