Correctly marshal structure return values in member functions on Win-x64 and Win-x86 (#23145)

* In Windows-x64, if we have a native member function signature with a struct return type, we need to do a by-ref return.

* Implement support for marshalling structure return values via a return buffer argument correctly in instance signatures on AMD64-Windows.

* Change field initialization ordering to satisfy warning.

* Try to narrow down the conditions that trigger these changes to just COM methods.

* Don't bash the return type on AMD64 Windows. Only treat it as a byref return buffer.

* PR feedback.

* Enable returning structs from COM methods via a return buffer on x86 for structs <= 8 bytes.

* Add test for struct returns with ThisCall. Extend the "struct return buffer" fix to functions marked as unmanaged thiscall since they all must be instance methods

* Don't include the return-type-bashing switch on AMD64 platforms.

* Don't do the signature swapping/copy on non-instance functions with struct returns.

* Cast the return type of GetStubTargetCallingConv to the right calling convention enum type.

* If we're doing a thiscall, marshal the "this" parameter before the return buffer (if the return buffer exists) on all platforms.

* Remove temporary logging code I added in for debugging.

* Clean up class naming.

* Try using a vtable instead of a pointer-to-member-function.

* Remove delete of class with non-virtual destructor

1 files changed, 5 insertions, 0 deletions

diff --git a/src/vm/dllimport.h b/src/vm/dllimport.h
index b88d339a84..a6967ebfb4 100644
--- a/src/vm/dllimport.h
+++ b/src/vm/dllimport.h
@@ -494,6 +494,10 @@ public:
 
     void    SetInteropParamExceptionInfo(UINT resID, UINT paramIdx);
     bool    HasInteropParamExceptionInfo();
+    bool    TargetHasThis()
+    {
+        return m_targetHasThis == TRUE;
+    }
 
     void ClearCode();
 
@@ -556,6 +560,7 @@ protected:
     BOOL                m_fHasCleanupCode;
     BOOL                m_fHasExceptionCleanupCode;
     BOOL                m_fCleanupWorkListIsSetup;
+    BOOL                m_targetHasThis;
     DWORD               m_dwThreadLocalNum;                 // managed-to-native only
     DWORD               m_dwArgMarshalIndexLocalNum;
     DWORD               m_dwCleanupWorkListLocalNum;