Use runtest.py to run tests for all platforms (#19213)

Change build-test.sh to always build the xunit wrappers. Before it would drop a token and check the existence of the token.

Unify x64 linux/OSX/Windows excludes into one file, issues.targets. Includes different locations in the file which show where to put excludes.

Remove all target specific aspects of issues.targets, all tests are excluded now via wildcard, this allows expanding to .cmd and .sh based on the built platform.

Unify path separators to forward slash(/) in issues.targets to support both platforms

Clean up issues.targets by removing long standing exclude tests, specifically tests that have been excluded due to missing features like rva_statics.
Add DisableProjectBuild to tests which have been removed from issues.targets

Conditionally add DisableProjectBuild to tests which have been marked as unsupported on unix. This is mostly a port of the unsupportedOnUnix.txt list. Instead of excluding the tests, unix will simply not build them. If tests are built on windows, they will be run but they will return pass, the test wrapper will check return instantly.

All exclusions ported to issues.targets for linux targets.
Expand runtest.py, this includes simple issues that made it past the original CR. In addition it adds more optional features to help with inner loop dev work such as: creating a repro folder under bin/repro/.. which sets up the env and calls the failing test. In addition a launch.json will now be created under bin/repro/.. which can be used to easily debug using vscode. More logging, such as printing failures, longest running tests ect.
Initial excludes ported for arm64 windows

Arm64 linux, armhf unix excludes and enables running runtest.sh for these targets.
arm64 windows and arm32 windows excludes and enables running runtest.cmd on arm64 targets
init-tools.sh changes to pull armhf and aarch64 dotnetcli

init-tools.cmd changes to pull x86 packages for dotnetcli for arm64 windows
runtest.cmd for almost all scenarios will call runtest.py
runtest.sh for almsot all scenarios will call runtest.py
Removes all logic for running tests using runtest.sh

3 files changed, 30 insertions, 55 deletions

diff --git a/Documentation/building/test-configuration.md b/Documentation/building/test-configuration.md
index 14fbb4e3d6..d799bfdaad 100644
--- a/Documentation/building/test-configuration.md
+++ b/Documentation/building/test-configuration.md
@@ -39,15 +39,17 @@ Test cases are categorized by priority level. The most important subset should b
     ```
 * Disable building of a test by conditionally setting the `<DisableProjectBuild>` property.
 	* e.g. `<DisableProjectBuild Condition=" '$(Platform)' == 'arm64' ">true</DisableProjectBuild>`
+* Exclude test from GCStress runs by adding the following to the csproj:
+    * `<GCStressIncompatible>true</GCStressIncompatible>`
+* Exclude test from JIT stress runs runs by adding the following to the csproj:
+    * `<JitOptimizationSensitive>true</JitOptimizationSensitive>`
 * Add NuGet/MyGet references by updating the following [test project](https://github.com/dotnet/coreclr/blob/master/tests/src/Common/test_dependencies/test_dependencies.csproj).
 * Build against the `mscorlib` facade by adding `<ReferenceLocalMscorlib>true</ReferenceLocalMscorlib>` to the test project.
 * Update relevent exclusion lists:
-  * Note that there are two build pipelines Jenkin's CI and nightly Helix - both must be updated for expected exclusion.
-    * Jenkin's CI build - see the associated `*.txt` files under `tests/` (e.g. `tests/testsUnsupportedOutsideWindows.txt`).
-    * Official nightly Helix build - see the `tests/issues.targets` file.
-  * ARM/ARM64
-    * `tests/arm/Tests.lst` and `tests/arm64/Tests.lst` are used to define the tests to run due to limitations with XUnit.
-    * These files can be manually edited or the generated using `tests/scripts/lst_creator.py`.
+  There are currently three different exclude lists. Tests.lst is currently only used by CI.
+    - `tests/issues.targets`
+    - `tests/arm/Tests.lst` - Used by Windows arm32 testing
+    - `tests/arm64/Tests.lst` - Used by Windows arm64 testing
 
 ### Creating a C# test project
 
diff --git a/Documentation/building/unix-test-instructions.md b/Documentation/building/unix-test-instructions.md
index 89fead7aa6..9b0190c4a0 100644
--- a/Documentation/building/unix-test-instructions.md
+++ b/Documentation/building/unix-test-instructions.md
@@ -81,50 +81,15 @@ sudo apt-get install libunwind8:armhf libunwind8-dev:armhf libicu-dev:armhf libl
 The following instructions assume that on the Unix machine:
 - The CoreCLR repo is cloned at `/mnt/coreclr`
 
-If DotNet is unsupported
-- The CoreFX repo is cloned at `/mnt/corefx`
-- The other platform's clone of the CoreCLR repo is mounted at `/media/coreclr`
-
-The following steps are different if DotNet is supported or not on your arch and os.
-
-### DotNet is supported
-
-build-test.sh will have setup the Core_Root directory correctly after the test build. If this was either skipped or needs to be regenerated use:
-
->`build-test.sh generatelayoutonly`
-
-To run the tests run with the --coreOverlayDir path
+build-test.sh will have setup the Core_Root directory correctly after the test build.
 
 ```bash
-~/coreclr$ tests/runtest.sh
-    --testRootDir=/mnt/coreclr/bin/tests/Linux.x64.Debug
-    --testNativeBinDir=/mnt/coreclr/bin/obj/Linux.x64.Debug/tests
-    --coreOverlayDir=/mnt/coreclr/bin/tests/Linux.x64.Debug/Tests/Core_Root
-    --copyNativeTestBin
+~/coreclr$ tests/runtest.sh x64 checked
 ```
 
-### DotNet is not supported
-
-Tests need to be built on another platform and copied over to the Unix machine for testing. Copy the test build over to the Unix machine:
-
-> `cp --recursive /media/coreclr/bin/tests/Windows_NT.x64.Debug /mnt/test/`
-
-See `runtest.sh` usage information:
-
-> `/mnt/coreclr$ tests/runtest.sh --help`
-
-Run tests (`Debug` may be replaced with `Release` or `Checked`, depending on which Configuration you've built):
+Please use the following command for help.
 
-```bash
-/mnt/coreclr$ tests/runtest.sh
-    --testRootDir=/mnt/test/Windows_NT.x64.Debug
-    --testNativeBinDir=/mnt/coreclr/bin/obj/Linux.x64.Debug/tests
-    --coreClrBinDir=/mnt/coreclr/bin/Product/Linux.x64.Debug
-    --mscorlibDir=/mnt/coreclr/bin/Product/Linux.x64.Debug
-    --coreFxBinDir=/mnt/corefx/bin/runtime/netcoreapp-Linux-Debug-x64
-```
-
-The method above will copy dependencies from the set of directories provided to create an 'overlay' directory.
+>./tests/runtest.sh -h
 
 ### Results
 
@@ -134,13 +99,21 @@ Test results will go into:
 
 ### Unsupported and temporarily disabled tests
 
-These tests are skipped by default:
-- Tests that are not supported outside Windows, are listed in:
-    > `~/coreclr/tests/testsUnsupportedOutsideWindows.txt`
-- Tests that are temporarily disabled outside Windows due to unexpected failures (pending investigation), are listed in:
-    > `~/coreclr/tests/testsFailingOutsideWindows.txt`
+Unsupported tests outside of Windows have two annotations in their csproj to
+ignore them when run.
+
+```
+<TestUnsupportedOutsideWindows>true</TestUnsupportedOutsideWindows>
+```
+
+This will write in the bash target to skip the test by returning a passing value if run outside Windows.
+
+In addition:
+```
+<DisableProjectBuild Condition="'$(TargetsUnix)' == 'true'">true</DisableProjectBuild>
+```
 
-To run only the set of temporarily disabled tests, pass in the `--runFailingTestsOnly` argument to `runtest.sh`.
+Is used to disable the build, that way if building on unix cycles are saved building/running.
 
 PAL tests
 ---------
diff --git a/Documentation/building/windows-test-instructions.md b/Documentation/building/windows-test-instructions.md
index 631f036da9..de422ed571 100644
--- a/Documentation/building/windows-test-instructions.md
+++ b/Documentation/building/windows-test-instructions.md
@@ -21,9 +21,9 @@ This will use `crossgen.exe` to precompile the test executables before they are
 
 ## Building Other Priority Tests
 
-    C:\git\coreclr>build-test.cmd -priority=2
+    C:\git\coreclr>build-test.cmd -priority=1
 
-The number '2' is just an example. The default value (if no priority is specified) is 0. To clarify, if '2' is specified, all tests with CLRTestPriorty 0, 1 **and** 2 will be built and consequently run.
+The number '1' is just an example. The default value (if no priority is specified) is 0. To clarify, if '1' is specified, all tests with CLRTestPriorty 0 **and** 1 will be built and consequently run.
 
 ## Examples
 
@@ -35,7 +35,7 @@ To run a clean, priority 1, crossgen test pass:
 
 ### Building Individual Tests
 
-Note: buildtest.cmd or build.cmd skipnative skipmscorlib needs to be run atleast once
+Note: build-test.cmd or build.cmd skipnative needs to be run atleast once
 
 * Native Test: Build the generated Visual Studio solution or make file corresponding to Test cmake file.
   
@@ -47,7 +47,7 @@ Note: buildtest.cmd or build.cmd skipnative skipmscorlib needs to be run atleast
 
 For example to run all of the tests using your checked build:
 
-     <repo_root>\tests\runtest.cmd -checked
+     <repo_root>\tests\runtest.cmd checked
 
 This will generate a report named as `TestRun_<arch>_<flavor>.html` (e.g. `TestRun_Windows_NT__x64__Checked.html`) in the subdirectory `<repo_root>\bin\Logs`. Any tests that failed will be listed in `TestRunResults_Windows_NT__x64__Checked.err`.