Update documentation for testing (#19063)

* Update documentation for testing

3 files changed, 94 insertions, 97 deletions

diff --git a/Documentation/building/test-configuration.md b/Documentation/building/test-configuration.md
index 127495978e..14fbb4e3d6 100644
--- a/Documentation/building/test-configuration.md
+++ b/Documentation/building/test-configuration.md
@@ -32,23 +32,54 @@ Test cases are categorized by priority level. The most important subset should b
 ## Adding Test Guidelines
 
 * All test source files should include the following banner:
-```
-    // Licensed to the .NET Foundation under one or more agreements.
-    // The .NET Foundation licenses this file to you under the MIT license.
-    // See the LICENSE file in the project root for more information.
-```
+    ```
+        // Licensed to the .NET Foundation under one or more agreements.
+        // The .NET Foundation licenses this file to you under the MIT license.
+        // See the LICENSE file in the project root for more information.
+    ```
 * Disable building of a test by conditionally setting the `<DisableProjectBuild>` property.
 	* e.g. `<DisableProjectBuild Condition=" '$(Platform)' == 'arm64' ">true</DisableProjectBuild>`
-* Add NuGet/MyGet references by updating the following [project file](https://github.com/dotnet/coreclr/blob/master/tests/src/Common/test_dependencies/test_dependencies.csproj).
+* Add NuGet/MyGet references by updating the following [test project](https://github.com/dotnet/coreclr/blob/master/tests/src/Common/test_dependencies/test_dependencies.csproj).
 * Build against the `mscorlib` facade by adding `<ReferenceLocalMscorlib>true</ReferenceLocalMscorlib>` to the test project.
+* Update relevent exclusion lists:
+  * Note that there are two build pipelines Jenkin's CI and nightly Helix - both must be updated for expected exclusion.
+    * Jenkin's CI build - see the associated `*.txt` files under `tests/` (e.g. `tests/testsUnsupportedOutsideWindows.txt`).
+    * Official nightly Helix build - see the `tests/issues.targets` file.
+  * ARM/ARM64
+    * `tests/arm/Tests.lst` and `tests/arm64/Tests.lst` are used to define the tests to run due to limitations with XUnit.
+    * These files can be manually edited or the generated using `tests/scripts/lst_creator.py`.
 
-### Creating a new C# test project
+### Creating a C# test project
 
-**TODO**
+1. Use an existing test such as `<repo_root>\tests\src\Exceptions\Finalization\Finalizer.csproj` as a template and copy it to a new folder under `<repo_root>\tests\src`.
+1. Be sure that the `<AssemblyName>` property has been removed
 
-### Converting an existing C# project
-  * Remove the `<AssemblyName>` property
-  * Import `dir.props`
-  * Import `dir.targets`
-  * Assign a `<CLRTestKind>`
-  * (optional) Assign a priority value
\ No newline at end of file
+    * Not removing this can cause confusion with the way tests are generally handled behind the scenes by the build system.
+
+1. Set the `<CLRTestKind>`/`<CLRTestPriority>` properties.
+1. Add source files to new project.
+1. Indicate the success of the test by returning `100`. Failure can be indicated by any non-`100` value.
+
+    Example:
+    ```
+        static public int Main(string[] notUsed)
+        {
+            try
+            {
+                // Test scenario here
+            }
+            catch (Exception e)
+            {
+                Console.WriteLine($"Test Failure: {e.Message}");
+                return 101;
+            }
+
+            return 100;
+        }
+    ```
+
+1. Add any other projects as a dependency, if needed.
+    * Managed reference: `<ProjectReference Include="../ManagedDll.csproj" />`
+    * Native reference: `<ProjectReference Include="../NativeDll/CMakeLists.txt" />`
+1. Build the test.
+1. Follow the steps to re-run a failed test to validate the new test.
diff --git a/Documentation/building/unix-test-instructions.md b/Documentation/building/unix-test-instructions.md
index bd15f173aa..89fead7aa6 100644
--- a/Documentation/building/unix-test-instructions.md
+++ b/Documentation/building/unix-test-instructions.md
@@ -4,11 +4,11 @@ Building and running tests on Linux, OS X, and FreeBSD
 CoreCLR tests
 -------------
 
-**Building**
+## Building
 
 Build CoreCLR on [Unix](https://github.com/dotnet/coreclr/blob/master/Documentation/building/linux-instructions.md).
 
-**Building the Tests**
+## Building the Tests
 
 DotNet is required to build the tests, this can be done on any platform then copied over if the arch or os does not support DotNet. If DotNet is not supported, [CoreFX](https://github.com/dotnet/corefx/blob/master/Documentation/building/unix-instructions.md) is also required to be built.
 
@@ -16,27 +16,19 @@ To build the tests on Unix:
 
 > `./build-test.sh`
 
-To build on Windows:
-
-> `C:\coreclr>build-test.cmd`
-
 Please note that this builds the Priority 0 tests. To build priority 1:
 
 > `build-test.sh -priority 1`
 
+## Building Individual Tests
 
-**Building Individual Tests**
+During development there are many instances where building an individual test is fast and necessary. All of the necessary tools to build are under `coreclr/Tools`. It is possible to use `coreclr/Tools/MSBuild.dll` as you would normally use MSBuild with a few caveats.
 
-During development there are many instances where building an individual test is fast and necessary. All of the necessary tools to build are under coreclr/Tools. It is possible to use coreclr/Tools/MSBuild.dll as you would normally use MSBuild with a few caveats.
-
-Note that coreclr/Tools/msbuild.sh exists as well to make the call shorter.
+Note that `coreclr/Tools/msbuild.sh` exists as well to make the call shorter.
 
 **!! Note !! -- Passing /p:__BuildOs=[OSX|Linux] is required.** 
 
->If you omit it you will get the following error: `error MSB4801: The task factory "CodeTaskFactory" could not be loaded because this version of MSBuild does not support it.`
----
-
-**Building an Individual Test Example**
+## Building an Individual Test Example
 
 >`coreclr/Tools/msbuild.sh /maxcpucount  coreclr/tests/src/JIT/CodeGenBringUpTests/Array1.csproj /p:__BuildType=Release /p:__BuildOS=OSX`
 
@@ -44,8 +36,7 @@ Or
 
 >`coreclr/Tools/dotnetcli/dotnet coreclr/Tools/MSBuild.dll /maxcpucount coreclr/tests/src/JIT/CodeGenBringUpTests/Array1.csproj /p:__BuildType=Release /p:__BuildOS=OSX`
 
-
-**Aarch64/armhf multiarch**
+## Aarch64/armhf multi-arch
 
 For machines that have aarch64/armhf support, all the armhf packages will need to also be downloaded. Please note you will need to enable multiplatform support as well. Check with your distro provider or kernel options to see if this is supported. For simplicity, these instructions relate to aarch64 ubuntu enabling arm32 (hf) coreclr runs.
 
@@ -67,7 +58,7 @@ Linux tegra-ubuntu 4.4.38-tegra #1 SMP PREEMPT Thu Jul 20 00:41:06 PDT 2017 aarc
 [ubuntu:~]: sudo apt-get install libstdc++6:armhf
 ````
 
-At this point you should be able to run a 32bit corerun. You can verify this by downloading and running a recently built arm32 coreclr.
+At this point you should be able to run a 32-bit `corerun`. You can verify this by downloading and running a recently built arm32 coreclr.
 
 ```
 [ubuntu:~]: wget https://ci.dot.net/job/dotnet_coreclr/job/master/job/armlb_cross_checked_ubuntu/lastSuccessfulBuild/artifact/*zip*/archive.zip --no-check-certificate
@@ -85,7 +76,7 @@ Now download the coreclr armhf dependencies.
 sudo apt-get install libunwind8:armhf libunwind8-dev:armhf libicu-dev:armhf liblttng-ust-dev:armhf libcurl4-openssl-dev:armhf libicu-dev:armhf libssl-dev libkrb5-dev:armhf
 ```
 
-**Running tests**
+## Running Tests
 
 The following instructions assume that on the Unix machine:
 - The CoreCLR repo is cloned at `/mnt/coreclr`
@@ -96,7 +87,7 @@ If DotNet is unsupported
 
 The following steps are different if DotNet is supported or not on your arch and os.
 
-**DotNet is supported**
+### DotNet is supported
 
 build-test.sh will have setup the Core_Root directory correctly after the test build. If this was either skipped or needs to be regenerated use:
 
@@ -104,50 +95,50 @@ build-test.sh will have setup the Core_Root directory correctly after the test b
 
 To run the tests run with the --coreOverlayDir path
 
-> ```bash
-> ~/coreclr$ tests/runtest.sh
->     --testRootDir=/mnt/coreclr/bin/tests/Linux.x64.Debug
->     --testNativeBinDir=/mnt/coreclr/bin/obj/Linux.x64.Debug/tests
->     --coreOverlayDir=/mnt/coreclr/bin/tests/Linux.x64.Debug/Tests/Core_Root
->     --copyNativeTestBin
-> ```
+```bash
+~/coreclr$ tests/runtest.sh
+    --testRootDir=/mnt/coreclr/bin/tests/Linux.x64.Debug
+    --testNativeBinDir=/mnt/coreclr/bin/obj/Linux.x64.Debug/tests
+    --coreOverlayDir=/mnt/coreclr/bin/tests/Linux.x64.Debug/Tests/Core_Root
+    --copyNativeTestBin
+```
 
-**DotNet is not supported**
+### DotNet is not supported
 
 Tests need to be built on another platform and copied over to the Unix machine for testing. Copy the test build over to the Unix machine:
 
 > `cp --recursive /media/coreclr/bin/tests/Windows_NT.x64.Debug /mnt/test/`
 
-See runtest.sh usage information:
+See `runtest.sh` usage information:
 
 > `/mnt/coreclr$ tests/runtest.sh --help`
 
 Run tests (`Debug` may be replaced with `Release` or `Checked`, depending on which Configuration you've built):
 
-> ```bash
-> /mnt/coreclr$ tests/runtest.sh
->     --testRootDir=/mnt/test/Windows_NT.x64.Debug
->     --testNativeBinDir=/mnt/coreclr/bin/obj/Linux.x64.Debug/tests
->     --coreClrBinDir=/mnt/coreclr/bin/Product/Linux.x64.Debug
->     --mscorlibDir=/mnt/coreclr/bin/Product/Linux.x64.Debug
->     --coreFxBinDir=/mnt/corefx/bin/runtime/netcoreapp-Linux-Debug-x64
-> ```
+```bash
+/mnt/coreclr$ tests/runtest.sh
+    --testRootDir=/mnt/test/Windows_NT.x64.Debug
+    --testNativeBinDir=/mnt/coreclr/bin/obj/Linux.x64.Debug/tests
+    --coreClrBinDir=/mnt/coreclr/bin/Product/Linux.x64.Debug
+    --mscorlibDir=/mnt/coreclr/bin/Product/Linux.x64.Debug
+    --coreFxBinDir=/mnt/corefx/bin/runtime/netcoreapp-Linux-Debug-x64
+```
 
 The method above will copy dependencies from the set of directories provided to create an 'overlay' directory.
 
-**Results**
+### Results
 
 Test results will go into:
 
 > `~/test/Windows_NT.x64.Debug/coreclrtests.xml`
 
-**Unsupported and temporarily disabled tests**
+### Unsupported and temporarily disabled tests
 
 These tests are skipped by default:
 - Tests that are not supported outside Windows, are listed in:
->> `~/coreclr/tests/testsUnsupportedOutsideWindows.txt`
+    > `~/coreclr/tests/testsUnsupportedOutsideWindows.txt`
 - Tests that are temporarily disabled outside Windows due to unexpected failures (pending investigation), are listed in:
->> `~/coreclr/tests/testsFailingOutsideWindows.txt`
+    > `~/coreclr/tests/testsFailingOutsideWindows.txt`
 
 To run only the set of temporarily disabled tests, pass in the `--runFailingTestsOnly` argument to `runtest.sh`.
 
diff --git a/Documentation/building/windows-test-instructions.md b/Documentation/building/windows-test-instructions.md
index 99aa11e613..631f036da9 100644
--- a/Documentation/building/windows-test-instructions.md
+++ b/Documentation/building/windows-test-instructions.md
@@ -1,39 +1,39 @@
 Building and running tests on Windows
 =====================================
 
-**Building Tests**        
+## Building Tests
 
 To build the tests simply navigate to the tests directory above the repo and run,
 
     C:\git\coreclr>build-test.cmd
 
-*Cleaning Tests*
+## Cleaning Tests
 
 **Note:** Cleaning should be done before all tests to be sure that the test assets are initialized correctly. To do a clean build of the tests, in a clean command prompt, issue the following command: 
 
     C:\git\coreclr>build-test.cmd -rebuild
 
-*Building tests that will be precompiled*
+## Building Tests that will be precompiled
 
     C:\git\coreclr>build-test.cmd crossgen
 
-This will use crossgen.exe to precompile the test executables before they are executed.
+This will use `crossgen.exe` to precompile the test executables before they are executed.
 
-*Building Other Priority Tests*
+## Building Other Priority Tests
 
     C:\git\coreclr>build-test.cmd -priority=2
 
-The number '2' is just an example. The default value (if no priority is specified) is 0. To clarify, if '2' is specified, all tests with CLRTestPriorty 0, 1 AND 2 will be built and consequently run.
+The number '2' is just an example. The default value (if no priority is specified) is 0. To clarify, if '2' is specified, all tests with CLRTestPriorty 0, 1 **and** 2 will be built and consequently run.
 
-**Example**
+## Examples
 
 To run a clean, priority 1, crossgen test pass:
 
     C:\git\coreclr>build-test.cmd -rebuild crossgen -priority=1
 
-**buildtest /?** will list additional supported parameters.
+`build-test.cmd /?` - will list additional supported parameters.
 
-**Building Individual Tests**
+### Building Individual Tests
 
 Note: buildtest.cmd or build.cmd skipnative skipmscorlib needs to be run atleast once
 
@@ -41,66 +41,41 @@ Note: buildtest.cmd or build.cmd skipnative skipmscorlib needs to be run atleast
   
 * Managed Test: You can invoke msbuild on the project directly from Visual Studio Command Prompt.
 
-**Running Tests**
+### Running Tests
 
-**runtest /?** will list supported parameters.
+`runtest.cmd /?` - will list supported parameters.
 
 For example to run all of the tests using your checked build:
 
-`<repo_root>\tests\runtest.cmd checked`
+     <repo_root>\tests\runtest.cmd -checked
 
 This will generate a report named as `TestRun_<arch>_<flavor>.html` (e.g. `TestRun_Windows_NT__x64__Checked.html`) in the subdirectory `<repo_root>\bin\Logs`. Any tests that failed will be listed in `TestRunResults_Windows_NT__x64__Checked.err`.
 
-**Investigating Test Failures**
+### Investigating Test Failures
 
 Upon completing a test run, you may find one or more tests have failed.
 
-The output of the Test will be available in Test reports directory, but the default the directory would be something like is `<repo_root>\bin\tests\Windows_NT.x64.Checked\Reports\Exceptions\Finalization`.  
+The output of the Test will be available in Test reports directory, but the default the directory would be something like is `<repo_root>\bin\tests\Windows_NT.x64.Checked\Reports\Exceptions\Finalization`.
 
 There are 2 files of interest: 
 
 - `Finalizer.output.txt` - Contains all the information logged by the test.
 - `Finalizer.error.txt`  - Contains the information reported by CoreRun.exe (which executed the test) when the test process crashes.
 
-**Rerunning a failed test**
+### Rerunning a failed test
 
 If you wish to re-run a failed test, please follow the following steps:
 
 1. Set an environment variable, `CORE_ROOT`, pointing to the path to product binaries that was passed to runtest.cmd.
 For example using a checked build the location would be: `<repo_root>\bin\tests\Windows_NT.x64.Checked\Tests\Core_Root`
 
-2. Next, run the failed test, the command to which is also present in the test report for a failed test. It will be something like `<repo_root>\bin\tests\Windows_NT.x64.Checked\Exceptions\Finalization\Finalizer.cmd`.
+1. Next, run the failed test, the command to which is also present in the test report for a failed test. It will be something like `<repo_root>\bin\tests\Windows_NT.x64.Checked\Exceptions\Finalization\Finalizer.cmd`.
 
 If you wish to run the test under a debugger (e.g. [WinDbg](http://msdn.microsoft.com/en-us/library/windows/hardware/ff551063(v=vs.85).aspx)), append `-debug <debuggerFullPath>` to the test command. For example:
 
      <repo_root>\bin\tests\Windows_NT.x64.Checked\Exceptions\Finalization\Finalizer.cmd -debug <debuggerFullPath>
-    
-**Modifying a test**
 
-If test changes are needed, make the change and build the test project. This will binplace the binaries in test binaries folder (e.g. `<repo_root>\bin\tests\Windows_NT.x64.Checked\Exceptions\Finalization`). At this point, follow the steps to re-run a failed test to re-run the modified test.
-
-**Authoring Tests (in VS)**
-
-
-1. Use an existing test such as `<repo_root>\tests\src\Exceptions\Finalization\Finalizer.csproj` as a template and copy it to a new folder under `<repo_root>\tests\src`.
-2. Be sure that the AssemblyName has been removed (this causes confusion with the way tests are generally handled behind the scenes by the build system). 
-3. [Assign a CLRTestKind/CLRTestPriority.](test-configuration.md)
-4. Add source files to this newly added project.
-5. Indicate the success of the test by returning `100`.
-6. Add any other projects as a dependency, if needed.
-7. Build the test.
-8. Follow the steps to re-run a failed test to validate the new test.
+### Modifying a test
 
-Note:
-
-1. You can disable building of a test per architecture or configuration by using DisableProjectBuild tag in the project. for example:
-
-  ```xml
-  <PropertyGroup>
-    <DisableProjectBuild Condition=" '$(Platform)' == 'arm64' ">true</DisableProjectBuild>
-  </PropertyGroup>
-  ```
-
-2. To add Nuget/MyGet references use this (project file)[https://github.com/dotnet/coreclr/blob/master/tests/src/Common/test_dependencies/test_dependencies.csproj].
+If test changes are needed, make the change and build the test project. This will binplace the binaries in test binaries folder (e.g. `<repo_root>\bin\tests\Windows_NT.x64.Checked\Exceptions\Finalization`). At this point, follow the steps to re-run a failed test to re-run the modified test.
 
-3. To build against the mscorlib facade add `<ReferenceLocalMscorlib>true</ReferenceLocalMscorlib>` to your project.