diff options
| -rw-r--r-- | Documentation/building/test-configuration.md | 41 | ||||
| -rw-r--r-- | Documentation/building/windows-test-instructions.md | 47 | ||||
| -rw-r--r-- | Documentation/project-docs/tests.md | 8 | ||||
| -rw-r--r-- | tests/buildtest.cmd | 11 | ||||
| -rw-r--r-- | tests/src/CLRTest.Execute.Batch.targets | 51 | ||||
| -rw-r--r-- | tests/src/dir.props | 1 |
6 files changed, 134 insertions, 25 deletions
diff --git a/Documentation/building/test-configuration.md b/Documentation/building/test-configuration.md new file mode 100644 index 0000000000..a1e67a6fef --- /dev/null +++ b/Documentation/building/test-configuration.md @@ -0,0 +1,41 @@ +## General Test Infrastructure Notes ## + +### Kinds of Build Properties ### +* Build Only +> `<CLRTestKind>BuildOnly</CLRTestKind>` + + * Builds an executable. + * Will not execute it. + +* Run Only +> `<CLRTestKind>RunOnly</CLRTestKind>` + + * Will only execute another assembly. +* Build and Run +> `<CLRTestKind>BuildAndRun</CLRTestKind>` + + * Builds an executable. + * Will execute said executable. +* Shared Libraries +> `<CLRTestKind>SharedLibrary</CLRTestKind>` + + * For building libraries common to zero or more tests. + + +By default (i.e. if not specified explicitly) a project file is BuildAndRun. + +### Priority ### +Testcases are categorized by their priority levels. The most important subset should be and is the smallest subset. This subset is called priority 0. + * By default, a testcase is priority 0. You must elect to de-prioritize a test. + * To de-prioritize a test, add a property _CLRTestPriority_ to the test's project file. +> `<CLRTestPriority>2</CLRTestPriority>` + * Lower priority values are always run in conjunction when running higher priority value tests. I.e. if a developer elects to do a priority 2 test run, then all priority 0, 1 and 2 tests are run. + +### Adding Tests ### +#### Converting an existing C# project #### + * Remove AssemblyName + * Swap in dir.props + * Swap in dir.targets + * Assign a CLRTestKind + * (optional) Assign a priority value + diff --git a/Documentation/building/windows-test-instructions.md b/Documentation/building/windows-test-instructions.md index 1935ce550c..6b74834048 100644 --- a/Documentation/building/windows-test-instructions.md +++ b/Documentation/building/windows-test-instructions.md @@ -3,15 +3,37 @@ Building and running tests on Windows **Building Tests** -In a clean command prompt, issue the following command: +To build the tests simply navigate to the tests directory above the repo and run, + + C:\git\coreclr>tests\buildtest.cmd + +*Cleaning Tests* + +**Note:** Cleaning should be done before all tests to be sure that the test assets are initialized correctly. To do a clean build of the tests, in a clean command prompt, issue the following command: C:\git\coreclr>tests\buildtest.cmd clean -**buildtest /?** will list supported parameters. +*Building tests that will CrossGen* + + C:\git\coreclr>tests\buildtest.cmd crossgen + +This will enable crossgen.exe to be run against test executables before they are executed. + +*Building Other Priority Tests* -**Note:** The above command (or building from the repo_root) must be done once, at the least, to ensure that all test dependencies are initialized correctly. + C:\git\coreclr>tests\buildtest.cmd priority 2 -In Visual Studio, open `<repo_root>\tests\src\AllTestProjects.sln`, build all the test projects or the one required. +The number '2' is just an example. The default value (if no priority is specified) is 0. To clarify, if '2' is specified, all tests with CLRTestPriorty 0, 1 AND 2 will be built and consequently run. + +**Example** + +To run a clean, priority 1, crossgen test pass: + + C:\git\coreclr>tests\buildtest.cmd clean crossgen priority 1 + +**buildtest /?** will list additional supported parameters. + +Additionally, there is a Visual Studio solution, `<repo_root>\tests\src\AllTestProjects.sln`, where users can build a particular testcase, or all priority 0 testcases that are within it. **Running Tests** @@ -50,11 +72,14 @@ If test changes are needed, make the change and build the test project. This wil **Authoring Tests (in VS)** + 1. Use an existing test such as `<repo_root>\tests\src\Exceptions\Finalization\Finalizer.csproj` as a template and copy it to a new folder under `<repo_root>\tests\src`. -2. Add the project of the new test to `<repo_root>\tests\src\AllTestProjects.sln` in VS -3. Add source files to this newly added project. -4. Indicate the success of the test by returning `100`. -5. Add the .NET CoreFX contract references, as required, via the Nuget Package Manager in Visual Studio. *Make sure this does not change the csproj. If it does, then undo the change in the csproj.* -6. Add any other projects as a dependency, if needed. -7. Build the test. -8. Follow the steps to re-run a failed test to validate the new test. +2. Be sure that the AssemblyName has been removed (this causes confusion with the way tests are generally handled behind the scenes by the build system). +3. [Assign a CLRTestKind/CLRTestPriority.](test-configuration.md) +4. Add the project of the new test to `<repo_root>\tests\src\AllTestProjects.sln` in VS +5. Add source files to this newly added project. +6. Indicate the success of the test by returning `100`. +7. Add the .NET CoreFX contract references, as required, via the Nuget Package Manager in Visual Studio. *Make sure this does not change the csproj. If it does, then undo the change in the csproj.* +8. Add any other projects as a dependency, if needed. +9. Build the test. +10. Follow the steps to re-run a failed test to validate the new test. diff --git a/Documentation/project-docs/tests.md b/Documentation/project-docs/tests.md index a1e67a6fef..ca6a5ef740 100644 --- a/Documentation/project-docs/tests.md +++ b/Documentation/project-docs/tests.md @@ -1,4 +1,3 @@ -## General Test Infrastructure Notes ## ### Kinds of Build Properties ### * Build Only @@ -31,11 +30,4 @@ Testcases are categorized by their priority levels. The most important subset sh > `<CLRTestPriority>2</CLRTestPriority>` * Lower priority values are always run in conjunction when running higher priority value tests. I.e. if a developer elects to do a priority 2 test run, then all priority 0, 1 and 2 tests are run. -### Adding Tests ### -#### Converting an existing C# project #### - * Remove AssemblyName - * Swap in dir.props - * Swap in dir.targets - * Assign a CLRTestKind - * (optional) Assign a priority value diff --git a/tests/buildtest.cmd b/tests/buildtest.cmd index 92d6dc0c1b..6d4f16c1b6 100644 --- a/tests/buildtest.cmd +++ b/tests/buildtest.cmd @@ -29,11 +29,16 @@ if /i "%1" == "clean" (set __CleanBuild=1&shift&goto Arg_Loop) if /i "%1" == "vs2013" (set __VSVersion=%1&shift&goto Arg_Loop) if /i "%1" == "vs2015" (set __VSVersion=%1&shift&goto Arg_Loop) +if /i "%1" == "crossgen" (set _crossgen=true&shift&goto Arg_Loop) +if /i "%1" == "priority" (set _priorityvalue=%2&shift&shift&goto Arg_Loop) + goto Usage :ArgsDone +if defined _crossgen echo Building tests with CrossGen enabled.&set _buildParameters=%_buildParameters% /p:CrossGen=true +if defined _priorityvalue echo Building Test Priority %_priorityvalue%&set _buildParameters=%_buildParameters% /p:CLRTestPriorityToBuild=%_priorityvalue% if not defined __BuildArch set __BuildArch=x64 if not defined __BuildType set __BuildType=Debug @@ -187,7 +192,7 @@ exit /b %ERRORLEVEL% :build -%_buildprefix% %_msbuildexe% "%__ProjectFilesDir%\build.proj" %__MSBCleanBuildArgs% /nologo /maxcpucount /verbosity:minimal /nodeReuse:false /fileloggerparameters:Verbosity=diagnostic;LogFile="%__TestManagedBuildLog%";Append %* %_buildpostfix% +%_buildprefix% %_msbuildexe% "%__ProjectFilesDir%\build.proj" %__MSBCleanBuildArgs% /nologo /maxcpucount /verbosity:minimal /nodeReuse:false %_buildParameters% /fileloggerparameters:Verbosity=diagnostic;LogFile="%__TestManagedBuildLog%";Append %* %_buildpostfix% IF ERRORLEVEL 1 echo Test build failed. Refer to !__TestManagedBuildLog! for details && exit /b 1 exit /b 0 @@ -200,10 +205,12 @@ exit /b 0 :Usage echo. echo Usage: -echo %0 BuildArch BuildType [clean] [vsversion] where: +echo %0 BuildArch BuildType [clean] [vsversion] [crossgen] [priority N] where: echo. echo BuildArch can be: x64 echo BuildType can be: Debug, Release echo Clean - optional argument to force a clean build. echo VSVersion - optional argument to use VS2013 or VS2015 (default VS2015) +echo CrossGen - Enables the tests to run crossgen on the test executables before executing them. +echo Priority (N) where N is a number greater than zero that signifies the set of tests that will be built and consequently run. exit /b 1 diff --git a/tests/src/CLRTest.Execute.Batch.targets b/tests/src/CLRTest.Execute.Batch.targets index acc1eaf8eb..88ca8797f1 100644 --- a/tests/src/CLRTest.Execute.Batch.targets +++ b/tests/src/CLRTest.Execute.Batch.targets @@ -18,6 +18,30 @@ WARNING: When setting properties based on their current state (for example: --> <Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> + <ItemGroup> + <CLRTestEnvironmentVariable Condition="'$(CrossGen)' == 'true'" Include="set complus_zaprequire=2" /> + <CLRTestEnvironmentVariable Condition="'$(CrossGen)' == 'true'" Include="set complus_zaprequireexcludelist=corerun" /> + </ItemGroup> + <!-- + Target: GetBatchCrossgenScript + This returns the portion of the execution script that generates the required lines to crossgen the test executable. + --> + <Target + Name="GetCrossgenBatchScript" + Returns="$(CrossgenBatchScript)"> + + <PropertyGroup> +<!-- CrossGen will create output if it needs to crossgen. Otherwise there will be silence. --> + <CrossgenBatchScript Condition="'$(CLRTestKind)' == 'BuildOnly' + OR '$(CLRTestKind)' == 'BuildAndRun' + OR '$(CLRTestKind)' == 'SharedLibrary'"> + <![CDATA[ +if not exist "$(OutputPath)\$(MSBuildProjectName).ni.$(OutputType)" "%CORE_ROOT%\crossgen.exe" /Platform_Assemblies_Paths %CORE_ROOT%%3B. "$(OutputPath)\$(MSBuildProjectName).$(OutputType)" + ]]> + </CrossgenBatchScript> + </PropertyGroup> + </Target> + <!-- Target: GetExecuteCmdFullPath Return Executed Cmd Relative Full Path @@ -49,9 +73,11 @@ WARNING: When setting properties based on their current state (for example: Inputs="$(MSBuildProjectFullPath)" Outputs="$(OutputPath)\$(MSBuildProjectName).cmd"> - <PropertyGroup> + <CallTarget Targets="GetCrossgenBatchScript" Condition="'$(CrossGen)' == 'true'"> + <Output TaskParameter="TargetOutputs" PropertyName="CrossgenBatchScript"/> + </CallTarget> - <!-- TODO:2 build in debugger experience support --> + <PropertyGroup> <BatchCLRTestExitCodePrep Condition="$(_CLRTestNeedsToRun)"><![CDATA[ set CLRTestExpectedExitCode=$(CLRTestExitCode) @@ -103,6 +129,13 @@ IF NOT "%CLRTestExitCode%"=="%CLRTestExpectedExitCode%" ( <Message Text="Project depends on $(_CLRTestToRunFileFullPath)." Condition="'$(_CLRTestNeedsProjectToRun)' == 'True'" /> + <ItemGroup> + <CLRTestEnvironmentVariable Condition="'$(CrossGen)' == 'true' AND '$(_CLRTestNeedsProjectToRun)' == 'true'" Include="set complus_zaprequirelist=$(_CLRTestToRunFileFullPath)" /> + <CLRTestEnvironmentVariable Condition="'$(CrossGen)' == 'true' AND '$(_CLRTestNeedsProjectToRun)' == 'false'" Include="set complus_zaprequirelist=$(MSBuildProjectName)" /> + </ItemGroup> + + + <PropertyGroup> <_CLRTestRunFile Condition="'$(_CLRTestNeedsProjectToRun)' == 'true'">"$(_CLRTestToRunFileFullPath)"</_CLRTestRunFile> <_CLRTestRunFile Condition=" '$(CLRTestIsHosted)'=='true' And $(_CLRTestNeedsProjectToRun) ">"%CORE_ROOT%\corerun.exe" $(_CLRTestToRunFileFullPath)</_CLRTestRunFile> @@ -115,9 +148,13 @@ ECHO $(_CLRTestRunFile) %CLRTestExecutionArguments% %Host_Args% %_DebuggerFullPath% $(_CLRTestRunFile) %CLRTestExecutionArguments% %Host_Args% set CLRTestExitCode=%ERRORLEVEL% ]]></BatchCLRTestLaunchCmds> - + </PropertyGroup> + <PropertyGroup> + <BatchEnvironmentVariables> +@(CLRTestEnvironmentVariable -> '%(Identity)', '%0d%0a') + </BatchEnvironmentVariables> </PropertyGroup> - + <Message Text="MSBuildProjectDirectory:$(MSBuildProjectDirectory)" /> <Message Text="_CLRTestToRunFileFullPath:$(_CLRTestToRunFileFullPath)"/> <Message Text="_CLRTestRunFile:$(_CLRTestRunFile)" /> @@ -193,6 +230,12 @@ $(BatchCLRTestExitCodePrep) REM The __TestEnv variable may be used to specify something to run before the test. IF NOT "%__TestEnv%"=="" call %__TestEnv% +REM Environment Variables +$(BatchEnvironmentVariables) + +REM CrossGen +$(CrossgenBatchScript) + REM Precommands $(CLRTestBatchPreCommands) REM Launch diff --git a/tests/src/dir.props b/tests/src/dir.props index 6fe1cd2d1c..8803c7fafd 100644 --- a/tests/src/dir.props +++ b/tests/src/dir.props @@ -11,6 +11,7 @@ <NoWarn>78,162,164,168,169,219,251,252,414,429,642,649,652,675,1691,1717,1718,3001,3002,3003,3005,3008</NoWarn> <GenerateAssemblyInfo>false</GenerateAssemblyInfo> <CLRTestKind>BuildAndRun</CLRTestKind> + <SkipSigning Condition="'$(CrossGen)' == 'true'">true</SkipSigning> </PropertyGroup> </Project> |