| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
For corefx testing, use a matching corefx repo version git hash
|
|
* Convert perf.groovy to use run-xunit-perf.py
Also removes run-xunit-perf.cmd and run-xunit-perf.sh. This change moves us to using build-tests.sh to generate the layout (ie pull down corefx), so this change fixes #12143.
|
|
If the user doesn't specify a specific corefx commit hash to use,
try to find the matching commit hash in the coreclr repro. If the
matching hash can't be found, use 'HEAD'.
We find the matching corefx commit hash by first parsing file
'dependencies.props' at the root of the coreclr repro, looking for this:
<MicrosoftPrivateCoreFxNETCoreAppPackageVersion>4.5.0-preview1-26112-01</MicrosoftPrivateCoreFxNETCoreAppPackageVersion>
This determines the corefx package version that matches. Next, we look for the version.txt
file in the package cache, e.g.,
<coreclr_root>\packages\microsoft.private.corefx.netcoreapp\4.5.0-preview1-26112-01\version.txt
The contents of this file is exactly the git commit hash
we need to use, e.g.:
197a0699b08087ea85581679afdd9fd7b5c465c3
The version.txt file is created when the corefx package is restored,
which happens when doing one of:
Windows: tests\runtests.cmd GenerateLayoutOnly
non-Windows: build-test.sh generatelayoutonly
It would also be possible to not depend on the package already
being downloaded, but instead download the correct package here,
using the determined "MicrosoftPrivateCoreFxNETCoreAppPackageVersion"
package version, e.g.:
https://dotnet.myget.org/F/dotnet-core/api/v2/package/Microsoft.Private.CoreFx.NETCoreApp/4.5.0-preview1-26112-01
and then extracting the ZIP archive to find the version.txt file.
This might get easier if the corefx commit hash is added directly to dependencies.props, as
discussed in https://github.com/dotnet/buildtools/issues/1141.
|
|
* Convert run-xunit-perf to python script
This change merges the two run-xunit-perf scripts (.sh and .cmd) into
one unified python script and updates the pipeline job to use the
pythong script. This change also updates the linux jobs to use the new
build-tests.sh generatelayoutonly command so that we don't need to pull
down corefx from the cloud anymore. The unified python script enables us
to more easily update both linux scripting and windows scripting at the
same time so that one does not lag behind the other (such as when we add
new configurations or options like slicing). This change also turns
linux testing back on by default for PRs.
|
|
- Issue was fixed with this: https://github.com/aspnet/JitBench/pull/82
- Set the UseSharedCompilation=false to avoid that the `VBCSCompiler.exe` stays running.
|
|
* Change name fo perf jobs to reflect the actual OS they run on
* readd Ubuntu 14.04
|
|
|
|
-NoProfile (#15164)
Code execution in several scripts and tests can be affected by powershell profile
execution at startup. Adding -NoProfile option to all powershell invocations which
may be affected by execution of profile startup code.
|
|
- Added stability prefix to the scenario benchmark (JitBench)
- Specify output directory to the `run-xunit-perf.cmd` script and avoid the extra step to xcopy files to the archive folder.
- Added a command line parser class to the illink scenario, and changed its behavior where it used to fail when a new command line option passed to xUnit was not recognized.
- Save the output log of the tests into the sandbox-logs folder.
- Updating the label of the machine pool used by the illink scenario
|
|
For altjit, we can't run managed code from .NET installs that
are not under test. The XSLT Compiler tests invoke an XSLT
Compiler from the machine installed desktop framework. When it
sees the AltJit variables, but no altjit compiler dll, it dies.
So, don't run those tests. They have been annotated with a new
attribute to allow this exclusion.
|
|
There is a missing ! before checking to see if the throughput benchmarks
have already been downloaded. Currently, we download them if they
already exist, where we want to download them if they don't already
exist. This change fixes that.
|
|
This change converts our perf testing to use pipeline jobs. Pipeline
jobs allow us to do the following:
1) Test on the same commit for each of the test legs
2) Parallelize the build and test steps.
3) Separate the build and test steps from one another. This gives us the
ability to use the same build assets for all of the test legs of the
same configuration. It also allows us to build on virtual machines and
test on perf machines, so we only use the perf resources for testing.
4) Have different test scenarios for PRs and rolling. This isn't
strictly a benefit of pipeline jobs, but certainly is made easier by
them.
5) Allows us to have one trigger for PR jobs which will get us all the
perf testing scenarios.
This change also cleans up the groovy scripting for perf testing.
|
|
- Adding 'tiered' jitting
- Enable JitBench to capture Pmc data (Profile=On)
- Fixed some xUnit performance warnings
- Enforce DOTNET_MULTILEVEL_LOOKUP environment to be turned off
- Specify `dotnet publish` output directory.
|
|
Need 2.0 SDK now. Not sure of the best URLs but these seem to work.
Also shorten path so we don't run into windows pathlength issues.
|
|
Fix corefx testing invocation
|
|
In run-throughput-perf.py, we were calling submission.py with --arch,
but submission.py requires -arch or --architecture. This recently made
submission.py start failing in throughput jobs.
This change also fixes run-throughput-perf so that if submission.py or
upload.py fail, the script fails so we see these jobs as failing in the
lab.
|
|
Instead of executing the environment-setting script before calling
build-tests, pass the `/p:PreExecutionTestScript=<env_script>`
msbuild property, which will write the environment script directly
into the generated corefx RunTests.cmd scripts. Thus, the
environment changes will be made immediately before the test is run.
Depends on https://github.com/dotnet/buildtools/pull/1723.
|
|
* [Tizen/ARM32] Update armel rootfs for Tizen
- Add lttng-ust-dev to Tizen rootfs for cross build
- Use rootfs of the latest Tizen 4.0
Signed-off-by: Hyung-Kyu Choi <hk0110.choi@samsung.com>
* [Tizen/ARM32] Update Tizen CI script
- Update arm32 CI script for Tizen to use the latest environment
Signed-off-by: Hyung-Kyu Choi <hk0110.choi@samsung.com>
|
|
|
|
Add support for priority and update x64 client
|
|
|
|
For JitStressRegs, allow 0x10, 0x80, 0x1000; don't use simple 10 or 80
in names (to match netci.groovy).
Remove "tieredcompilation" from allowed modes; it is not supported in the
CI smarty helper currently.
|
|
Renames arm legacy backend jobs to "armlb" and keeps "arm"
for RyuJIT/arm32 testing.
|
|
Add OptLevel and JitName configurations
|
|
To unify the configuration names in benchview, add OptLevel and JitName
configurations in run-xunit-perf.sh. This change does not add min_opts
to perf.groovy.
|
|
|
|
This change includes:
1) lst_creator updates to allow adding priority tags automatically
2) arm32 lstFile updates: 29 new tests, 50 removed
3) arm64 lstFile updates: 80 new tests, 55 removed
|
|
The loop was iterating through fixed file name and a pattern. The fixed file name did not exist, and the whole function failed.
With this change, we loop through the files if they exist.
|
|
|
|
Fix running corefx tests
|
|
- IlLink scenario will run on a VM (currently it is only measuring MSIL size)
- Updated the patterns of files to search for on scenario runs
- Updated the JitBench commit hash being used ()
- Disable min_opt from the PR correctness mode.
- String was not being interpolated.
|
|
Running CoreFX tests was supposed to use a batch file (env_script) to run CoreFX tests under different environment configurations. However due to subtlties in how the command line is parsed none of the environment variables set by this script would actually persist for the execution of the tests. As best I can tell prior to this fix specifying a env_script had no effect whatsoever on windows, and our CI corefx test stress scenarios weren't actually testing the scenario they claimed to be testing.
cmd /c "Set.bat&&Check.bat" will propagate env vars set in Set.bat so that they are visible in Check.bat because both batch files execute within the context of the newly invoked cmd.exe process. However cmd /c Set.bat&&Check.bat (no quotes) will run Set.bat in its own cmd.exe process, then exit that process and run Check.bat in the parent shell process. The parent process does not have any environment variable changes that were made by Set.bat.
|
|
|
|
- Move xUnit Perf Api output files to a folder `Logs` under sandbox
- Made necessary changes to `perf.groovy` to archive artifacts
- Added a switch -outputdir to the run-xunit-perf* scripts
- Enable running all CoreClr scenarios (JitBench & IlLink) despite failure
- Move sandbox to bin folder
- Use the default output dir: `.\bin\sandbox`
|
|
This change adds an opt_level throughput option so that we can run
throughput jobs in minopts mode. This also adds jobs to the lab for min
opts runs
|
|
|
|
|
|
Note this will add a new lst file and leverage the old existing arm64
infrastructure.
|
|
Fixes #12171.
* bootstrap.py was printing an improperly-cased path to put PGO/IBC
training data, which breaks on case-sensitive file systems, i.e., on
Linux.
* On Linux, `skiprestoreoptdata` was improperly skipping PGO even if
optdata was already present due to some of the conditional checks in
build.sh. This cleans that up to better match the intent of the
switch, and to fix Linux behavior to better match Windows.
|
|
There are many assemblies in our throughput benchmark package that are
reference or forwarding assemblies. There is no jit work in these
assemblies, so we want to exclude them.
This change also adds an iterations parameter so users can specify a
specific number of iterations for testing.
|
|
* Replace ScriptGenerator project.json with csproj
* Update project.json occurrences in docs
Fix few typos.
* Remove project.json comments
|
|
On Windows, we download a set of assemblies that we have uploaded to
MyGet. On Linux, we were building corefx each run to get the benchmarks.
This change updates Linux to pull down the same benchmarks Windows uses,
using curl to grab the nuget package from MyGet.
Note: after this commit, we will have data for fewer assemblies on
Linux, but we will now have the same set for both Windows and Linux.
Some data in Benchview may be affected.
|
|
|
|
SPC and the jit/coreclr are tightly coupled. With the changes to
crossgen into the dll, rather than the ni.dll, the throughput jobs are
now using the non-crossgen'd version of SPC on TP runs, which led to a
regression in TP on Linux because our comparison was not the same. This
change updates testing to replace the System.Private.CoreLib in the
throughput directory with the System.Private.CoreLib built by the
coreclr build.
|
|
* Return 0 on success from GC Reliability Framework scripts
* Don't save standard out, Jenkins already does this
|
|
Fix optdata restore functionality (#11935)
Bring back optdata restore functionality following the move to 2.0
BuildTools and csproj-based CLI. Disable a harmless warning that broke
the Linux build when consuming PGO counts due to -Werr.
Fix #11796 for master
|
|
- This update contains a fix for this issue: https://github.com/Microsoft/xunit-performance/issues/230
- Consolidate the Api version in two places: `dependencies.prop` and `PerfHarness.csproj`
- Move performance log files to root directory where Jenkins can archive them
- Fixed the output to console, so it is easier to see on the Jenkins job the sccript steps live and get an exact repro step if needed
|
|
- Removing `[assembly: MeasureInstructionsRetired]` from performance tests.
- Due to this issue: https://github.com/Microsoft/xunit-performance/issues/231, we are blocked from running on performance tests on Linux.
The tag is not used by infrastructure and it is passed during runtime when profiling on Windows.
- Adding the option to execute CoreRun with the stability prefix.
- Moving file to be archived by Jenkins and updating groovy file.
- Setting the maximum number of iterations.
|
|
|
|
Tizen basically requires GDBJIT enabled libraries.
Enable GDBJIT feature to prevent related build fail issues in advance.
|
