| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
In anticipation of taking a dependence on netcoreapp2.1 for the
forthcoming PMI work from jitutils.
Also make sure to shut down build servers before cleaning up the SDK.
|
|
* Add Word2Vec Benchmark Harness
This change adds an additional scenario benchmark, the Word2Vec
benchmark. The harness pulls down Word2Vec.Net from eabdullin, applies a
patch of changes that we made to work with netcoreapp21, harness the
word training and search, and then runs the benchmark. It also updates
the timeout for running benchmarks, since the training scenario on a
100M file takes about 7 minutes locally.
|
|
Things have progressed far enough that its time to use a friendlier name. The feature still still has performance aspects that need to be investigated and improved, but I don't want to scare people off simply because it isn't as fast as it could be.
This also updates to use a newer CoreFX version for JitBench since that appeared to be broken, and updated some comments and usage of the tieredcompilation variable.
|
|
Fix run-corefx-tests.py for Arm64 Linux cross build
|
|
Enable cross build of corefx on x64, creates runtime for arm64
and builds corefx tests.
|
|
Add support for arm Ubuntu testing on hardware. Most JIT stress jobs are enabled.
GC stress jobs are not enabled currently (they haven't been tried yet).
Most jobs have both PR trigger jobs as well as weekly "cron" jobs.
Job frequency will be adjusted as we gain experience with the load.
Initially, there are no automatically triggered PR jobs, but I plan to add
one or more soon.
Moved existing arm Ubuntu (on emulator) and arm Tizen (on emulator)
to use new "armem" pseudo-architecture.
The arm Ubuntu jobs have the product cross-built on Ubuntu/x64 in Docker,
use tests built in a new Windows_NT x86 BuildOnly (`_bld`) job, and are run
on hardware after copying over all the correct artifacts.
Refactored lots of code related to determining if a particular combination
of os/architecture/configuration/isPR/etc. should create a job.
Refactored test/flow job creation.
runtest.sh now creates three files based on test results that can be fed
back to runtest.sh using the `--playlist` argument: coreclrtests.pass.txt,
coreclrtests.fail.txt, coreclrtests.skip.txt.
New script `runtesttilstable.sh` runs `runtest.sh` in a loop, rerunning failing
tests until no tests fail, or a maximum iteration count is reached (default: 4).
This works around existing issues with flaky test failures on Linux/arm32 hardware.
Some extraneous unused jobs have been removed, such as unused `_bld` jobs
and non-PR triggered `innerloop` jobs.
|
|
This reverts commit 0598b6b8af0fb560837ce0bb8f3c8236c05bbdc9.
Updates to xunit console runner break things in Helix. So reverting
until we can get that part sorted.
Also undoes version updates from #16597; we can't move these forward
yet as we're back to netstandard 1.4 for the time being.
|
|
Remove the special benchmark configs and update benchmark projects accordingly.
Also update other random projects that were referencing benchmark configs.
Benchmarks now build against the default standard.
Addresses #14124, #16126.
|
|
* Add more JitBench Benchmarks
a) JitBench includes several more benchmarks and a refactored runner to make it easier to add more
b) A new JitBench/unofficial_dotnet SDK style build project offers simpler developer workflow at the command line and in VS when working with these Benchmarks
c) Misc build issues with local dotnet build in the test tree have workarounds now, but official CI builds still don't support it
d) run-xunit-perf.py has support to run a specific assembly from the testBinLoc folder intended to handle selecting a binary from the folder of binaries produced in the .Net Core SDK project build process. However until CI build can support building projects that way the xunit-perf support is unused.
|
|
* Add CI perf repro script
There wasn't any good way to reproduce what happens in CI for perf jobs because groovy can't be directly run. As a first step to solving this I created a python script that reproduces the perf leg I needed.
|
|
Instead of using the `/p:CoreCLROverridePath` switch when building
corefx, to bring in coreclr, build corefx normally. This builds
against the coreclr packages that corefx has already been validated
against in the CI, so the build should always work. Then, copy
over our built coreclr bits before testing. This method is expected
to have better behavior in most breaking change scenarios.
|
|
|
|
Enable Windows ARM32 corefx testing
|
|
The value returned from system() is encoded bag of bits that gets
misinterpretted once it starts propagate through the system. Normalize
it to 0/1.
|
|
We create a flow job for each arm32 corefx mode, e.g.
arm_cross_checked_windows_nt_corefx_jitstress1_flow_prtest. This
depends on a build job that is specific to the stress mode, e.g.
arm_cross_checked_windows_nt_corefx_jitstress1_bld_prtest, which
(on x64) builds CoreCLR, then clones and builds CoreFX using run-corefx-tests.py.
In particular, it only builds the CoreFX tests; it doesn't run them.
Note that because the CoreFX test build embeds the stress mode environment
variables in its generated RunTests.cmd scripts, we need a different
corefx build dependency for each corefx test run; we can't share the
builds. The build script then ZIPs up the CoreFX tests and generated
CoreFX runtime (which will include the coreclr runtime because we built
CoreFX with the `/p:CoreCLROverridePath` argument), and archives these.
The test job, which runs on an arm64 machine, then copies the ZIPed
tests and runtime, unzips them, and runs a batch script to run each
RunTests.cmd file. Note that we don't use any existing mechanism to
run each test (such as msbuild), and I believe the CoreFX msbuild
harness doesn't have any mechanism to just run a previously built set of tests.
There is a very simple test exclusion mechanism: an entire test assembly
can be excluded by putting its name (e.g., System.IO.Ports.Tests), in
a file (e.g., tests\arm\corefx_test_exclusions.txt).
Note that this corefx testing mechanism is only enabled for arm
(aka arm32), not armlb or arm64.
|
|
|
|
For corefx testing, use a matching corefx repo version git hash
|
|
* Convert perf.groovy to use run-xunit-perf.py
Also removes run-xunit-perf.cmd and run-xunit-perf.sh. This change moves us to using build-tests.sh to generate the layout (ie pull down corefx), so this change fixes #12143.
|
|
If the user doesn't specify a specific corefx commit hash to use,
try to find the matching commit hash in the coreclr repro. If the
matching hash can't be found, use 'HEAD'.
We find the matching corefx commit hash by first parsing file
'dependencies.props' at the root of the coreclr repro, looking for this:
<MicrosoftPrivateCoreFxNETCoreAppPackageVersion>4.5.0-preview1-26112-01</MicrosoftPrivateCoreFxNETCoreAppPackageVersion>
This determines the corefx package version that matches. Next, we look for the version.txt
file in the package cache, e.g.,
<coreclr_root>\packages\microsoft.private.corefx.netcoreapp\4.5.0-preview1-26112-01\version.txt
The contents of this file is exactly the git commit hash
we need to use, e.g.:
197a0699b08087ea85581679afdd9fd7b5c465c3
The version.txt file is created when the corefx package is restored,
which happens when doing one of:
Windows: tests\runtests.cmd GenerateLayoutOnly
non-Windows: build-test.sh generatelayoutonly
It would also be possible to not depend on the package already
being downloaded, but instead download the correct package here,
using the determined "MicrosoftPrivateCoreFxNETCoreAppPackageVersion"
package version, e.g.:
https://dotnet.myget.org/F/dotnet-core/api/v2/package/Microsoft.Private.CoreFx.NETCoreApp/4.5.0-preview1-26112-01
and then extracting the ZIP archive to find the version.txt file.
This might get easier if the corefx commit hash is added directly to dependencies.props, as
discussed in https://github.com/dotnet/buildtools/issues/1141.
|
|
* Convert run-xunit-perf to python script
This change merges the two run-xunit-perf scripts (.sh and .cmd) into
one unified python script and updates the pipeline job to use the
pythong script. This change also updates the linux jobs to use the new
build-tests.sh generatelayoutonly command so that we don't need to pull
down corefx from the cloud anymore. The unified python script enables us
to more easily update both linux scripting and windows scripting at the
same time so that one does not lag behind the other (such as when we add
new configurations or options like slicing). This change also turns
linux testing back on by default for PRs.
|
|
- Issue was fixed with this: https://github.com/aspnet/JitBench/pull/82
- Set the UseSharedCompilation=false to avoid that the `VBCSCompiler.exe` stays running.
|
|
* Change name fo perf jobs to reflect the actual OS they run on
* readd Ubuntu 14.04
|
|
|
|
-NoProfile (#15164)
Code execution in several scripts and tests can be affected by powershell profile
execution at startup. Adding -NoProfile option to all powershell invocations which
may be affected by execution of profile startup code.
|
|
- Added stability prefix to the scenario benchmark (JitBench)
- Specify output directory to the `run-xunit-perf.cmd` script and avoid the extra step to xcopy files to the archive folder.
- Added a command line parser class to the illink scenario, and changed its behavior where it used to fail when a new command line option passed to xUnit was not recognized.
- Save the output log of the tests into the sandbox-logs folder.
- Updating the label of the machine pool used by the illink scenario
|
|
For altjit, we can't run managed code from .NET installs that
are not under test. The XSLT Compiler tests invoke an XSLT
Compiler from the machine installed desktop framework. When it
sees the AltJit variables, but no altjit compiler dll, it dies.
So, don't run those tests. They have been annotated with a new
attribute to allow this exclusion.
|
|
There is a missing ! before checking to see if the throughput benchmarks
have already been downloaded. Currently, we download them if they
already exist, where we want to download them if they don't already
exist. This change fixes that.
|
|
This change converts our perf testing to use pipeline jobs. Pipeline
jobs allow us to do the following:
1) Test on the same commit for each of the test legs
2) Parallelize the build and test steps.
3) Separate the build and test steps from one another. This gives us the
ability to use the same build assets for all of the test legs of the
same configuration. It also allows us to build on virtual machines and
test on perf machines, so we only use the perf resources for testing.
4) Have different test scenarios for PRs and rolling. This isn't
strictly a benefit of pipeline jobs, but certainly is made easier by
them.
5) Allows us to have one trigger for PR jobs which will get us all the
perf testing scenarios.
This change also cleans up the groovy scripting for perf testing.
|
|
- Adding 'tiered' jitting
- Enable JitBench to capture Pmc data (Profile=On)
- Fixed some xUnit performance warnings
- Enforce DOTNET_MULTILEVEL_LOOKUP environment to be turned off
- Specify `dotnet publish` output directory.
|
|
Need 2.0 SDK now. Not sure of the best URLs but these seem to work.
Also shorten path so we don't run into windows pathlength issues.
|
|
Fix corefx testing invocation
|
|
In run-throughput-perf.py, we were calling submission.py with --arch,
but submission.py requires -arch or --architecture. This recently made
submission.py start failing in throughput jobs.
This change also fixes run-throughput-perf so that if submission.py or
upload.py fail, the script fails so we see these jobs as failing in the
lab.
|
|
Instead of executing the environment-setting script before calling
build-tests, pass the `/p:PreExecutionTestScript=<env_script>`
msbuild property, which will write the environment script directly
into the generated corefx RunTests.cmd scripts. Thus, the
environment changes will be made immediately before the test is run.
Depends on https://github.com/dotnet/buildtools/pull/1723.
|
|
* [Tizen/ARM32] Update armel rootfs for Tizen
- Add lttng-ust-dev to Tizen rootfs for cross build
- Use rootfs of the latest Tizen 4.0
Signed-off-by: Hyung-Kyu Choi <hk0110.choi@samsung.com>
* [Tizen/ARM32] Update Tizen CI script
- Update arm32 CI script for Tizen to use the latest environment
Signed-off-by: Hyung-Kyu Choi <hk0110.choi@samsung.com>
|
|
|
|
Add support for priority and update x64 client
|
|
|
|
For JitStressRegs, allow 0x10, 0x80, 0x1000; don't use simple 10 or 80
in names (to match netci.groovy).
Remove "tieredcompilation" from allowed modes; it is not supported in the
CI smarty helper currently.
|
|
Renames arm legacy backend jobs to "armlb" and keeps "arm"
for RyuJIT/arm32 testing.
|
|
Add OptLevel and JitName configurations
|
|
To unify the configuration names in benchview, add OptLevel and JitName
configurations in run-xunit-perf.sh. This change does not add min_opts
to perf.groovy.
|
|
|
|
This change includes:
1) lst_creator updates to allow adding priority tags automatically
2) arm32 lstFile updates: 29 new tests, 50 removed
3) arm64 lstFile updates: 80 new tests, 55 removed
|
|
The loop was iterating through fixed file name and a pattern. The fixed file name did not exist, and the whole function failed.
With this change, we loop through the files if they exist.
|
|
|
|
Fix running corefx tests
|
|
- IlLink scenario will run on a VM (currently it is only measuring MSIL size)
- Updated the patterns of files to search for on scenario runs
- Updated the JitBench commit hash being used ()
- Disable min_opt from the PR correctness mode.
- String was not being interpolated.
|
|
Running CoreFX tests was supposed to use a batch file (env_script) to run CoreFX tests under different environment configurations. However due to subtlties in how the command line is parsed none of the environment variables set by this script would actually persist for the execution of the tests. As best I can tell prior to this fix specifying a env_script had no effect whatsoever on windows, and our CI corefx test stress scenarios weren't actually testing the scenario they claimed to be testing.
cmd /c "Set.bat&&Check.bat" will propagate env vars set in Set.bat so that they are visible in Check.bat because both batch files execute within the context of the newly invoked cmd.exe process. However cmd /c Set.bat&&Check.bat (no quotes) will run Set.bat in its own cmd.exe process, then exit that process and run Check.bat in the parent shell process. The parent process does not have any environment variable changes that were made by Set.bat.
|
|
|
|
- Move xUnit Perf Api output files to a folder `Logs` under sandbox
- Made necessary changes to `perf.groovy` to archive artifacts
- Added a switch -outputdir to the run-xunit-perf* scripts
- Enable running all CoreClr scenarios (JitBench & IlLink) despite failure
- Move sandbox to bin folder
- Use the default output dir: `.\bin\sandbox`
|
