Age | Commit message (Collapse) | Author | Files | Lines |
|
We are still seeing issues with running out of space on the archive
machines due to the size of the perf archives. This change reduces the
number of days we store the archives to 14 and the number of archives to
keep to 100.
|
|
* Restrict what we archive for perf testing
* change all perf legs archival
* Change txt save to *_log.txt
* Reduce number of days to keep artifacts
|
|
|
|
|
|
|
|
|
|
We need to disable these jobs on commit until dotnet/core-eng#2808 is fixed.
|
|
* Archive CI perf results
I need to diagnose a failure that only reproes in CI but the CI scripts don't archive the logs on failure. This change should fix that.
* Adding pipeline build per feedback
|
|
We were uploading windows results as win32 instead of Windows_NT.
|
|
|
|
* Remove hardcoded paths in linkbench scripts
Linkbench has hardcoded paths based off of VS140COMNTOOLS, which not all
machines will have (ie, machines with only VS2017 installed). This
change removes the hardcoded paths, and replaces them with checks to
make sure the tool is on the path (which they will be if we are in a VS
environment of any kind - which we would have been in if
VS140COMNTOOLS was already set).
|
|
* Convert perf.groovy to use run-xunit-perf.py
Also removes run-xunit-perf.cmd and run-xunit-perf.sh. This change moves us to using build-tests.sh to generate the layout (ie pull down corefx), so this change fixes #12143.
|
|
We don't need the perf results for min-opt/no-pgo runs as frequently as
we need full-opt pgo runs (which we want to get per commit, if
possible), so move them to only running once daily.
|
|
* Change name fo perf jobs to reflect the actual OS they run on
* readd Ubuntu 14.04
|
|
-NoProfile (#15164)
Code execution in several scripts and tests can be affected by powershell profile
execution at startup. Adding -NoProfile option to all powershell invocations which
may be affected by execution of profile startup code.
|
|
|
|
|
|
- Added stability prefix to the scenario benchmark (JitBench)
- Specify output directory to the `run-xunit-perf.cmd` script and avoid the extra step to xcopy files to the archive folder.
- Added a command line parser class to the illink scenario, and changed its behavior where it used to fail when a new command line option passed to xUnit was not recognized.
- Save the output log of the tests into the sandbox-logs folder.
- Updating the label of the machine pool used by the illink scenario
|
|
This change converts our perf testing to use pipeline jobs. Pipeline
jobs allow us to do the following:
1) Test on the same commit for each of the test legs
2) Parallelize the build and test steps.
3) Separate the build and test steps from one another. This gives us the
ability to use the same build assets for all of the test legs of the
same configuration. It also allows us to build on virtual machines and
test on perf machines, so we only use the perf resources for testing.
4) Have different test scenarios for PRs and rolling. This isn't
strictly a benefit of pipeline jobs, but certainly is made easier by
them.
5) Allows us to have one trigger for PR jobs which will get us all the
perf testing scenarios.
This change also cleans up the groovy scripting for perf testing.
|
|
- Adding 'tiered' jitting
- Enable JitBench to capture Pmc data (Profile=On)
- Fixed some xUnit performance warnings
- Enforce DOTNET_MULTILEVEL_LOOKUP environment to be turned off
- Specify `dotnet publish` output directory.
|
|
- Recent changes to enforce data validation uncovered bug on the legacy backend runs. Now, we do not allow to upload tests with no performance data.
CQ run bug tracked by: #14463
JitBench run bug tracked by: #14477
|
|
- In the lab, the path where the JitBench sources are downloaded is already too long, which eventially causes `dotnet-install.ps1` to fail. Part of the changes are to reduce the PATH where the tests are placed.
- Updated the JitBench version/commit being used. This one is already on 2.1
- Fixing typo on `perf.groovy`
- Increasing the timeout value as JitBench keeps failing on the lab.
|
|
- `submission-metadata` option is `--user-email`, not `--user`
- We need to escape user input. On batch, " -> ""
|
|
* [WIP] First version of SoDBench. Contains bugs
* [WIP] Move SoDBench files and increase error checking
* [WIP] Add NugetConfig to enable pulling packages from myget
* [WIP] Remove unhelpful templates and add back oses
* [WIP] Add ability to specify channel
* [WIP] Improve CSV writing
* [WIP] Improve options parsing
* Fix syntax error
* [WIP] Add test leg to perf.groovy
* [WIP] Adjust label to target an existing machine pool
* Change label to run on virtual machine
* Use setMachineAffinity
* Add ASP.NET-Core feed and deafult to master as the channel
* Change channel to master in perf.groovy
* Move NuGet.Config up a directory so it only needs to be written once
* Add CommandLine to external dependencies
* Remove CommandLine as it is now in external.depproj
* Adjust wget command to work more consistantly
* Change calculation of relative paths for clarity
* Set job to run daily instead of on push/PR
* Build sodbench on job run
* Remove MSBuild from job
* Fix quote placement
* Change metadata to be more accurate
* Add rollup totals for each measured category
* Refactor to use a tree rather than a dictionary
* Limit report size
* Publish in release configuration
|
|
|
|
- Previous change did not fix archiving artifacts
|
|
destination folder (#13858)
|
|
|
|
|
|
#13638 and #13647 attempted to replace the generic 'dotnet-bot' e-mail
address with PR authors' e-mail addresses when uploading perf data to
BenchView. Unfortunately, these changes are still broken for some users
(specifically, if a user's e-mail address is not published/visible on
their GitHub profile).
There is no clean way to implement the proper fix, and the right fix
will change once pipeline support is available. Rather than putting in
something hacky now, I'm reverting these changes to unblock PRs. We
should revisit these changes after pipeline jobs are available.
|
|
|
|
- CoreClr Scenario and CoreClr IlLink jobs were aliased on CI.
|
|
Perf smoke tests have been broken since #13638. The issue appears to be
that `ghprbTriggerAuthorEmail` is an empty string in most cases, so its
value is not expanded.
The attempted fix is to use `ghprbPullAuthorEmail` instead, which is
always defined (since there's always an owner to a PR).
|
|
User the PR trigger email for benchview submit
|
|
When users trigger perf runs for PRs, the alias used to submit in
submission-metadata.json should be that user's email. We should only
use dotnet-bot for official runs.
|
|
- IlLink scenario will run on a VM (currently it is only measuring MSIL size)
- Updated the patterns of files to search for on scenario runs
- Updated the JitBench commit hash being used ()
- Disable min_opt from the PR correctness mode.
- String was not being interpolated.
|
|
- Increase how long performance artifacts are kept.
- Added an extra step to move files to be archived into a different folder.
|
|
|
|
- Files were not being archived after last changes.
|
|
- Move xUnit Perf Api output files to a folder `Logs` under sandbox
- Made necessary changes to `perf.groovy` to archive artifacts
- Added a switch -outputdir to the run-xunit-perf* scripts
- Enable running all CoreClr scenarios (JitBench & IlLink) despite failure
- Move sandbox to bin folder
- Use the default output dir: `.\bin\sandbox`
|
|
- Currently ILLink scenario is broken and the x64 scenario leg does not run JitBench
By swaping the scenario it will let JitBench run while ILLink is being fixed.
|
|
|
|
This change adds an opt_level throughput option so that we can run
throughput jobs in minopts mode. This also adds jobs to the lab for min
opts runs
|
|
|
|
On Windows, we download a set of assemblies that we have uploaded to
MyGet. On Linux, we were building corefx each run to get the benchmarks.
This change updates Linux to pull down the same benchmarks Windows uses,
using curl to grab the nuget package from MyGet.
Note: after this commit, we will have data for fewer assemblies on
Linux, but we will now have the same set for both Windows and Linux.
Some data in Benchview may be affected.
|
|
Sudo is not necessary for running throughput testing, so remove sudo so
we don't run into requests for passwords if the ubuntu machines are not
set up correctly.
|
|
Fix perf.groovy so that ILLink perf tests can be triggered.
|
|
Our Ubuntu perf testing has been creating Linux CoreCLR builds by
creating the build locally. This fits the local developer scenario, but
doesn't actually test the bits that ship; the 'official' product
configuration is generated by building on RHEL 7.2 and packaging those
bits for various distros, including Ubuntu.
In order to more accurately measure end-user-perceived performance, we
should do the same thing official builds do: build on RHEL7.2, then copy
those bits onto other OSes to test perf on those OSes.
This PR makes that happen by structuring Linux perf and throughput jobs
as flow jobs, with builds up front on a RHEL VM, before the bits are
dropped onto Ubuntu perf test machines.
|
|
When you use the help from dotnet bot you do not get any of the
performance runs in the help. This should add those in.
|
|
This changes ILLink performance testing to its own leg.
With this change, ILLink perf test will not be run for every PR by default.
It can be requested explicitly on a PR, and will run on the rolling builds.
JitBench build breaks have broken the PR builds a few times, so this change
moves ILLink perf tests out of the PR job, so that MusicStore testing can be
enabled again.
|