Age | Commit message (Collapse) | Author | Files | Lines |
|
Summary:
Continuing pjh5's work to remove FULL_CAFFE2 flag completely.
With these changes you'll be able to also do something like
```
NO_TEST=1 python setup.py build_deps
```
and this will skip building tests in caffe2, aten, and c10d. By default the tests are built.
cc mingzhe09088 Yangqing
Pull Request resolved: https://github.com/pytorch/pytorch/pull/11321
Reviewed By: mingzhe09088
Differential Revision: D9694950
Pulled By: orionr
fbshipit-source-id: ff5c4937a23d1a263378a196a5eda0cba98af0a8
|
|
Summary:
This PR adds all PyTorch and Caffe2 job configs to CircleCI.
Steps for the CircleCI mini-trial:
- [ ] Make sure this PR passes Jenkins CI and fbcode internal tests
- [x] Approve this PR
- [ ] Ask CircleCI to turn up the number of build machines
- [ ] Land this PR so that the new `.circleci/config.yml` will take effect
Several Caffe2 tests are flaky on CircleCI machines and hence skipped when running on CircleCI. A proper fix for them will be worked on after a successful mini-trial.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/11264
Differential Revision: D9656793
Pulled By: yf225
fbshipit-source-id: 7832e90018f3dff7651489c04a179d6742168fe1
|
|
Summary:
cc goldsborough
Pull Request resolved: https://github.com/pytorch/pytorch/pull/11090
Reviewed By: soumith
Differential Revision: D9582525
Pulled By: apaszke
fbshipit-source-id: 5d2c6e930e7b09f0ed5a35fbf4fe36b8845a2580
|
|
Summary:
This completely removes BUILD_CAFFE2 from CMake. There is still a little bit of "full build" stuff in setup.py that enables USE_CUDNN and BUILD_PYTHON, but otherwise everything should be enabled for PyTorch as well as Caffe2. This gets us a lot closer to full unification.
cc mingzhe09088, pjh5, ezyang, smessmer, Yangqing
Pull Request resolved: https://github.com/pytorch/pytorch/pull/8338
Reviewed By: mingzhe09088
Differential Revision: D9600513
Pulled By: orionr
fbshipit-source-id: 9f6ca49df35b920d3439dcec56e7b26ad4768b7d
|
|
Summary:
Generate serialized test inputs/outputs/backward graphs of tests inside `caffe2/python/operator_test` that call assertSerializedOperatorCheck(). Tests should be decorated with serialized_test.collect_tests.given_and_seeded to run hypothesis tests that are actually random and a single fixed seeded hypothesis tests.
To use:
1. Refactor your test to be a SerializedTestCase
1a. Decorate it with given_and_seeded
1b. Call testWithArgs in main
2. Run your test with -g to generate the output. Check it in.
3. Subsequent runs of the test without generating the output will check against the checked in test case.
Details:
Run your test with `python caffe2/python/operator_test/[your_test].py -g`
Outputs are in `caffe2/python/serialized_test/data`. The operator tests outputs are in a further subdirectory `operator_test`, to allow for other tests in the future (model zoo tests?)
Currently, we've only refactored weighted_sum_test to use this, but in the next diff, we'll refactor as many as possible. The directory structure may also change as usually there are multiple tests in a single file, so we may create more structure to account for that.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10594
Reviewed By: ezyang
Differential Revision: D9370359
Pulled By: ajyu
fbshipit-source-id: 2ce77389cd8bcc0255d3bccd61569833e545ede8
|
|
CI tests (#10871)
Summary:
The PR includes:
(1) torch.distributed.c10d, which now includes the complete backward compatible frontend API for `torch.distributed`
(2) `env://` init method functionality
(3) Minor change to `test_distributed.py`, which is now a test for `torch.distributed.c10d`.
(4) The old `test_distributed.py' is now moved to `test_distributed_thd`
(5) Miscellaneous bug fixes.
(6) DDP CPU test is removed since c10d doesn't have this support yet, but this is a very easy test after moving DDP CPU's dependency to torch.distributed.c10d.
(7) CI config to test MPI, NCCL, and Gloo backend of c10d
**Now all the distributed test including c10d DDP can pass with the c10d frontend API**
TODO: (in a separate PR)
MPI subgroup support, once this is added, CI group test will be enabled.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10871
Differential Revision: D9554514
Pulled By: teng-li
fbshipit-source-id: fb686ad42258526c8b4372148e82969fac4f42dd
|
|
(#10932)
Summary:
The previous NCCL all gather doesn't work as expected. This is a fully working async version. Tested on both C++ and Python Frontend.
Multi-node:
```
tengli@learnfair042:~/new_pytorch/pytorch/torch/lib/build/c10d/test$ TMPFILE="/private/home/tengli/temp/tengli-test" RANK=0 WORLD_SIZE=2 ./ProcessGroupNCCLTest
Multi-node world size: 2 rank: 0
Allreduce test successful
Broadcast test successful
Reduce test successful
Allgather test successful
tengli@learnfair117:~/new_pytorch/pytorch/torch/lib/build/c10d/test$ TMPFILE="/private/home/tengli/temp/tengli-test" RANK=1 WORLD_SIZE=2 ./ProcessGroupNCCLTest
Multi-node world size: 2 rank: 1
Allreduce test successful
Broadcast test successful
Reduce test successful
Allgather test successful
```
CI test:
```
test_set_get (__main__.FileStoreTest) ... ok
test_set_get (__main__.PrefixFileStoreTest) ... ok
test_set_get (__main__.PrefixTCPStoreTest) ... ok
test_allreduce_ops (__main__.ProcessGroupGlooTest) ... ok
test_broadcast_ops (__main__.ProcessGroupGlooTest) ... ok
test_allgather_ops (__main__.ProcessGroupNCCLTest) ... ok
test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... ok
test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... ok
test_reduce_ops (__main__.ProcessGroupNCCLTest) ... ok
test_common_errors (__main__.RendezvousFileTest) ... ok
test_nominal (__main__.RendezvousFileTest) ... ok
test_common_errors (__main__.RendezvousTCPTest) ... ok
test_nominal (__main__.RendezvousTCPTest) ... ok
test_unknown_handler (__main__.RendezvousTest) ... ok
test_set_get (__main__.TCPStoreTest) ... ok
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10932
Differential Revision: D9542067
Pulled By: teng-li
fbshipit-source-id: 25513eddcc3119fd736875d69dfb631b10f4ac86
|
|
Summary:
FULL_CAFFE2=1 python setup.py (install | build_deps develop) should be all anyone needs.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10734
Reviewed By: orionr
Differential Revision: D9439354
Pulled By: pjh5
fbshipit-source-id: 0169afcda4f8f38c57498ba2151f7654ecce6070
|
|
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/10758
Differential Revision: D9467554
Pulled By: bddppq
fbshipit-source-id: 6853ccd96ac3209e062c110913ea37d6840c8134
|
|
Summary:
Breaking out of #8338 to completely remove build_aten and use_aten.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10469
Reviewed By: orionr
Differential Revision: D9413639
Pulled By: mingzhe09088
fbshipit-source-id: b7203aa4f5f2bb95c504c8dc187a3167f2570183
|
|
Summary:
delete build_caffe2.sh, replace with build_libtorch.py as suggested by peter (and copy-pasted from his draft PR). This ensures that all consumers of the torch CMake file go through as unified a path as possible.
In order to change the surrounding infrastructure as little as possible, I made some tweaks to enable build_pytorch_libs.sh to generate the test binaries relative to the current directory, rather than hardcoding to pytorch/build.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10508
Differential Revision: D9354398
Pulled By: anderspapitto
fbshipit-source-id: 05b03df087935f88fca7ccefc676af477ad2d1e9
|
|
Summary:
This should make ASAN tests run faster.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9902
Differential Revision: D9032986
Pulled By: yf225
fbshipit-source-id: 3d2edec2d7ce78bc995d25865aa82ba6d3f971d0
|
|
Summary:
Since we can't specify version number to `choco install curl`, we should not assume that `7.57.0` is the curl version that's in the Windows AMI.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10476
Differential Revision: D9303129
Pulled By: yf225
fbshipit-source-id: 198544be68330860fbcf93c99bc995f4e280bda7
|
|
Summary:
Initial jenkins builds / test scripts for ppc64le.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10257
Differential Revision: D9331278
Pulled By: ezyang
fbshipit-source-id: 6d9a4f300a0233faf3051f8151beb31786dcd838
|
|
Summary:
* some small leftovers from the last PR review
* enable more unit test sets for CI
* replace use of hcRNG w/ rocRAND (docker image was already updated w/ newer rocRAND)
* use rocBLAS instead of hipBLAS to allow convergence w/ Caffe2
* use strided_batched gemm interface also from the batched internal interface
* re-enable Dropout.cu as we now have philox w/ rocRAND
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10406
Reviewed By: Jorghi12
Differential Revision: D9277093
Pulled By: ezyang
fbshipit-source-id: 7ef2f6fe4ead77e501ed7aea5c3743afe2466ca2
|
|
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/10328
Reviewed By: pjh5
Differential Revision: D9275809
Pulled By: ezyang
fbshipit-source-id: c22cb1570c67199b74b2188ad83b1e4828e11911
|
|
Summary:
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10320
Reviewed By: jerryzh168
Differential Revision: D9198902
Pulled By: ezyang
fbshipit-source-id: 8af28d607735e5f4450c40127c1f8c262ea602ce
|
|
(#10285)
Summary:
…d_libtorch entrypoint (#9836)"
This reverts commit 62e23a1ee47eb66056e6695cefef4e42599f8bd0.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10285
Differential Revision: D9193107
Pulled By: ezyang
fbshipit-source-id: de96dce12fdf74410413ae18feee5caf0bed0025
|
|
Summary:
This PR for the ROCm target does the following:
* enable some unit tests on ROCm
* fix a missing static_cast that breaks BatchNorm call on ROCm
* fix BatchNorm to work on ROCm w/ ROCm warp sizes etc
* improve the pyhipify script by introducing kernel scope to some transpilations and other improvements
* fix a linking issue on ROCm
* for more unit test sets: mark currently broken tests broken (to be fixed)
* enable THINLTO (phase one) to parallelize linking
* address the first failing of the elementwise kernel by removing non-working ROCm specialization
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10266
Differential Revision: D9184178
Pulled By: ezyang
fbshipit-source-id: 03bcd1fe4ca4dd3241f09634dbd42b6a4c350297
|
|
entrypoint (#9836)
Summary:
the new entrypoint is `./tools/build_pytorch_libs.sh caffe2`
this will also speed up CI builds a bit, since we will no longer be compiling all of libtorch twice
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9836
Differential Revision: D9182634
Pulled By: anderspapitto
fbshipit-source-id: 0b9a20ab04f5df2d5c4e7777e4dc468ab25b9ce2
|
|
Summary:
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10211
Differential Revision: D9150687
Pulled By: ezyang
fbshipit-source-id: 4cd458d19a34788c8897905a87d1b52229f67f90
|
|
Summary:
In this changeset:
* improvements to `hipify-python.py`
* marking unit tests broken for ROCm
* reducing the number of jobs for the built to avoid out of memory issues
* switch to Thrust/cub-hip master for the CI
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9653
Differential Revision: D9117791
Pulled By: ezyang
fbshipit-source-id: a6c3c7b81f2bda9825974bf9bf89a97767244352
|
|
Summary:
petrex
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10144
Differential Revision: D9125707
Pulled By: bddppq
fbshipit-source-id: 8ef8f3da6ceb1855f28fc24be621b9b4854ff7f9
|
|
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/10060
Differential Revision: D9081387
Pulled By: bddppq
fbshipit-source-id: 13cbff63df5241e013d4ebacfcd6da082e7196f6
|
|
Summary:
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10005
Differential Revision: D9066107
Pulled By: ezyang
fbshipit-source-id: 346f654214cff1c956a4022173347d95657ee9d4
|
|
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/10064
Differential Revision: D9082082
Pulled By: gchanan
fbshipit-source-id: ae49470f5b4c89b13beb55fd825de1ba05b6a4fa
|
|
Summary:
zrphercule
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10014
Reviewed By: houseroad
Differential Revision: D9061842
Pulled By: bddppq
fbshipit-source-id: 1e1c2aeae62dd2cc5c6a8d5e1d395ea5cf882734
|
|
Summary:
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/10012
Differential Revision: D9057766
Pulled By: ezyang
fbshipit-source-id: 12148a8cf3061423c61b3e7b36864dfcdb1138a1
|
|
Summary:
Unblock https://github.com/pytorch/pytorch/pull/8469
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9909
Differential Revision: D9023650
Pulled By: houseroad
fbshipit-source-id: 7682a9cd7905e98c802b820ad59745672b32970d
|
|
Summary:
Somehow we have Unique operator tests in two places test_unqiue_ops.py and hypothesis_test.py
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9892
Reviewed By: houseroad
Differential Revision: D9017631
Pulled By: bddppq
fbshipit-source-id: 1f9e40e4953afca26141ef4581202b9b9fce0ae9
|
|
Summary:
petrex
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9862
Differential Revision: D9012520
Pulled By: bddppq
fbshipit-source-id: cdcc184e23befa8dbd1bc44d59bd25766aac33d0
|
|
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/9861
Differential Revision: D9011458
Pulled By: bddppq
fbshipit-source-id: 7ab1b1763d56f1290ade7a99682ad461c97f807b
|
|
Summary:
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9856
Differential Revision: D9009100
Pulled By: ezyang
fbshipit-source-id: 28f34128fcb7c3d6a115884bf28dc2a6bde5aed6
|
|
Summary:
We think this will band-aid some of the new Caffe2 test failures.
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9830
Differential Revision: D9008052
Pulled By: ezyang
fbshipit-source-id: 84f1c0faea429d758d760965d6cbfe9e4c72eb19
|
|
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/9841
Differential Revision: D9008030
Pulled By: ezyang
fbshipit-source-id: 51cac3c75fc52658b22a10a6bf8a479bcf803fb2
|
|
Summary:
Temporarily need this to prevent sccache from breaking when I move sccache install to the DockerFile.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9810
Differential Revision: D8991684
Pulled By: Jorghi12
fbshipit-source-id: 14cd0278f53a72372f9bbe27b228980f8d3c1d4a
|
|
Summary:
petrex
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9616
Differential Revision: D8960623
Pulled By: bddppq
fbshipit-source-id: bde93bda6230094e6bf4badd8ee79f0688ae1993
|
|
Summary:
This is a modification of the strategy from https://github.com/pytorch/pytorch/pull/8919 and https://github.com/pytorch/pytorch/pull/9579.
```
Previously, the CPU architecture-specific kernels self-registered with
the DispatchStub. When linking as part of a static library, this requires
the flag --whole-archive to be passed to the linker to ensure that the
object files for the kernels are included. Caffe2 and TensorFlow use that
strategy.
We ran into some issues with --whole-archive blowing up the binary size
of some downstream projects in Facebook. This PR avoids --whole-archive
for CPU kernels. The downside is that the generic code needs to be aware
of whether kernels are compiled with AVX and with AVX2 (via
HAVE_AVX_CPU_DEFINITION and HAVE_AVX2_CPU_DEFINITION).
The CUDA kernels still self-register with DispatchStub because the CPU
library is not aware of whether the CUDA library will be available at
runtime.
There are a few major changes to DispatchStub
- The environment variable ATEN_CPU_CAPABILITY overrides the CPU
capability detection code (Previous ATEN_DISABLE_AVX/AVX2)
- DispatchStub is defined in the generic native code instead of the
CPU_CAPABILITY_DEFAULT kernel.
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9664
Differential Revision: D8943350
Pulled By: colesbury
fbshipit-source-id: 329229b0ee9ff94fc001b960287814bd734096ef
|
|
Summary:
This reverts commit bcf0bf42a1727c8ee788f733c28579d0e36a387c.
The commit was causing issues for some internal FB projects.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9614
Reviewed By: Yangqing
Differential Revision: D8929552
Pulled By: colesbury
fbshipit-source-id: ae9026ad8762a4c5de401273694b4c878fc241a6
|
|
Summary:
This is a few files taken from https://github.com/pytorch/pytorch/pull/8919. They're unchanged from the latest versions of that PR.
```
This is part of https://github.com/pytorch/pytorch/pull/8919. It's
separated to make it easier to merge the PR in pieces.
There are a few major changes to DispatchStub
- The environment variable ATEN_CPU_CAPABILITY overrides the CPU
capability detection code (Previous ATEN_DISABLE_AVX/AVX2)
- DispatchStub is defined in the generic native code instead of the
CPU_CAPABILITY_DEFAULT kernel.
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9579
Differential Revision: D8909000
Pulled By: colesbury
fbshipit-source-id: fdeb606270b06acdab3c01dba97ec9d81584ecc0
|
|
Summary:
The goal of this PR is to add an infrastructure; to convert(hipify) CUDA ops into [HIP](https://github.com/ROCm-Developer-Tools/HIP) ops , at **compile** time.
Note that HIP ops, which are portable c++ code, can run on AMD and NVIDIA platform.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9322
Differential Revision: D8884707
Pulled By: bddppq
fbshipit-source-id: dabc6319546002c308c10528238e6684f7aef0f8
|
|
Summary:
Prior to this diff, there have been two ways of compiling the bulk of the torch codebase. There was no interaction between them - you had to pick one or the other.
1) with setup.py. This method
- used the setuptools C extension functionality
- worked on all platforms
- did not build test_jit/test_api binaries
- did not include the C++ api
- always included python functionality
- produced _C.so
2) with cpp_build. This method
- used CMake
- did not support Windows or ROCM
- was capable of building the test binaries
- included the C++ api
- did not build the python functionality
- produced libtorch.so
This diff combines the two.
1) cpp_build/CMakeLists.txt has become torch/CMakeLists.txt. This build
- is CMake-based
- works on all platforms
- builds the test binaries
- includes the C++ api
- does not include the python functionality
- produces libtorch.so
2) the setup.py build
- compiles the python functionality
- calls into the CMake build to build libtorch.so
- produces _C.so, which has a dependency on libtorch.so
In terms of code changes, this mostly means extending the cmake build to support the full variety of environments and platforms. There are also a small number of changes related to the fact that there are now two shared objects - in particular, windows requires annotating some symbols with dllimport/dllexport, and doesn't allow exposing thread_local globals directly.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/8792
Reviewed By: ezyang
Differential Revision: D8764181
Pulled By: anderspapitto
fbshipit-source-id: abec43834f739049da25f4583a0794b38eb0a94f
|
|
Summary:
….txt setting
In the ROCm branches we will experiment with turning this on.
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9543
Differential Revision: D8897990
Pulled By: ezyang
fbshipit-source-id: ae9d25d1b79ee421d49436593edf8c7e49b3a4e5
|
|
Summary:
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9467
Reviewed By: houseroad
Differential Revision: D8860794
Pulled By: ezyang
fbshipit-source-id: 9b11475d9bb4b3361973865d7f68e562bffbf9d8
|
|
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/9354
Reviewed By: orionr
Differential Revision: D8810415
Pulled By: pjh5
fbshipit-source-id: 792b0dc6f6a4fabde38e2ad4475963526204914c
|
|
Summary:
Breaking this out of #8338
This fixes some CUDA related build and runtime issues after BUILD_CAFFE2 and BUILD_ATEN are removed.
cc orionr
Pull Request resolved: https://github.com/pytorch/pytorch/pull/9347
Reviewed By: orionr
Differential Revision: D8806954
Pulled By: mingzhe09088
fbshipit-source-id: 9f8e3feee06478d1ac2deb30796939453352d388
|
|
Summary:
As discussed in call, this will allow us to keep this integral part of the effort to run PyTorch on ROCm in sync with the main code.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/8812
Reviewed By: ezyang
Differential Revision: D8796245
Pulled By: bddppq
fbshipit-source-id: 8e12c2acf6a7e0740f31b21e50be74e10ed8b12c
|
|
Summary:
Copy of https://github.com/pytorch/pytorch/pull/8802
Closes https://github.com/pytorch/pytorch/pull/8813
Differential Revision: D8707364
Pulled By: yf225
fbshipit-source-id: bc201980b50e9fb44c42a17f898b50d3558fc417
|
|
Summary:
1. Escape quotes
2. Use file exist logic to determine build success/failure
Closes https://github.com/pytorch/pytorch/pull/9060
Differential Revision: D8707290
Pulled By: soumith
fbshipit-source-id: a34265f46725eaaf9489bc38546200aeae75e8a9
|
|
Summary:
closes #7929
Closes https://github.com/pytorch/pytorch/pull/8018
Differential Revision: D8682540
Pulled By: li-roy
fbshipit-source-id: 649170dd1a7f373151c1d4e949838bd1c5651936
|