Frequently Asked Questions
We list some common troubles faced by many users and their corresponding solutions here. Feel free to enrich the list if you find any frequent issues and have ways to help others to solve them. If the contents here do not cover your issue, please create an issue using the provided templates and make sure you fill in all required information in the template.
MMCV Installation
- Compatibility issue between MMCV and MMDetection; "ConvWS is already registered in conv layer"; "AssertionError: MMCV==xxx is used but incompatible. Please install mmcv>=xxx, <=xxx."
Compatible MMCV, MMDetection and MMRotate versions are shown as below. Please install the correct version of them to avoid installation issues.
MMRotate version |
MMCV version |
MMDetection version |
main |
mmcv-full>=1.5.3, <1.8.0 |
mmdet >= 2.25.1, <3.0.0 |
0.3.4 |
mmcv-full>=1.5.3, <1.8.0 |
mmdet >= 2.25.1, <3.0.0 |
0.3.3 |
mmcv-full>=1.5.3, <1.7.0 |
mmdet >= 2.25.1, <3.0.0 |
0.3.2 |
mmcv-full>=1.5.3, <1.7.0 |
mmdet >= 2.25.1, <3.0.0 |
0.3.1 |
mmcv-full>=1.4.5, <1.6.0 |
mmdet >= 2.22.0, <3.0.0 |
0.3.0 |
mmcv-full>=1.4.5, <1.6.0 |
mmdet >= 2.22.0, <3.0.0 |
0.2.0 |
mmcv-full>=1.4.5, <1.5.0 |
mmdet >= 2.19.0, <3.0.0 |
0.1.1 |
mmcv-full>=1.4.5, <1.5.0 |
mmdet >= 2.19.0, <3.0.0 |
0.1.0 |
mmcv-full>=1.4.5, <1.5.0 |
mmdet >= 2.19.0, <3.0.0 |
PyTorch/CUDA Environment
-
"invalid device function" or "no kernel image is available for execution".
- Check if your cuda runtime version (under
/usr/local/
), nvcc --version
and conda list cudatoolkit
version match.
- Run
python mmdet/utils/collect_env.py
to check whether PyTorch, torchvision, and MMCV are built for the correct GPU architecture.
You may need to set TORCH_CUDA_ARCH_LIST
to reinstall MMCV.
The GPU arch table could be found here,
i.e. run TORCH_CUDA_ARCH_LIST=7.0 pip install mmcv-full
to build MMCV for Volta GPUs.
The compatibility issue could happen when using old GPUS, e.g., Tesla K80 (3.7) on colab.
- Check whether the running environment is the same as that when mmcv/mmdet has compiled.
For example, you may compile mmcv using CUDA 10.0 but run it on CUDA 9.0 environments.
-
"undefined symbol" or "cannot open xxx.so".
- If those symbols are CUDA/C++ symbols (e.g., libcudart.so or GLIBCXX), check whether the CUDA/GCC runtimes are the same as those used for compiling mmcv,
i.e. run python mmdet/utils/collect_env.py
to see if "MMCV Compiler"
/"MMCV CUDA Compiler"
is the same as "GCC"
/"CUDA_HOME"
.
- If those symbols are PyTorch symbols (e.g., symbols containing caffe, aten, and TH), check whether the PyTorch version is the same as that used for compiling mmcv.
- Run
python mmdet/utils/collect_env.py
to check whether PyTorch, torchvision, and MMCV are built by and running on the same environment.
-
"setuptools.sandbox.UnpickleableException: DistutilsSetupError("each element of 'ext_modules' option must be an Extension instance or 2-tuple")"
- If you are using miniconda rather than anaconda, check whether Cython is installed as indicated in #3379.
You need to manually install Cython first and then run command pip install -r requirements.txt
.
- You may also need to check the compatibility between the
setuptools
, Cython
, and PyTorch
in your environment.
-
"Segmentation fault".
-
Check you GCC version and use GCC 5.4. This usually caused by the incompatibility between PyTorch and the environment (e.g., GCC < 4.9 for PyTorch). We also recommend the users to avoid using GCC 5.5 because many feedbacks report that GCC 5.5 will cause "segmentation fault" and simply changing it to GCC 5.4 could solve the problem.
-
Check whether PyTorch is correctly installed and could use CUDA op, e.g. type the following command in your terminal.
python -c 'import torch; print(torch.cuda.is_available())'
And see whether they could correctly output results.
-
If Pytorch is correctly installed, check whether MMCV is correctly installed.
python -c 'import mmcv; import mmcv.ops'
If MMCV is correctly installed, then there will be no issue of the above two commands.
-
If MMCV and Pytorch is correctly installed, you man use ipdb
, pdb
to set breakpoints or directly add 'print' in mmdetection code and see which part leads the segmentation fault.
E2CNN
Training
Evaluation
- COCO Dataset, AP or AR = -1
- According to the definition of COCO dataset, the small and medium areas in an image are less than 1024 (32*32), 9216 (96*96), respectively.
- If the corresponding area has no object, the result of AP and AR will set to -1.