MagNet-main， 是一种用于生成对抗网络（GAN）训练的模型，主要用来提升生成图像的质量并解决生成模型中存在的一些挑战，...

<p align="center"> <img width="150" alt="logo" src="https://i.imgur.com/0OaOlKO.png"> </p> # Progressive Semantic Segmentation (MagNet) [](https://colab.research.google.com/drive/1WTdfIQIEQrnoX40YIzs3HqeIKSZD_iPG?usp=sharing) [](https://arxiv.org/abs/2104.03778) [](https://youtu.be/40Kw8QLym7E) [**MagNet**](https://github.com/VinAIResearch/MagNet), a multi-scale framework that resolves local ambiguity by looking at the image at multiple magnification levels, has multiple processing stages, where each stage corresponds to a magnification level, and the output of one stage is fed into the next stage for coarse-to-fine information propagation. Experiments on three high-resolution datasets of urban views, aerial scenes, and medical images show that MagNet consistently outperforms the state-of-the-art methods by a significant margin.  Details of the MagNet model architecture and experimental results can be found in our [following paper](https://arxiv.org/abs/2104.03778): ``` @inproceedings{m_Huynh-etal-CVPR21,   author = {Chuong Huynh and Anh Tran and Khoa Luu and Minh Hoai},   title = {Progressive Semantic Segmentation},   year = {2021},   booktitle = {Proceedings of the {IEEE} Conference on Computer Vision and Pattern Recognition (CVPR)}, } ``` **Please CITE** our paper when MagNet is used to help produce published results or incorporated into other software. ## Datasets This current code provides configurations to train, evaluate on two datasets: **Cityscapes** and **DeepGlobe**. To prepare the datasets, in the `./data` directory, please do following steps: ### For Cityscapes 1. Register an account on this [page](https://www.cityscapes-dataset.com/) and log in. 2. Download `leftImg8bit_trainvaltest.zip` and `gtFine_trainvaltest.zip`. 3. Run the script below to extract zip files to correct locations: ```bash sh ./prepare_cityscapes.sh ``` ### For DeepGlobe 1. Register an account on this [page](https://competitions.codalab.org/competitions/18468) and log in. 2. Go to this [page]() and download **Starting Kit** of the `#1 Development` Phase. 3. Run the script below to extract zip files to correct locations: ```bash sh ./prepare_deepglobe.sh ``` If you want to train/evaluate with your dataset, follow the steps in this [document](CUSTOM_DATA.md) ## Getting started ### Requirements The framework is tested on machines with the following environment: - Python >= 3.6 - CUDA >= 10.0 To install dependencies, please run the following command: ```bash pip install -r requirements.txt ``` ### Pretrained models Performance of pre-trained models on datasets: | Dataset | Backbone | Baseline IoU (%) | MagNet IoU (%) | MagNet-Fast IoU (%) | Download | | -------- | -------- | -------- | -------- | -------- | -------- | | Cityscapes | HRNetW18+OCR | 63.24 | 68.20 | 67.37 |[backbone](https://public.vinai.io/magnet_cvpr2021/cityscapes_hrnet.pth)<br>[refine_512x256](https://public.vinai.io/magnet_cvpr2021/cityscapes_refinement_512.pth)<br>[refine_1024x512](https://public.vinai.io/magnet_cvpr2021/cityscapes_refinement_1024.pth)<br>[refine_2048x1024](https://public.vinai.io/magnet_cvpr2021/cityscapes_refinement_2048.pth) | | DeepGlobe | Resnet50-FPN | 67.22 | 72.10 | 68.22 | [backbone](https://public.vinai.io/magnet_cvpr2021/deepglobe_fpn.pth)<br>[refine](https://public.vinai.io/magnet_cvpr2021/deepglobe_refinement.pth) Please manually download pre-trained models to `./checkpoints` or run the script below: ```bash cd checkpoints sh ./download_cityscapes.sh # for Cityscapes # or sh ./download_deepglobe.sh # for DeepGlobe ``` ### Usage You can run this [Google Colab Notebook](https://colab.research.google.com/drive/1WTdfIQIEQrnoX40YIzs3HqeIKSZD_iPG?usp=sharing) to test our pre-trained models with street-view images. Please follow the instructions in the notebook to experience the performance of our network. If you want to test our framework on your local machine: 1. To test with a Cityscapes image, e.g `data/frankfurt_000001_003056_leftImg8bit.png`: - With MagNet refinement: ```bash python demo.py --dataset cityscapes \ --image data/frankfurt_000001_003056_leftImg8bit.png \ --scales 256-128,512-256,1024-512,2048-1024 \ --crop_size 256 128 \ --input_size 256 128 \ --model hrnet18+ocr \ --pretrained checkpoints/cityscapes_hrnet.pth \ --pretrained_refinement checkpoints/cityscapes_refinement_512.pth checkpoints/cityscapes_refinement_1024.pth checkpoints/cityscapes_refinement_2048.pth \ --num_classes 19 \ --n_points 32768 \ --n_patches -1 \ --smooth_kernel 5 \ --save_pred \ --save_dir test_results/demo # or in short, you can run sh scripts/cityscapes/demo_magnet.sh data/frankfurt_000001_003056_leftImg8bit.png ``` - With MagNet-Fast refinement ```bash python demo.py --dataset cityscapes \ --image frankfurt_000001_003056_leftImg8bit.png \ --scales 256-128,512-256,1024-512,2048-1024 \ --crop_size 256 128 \ --input_size 256 128 \ --model hrnet18+ocr \ --pretrained checkpoints/cityscapes_hrnet.pth \ --pretrained_refinement checkpoints/cityscapes_refinement_512.pth checkpoints/cityscapes_refinement_1024.pth checkpoints/cityscapes_refinement_2048.pth \ --num_classes 19 \ --n_points 0.9 \ --n_patches 4 \ --smooth_kernel 5 \ --save_pred \ --save_dir test_results/demo # or in short, you can run sh scripts/cityscapes/demo_magnet_fast.sh data/frankfurt_000001_003056_leftImg8bit.png ``` All results will be stored at `test_results/demo/frankfurt_000001_003056_leftImg8bit` 2. To test with a Deepglobe image, e.g `data/639004_sat.jpg`: - With MagNet refinement: ```bash python demo.py --dataset deepglobe \ --image data/639004_sat.jpg \ --scales 612-612,1224-1224,2448-2448 \ --crop_size 612 612 \ --input_size 508 508 \ --model fpn \ --pretrained checkpoints/deepglobe_fpn.pth \ --pretrained_refinement checkpoints/deepglobe_refinement.pth \ --num_classes 7 \ --n_points 0.75 \ --n_patches -1 \ --smooth_kernel 11 \ --save_pred \ --save_dir test_results/demo # or in short, you can run sh scripts/deepglobe/demo_magnet.sh data/639004_sat.jpg ``` - With MagNet-Fast refinement ```bash python demo.py --dataset deepglobe \ --image data/639004_sat.jpg \ --scales 612-612,1224-1224,2448-2448 \ --crop_size 612 612 \ --input_size 508 508 \ --model fpn \ --pretrained checkpoints/deepglobe_fpn.pth \ --pretrained_refinement checkpoints/deepglobe_refinement.pth \ --num_classes 7 \ --n_points 0.9 \ --n_patches 3 \ --smooth_kernel 11 \ --save_pred \ --save_dir test_results/demo # or in short, you can run sh scripts/deepglobe/demo_magnet_fast.sh data/639004_sat.jpg ``` All results will be stored at `test_results/demo/639004_sat` ## Training ### Training backbone networks We customize the training script from [HRNet repository](https://github.com/HRNet/HRNet-Semantic-Segmentation) to train our backbones. Please first go to this directory `./backbone` and run the following scripts: #### HRNetW18V2+OCR for Cityscapes Download pre-trained weights on ImageNet: ``` # In ./backbone cd pretrained_wei