feat: 🚩 add datasets PIQ2023, GFIQA and metric topiq_nr-face.

README.md

@@ -36,6 +36,7 @@ This is a image quality assessment toolbox with **pure python and pytorch**. We
 ---
 
 ### :triangular_flag_on_post: Updates/Changelog
+- **Oct 09, 2023**. Add datasets: [PIQ2023](https://github.com/DXOMARK-Research/PIQ2023), [GFIQA](http://database.mmsp-kn.de/gfiqa-20k-database.html). Add metric `topiq_nr-face`. 
 - **Aug 15, 2023**. Add `st-lpips` and `laion_aes`. Refer to official repo at [ShiftTolerant-LPIPS](https://github.com/abhijay9/ShiftTolerant-LPIPS) and [improved-aesthetic-predictor](https://github.com/christophschuhmann/improved-aesthetic-predictor)
 - **Aug 05, 2023**. Add our work [TOPIQ](https://arxiv.org/abs/2308.03060) with remarkable performance on almost all benchmarks via efficient Resnet50 backbone. Use it with `topiq_fr, topiq_nr, topiq_iaa` for Full-Reference, No-Reference and Aesthetic assessment respectively.
 - **March 30, 2023**. Add [URanker](https://github.com/RQ-Wu/UnderwaterRanker) for IQA of under water images. 

docs/Dataset_Preparation.md

@@ -16,11 +16,12 @@
 | PieAPP     | *2AFC*      | AVA              | *Aesthetic*        |
 | KADID-10k  |             | KonIQ-10k(++)    |                    |
 | LIVEM      |             | LIVEChallange    |                    |
-| LIVE       |             |                  |                    |
-| TID2013    |             |                  |                    |
+| LIVE       |             | [PIQ2023](https://github.com/DXOMARK-Research/PIQ2023)| Portrait dataset   |
+| TID2013    |             | [GFIQA](http://database.mmsp-kn.de/gfiqa-20k-database.html)| Face IQA Dataset   |
 | TID2008    |             |                  |                    |
 | CSIQ       |             |                  |                    |
 
+Please see more details at [Awesome Image Quality Assessment](https://github.com/chaofengc/Awesome-Image-Quality-Assessment)
 
 ## Resources
 

docs/ModelCard.md

@@ -2,49 +2,53 @@
 
 ## General FR/NR Methods
 
-<table>
-<tr><td>
+List all model names with:
+```
+import pyiqa
+print(pyiqa.list_models())
+```
 
-| FR Method                | Backward           |
-| ------------------------ | ------------------ |
-| AHIQ                     | :white_check_mark: |
-| PieAPP                   | :white_check_mark: |
-| LPIPS                    | :white_check_mark: |
-| DISTS                    | :white_check_mark: |
-| WaDIQaM                  | :white_check_mark: |
-| CKDN<sup>[1](#fn1)</sup> | :white_check_mark: |
-| FSIM                     | :white_check_mark: |
-| SSIM                     | :white_check_mark: |
-| MS-SSIM                  | :white_check_mark: |
-| CW-SSIM                  | :white_check_mark: |
-| PSNR                     | :white_check_mark: |
-| VIF                      | :white_check_mark: |
-| GMSD                     | :white_check_mark: |
-| NLPD                     | :white_check_mark: |
-| VSI                      | :white_check_mark: |
-| MAD                      | :white_check_mark: |
+| FR Method                | Model names | Description
+| ------------------------ | ------------------ | ------------ |
+| TOPIQ |  `topiq_fr`, `topiq_fr-pipal` | Proposed in [this paper](https://arxiv.org/abs/2308.03060) | 
+| AHIQ                     |  `ahiq` |
+| PieAPP                   |  `pieapp` |
+| LPIPS                    |  `lpips`, `lpips-vgg`, `stlpips`, `stlpips-vgg`  |
+| DISTS                    |  `dists` |
+| WaDIQaM                  |  | *No pretrain models* |
+| CKDN<sup>[1](#fn1)</sup> |  `ckdn` |
+| FSIM                     |  `fsim` |
+| SSIM                     |  `ssim`, `ssimc` | Gray input (y channel), color input
+| MS-SSIM                  |  `ms_ssim` |
+| CW-SSIM                  |  `cw_ssim` |
+| PSNR                     |  `psnr`, `psnry` | Color input, gray input (y channel)
+| VIF                      |  `vif` |
+| GMSD                     |  `gmsd` |
+| NLPD                     |  `nlpd` |
+| VSI                      |  `vsi` |
+| MAD                      |  `mad` |
 
-</td><td>
-
-| NR Method                    | Backward                 |
-| ---------------------------- | ------------------------ |
-| FID                          | :heavy_multiplication_x: |
-| CLIPIQA(+)                   | :white_check_mark:       |
-| MANIQA                       | :white_check_mark:       |
-| MUSIQ                        | :white_check_mark:       |
-| DBCNN                        | :white_check_mark:       |
-| PaQ-2-PiQ                    | :white_check_mark:       |
-| HyperIQA                     | :white_check_mark:       |
-| NIMA                         | :white_check_mark:       |
-| WaDIQaM                      | :white_check_mark:       |
-| CNNIQA                       | :white_check_mark:       |
-| NRQM(Ma)<sup>[2](#fn2)</sup> | :heavy_multiplication_x: |
-| PI(Perceptual Index)         | :heavy_multiplication_x: |
-| BRISQUE                      | :white_check_mark:       |
-| ILNIQE                       | :white_check_mark:       |
-| NIQE                         | :white_check_mark:       |
-</tr>
-</table>
+| NR Method                    | Model names | Description |
+| ---------------------------- | ------------------------ | ------ |
+| TOPIQ | `topiq_nr`, `topiq_nr-flive`, `topiq_nr-spaq` | [TOPIQ](https://arxiv.org/abs/2308.03060) with different datasets, `koniq` by default |
+| TReS | `tres`, `tres-koniq`, `tres-flive` | TReS with different datasets, `koniq` by default |
+| FID                          | `fid` | Statistic distance between two datasets |
+| CLIPIQA(+)                   |  `clipiqa`, `clipiqa+`, `clipiqa+_vitL14_512`,`clipiqa+_rn50_512`  | CLIPIQA(+) with different backbone, RN50 by default |
+| MANIQA                       | `maniqa`, `maniqa-kadid`, `maniqa-koniq`, `maniqa-pipal` |MUSIQ with different datasets, `koniq` by default |
+| MUSIQ                        | `musiq`, `musiq-koniq`, `musiq-spaq`, `musiq-paq2piq`, `musiq-ava` | MUSIQ with different datasets, `koniq` by default |
+| DBCNN                        | `dbcnn` |
+| PaQ-2-PiQ                    | `paq2piq` |
+| HyperIQA                     |  `hyperiqa` |
+| NIMA                         |  `nima`, `nima-vgg16-ava` | Aesthetic metric trained with AVA dataset |
+| WaDIQaM                      |  | *No pretrain models*
+| CNNIQA                       |  `cnniqa` |
+| NRQM(Ma)<sup>[2](#fn2)</sup> |  `nrqm` | No backward |
+| PI(Perceptual Index)         |  `pi` | No backward |
+| BRISQUE                      | `brisque` | No backward |
+| ILNIQE                       | `ilniqe` | No backward |
+| NIQE                         | `niqe` | No backward |
+<!-- </tr>
+</table> -->
 
 <a name="fn1">[1]</a> This method use distorted image as reference. Please refer to the paper for details.<br>
 <a name="fn2">[2]</a> Currently, only naive random forest regression is implemented and **does not** support backward.
@@ -53,6 +57,7 @@
 
 | Task           | Method  | Description                                                                                                                                                                 |
 | -------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Face IQA | `topiq_face` | TOPIQ model trained with face IQA dataset (GFIQA) |
 | Underwater IQA | URanker | A ranking-based underwater image quality assessment (UIQA) method, AAAI2023, [Arxiv](https://arxiv.org/abs/2208.06857), [Github](https://github.com/RQ-Wu/UnderwaterRanker) |
 
 ## Outputs of Different Metrics 

docs/index.rst

@@ -6,6 +6,11 @@
 Welcome to pyiqa's documentation!
 =================================
 
+``pyiqa`` is a image quality assessment toolbox **with pure python and pytorch**. We provide reimplementation of many mainstream full reference (FR) and no reference (NR) metrics (results are calibrated with official matlab scripts if exist). **With GPU acceleration, most of our implementations are much faster than Matlab.**
+
+Basic Information 
+-------------------------
+
 .. toctree::
    :maxdepth: 1
 
@@ -14,13 +19,44 @@ Welcome to pyiqa's documentation!
    ModelCard
    benchmark
 
+API Tools and References
+-------------------------
+
 .. toctree::
    :maxdepth: 2
 
    api_entries
    metrics_implement
    training_tools
    Dataset_Preparation
+
+
+Citation
+==================================
+
+If you find our codes helpful to your research, please consider to use the following citation:
+::
+
+   @misc{pyiqa,
+     title={{IQA-PyTorch}: PyTorch Toolbox for Image Quality Assessment},
+     author={Chaofeng Chen and Jiadi Mo},
+     year={2022},
+     howpublished = "[Online]. Available: \url{https://github.com/chaofengc/IQA-PyTorch}"
+   }
+
+
+Please also consider to cite our new work **TOPIQ** if it is useful to you:
+::
+
+   @misc{chen2023topiq,
+         title={TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessment}, 
+         author={Chaofeng Chen and Jiadi Mo and Jingwen Hou and Haoning Wu and Liang Liao and Wenxiu Sun and Qiong Yan and Weisi Lin},
+         year={2023},
+         eprint={2308.03060},
+         archivePrefix={arXiv},
+         primaryClass={cs.CV}
+   }
+
    
 Indices and tables
 ==================

options/default_dataset_opt.yml

@@ -139,3 +139,17 @@ bapps:
   type: BAPPSDataset
   dataroot_target: './datasets/PerceptualSimilarity/dataset'
   meta_info_file: './datasets/meta_info/meta_info_BAPPSDataset.csv'
+
+piq:
+  name: PIQ2023
+  type: PIQDataset 
+  dataroot_target: ./datasets/PIQ
+  meta_info_file: ./datasets/meta_info/meta_info_PIQDataset.csv
+  split_index: 1 
+
+gfiqa:
+  name: GFIQA 
+  type: GeneralNRDataset
+  dataroot_target: ./datasets/GFIQA/image
+  meta_info_file: ./datasets/meta_info/meta_info_GFIQADataset.csv
+  split_file: ./datasets/meta_info/gfiqa_seed123.pkl

options/train/TOPIQ/train_TOPIQ_res50_gfiqa.yml

@@ -0,0 +1,116 @@
+name: 002_CFANet_Res50_gfiqa
+# name: debug_model
+model_type: GeneralIQAModel 
+num_gpu: 1  # set num_gpu: 0 for cpu mode
+manual_seed: 123 
+
+define: &img_size_oneside 512 
+define: &img_size [*img_size_oneside, *img_size_oneside]
+
+define: &backbone resnet50
+
+define: &train_batch_size 16 
+define: &test_batch_size 1 
+
+# dataset and data loader settings
+datasets:
+  train:
+    name: GFIQA 
+    type: GeneralNRDataset
+    dataroot_target: ./datasets/GFIQA/image
+    meta_info_file: ./datasets/meta_info/meta_info_GFIQADataset.csv
+    split_file: ./datasets/meta_info/gfiqa_seed123.pkl
+
+    augment:
+      hflip: true
+    img_range: 1
+
+    # data loader
+    use_shuffle: true
+    num_worker_per_gpu: 4
+    batch_size_per_gpu: *train_batch_size
+    dataset_enlarge_ratio: 1
+    prefetch_mode: ~
+
+  val:
+    name: GFIQA 
+    type: GeneralNRDataset
+    dataroot_target: ./datasets/GFIQA/image
+    meta_info_file: ./datasets/meta_info/meta_info_GFIQADataset.csv
+    split_file: ./datasets/meta_info/gfiqa_seed123.pkl
+
+
+# network structures
+network:
+  type: CFANet 
+  use_ref: false
+  pretrained: false
+  num_crop: 1 
+  num_attn_layers: 1
+  crop_size: *img_size
+  semantic_model_name: *backbone
+  block_pool: weighted_avg
+
+# path
+path:
+  strict_load_g: true
+  resume_state: ~
+
+# training settings
+train:
+  optim:
+    type: AdamW
+    lr: !!float 3e-5 
+    weight_decay: !!float 1e-5
+
+  scheduler:
+    type: CosineAnnealingLR
+    T_max: 50 
+    eta_min: 0 
+    # type: StepLR 
+    # step_size: !!float 1e9
+    # gamma: 1.0 
+
+  total_iter: 20000 
+  total_epoch: 200
+  warmup_iter: -1  # no warm up
+
+  # losses
+  mos_loss_opt:
+    type: MSELoss 
+    loss_weight: !!float 1.0
+
+  metric_loss_opt:
+    type: NiNLoss 
+    loss_weight: !!float 1.0
+
+# validation settings
+val:
+  val_freq: !!float 800 
+  save_img: false
+  pbar: true
+
+  key_metric: srcc # if this metric improve, update all metrics. If not specified, each best metric results will be updated separately 
+  metrics:
+    srcc: 
+      type: calculate_srcc
+
+    plcc: 
+      type: calculate_plcc
+
+# logging settings
+logger:
+  print_freq: 100
+  save_latest_freq: !!float 500 
+  log_imgs_freq: 1000
+  use_tb_logger: true
+  wandb:
+    project: ~
+    resume_id: ~
+
+# dist training settings
+dist_params:
+  backend: nccl
+  port: 29500
+
+find_unused_parameters: True

pyiqa/archs/topiq_arch.py

@@ -32,6 +32,7 @@
     'cfanet_nr_spaq_res50': 'https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/cfanet_nr_spaq_res50-a7f799ac.pth',
     'cfanet_iaa_ava_res50': 'https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/cfanet_iaa_ava_res50-3cd62bb3.pth',
     'cfanet_iaa_ava_swin': 'https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/cfanet_iaa_ava_swin-393b41b4.pth',
+    'topiq_nr_gfiqa_res50': 'https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/topiq_nr_gfiqa_res50-d76bf1ae.pth',
 }
 
 
@@ -186,7 +187,7 @@ def __init__(self,
                  pretrained_model_path=None,
                  out_act=False,
                  block_pool='weighted_avg',
-                 iaa_img_size=384,
+                 test_img_size=None,
                  default_mean=IMAGENET_DEFAULT_MEAN,
                  default_std=IMAGENET_DEFAULT_STD,
                  ):
@@ -202,7 +203,7 @@ def __init__(self,
 
         self.num_class = num_class 
         self.block_pool = block_pool
-        self.iaa_img_size = iaa_img_size
+        self.test_img_size = test_img_size 
 
         # =============================================================
         # define semantic backbone network
@@ -357,12 +358,12 @@ def dist_func(self, x, y, eps=1e-12):
 
     def forward_cross_attention(self, x, y=None):
 
-        # resize image when testing IAA model
+        # resize image when testing 
         if not self.training:
-            if self.model_name == 'cfanet_iaa_ava_res50':
-                x = TF.resize(x, self.iaa_img_size, antialias=True)  # keep aspect ratio for CNN backbone 
-            elif self.model_name == 'cfanet_iaa_ava_swin':
+            if self.model_name == 'cfanet_iaa_ava_swin':
                 x = TF.resize(x, [384, 384], antialias=True)  # swin require square inputs
+            elif self.test_img_size is not None:
+                x = TF.resize(x, self.test_img_size, antialias=True)  
 
         x = self.preprocess(x)
         if self.use_ref:

pyiqa/data/base_iqa_dataset.py

@@ -61,12 +61,14 @@ def mos_normalize(self, opt):
 
             def normalize(mos_label):
                 mos_label = (mos_label - mos_range[0]) / (mos_range[1] - mos_range[0])
+                # convert to higher better if lower better is true
                 if mos_lower_better:
                     mos_label = 1 - mos_label
                 return mos_label
 
-            self.paths_mos = [(p, normalize(m)) for p, m in self.paths_mos]
-            self.logger.info(f'mos_label is normalized from {mos_range}, lower_better[{mos_lower_better}] to [0, 1], higher better.')
+            for item in self.paths_mos:
+                item[1] = normalize(float(item[1]))
+            self.logger.info(f'mos_label is normalized from {mos_range}, lower_better[{mos_lower_better}] to [0, 1], lower_better[False(higher better)].')
 
     def get_transforms(self, opt):
         transform_list = []

pyiqa/data/general_nr_dataset.py

@@ -17,7 +17,7 @@ def init_path_mos(self, opt):
     def __getitem__(self, index):
 
         img_path = self.paths_mos[index][0]
-        mos_label = self.paths_mos[index][1]
+        mos_label = float(self.paths_mos[index][1])
         img_pil = Image.open(img_path).convert('RGB')
 
         img_tensor = self.trans(img_pil) * self.img_range