@@ -22,9 +22,11 @@ Safety and security are critical issues in maritime environment. Automatic and r
Fig. 2. Proposed RetinaNet based fusion framework. The original input images are of size 3240 _ 944 pixels. They are fused using VSM-WLS in order to provide complementary information for object detection. Then, the fused image is processed by RetinaNet in order to detect and localize objects around the vessel.
# Deep Convolutional Neural Network-based Fusion of RGB and IR Images in Marine Environment [4]:
According to the weakness and strengths of color camera and IR, this idea arises from the intuition that an improved solution would combine data from both homogenous sensors to produce more accurate and reliable performance. However, there is no general guideline for network architecture design, and questions of “what to fuse?”, “when to fuse?”, and “how to fuse?” remain open. Inspired by this consideration, we investigated how IR and camera images can be integrated for carrying out object detection. Beside the middle fusion framework in the previous work, we proposed a late multi-modal fusion to provide complimentary information from RGB and thermal infrared cameras in order to improve the detection performance. This framework (Fig.3) first employs RetinaNet as a dense simple deep model for each input image separately to extract possible candidate proposals which likely contain the targets of interest. Then, all proposals are generated by concatenating the obtained proposals from two modalities. Finally, redundant proposals are removed by Non-Maximum Suppression (NMS).
Fig. 3. An overview of the proposed late fusion framework. Our framework has two feature extractor: (A) a RetinaNet for process RGB input image and (B) a RetinaNet for extracting features from the corresponding input IR image. (C) The framework concatenates outputs of RetinaNet networks (ORGB,OIR), and then a final set of target proposals is obtained after none-maximum suppression. (D) The final output containing predicted bounding boxes which are associated with a category label and a objectness score.
# References
1. F. Farahnakian, M.Haghbayan, J. Poikonen, M. Laurinen, P. Nevalainen and J. Heikkonen, “Object Detection based on Multi-sensor Proposal Fusion in Maritime Environment”, The 17th IEEE International Conference on Machine Learning and Applications (ICMLA), 2018, US.
