| Applications | |
Intelligent airborne monitoring of irregularly shaped man-made marine objects using statistical Machine Learning techniques
The objective of this study is to create a new platform for the automated detection of irregularly shaped man-made marine objects (ISMMMOs) in large datasets derived from marine aerial survey imagery. Readers may recall that we published the first part of the paper in October issue. We present here the concluding part |
 |
|
 |
|
 |
|
 |
|
 |
|
3. Experimental design
Offshore digital wildlife surveys for the offshore renewables sector are performed by APEM, capturing high quality images year round in all light conditions and up to four different sea states. The data is recorded using a wide range of advanced, high-resolution photogrammetry sensor technologies, including 35 mm and medium format sensors from a variety of manufacturers, in either multiple camera or a single camera congurations, subject to the scope of the project. These high-tech cameras, enabling a very high resolution ranging from 35MP to 50MP, are mounted in a tiny twin engine aircraft (e.g., Fig. 17) on a route where all areas of interest are monitored with geospatial data (i.e., latitude, longitude, and altitude). It is noteworthy to emphasise that we have followed the standardised way of constructing applications for real-world uses with the development phases of i) build the model using a dataset and move to the second phase if the test results are satisfactory ii) test/evaluate the model using another dataset completely different from the rst dataset to observe if the test results are satisfactory without overtting, and nally iii) let eld experts evaluate the model with a completely new dataset independent from the rst and second datasets. The model can be deployed if it passes these three phases successfully. These phases are outlined in Fig. 3. The obtained results as well as their evaluation are provided in the following section. The experimental design of data utilisation and data processing phases with their targeted objectives are outlined in Table 3 regarding the APEM‘s database. The viability of the methodology was ensured in 4 phases.
Phase I. Model construction (Fig. 3 I): The proposed methodology was established using 145 images with ISMMMOs and 5000 images with no ISMMMOs acquired from the 22 surveys between 2014 and 2017, with around 250 samples from each survey. The sub samples of these surveys have around 3 million large-scale images that have been obtained from the various areas of the world in all seasons and numerous time zones. This large number of surveys enabled us to identify the broad features and parameters of aerial surveys and apply these parameters to make our methodology robust. All the steps of the model construction phase are explored in the sections above in detail. Phase II was conducted after the successful execution of Phase I by realising the targeted objectives, which is elaborated as follows.
Phase II. Test of the model (Fig. 3 II): In addition to the dataset used for the establishment of the methodology, a test dataset was prepared. This set was composed of 55 images with ISMMMOs and 5000 images with no ISMMMOs, The test results are displayed in Table 10 A. We moved to the next phase to evaluate the system using independent datasets after the satisfactory results (Se, Sp, PPV, NPV, and ACC > 0.95) obtained in this phase.
Phase III. Evaluation using recent surveys (Fig. 3 III): A dataset was prepared to evaluate the eligibility of the methodology. This set consists of 57 images with ISMMMOs and 5000 images with no ISMMMOs. This set is not included in the dataset used for the establishment of the methodology to observe if the methodology works as desired for other independent datasets. The test results are displayed in Table 10 B. We moved to the next phase to verify the system with eld experts using other independent datasets after the satisfactory results (Se, Sp, PPV, NPV, and ACC > 0.95) obtained in this phase.
Phase IV. Validation by field experts using the most recent surveys (Fig. 3 IV): Furthermore, in an independent verication dataset, 9 more images with ISMMMOs and 50 images with no ISMMMOs in different surveys from the surveys on which the methodology was established were provided by APEM for afrming the viability of the system to observe if the methodology can work as desired for any aerial datasets. Two eld experts from APEM Ltd. conrmed that the established system can meet their needs to detect ISMMMOs while performing surveys. The test results are displayed in Table 10 C. The results (Se, Sp, PPV, NPV, and ACC > 0.95) obtained in this phase were found to be highly satisfactory by the eld experts.
4. Results
The effectiveness of the proposed methodology in detecting images with ISMMMOs is demonstrated by several experiments performed on many aerial survey images as elaborated in Section 3. The results of these experiments are outlined in Table 10 and they are summarised in Table 11. The numerous tangible outcomes of these successful results are demonstrated in the supplementary technical reports of the paper and in Figs. 11f, 12f, 13f, 14f, 15f. With this approach, ISMMMOs can be captured with Se, Sp, PPV, NPV, and ACC values over 0.95. More specically, 140 images out of 145, 55 images out of 57 and 9 images out of 9 in the test, evaluation and validation phases (Fig. 3) are tagged as the images with ISMMMOs successfully with a high Se over the targeted value (> 0.95) in the research, which indicates that the methodology is strong in separating positive images from negative ones in situations where it is preferable to not miss positive images. The particular results of these phases are averaged at the bottom column in Table 11. All the averages are higher than 0.95, which indicates that the required phases were completed successfully and the application is ready for real world deployment (Fig. 3). It is noteworthy to emphasise that Precision (Pr), i. e., Positive Predictive Value – PPV = Pr = TP / (TP + FP) – is 0.9813 for the average. This high value demonstrates that the model is highly successful in assigning “Positive” images to the “Positive” class while the ‘Negative” images are assigned to the “Negative” class effectively with a NPV of 0.9995. Most importantly, we calculated Matthews Correlation Coefcient (MCC) due to an unbalanced number in the classes where the number of negative values were high, which may yield misleading ACC values. The MCC, ranging from – 1 to 1, was found to be 0.971, which indicates that the model is very close to a perfect prediction (i.e., 1).
During the implementation of the methodology, testing, and evaluation, it was observed that the ISMMMOs with completely white features (i.e., R = 255, G = 255, B = 255) had difculties being detected by our methodology since they have the same characteristics as waves with respect to HSV conversion, where zero is assigned to the S component during the conversion from RGB to HSV mode (Table 1, Table 2). In this respect, it is worth noting that the images with ISMMMOs that could not be detected during the design and development phases (Fig. 3) are these types of images. Examples of these images are presented in Appendix B (Fig. 18). We refer the readers to Fig. 13 to observe how the white parts of the wind turbine cannot be detected adequately.
5. Discussion
Gibert et al. (Gibert et al., 2018) dened Data Science as a multi disciplinary eld that is a combination of data analysis, data processing techniques, and domain knowledge that transforms data into comprehensible and actionable insights relevant to making informed decisions. Within this context, the objective of this study is to create a new environmental platform for the monitoring of the maritime environment by combining domain knowledge and data scientists in a productive collaboration and perform the detection of mobile and stationary ISMMMOs in an automated manner with their geospatial coordinate system. Changes in the marine ecosystem, such as habitat loss or population decreases in marine organisms, may not be readily foreseeable and it requires long term studies to reveal the environmental changes and impacts on the ecosystem and consequently to determine the required policies accordingly. Studies in marine environments, especially far offshore, are comparatively costly and require the employment of new automatic techniques and merge of different studies for eld researchers. In this sense, this study intends to help authorities and researchers with the automatic detection of offshore ISMMMOs using an advanced platform to ll some of this gap.
The robustness of the platform was validated on a wide range of aerial maritime domains, providing a high level of empirical proof of concept with successful results (Table 11). Strictly speaking, the experimental results show that the proposed approach is efcient and effective for the detection and the segmentation of ISMMMOs in large scale aerial images. More specically, the dynamic thresholding approach employed in the methodology increases Se from 0.85 to 0.97 and Sp from 0.82 to 0.99 when compared to the static optimum threshold value as displayed in Table 4. This increase is statistically signicant (p<0.01) by rejecting the null hypothesis (i.e., there is no signicant difference between two results) using a paired-samples t-test. The ISMMMOs not detected by the methodology are all complete white objects. This issue is specied in Section 7 as a limitation of the study. Furthermore, the evaluation and validation results using the new data-sets (Table 10) that were not in the surveys used during the establishment of the methodology (Fig. 3) demonstrate that the methodology can work effectively on any aerial survey with high accuracy rates. In other words, during the evaluation phase, 55 out of 57 images with ISMMMOs were put into the positive folder and 4998 out of 5000 images with no ISMMMOs were placed into the negative folder. During the validation by eld experts, 9 out of 9 images with ISMMMOs were put into the positive folder with all objects detected successfully and 50 out of 50 images with no ISMMMOs were placed into the negative folder successfully. It must be noted that the developed methodology neither classies the detected ISMMMOs into groups nor determines the recognition of them, such as “ship”, “wind turbine” etc. Particular classication tools need to be developed to group the ISMMMOs that are placed in the positive directory by the proposed technique in this study, which is proposed as a future work in Section 6. Bespoke semi-supervised ML approaches (e.g., SelfMatch in (Xing et al., 2022a)) can be a good candidate for addressing this type of research question by extracting features from labelled data and comparing them with the features that are obtained from detected ISMMMOs based on the semantic information (e.g., (Xiao et al., 2022) and a feature/distance based matching scheme (e.g., (Xing et al., 2022b)) considering various pose compositions (e.g., (Çalıs¸kan, 2023)). The methodology not only distinguishes ISMMMOs from the blank background (sea canvas with waves in many different shapes), but also from other objects (e.g., different types of ying birds, sitting birds, big mammals (e.g., whales, dolphins), sharks, turtles, rays) in images with various shapes and characteristics successfully for which several examples can be reached in the technical report (e.g., Fig. 3 in MarineObjects_Man-made_ Supplement_1.pdf) in the supplements. The results suggest that the saturation of maritime natural objects is significantly different from ISMMMOs. The processing time for each image varies from 7 s to 16 s, depending on the image size and the number of objects in the image and their sizes, which is a very fast processing time for high-pixels-per-image (HPP) images up to 50 MB based on the camera system that is explained in Section 3. The overall computational complexity of the developed algorithms is O(n log n). It is important to point out that the supervised DL and ML approaches, designed by us in our previous work in (Kuru et al., 2023), that runs on similar images in the same surveys, can detect specic marine small natural objects (e.g., birds) in a few seconds (i.e., between 2 and 4 s). In this sense, we can conclude that DL and ML techniques slightly outperform the proposed nonsupervised technique developed in this study considering the processing time.
The current rate of global environmental alteration necessitates the quantication of impacts in species abundance in order to evaluate the effects on the ecosystem. To assess the extent of the decline, effective long-term surveillance of populations and trends is required, which is rarely the case for most species (Rosenberg et al., 2019). Environmental models work better when they are based on the ndings of more up-to- date data analysis on specic domains. It is essential to continuously monitor species and ISMMMOs in an automated manner cost-efciently, which necessitates the utilisation of sophisticated equipment with effective intelligent surveillance methods. In this regard, WILDetect, which is a new non-parametric platform by utilising a combination of supervised ML and Reinforcement Learning (RL) methods, was built in our previous work in (Kuru et al., 2023) to carry out automated wildlife censuses in highly dynamic marine environments. With similar automated platforms, one of which is proposed in this research for detecting ISMMMOs, existing labour-intensive and costly censuses performed over long periods of time can be replaced by cost-efcient and highly automated computerised systems and they can be repeated automatically in regular, shorter periods. In this way, the environmental models, equipped with near-real-time outcomes for both marine wildlife and man- made presence, can foretell future trends with more realistic projections based on human footprint, which, in turn, help mitigate the potential damaging effects of human footprint.
6. Conclusions and future work
A novel methodology, the so called ISMMMOD, that detects and splits ISMMMOs automatically in largescale images in typical large marine surveys is built. The ISMMMOD is developed using the HSV colour space and statistical analysis of histograms of the channels in this space based on the ROC curve analysis. The techniques in the methodology differ man-made structures from natural maritime habitats (i.e., waves, sea animals, birds, seawater) in various aspects, in particular, composition, features of the surface and saturation of light. The large number of surveys, that were conducted in the various geographical regions and in the various time zones and seasons, on which our methodology was built, represent the key features of aerial surveys, which made our approach powerful and resilient in detecting ISMMMOs with very high accuracy rates. The successful results obtained in this research (Table 11) is an indication that using an automated computer- based system could be an effective alternative to labour-intensive approaches. The approach built in this study can be employed for several reasons, in particular, will provide researchers and policymakers with the ability to monitor maritime industries and ensure their proper deployment through the implementation of a suitable legal and regulatory framework that takes into account the changing dynamics of marine ecosystems. Additionally, this study will direct the researchers who would like to establish similar systems using unsupervised approaches. The proposed method was tested on largescale aerial images acquired by aeroplanes and we would like to observe the results of our method on satellite images wherever datasets are available, which may reduce the cost signicantly regarding the detection of ISMMMOs and may provide a real-time and quick evaluation of ISMMMOs in marine ecosystems. This study may direct other studies about the automatic classication of marine ISMMMOs. We will be developing other novel nonparametric approaches to detect maritime life (e.g., different types of ying birds, sitting birds, big mammals (e.g., whales, dolphins), sharks, turtles, rays) automatically in large number of images in surveys using supervised approaches (e.g., (Kuru et al., 2023)) to help evaluate the maritime industry and natural ecosystem together within well- prepared models. We aim to incorporate the built methodology with camera systems used in aeroplanes and unmanned aerial vehicles (UAVs) and to employ it in real-time rescue missions on high seas and open oceans as a future work, in particular, after aeroplane crashes and maritime accidents to nd wreckages and survivals.
7. Limitations of the study
Complete white ISMMMOs as displayed in Appendix B (Fig. 18) have the same characteristics as waves (R = 255, G = 255, B = 255) with respect to HSV conversion, in particular, saturation. As it can be readily noticed in Table 1, zero is assigned to the hue and saturation during the conversion from RGB to HSV colour space when the values for the RGB colour space is 255, 255, 255 for the three channels. Our techniques perform successfully where the hue and saturation values are distinctive (i.e., H > 0 and S > 0) and therefore, these types of objects can not be detected using the approach built in this study.
Declaration of Competing Interest
The authors declare that they have no known competing nancial interests or personal relationships that could have appeared to inuence the work reported in this paper.
Data availability
Data will be made available on request.
Appendix C. Supplementary data
Supplementary data to this article can be found online at https://doi. org/10.1016/j.ecoinf.2023.102285.
End notes
1 https://apem-inc.com
2 https://www.apemltd.co.uk
3 APEM Ltd. is an environmental company and proposes novel solutions for environmental problems (https://www.apemltd.co.uk).
4 The reports from 1 to 7 titled as MarineObjects_Man-made_Supplement are for ISMMMOs and the reports from 1 to 5 titled as MarineObjects_Man-made_ Supplement_Blank are for blank images.
References
Abu, A., Diamant, R., 2022. Feature set for classication of man-made underwater objects in optical and sas data. IEEE Sensors J. 22 (6), 6027–6041. http:// dx.doi.org/ 10.1109/JSEN.2022.3148530.
Bibby, C., Jones, M., Marsden, S., 1998. Expedition Field Techniques: Bird Surveys. Royal Geographical Society, London.
Çalıs¸kan, A., 2023. Detecting human activity types from 3d posture data using deep learning models. Biomed. Signal Process. Control 81, 104479. http:// dx.doi.org/ 10.1016/j.bspc.2022.104479.
Clements, N., Robinson, W., 2022. A resurvey of winter bird communities in the Oregon coast range, USA, initially surveyed in 1968-1970. Biodiers. Data J. http://dx.doi. org/10.3897/ arphapreprints.e91575. Aug.
Davis, K.L., Silverman, E.D., Sussman, A.L., Wilson, R.R., Zipkin, E.F., 2022. Errors in aerial survey count data: identifying pitfalls and solutions. Ecol. Evol. 12 (3), e8733 http:// dx.doi.org/10.1002/ece3.8733.
Elrick-Barr, C.E., Zimmerhackel, J.S., Hill, G., Clifton, J., Ackermann, F., Burton, M., Harvey, E.S., 2022. Man-made structures in the marine environment: a review of stakeholders’ social and economic values and perceptions. Environ. Sci. Pol. 129, 12–18. http:// dx.doi.org/10.1016/j.envsci.2021.12.006.
Gibert, K., Horsburgh, J.S., Athanasiadis, I.N., Holmes, G., 2018. Environmental data science. Environ. Model Softw. 106, 4–12 special Issue on Environmental Data Science. Applications to Air quality and Water cycle. https://doi. org/10.1016/j.envsoft.2018.04.005.
Graber, J., 2011. Land-Based Infrared Imagery for Marine Mammal Detection. Master’s thesis,. University of Washington, USA.
Han, Z., Xing, J., Wang, X., Xue, F., Fan, J., 2022. A robust lcse-resnet for marine manmade target classification based on optical remote sensing imagery. Int. J. Artif. Intell. Tools 31 (06), 2240022. http:// dx.doi.org/10.1142/S021821302240022X.
Kuru, K., 2014. Optimization and enhancement of h&e stained microscopical images by applying bilinear interpolation method on lab color mode. Theor. Biol. Med. Mod. 11 (1), 9. http:// dx.doi.org/10.1186/1742-4682-11-9.
Kuru, K., Khan, W., 2018. Novel hybrid object-based non-parametric clustering approach for grouping similar objects in specific visual domains. Appl. Soft Comput. 62, 667–701. http://dx.doi. org/10.1016/j.asoc.2017.11.007.
Kuru, K., Yetgin, H., 2019. Transformation to advanced mechatronics systems within new industrial revolution: a novel framework in automation of everything (aoe). IEEE Access 7, 41395–41415. http://dx.doi. org/10.1109/ACCESS.2019.2907809
Kuru, K., Girgin, S., Arda, K., Bozlar, U., 2013. A novel report generation approach for medical applications: the sisds methodology and its applications. Int. J. Med. Inform. 82 (5), 435–447. http://dx.doi. org/10.1016/j.ijmedinf.2012.05.019.
Kuru, K., Clough, S., Ansell, D., McCarthy, J., McGovern, S., 2023. Wildetect: an intelligent platform to perform airborne wildlife census automatically in the marine ecosystem using an ensemble of learning techniques and computer vision. Expert Syst. Appl. 231, 120574 http://dx.doi. org/10.1016/j.eswa.2023.120574
Leira, F.S., Johansen, T.A., Fossen, T.I., 2015. Automatic detection, classification and tracking of objects in the ocean surface from uavs using a thermal camera. In: 2015 IEEE Aerospace Conference, pp. 1–10. http://dx.doi. org/10.1109/AERO.2015.7119238.
Loesdau, M., Chabrier, S., Gabillon, A., 2014. Hue and saturation in the RGB color space. In: Elmoataz, A., Lezoray, O., Nouboud, F., Mammass, D. (Eds.), Image and Signal Processing: 6th International Conference, ICISP 2014, Cherbourg, France, June 30–July 2, 2014. Proceedings. Springer International Publishing, Cham, pp. 203–212. http://dx.doi. org/10.1007/978-3-319-07998-1_23.
Lopez, J., Schoonmaker, J., Saggese, S., 2014. Automated detection of marine animals using multispectral imaging. In: 2014 Oceans – St. John’s, pp. 1–6. http://dx.doi.org/10.1109/ OCEANS.2014.7003132.
McIntosh, B.S., Alexandrov, G., Matthews, K., Mysiak, J., van Ittersum, M., 2011. Preface: thematic issue on the assessment and evaluation of environmental models and software.
Environ. Model Softw. 26 (3), 245– 246 thematic issue on the assessment and evaluation of environmental models and software. https://doi. org/10.1016/j.envsoft.2010.08.008.
Mehrnejad, M., Albu, A.B., Capson, D., Hoeberechts, M., 2013. Detection of stationary animals in deep-sea video. In: 2013 OCEANS – San Diego, pp. 1–5. http://dx.doi.org/10.23919/ OCEANS.2013.6741095.
Noe, A., Pessoa, L., Thompson, E., 2000. Beyond the grand illusion: what change blindness really teaches us about vision. Vis. Cogn. 7 (2), 93–106. Paleczny, M., Hammill, E., Karpouzi, V., Pauly, D., 2015. Population trend of the world’s monitored seabirds, 1950-2010. PLoS One 10 (6), e0129342. http://dx.doi. org/10.1371/journal.pone.0129342.
Riser, S.C., Freeland, H.J., Roemmich, D., Wijffels, S., Troisi, A., Belb´eoch, M., Gilbert, D., Xu, J., Pouliquen, S., Thresher, A., Traon, P.-Y.L., Maze, G., Klein, B., Ravichandran, M., Grant, F., Poulain, P.-M., Suga, T., Lim, B., Sterl, A., Sutton, P., Mork, K.-A., Velez-Belchí, P.J., Ansorge, I., King, B., Turton, J., Baringer, M., Jayne, S.R., 2016. Fifteen years of ocean observations with the global argo array. Nat. Clim. Chang. 6 (1), 145–153.
Rosebrock, A., 2017. Face alignment with opencv and python [cited 01.13.2022]. URL. http://www. pyimagesearch.com/2017/05/22/facealignment-with-opencv-and-python/.
Rosenberg, K.V., Dokter, A.M., Blancher, P.J., Sauer, J.R., Smith, A.C., Smith, P.A., Stanton, J.C., Panjabi, A., Helft, L., Parr, M., Marra, P.P., 2019. Decline of the north american avifauna. Science 366 (6461), 120–124. http://dx.doi. org/10.1126/ science.aaw1313.
S`anchez-Marr`e, M., Cort´es, U., Comas, J., 2004. Environmental sciences and artificial intelligence. Environ. Model Softw. 19 (9), 761–762 environmental Sciences and Artificial Intelligence. https:// doi.org/10.1016/j.envsoft.2003.08.009.
Saur, G., Estable, S., Zielinski, K., Knabe, S., Teutsch, M., Gabel, M., 2011. Detection and classification of man-made offshore objects in terrasar-x and rapideye imagery: Selected results of the demarine-deko project. In: Oceans 2011 IEEE – Spain, pp. 1–10. http://dx.doi.org/10.1109/ Oceans-Spain.2011.6003596.
Shi, Y., Shen, C., Fang, H., Li, H., 2017. Advanced control in marine mechatronic systems: a survey. IEEE/ASME Trans. Mech. 22 (3), 1121–1131. http://dx.doi. org/10.1109/TMECH.2017.2660528.
Xiao, Z., Zhang, H., Tong, H., Xu, X., 2022. An efficient temporal network with dual selfdistillation for electroencephalography signal classification. In: 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1759–1762. http://dx.doi. org/10.1109/BIBM55620.2022.9995049.
Xing, H., Xiao, Z., Zhan, D., Luo, S., Dai, P., Li, K., 2022a. Selfmatch: robust semisupervised time-series classification with self-distillation. Int. J. Intell. Syst. 37 (11), 8583–8610. http://dx.doi.org/10.1002/int.22957.
Xing, H., Xiao, Z., Qu, R., Zhu, Z., Zhao, B., 2022b. An efficient federated distillation learning system for multitask time series classification. IEEE Trans. Instrum. Meas. 71, 1–12. http://dx.doi. org/10.1109/TIM.2022.3201203.
Zhang, T., Tian, B., Sengupta, D., Zhang, L., Si, Y., 2021. Global offshore wind turbine dataset. Sci. Data 8 (1), 191
© 2023 The Author(s). Published by Elsevier B.V. in Ecological Informatics 78 (2023) 102285. This is an open access article under the CC BY license (http:// creativecommons.org/licenses/by/4.0/).
The paper is republished with authors’ permission.
(No Ratings Yet)
Loading...
Leave your response!