Paper: Python-Based GeoImagery Dataset Development for Deep Learning-Driven Forest Wildfire Detection #916
Conversation
|
Curvenote Preview
|
Co-authored-by: Rowan Cockett <[email protected]>
|
@valemar10 Thanks for mentioning the title changes :) Really appreciate it! |
|
Hi @valemar10 👋! I'll be a reviewer for your SciPy 2024 proceedings submission. You can expect my initial review in the form of PR review here by June 29th or hopefully sooner. |
|
Hi @kafitzgerald! I will look forward to seeing your review! Thank you very much for your help. |
|
@valemar10 - Hello, My name is Kalyan , a Data and AI scientist from India. With extensive experience in tackling machine learning and artificial intelligence challenges at the enterprise level across diverse domains, I am thrilled to review your paper for the SciPy Conference 2024. This is my second year in a row reviewing for the conference, and I look forward to offering valuable suggestions and inputs to refine your paper, ensuring it effectively reaches and benefits a broader audience. I've just begin the work. You can see my initial review comments in a couple of days! |
|
Hi @kalyan678! Thank you for taking the time to review our paper for SciPy. We are happy to have your expertise and knowledge in machine learning and artificial intelligence to help us improve our work. We look forward to receiving your initial review comments. Thank you! |
|
@valemar10 - I enjoyed reviewing your paper and work. Below are my detailed comments section by section. Abstract:Overall the abstract is good but can be improved. I think you can briefly describe the key steps in the methodology. You mentioned using Google Earth Engine’s Python API but adding more context on what steps were involved in building, labeling, processing, and evaluating the dataset could be helpful. The current conclusion does a good job mentioning how the study improves environmental monitoring, but it can be clearer and more specific. For eg- The pipeline described in this paper shows how Python can be used to build and process high-resolution satellite imagery datasets, leading to accurate wildfire detection and offering a robust tool for broader environmental monitoring. Introduction:In my opinion, the introduction provides background but doesn’t clearly state the purpose of your study. I think you can clearly mention that the paper presents a Python-based methodology for creating and using a high-resolution satellite imagery dataset for forest wildfire detection. Wdyt? Next, you mention how Google Earth Engine (GEE) and Python are useful, but it would be clearer if you directly connect them to solving the problem of creating and using satellite imagery datasets. Explain how GEE makes data collection and processing faster and easier, and how Python’s tools help handle the complex steps of developing and training deep learning models with this data. This will show how these technologies specifically help overcome the challenges you’re addressing. Finally, instead of just focusing on wildfire detection, it would be great if you can also highlight how your method can be used for many environmental monitoring tasks. This shows how your approach can help with different areas of studying and reacting to changes in the environment. Again , I am happy to hear if you have any alternative thoughts on this. Downloading the Imagery Data Using GEE’s Python API :I think it is good to clarify terms like "bands B4, B3, and B2" by briefly explaining their significance in satellite imagery. Creating the Ground Truth Wildfire Labels:The process is detailed and covers essential steps, such as accessing wildfire polygon data and rasterizing them onto satellite imagery. Good job! Code 2- The code seems to be promising and very clear. I believe it is error free! Image Segmentation and Data Preparation for Deep Learning Architectures:The process is well-described, emphasizing the importance of maintaining spatial resolution and efficiency in deep learning model training. Good job! Code 3- Make sure to add comments to explain what each part of your code does. Comments help others understand how your code works by explaining the purpose of each function and major section. This makes it easier for readers to follow along with your code and see how each step fits into the larger process. Data Augmentation for Wildfire Damage Detection:The explanation about the dataset imbalance and the need for augmentation is clear and understandable. Awesome!! VGG16 Implementation:I think it is good to explain how the GeoTIFF images are processed and fed into VGG16 Code 4- Can you explain how labels are assigned based on the presence of "/Damaged/" in the file paths.? Rest all code is fine and clear. Good explanation of how VGG16 is initialized with pre-trained ImageNet weights and why the convolutional base is frozen. It clarifies that the pre-trained features are preserved while new layers focus on learning specific to wildfire detection. The choice of Adam optimizer is justified well, highlighting its adaptive learning rate which is crucial for satellite image classification where conditions can vary widely. Binary cross-entropy loss is appropriate for this binary classification task. Great job!! Awesome 💯 Code 5- The code is pretty straightforward and aligns with the described approach. Code 6- Try adding more inline comments to explain complex parts of your code, especially where you're manipulating image data or handling batch processing. Can we test data generator with different batch sizes and datasets to ensure it handles varying conditions without errors? Code 7- Training your model for 13 epochs over 933 minutes (about 15.5 hours) shows that it requires a lot of computing time. Can we make this more efficient? One way I can think here is adjusting the batch sizes of data processed at once. Can we also evaluate model performance metrics (accuracy, precision, recall) on both training and validation sets to understand how well the model generalizes to new data and its robustness against overfitting. After looking at the model performance metrics details, I feel like we should find ways to optimize VGG16's performance, such as adjusting hyperparameters, experimenting with different architectures, or exploring distributed training options to reduce training time. Conclusion:The conclusion effectively summarizes key findings and emphasizes the methodology's effectiveness. Additionally, mentioning future research directions like exploring FCNs adds completeness to the conclusion and encourages further discussion about how to expand the methodology's usefulness. Awesome!! References:All reference details are very clear and easy to navigate. General suggestions:Avoid or simplify technical terms where possible. Provide brief explanations for any unavoidable technical jargon. This concludes my initial review. Happy to clarify/ discuss if you have any questions on my comments. Good luck! |
|
Hi @kalyan678! Thank you very much for taking the time to review my paper and provide your valuable feedback. I appreciate your detailed comments and suggestions. I will be working on addressing your comments this week to improve the quality of the paper. I am committed to making the revisions that are needed. Thank you again for your time. |
|
Thanks for your review, @kalyan678! Hi @kafitzgerald, in case a little extra time is needed, the initial complete review deadline has been extended to next Wednesday, July 3rd. |
|
Hi @kafitzgerald, the Proceedings Committee has extended the initial complete review deadline to Monday, July 8th. Thanks so much for volunteering to review and we look forward to seeing your comments! |
|
Hi @kalyan678 , I hope you're doing well. I wanted to let you know that I have implemented your reviews. Thank you for everything. I appreciate your detailed feedback and support. Please let me know if there are any further changes or if you have any additional comments. Could you please go ahead and check the paper? Below are my responses to your kind comments. ABSTRACTREVIEW:Overall the abstract is good but can be improved. I think you can briefly describe the key steps in the methodology. You mentioned using Google Earth Engine’s Python API but adding more context on what steps were involved in building, labeling, processing, and evaluating the dataset could be helpful. RESPONSE:I added a more detailed description of the key steps in the methodology, including some of the libraries needed, and used the suggested specific conclusion. INTRODUCTIONREVIEW:In my opinion, the introduction provides background but doesn’t clearly state the purpose of your study. I think you can clearly mention that the paper presents a Python-based methodology for creating and using a high-resolution satellite imagery dataset for forest wildfire detection. Wdyt? RESPONSE:I agree, thank you for your valuable suggestions I have now stated the purpose of the study right from the beginning. I have elaborated on how Google Earth Engine (GEE) makes data collection and processing faster and easier. I have explained how Python’s tools help handle the steps of developing and training deep learning (DL) models with this data. Specifically, I mentioned libraries like TensorFlow for model training and libraries like Rasterio for processing satellite images. I have emphasized that while the methodology is applied to create and validate a high-resolution dataset for forest wildfire detection, it can also be adapted for various environmental monitoring tasks. I believe these revisions address the aspects of your review. DOWNLOADING THE IMAGERY DATA USING GEE’S PYTHON API :REVIEW:I think it is good to clarify terms like "bands B4, B3, and B2" by briefly explaining their significance in satellite imagery. RESPONSE:By incorporating the feedback, terms like "bands B4, B3, and B2" are clarified by briefly explaining their significance in satellite imagery. A short description is added before the code snippet, explaining its purpose. More detailed comments and more descriptive variable names are included in the code to enhance clarity and readability. CREATING THE GROUND TRUTH WILDFIRE LABELS:REVIEW:The process is detailed and covers essential steps, such as accessing wildfire polygon data and rasterizing them onto satellite imagery. Good job! RESPONSE:Thank you! IMAGE SEGMENTATION AND DATA PREPARATION FOR DEEP LEARNING ARCHITECTURES:REVIEW:The process is well-described, emphasizing the importance of maintaining spatial resolution and efficiency in deep learning model training. Good job! RESPONSE:Thank you! I commented Code 3. DATA AUGMENTATION FOR WILDFIRE DAMAGE DETECTION:REVIEW:The explanation about the dataset imbalance and the need for augmentation is clear and understandable. Awesome!! RESPONSE:Thank you! I added the example of the data augmentation. VGG16 IMPLEMENTATION:REVIEW:I think it is good to explain how the GeoTIFF images are processed and fed into VGG16 RESPONSE:I added the explanation on how the labels are assigned. REVIEW:Good explanation of how VGG16 is initialized with pre-trained ImageNet weights and why the convolutional base is frozen. It clarifies that the pre-trained features are preserved while new layers focus on learning specific to wildfire detection. The choice of Adam optimizer is justified well, highlighting its adaptive learning rate which is crucial for satellite image classification where conditions can vary widely. Binary cross-entropy loss is appropriate for this binary classification task. Great job!! Awesome 💯 RESPONSE:Thank you! Code 5-The code is pretty straightforward and aligns with the described approach. RESPONSE:The models were executed on a MacBook Pro (2019) equipped with a 2.4 GHz Intel Core i9 processor and 16 GB of 2400 MHz DDR4 memory, so we had limited computational resources. Satellite image data requires significant storage and processing power. Handling and processing such large datasets can be computationally intensive and time-consuming. Moreover, each image contains a large amount of data across multiple spectral bands and store geographic information that is not directly used by deep learning models but needs to be preserved. Thus, our current resources constrain the scale and speed of model training. All these explanations are now added to the paper. Code 6-Try adding more inline comments to explain complex parts of your code, especially where you're manipulating image data or handling batch processing. Can we test data generator with different batch sizes and datasets to ensure it handles varying conditions without errors? RESPONSE:I have added inline comments. I can test the data generator with different batches, but it will take me some time to access better computational resources through my university, I can test it with a smaller percentage of data for now. EfficientNet ImplementationCode 7-Training your model for 13 epochs over 933 minutes (about 15.5 hours) shows that it requires a lot of computing time. Can we make this more efficient? One way I can think here is adjusting the batch sizes of data processed at once. RESPONSE:It does require a lot of computing time; I can test different batch sizes and as soon as I have that I will have a better response. REVIEW:Can we also evaluate model performance metrics (accuracy, precision, recall) on both training and validation sets to understand how well the model generalizes to new data and its robustness against overfitting. RESPONSE:I do have this information and it is now added as a table in the paper for your review with the EfficientNetB0 model. Please let me know what you think. REVIEW:After looking at the model performance metrics details, I feel like we should find ways to optimize VGG16's performance, such as adjusting hyperparameters, experimenting with different architectures, or exploring distributed training options to reduce training time. RESPONSE:I agree, I am working on testing EfficientNet for this application as VGG16 is not performing as much as you’d expect, and it shows a higher degree of overfitting. As soon I have better results I will update the paper and let you know. I don’t have access to GPU’s or TPU’s at the moment, so the distributed training options is a step I have to consider further along. CONCLUSION:REVIEW 1:The conclusion effectively summarizes key findings and emphasizes the methodology's effectiveness. Additionally, mentioning future research directions like exploring FCNs adds completeness to the conclusion and encourages further discussion about how to expand the methodology's usefulness. Awesome!! RESPONSE:Great! REFERENCES:REVIEW:All reference details are very clear and easy to navigate. RESPONSE:Thanks! GENERAL SUGGESTIONS:REVIEW:Avoid or simplify technical terms where possible. Provide brief explanations for any unavoidable technical jargon. RESPONSE:I have provided more explanations! Thank you for your help. |
|
Hi @kafitzgerald ! Thank you very much for your detailed review and valuable feedback. I appreciate your focus on the geospatial data and remote sensing aspects, as well as your general comments. Your insights have been helpful. I understand the importance of clear language and have worked on improving the clarity in the abstract, introduction, discussion, and conclusion sections. I have also addressed the specific comments you made to better convey the methodology, impact, and choices made in my work. You can now see the changes I’ve made based on your reviews in the paper. I have also replied to each of your comments above. Thanks! |
@valemar10, at first glance these changes look great and really help clarify the paper! I'll do another full read through tomorrow. |
papers/Valeria_Martin/main.tex
Outdated
| To address this imbalance and enhance the model's accuracy in detecting wildfire-affected areas, data augmentation techniques are implemented. Specifically, functions using the rasterio and NumPy \citep{numpy} libraries are developed to perform image transformations, which include rotating the GeoTIFF tiles by 90°, 180°, and 270°, and flipping them horizontally and vertically. These transformations are applied to the imagery and the mask tiles that contain fire polygons and to their corresponding pre-wildfire RGB GeoTIFF tiles. | ||
|
|
||
| Finally, the augmentation process increased the diversity of the training data and resulted in a total of 111,093 pairs of labeled RGB GeoTIFF image tiles. The final dataset consists of 82,011 negative instances (no wildfire damage) and 29,082 positive instances (wildfire damage), significantly improving the class balance and reducing the risk of overfitting. | ||
| Figure X illustrates the data augmentation process applied to the satellite imagery and its labels. The original image tile is shown alongside its augmented versions, which include rotations and flips. This augmentation increases the diversity of the dataset, helping to balance the classes and improve the model's ability to detect wildfire-affected areas accurately. |
There was a problem hiding this comment.
Did you mean to reference a specific figure here? Looks like there's still a placeholder "Figure X"
There was a problem hiding this comment.
I did! Thank you. I fixed this paragraph.
papers/Valeria_Martin/main.tex
Outdated
|
|
||
| \item Reading and Transforming Images: It reads each GeoTIFF file using rasterio. The images are then transformed to have the channels last (i.e., changing the format from \textbf{(channels, height, width)} to \textbf{(height, width, channels))} to use with Keras. | ||
|
|
||
| \item Labels are assigned based on the presence of "Damaged" in the file paths. Specifically, if the file path contains the string "/Damaged/", the image is labeled as 1 (indicating it shows a damaged area). If the file path does not contain this string, the image is labeled as 0 (indicating it shows an undamaged area). In this method, the ground-truth masks are not directly used but are leveraged to define the imagery file paths. Hence, if the mask of an image had at least one pixel with a value of one, the image was redirected to the "Damaged" files. |
There was a problem hiding this comment.
I wonder if this threshold (i.e. how many pixels or what percentage of pixels should be used to classify an image as "damaged") might be something you could experiment with in the future (perhaps you've already thought about this).
Using a threshold of 1 pixel seems like it could potentially lead to false positives and other issues.
There was a problem hiding this comment.
Thank you for your feedback. It is a good catch. I understand the concern regarding the threshold used to classify an image as "damaged." The one-pixel threshold was chosen based on the presence of a polygon indicating wildfire presence, which makes it less likely for these pixels to be isolated. We checked the masks and less than 10% of them have less than 1000 pixels with one in it. However, I agree that this choice could potentially lead to false positives. I can explore different thresholds, such as a specific percentage of damaged pixels within an image, to improve classification accuracy in future experiments.
There was a problem hiding this comment.
To be clear this was more of a suggestion to think about (sounds like you already have to some degree 😄 ) or maybe for future work.
papers/Valeria_Martin/main.tex
Outdated
| \subsection{VGG16 Implementation} | ||
| VGG16 \citep{Simonyan15} is a deep CNN designed for image processing and classification tasks. It consists of 13 convolutional layers, 5 pooling layers, and 3 fully connected layers. This model is adapted to train on the dataset and detect positive and negative instances of forest wildfires. The VGG16 architecture only trains on the post-wildfire 3-channel RGB imagery, which contains both. | ||
|
|
||
| To use this CNN architecture with GeoTIFF satellite images, which typically store geographic data not supported by most deep-learning libraries, a custom function is implemented to process and feed the data to the model using the rasterio library and the shuffle function from the Scikit-learn library \citep{sklearn1,sklearn2}. This function (see Code \ref{lst}) is crucial for handling the unique structure of GeoTIFF files and converting them into a format suitable for deep learning. It is designed to read batches of GeoTIFF files, shuffle them to ensure randomness, and process them into a three-dimensional array compatible with VGG16. Here is how it works: |
There was a problem hiding this comment.
It looks like your code reference here is broken.
There was a problem hiding this comment.
You are right, thank you. I believe it is now fixed.
papers/Valeria_Martin/main.tex
Outdated
| \item Yielding Batches: Finally, the function yields batches of images and labels as numpy arrays, which are then fed into the VGG16 model for training. | ||
| \end{itemize} | ||
|
|
||
| To use this CNN architecture with GeoTIFF satellite images, which typically store geographic data not supported by most deep-learning libraries, a custom function to feed the data to the model is built using the rasterio library and the shuffle function from the Scikit-learn library \citep{sklearn1,sklearn2} (see Code \ref{lst:custom_generator}). |
There was a problem hiding this comment.
Similarly, the code reference here seems to be broken.
papers/Valeria_Martin/main.tex
Outdated
|
|
||
| \item Labels are assigned based on the presence of "Damaged" in the file paths. Specifically, if the file path contains the string "/Damaged/", the image is labeled as 1 (indicating it shows a damaged area). If the file path does not contain this string, the image is labeled as 0 (indicating it shows an undamaged area). In this method, the ground-truth masks are not directly used but are leveraged to define the imagery file paths. Hence, if the mask of an image had at least one pixel with a value of one, the image was redirected to the "Damaged" files. | ||
|
|
||
| \item Yielding Batches: Finally, the function yields batches of images and labels as numpy arrays, which are then fed into the VGG16 model for training. |
There was a problem hiding this comment.
| \item Yielding Batches: Finally, the function yields batches of images and labels as numpy arrays, which are then fed into the VGG16 model for training. | |
| \item Yielding Batches: Finally, the function yields batches of images and labels as NumPy arrays, which are then fed into the VGG16 model for training. |
papers/Valeria_Martin/main.tex
Outdated
|
|
||
| Both models are trained with subsets of the CWGID dataset, with the 6-channel input EF EfficientNet model using 25\% of the data and the VGG16 model using 50\% of the data. | ||
|
|
||
| The 6-channel input EfficientNet achieved the lowest loss (0.178) and highest accuracy (92.6\%), making it the most effective model in terms of overall performance. Its precision was at 92.5\% and its recall was 81.9\%, indicating a strong ability to identify damaged areas correctly with some missed detections. VGG16, despite processing a larger dataset, had a higher loss (1.220) but maintained an accuracy of 83.2\%. Its precision was 74.9\% and its recall was 58.8\%. This architecture took a longer training time. The EfficientNet model shows a relatively faster training time when considering the epochs and the amount of data used. The 6-channel input EfficientNet demonstrated superior performance in accurately classifying areas affected by wildfires with high precision and accuracy, making it an efficient choice for applications where quick and accurate assessments are critical. Moreover, it is important to mention that VGG16 was tested with Six-Channel inputs but proved computationally intensive. |
There was a problem hiding this comment.
| The 6-channel input EfficientNet achieved the lowest loss (0.178) and highest accuracy (92.6\%), making it the most effective model in terms of overall performance. Its precision was at 92.5\% and its recall was 81.9\%, indicating a strong ability to identify damaged areas correctly with some missed detections. VGG16, despite processing a larger dataset, had a higher loss (1.220) but maintained an accuracy of 83.2\%. Its precision was 74.9\% and its recall was 58.8\%. This architecture took a longer training time. The EfficientNet model shows a relatively faster training time when considering the epochs and the amount of data used. The 6-channel input EfficientNet demonstrated superior performance in accurately classifying areas affected by wildfires with high precision and accuracy, making it an efficient choice for applications where quick and accurate assessments are critical. Moreover, it is important to mention that VGG16 was tested with Six-Channel inputs but proved computationally intensive. | |
| The 6-channel input EfficientNet achieved the lowest loss (0.178) and highest accuracy (92.6\%), making it the most effective model in terms of overall performance. Its precision was at 92.5\% and its recall was 81.9\%, indicating a strong ability to identify damaged areas correctly with some missed detections. VGG16, despite processing a larger dataset, had a higher loss (1.220) but maintained an accuracy of 83.2\%. Its precision was 74.9\% and its recall was 58.8\%. This architecture took a longer training time. The EfficientNet model shows a relatively faster training time when considering the epochs and the amount of data used. The 6-channel input EfficientNet demonstrated superior performance in accurately classifying areas affected by wildfires with high precision and accuracy, making it an efficient choice for applications where quick and accurate assessments are critical. Moreover, it is important to mention that VGG16 was tested with six-channel inputs but proved computationally intensive. |
Co-authored-by: Katelyn FitzGerald <[email protected]>
Co-authored-by: Katelyn FitzGerald <[email protected]>
|
Hi @kafitzgerald! I just added your suggestion and further improved the paper's introduction and tables. Please let me know if there is anything else. Thank you so much for your help! |
|
Looks great to me! |
|
Hi @kalyan678! I hope everything is going well. I just finished adding the training and test metrics for both models and I was able to reduce overfitting for VGG16 and improve the computing time using a smaller subset of the data. I tested different batch sizes but this didn't change the computing time much, it has to do with my computer resources and the type of data I am using. I also tried Google Colab's Pro+ GPU but it would constantly fail. Please let me know if you have any other comments or questions! Thanks! Valeria |
@valemar10 - Apologies for responding late. I can see that you’ve made the changes based on the feedback, and they look great. Overall, it's excellent work. Wishing you the best of luck moving forward! It was a pleasure working with you, and I learned some really fascinating things, especially about Deep Learning in Wildfire Detection. Keep up the great work, and I’m excited to see where your journey takes you! |
@valemar10 - Since you've successfully reduced overfitting and improved computing time by using a smaller subset of data, it shows that you're on the right track. Given your challenges with resource limitations, another approach could be to continue focusing on optimizing your current setup. You might want to experiment with more aggressive data pruning or using a simpler, yet efficient model architecture like EfficientNet (which you're already testing). Also, considering that Google Colab Pro+ hasn’t been reliable, looking into alternative cloud platforms like AWS , GCP or Azure might provide better performance and stability for your needs, but it's also okay to continue working within the constraints of your local setup until you’re ready to scale up. |
|
Hi @kalyan678 and @kafitzgerald - Do you feel that this paper is ready for inclusion in the Proceedings? |
Yes (see above for my detailed review), and my apologies if I missed a step in finalizing my review. |
@mepa - Yes all good from my end. Thank you! |
No step missed, and thanks very much for reviewing. The final reviewer decision deadline is September 9th so I'm just checking in to confirm reviewer decisions as that deadline approaches. |
NOTE: The title we used in our accepted proposal was "Development and Application of CWGID: the California Wildfire GeoImaging Dataset for Deep Learning-Driven Forest Wildfire Detection". We changed it to "Python-Based GeoImagery Dataset Development for Deep Learning-Driven Forest Wildfire Detection" to make it more aligned with the focus of the SciPy conference, highlighting the use of Python tools for environmental monitoring. The proposal contents remain unchanged.
If you are creating this PR in order to submit a draft of your paper, please name your PR with
Paper: <title>. An editor will then add apaperlabel and GitHub Actions will be run to check and build your paper.See the project readme for more information.
Editor: Meghann Agarwal @mepa
Reviewers: