Factor: Safety- Street lights/Nighttime lights #53
Comments
amyburness
added
the
🏙️ Place characterization
|
If data for streetlights or nighttime lights is unavailable, or if the user has disaggregated statistics, they should have the option to provide data on women's perceived safety at the municipal, district, state, or any other required level. The tool would then standardize these scores, percentages, or statistics on a scale from 0 to 5, where 5 indicates the lowest level of violence or the highest level of perceived safety. |
SAFStreetLight Function Overview:
Processes streetlight vector data to create a raster of safety scores based on coverage. For detailed explanation, see docs/dev_notes/README.md in the issue-53-factor-safety branch. Constants: Light area radius: 20 meters Function:
Notes:
Substitute Streetlights Data GenerationDue to the unavailability of actual streetlight location data in the sample dataset and limited time for sourcing, a method was devised to generate substitute data based on available road network information. 'Residential' roads from the 'highways' category This extraction process provided a foundation for estimating streetlight distributions.
Residential Streetlight Generation
Major Road Streetlight Generation The merged major roads were filtered based on segment lengths to exclude longer cross-country roads that typically have less frequent lighting.
Resulting Distribution
|
|
"Perceived Safety Data" handling is still outstanding |
Could you please elaborate on and/or provide examples of the file format of the
|
|
@osundwajeff @javaftw As mentioned yesterday, if streetlights data or nightlights data is not available, the user will have the option to input a safety score for the country or provide disaggregated scores by administrative units. The indicator used will be the number of homicides per 100,000 people, standardized based on the following scale: Score 5 (Safest): 0 to 1 homicides per 100,000 people |
Hi @carolinamayh (Hennie here), While the information about "the user will have the option to input a safety score for the country" is useful (there are UI elements in the plugin which performs this action), the part about "or provide disaggregated scores by administrative units" is unclear, despite the accompanying breakdown of the scoring mechanic. |
2024-08-07T12:32:58 INFO Results: {'OUTPUT': 'temp/countryUTMLayerBuf.shp'} Traceback (most recent call last): |
|
Input Type: Raster
Input Type: Shapefile (Points):
Input Type: Shapefile (Polygons) When the input is a polygon layer, the user may choose the field of interest within that layer to process. If the field contains text (and not numeric) values, the name of the field is appended with
If the user chooses a text element in the dropdown, values may be assigned to the unique text fields
The result of the above operation, using user-assigned text field values:
The results of using a numeric field:
Input Type: User Value If no input file is specified, the user value is evaluated:
|
|
Improvement of point rendering method
|
|
@osundwajeff @mvmaltitz we will provide detailed guidance on the processing of the NTL data for this factor today; |
Enhanced Night Time Light (NTL) Classification for GEESTApproach and RationaleWe propose refining the NTL classification in GEEST to better reflect local conditions, particularly for small island countries. Instead of using fixed thresholds or coverage percentages, we'll employ local statistics to create a more contextually relevant classification. This approach is supported by Elvidge et al. (2013), who emphasize the importance of considering local context in night-time light analysis. Standard Classification Procedure:
Standard Classification Scheme:
This classification approach aligns with the methods used by Chen & Nordhaus (2011) in using luminosity data as a proxy for socio-economic factors. QGIS Implementation:For implementation in QGIS, use the Raster Calculator with the following expression: Replace Example Python Script for Calculating Statistics:Here's a simple Python script to calculate the required statistics for a GeoTIFF raster: This script can be integrated into the GEEST QGIS plugin to automatically calculate the statistics needed for the classification. Benefits:
Additional Note: Handling Areas with Low NTL Values:If the NTL values within the Area of Interest (AOI) are predominantly or entirely below 0.05, which is our baseline for electricity access, we need to adjust our approach. Here's how to proceed:
Low NTL Classification Scheme:
This adaptive thresholding approach is conceptually similar to methods described by Otsu (1979) for image classification. QGIS Implementation:For implementation in QGIS, use the Raster Calculator with the following expression: Note: Replace "NTL@1" with our actual raster layer name. Additional Considerations:Temporal Variability:It's important to note that NTL data can vary seasonally and annually. As Xie et al. (2019) demonstrate, temporal variations in artificial nighttime lights can provide insights into urbanization patterns and socio-economic changes. For GEEST, consider using multi-year averaged NTL data to mitigate short-term fluctuations and capture more stable patterns of lighting infrastructure. Limitations and Caveats:Users of this enhanced NTL classification in GEEST should be aware of certain limitations:
References:
|
|
@osundwajeff @mvmaltitz please see above, grateful if you could implement the NTL processing accordingly. I will move this to Work in Progress, thank you! |
|
Good morning Benny,
Thank you for this information. Just a few quick questions:
- Can you confirm the source and resolution of the NTL data to be used? Is
it already available, or do I need to obtain it?
- Are there any specific guidelines for the statistics (esp. handling
outliers, excluding certain values)?
- What should be done if the calculated median/90th percentile is very
close or equal to the max value?
Kind regards,
Hennie Kotze
…On Wed, Aug 21, 2024 at 4:46 PM Benny Istanto ***@***.***> wrote:
Enhanced Night Time Light (NTL) Classification for GEEST Approach and
Rationale
We propose refining the NTL classification in GEEST to better reflect
local conditions, particularly for small island countries. Instead of using
fixed thresholds or coverage percentages, we'll employ local statistics to
create a more contextually relevant classification. This approach is
supported by Elvidge et al. (2013), who emphasize the importance of
considering local context in night-time light analysis.
Standard Classification Procedure:
1. Clip the global NTL raster to the country boundary.
2. Calculate statistics for the clipped raster: min, max, mean,
median, and 90th percentile.
- The use of percentiles in NTL analysis is supported by Levin &
Zhang (2017).
3. Apply the classification scheme based on local statistics.
Standard Classification Scheme:
GEEST Class Description NTL Value Range
0 No Access 0 - 0.05
1 Very Low > 0.05 - 0.25 * Median
2 Low > 0.25 * Median - 0.5 * Median
3 Moderate > 0.5 * Median - Median
4 High > Median - 90th percentile
5 Very High > 90th percentile
This classification approach aligns with the methods used by Chen &
Nordhaus (2011) in using luminosity data as a proxy for socio-economic
factors.
QGIS Implementation:
For implementation in QGIS, use the Raster Calculator with the following
expression:
(NTL <= 0.05) * 0 +
(NTL > 0.05 AND NTL <= [0.25 * median]) * 1 +
(NTL > [0.25 * median] AND NTL <= [0.5 * median]) * 2 +
(NTL > [0.5 * median] AND NTL <= [median]) * 3 +
(NTL > [median] AND NTL <= [90th_percentile]) * 4 +
(NTL > [90th_percentile]) * 5
Replace [median] and [90th_percentile] with the actual calculated values.
Example Python Script for Calculating Statistics:
Here's a simple Python script to calculate the required statistics for a
GeoTIFF raster:
import rasterio
import numpy as np
def calculate_raster_stats(raster_path):
with rasterio.open(raster_path) as src:
data = src.read(1)
data = data[data != src.nodata] # Remove no data values
min_val = np.min(data)
max_val = np.max(data)
mean_val = np.mean(data)
median_val = np.median(data)
percentile_90 = np.percentile(data, 90)
return {
'min': min_val,
'max': max_val,
'mean': mean_val,
'median': median_val,
'90th_percentile': percentile_90
}
# Usage
raster_path = 'path/to/your/iso3_ntl.tif'
stats = calculate_raster_stats(raster_path)
print(stats)
This script can be integrated into the GEEST QGIS plugin to automatically
calculate the statistics needed for the classification.
Benefits:
- Adapts to local lighting conditions
- Provides meaningful classification for areas with varying NTL
intensities
- Maintains consistency with the original GEEST approach while
improving granularity
Additional Note: Handling Areas with Low NTL Values:
If the NTL values within the Area of Interest (AOI) are predominantly or
entirely below 0.05, which is our baseline for electricity access, we need
to adjust our approach. Here's how to proceed:
1.
*Check the data range:*
After clipping the NTL raster to our AOI, examine the statistics,
particularly the maximum value.
2.
*If max value < 0.05:*
- This indicates that the entire area falls below our standard
threshold for electricity access.
- In this case, we need to create a relative scale based on the
available data. This approach is informed by Small et al. (2005), who
discuss methods for analyzing areas with low light emissions.
3.
*Adjusted classification for low-value areas:*
Use this modified classification scheme based on the local maximum
value:
Low NTL Classification Scheme:
GEEST Class Description NTL Value Range
0 No Light 0
1 Very Low > 0 - 0.2 * max_value
2 Low > 0.2 * max_value - 0.4 * max_value
3 Moderate > 0.4 * max_value - 0.6 * max_value
4 High > 0.6 * max_value - 0.8 * max_value
5 Highest > 0.8 * max_value - max_value
This adaptive thresholding approach is conceptually similar to methods
described by Otsu (1979) for image classification.
QGIS Implementation:
For implementation in QGIS, use the Raster Calculator with the following
expression:
with_variable('max_val', ***@***.***"),
(NTL = 0) * 0 +
(NTL > 0 AND NTL <= @max_val * 0.2) * 1 +
(NTL > @max_val * 0.2 AND NTL <= @max_val * 0.4) * 2 +
(NTL > @max_val * 0.4 AND NTL <= @max_val * 0.6) * 3 +
(NTL > @max_val * 0.6 AND NTL <= @max_val * 0.8) * 4 +
(NTL > @max_val * 0.8) * 5
)
Note: Replace ***@***.***" with our actual raster layer name.
Additional Considerations: Temporal Variability:
It's important to note that NTL data can vary seasonally and annually. As
Xie et al. (2019) demonstrate, temporal variations in artificial nighttime
lights can provide insights into urbanization patterns and socio-economic
changes. For GEEST, consider using multi-year averaged NTL data to mitigate
short-term fluctuations and capture more stable patterns of lighting
infrastructure.
Limitations and Caveats:
Users of this enhanced NTL classification in GEEST should be aware of
certain limitations:
- Overglow Effect: NTL data can suffer from the "overglow" effect,
where light appears to extend beyond its actual source (Small et al.,
2005). This may lead to overestimation of lit areas in some regions.
- Saturation in Urban Centers: In highly urbanized areas, NTL sensors
may saturate, leading to a loss of differentiation in the brightest areas
(Elvidge et al., 2013).
- Rural Underrepresentation: In sparsely populated or rural areas, the
resolution of NTL data may not capture small-scale lighting infrastructure
adequately.
References:
1. Elvidge, C. D., Baugh, K. E., Zhizhin, M., & Hsu, F.-C. (2013). Why
VIIRS data are superior to DMSP for mapping nighttime lights. Proceedings
of the Asia-Pacific Advanced Network, 35(0), 62.
https://doi.org/10.7125/APAN.35.7
2. Levin, N., & Zhang, Q. (2017). A global analysis of factors
controlling VIIRS nighttime light levels from densely populated areas.
Remote Sensing of Environment, 190, 366–382.
https://doi.org/10.1016/j.rse.2017.01.006
3. Small, C., Pozzi, F., & Elvidge, C. D. (2005). Spatial analysis of
global urban extent from DMSP-OLS night lights. Remote Sensing of
Environment, 96(3-4), 277-291.
https://doi.org/10.1016/j.rse.2005.02.002
4. Chen, X., & Nordhaus, W. D. (2011). Using luminosity data as a
proxy for economic statistics. Proceedings of the National Academy of
Sciences, 108(21), 8589-8594. https://doi.org/10.1073/pnas.1017031108
5. N. Otsu, "A Threshold Selection Method from Gray-Level Histograms,"
in IEEE Transactions on Systems, Man, and Cybernetics, vol. 9, no. 1, pp.
62-66, Jan. 1979, https://doi.org/10.1109/TSMC.1979.4310076
6. Xie, Y., Weng, Q., & Fu, P. (2019). Temporal variations of
artificial nighttime lights and their implications for urbanization in the
conterminous United States, 2013–2017. Remote Sensing of Environment, 225,
160-174. https://doi.org/10.1016/j.rse.2019.03.008
—
Reply to this email directly, view it on GitHub
<#53 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ANEBKFENYEJATB2UQINTIG3ZSSR23AVCNFSM6AAAAABJV65GZWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBSGIZTAOJRGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@osundwajeff, please note that the score 5 ( Very High) is > 75th percentile |
|
@osundwajeff @mvmaltitz @javaftw When raster data input (nighttime lights) it still gets: Traceback (most recent call last): Also for the points input it gets this:
where we can see that there are street lights but no data at all |
Hi Hennie Answering your question below:
You can use the clipped version below for LCA testing
graph TD
A[Download NTL data] --> B[Gunzip data]
B --> C[Clip using country boundary]
C --> D[Check max_value]
D -->|max_value > 0.05| E[Check median and 75th percentile]
D -->|max_value <= 0.05| F[Use modified classification scheme]
E -->|Difference > 5% of max_value| G[Use original classification scheme]
E -->|Difference <= 5% of max_value| F
G -->|Use median and percentile| H[Classify NTL data]
F -->|Use max_value| H
H --> I[Output classified NTL map]
style A fill:#f9f,stroke:#333,stroke-width:2px
style B fill:#bbf,stroke:#333,stroke-width:2px
style C fill:#bbf,stroke:#333,stroke-width:2px
style D fill:#fbb,stroke:#333,stroke-width:2px
style E fill:#fbb,stroke:#333,stroke-width:2px
style F fill:#bfb,stroke:#333,stroke-width:2px
style G fill:#bfb,stroke:#333,stroke-width:2px
style H fill:#fbf,stroke:#333,stroke-width:2px
style I fill:#ff9,stroke:#333,stroke-width:2px
This diagram illustrates the process as follows:
Original Classification Scheme:
Modified Classification Scheme:
|
mvmaltitz
changed the title
|
@dragosgontariu
Street lights (points)
Nighttime lights (raster)
|
|
@mvmaltitz I've obtained different results for the NTL classification for LCA. Here are my findings:
The resulting classification map is shown below: You can review the code I used to generate this map here. Could you please compare these results with your findings? I'm particularly interested in understanding any discrepancies in the classification thresholds or the final map output. |
|
@mvmaltitz @osundwajeff please ensure that the descriptive text on the tab is updated and explains briefly the processing of each data type. Also, please rename this factor "Safety" as per the indications in the Indicators excel --> remove "safe urban design" |
|
@bennyistanto Hi Benny Modified algo + original supplied input data data:Input:
Output:
Modified algo + downloaded input data data:(Data download source )
Output:
Code snippet: Using the same approach in both cases using different but very similar input data, the results appear to be significantly different. |
|
Hey Hennie @javaftw, I've figured out why our maps look different. The classification patterns we're seeing on the map are different because we're getting different statistical values, especially for the Here's what I'm guessing you got:
And here's what I'm seeing:
The reason for this difference? It's probably how we're dealing with those pesky When we clip the raster with the country boundary, we might end up with areas outside the boundary that aren't properly marked as I took a look at the GitHub code (https://github.com/kartoza/GEEST/blob/dev/gender_indicator_tool/gender_indicator_tool.py, lines 3640-3685), and I noticed it's following the Colab script. The thing is, Colab handles NaN values correctly during clipping, but in the gender_indicator_tool.py, especially in the gdal:cliprasterbymasklayer part, nodata is set to 0. I tried to make the Colab script work in the PyQGIS console by tweaking how we set the NODATA value. The goal was to make sure the areas outside the boundary are properly recognized as NODATA. I've got the full code below if you want to take a look.
After testing in both Colab and PyQGIS with this updated script, I'm getting the same stats for QGIS test import numpy as np
from qgis.core import QgsRasterLayer
# File path to your raster
raster_path = r'D:\temp\geest\factor_safety\ntl\lca_VNL_npp_2023_global_vcmslcfg_v2_c202402081600.average.dat.tif'
# Load the raster layer
raster_layer = QgsRasterLayer(raster_path, "NTL")
# Check if the raster layer is valid
if not raster_layer.isValid():
print("Error: Raster failed to load.")
else:
provider = raster_layer.dataProvider()
extent = raster_layer.extent()
cols = raster_layer.width()
rows = raster_layer.height()
block = provider.block(1, extent, cols, rows)
# Convert raster block to numpy array
data = np.array([[block.value(i, j) for j in range(cols)] for i in range(rows)])
# Handle nodata values (non-NaN)
nodata = provider.sourceNoDataValue(1)
valid_data = data[~np.isnan(data) & (data != nodata)]
if valid_data.size > 0:
# Calculate statistics
min_value = np.min(valid_data)
max_value = np.max(valid_data)
median = np.median(valid_data)
percentile_75 = np.percentile(valid_data, 75)
print(f"Min Value: {min_value:.6f}")
print(f"Max Value: {max_value:.6f}")
print(f"Median: {median:.6f}")
print(f"75th Percentile: {percentile_75:.6f}")
else:
print("No valid data found.")Colab test import numpy as np
import rasterio
def analyze_raster(raster_path):
# Open the raster file
with rasterio.open(raster_path) as src:
data = src.read(1) # Read the first band
# Retrieve the nodata value from the raster metadata
nodata = src.nodata
# Get valid data (non-NaN values and not nodata)
valid_data = data[~np.isnan(data) & (data != nodata)]
if valid_data.size > 0:
# Calculate statistics
min_value = np.min(valid_data)
max_value = np.max(valid_data)
median = np.median(valid_data)
percentile_75 = np.percentile(valid_data, 75)
print(f"Min Value: {min_value:.6f}")
print(f"Max Value: {max_value:.6f}")
print(f"Median: {median:.6f}")
print(f"75th Percentile: {percentile_75:.6f}")
else:
print("No valid data found.")
# Path to your raster file
raster_path = f'{dir}/ntl/lca_VNL_npp_2023_global_vcmslcfg_v2_c202402081600.average.dat.tif'
# Run the analysis
analyze_raster(raster_path)Both script returned the same value:
What do you think? Does this help explain the differences we're seeing? |
|
Safety trigger a warning on OGR when open the NTL data, although the data is available and loaded into map, dragged from Explorer window.
After Execute the button, the label next to the button appear with text: See below the python warning And below the Processing log And OGR log |
Mapillary
This indicator should be calculated by creating 20-meter buffers around streetlights. Raster cells in which 80-100% of their area intersects with the buffers should receive a score of 5. Rasters with 60-79% intersection should be scored as 4, 40-59% as 3, 20-39% as 2, and 1-19% as 1. Areas without any overlap with streetlight buffers should be scored as 0. NOTE: Use nighttime lights only if street lights are absent.
Nighttime Lights
Nighttime lights observation data from a Low light imaging sensor part of the Visible and Infrared Imaging Suite (VIIRS) Day Night Band (DNB). Unit - (average, average-masked, max, median, median-masked, min) nW/cm2/sr, CRS - EPSG:4326 (Geographic Latitude/Longitude), Resolution - 15 arc second (~500m at the Equator)
The text was updated successfully, but these errors were encountered: