diff --git a/README.md b/README.md index 863ba50..db7b407 100644 --- a/README.md +++ b/README.md @@ -1,43 +1,38 @@ -# FTOT - -Freight and Fuel Transportation Optimization Tool - -## Description: -FTOT is a flexible scenario-testing tool that optimizes the transportation of materials for future energy and freight -scenarios. FTOT models and tracks commodity-specific information and can take into account conversion of raw materials to products (e.g., crude oil to jet fuel and diesel) and the fulfillment of downstream demand. FTOT was developed at the US Dept. of Transportation's Volpe National Transportation Systems Center. - -## Installation: -See [FTOT Installation wiki](https://github.com/VolpeUSDOT/FTOT-Public/wiki/FTOT-Installation-Guide) for the detailed instructions. -* FTOT is a python based tool. -* Clone or download the repository. -* Install the required dependencies (including ESRI ArcGIS) -* Download the [documentation and scenario dataset](https://github.com/VolpeUSDOT/FTOT-Public/wiki/Documentation-and-Scenario-Datasets) - -## Usage: -* Usage is explained in the Quick Start documentation here: [Documentation and Scenario Dataset Wiki](https://github.com/VolpeUSDOT/FTOT-Public/wiki/Documentation-and-Scenario-Datasets) - -## Contributing: -* Add bugs and feature requests to the Issues tab for the Volpe Development Team to triage. - -## Credits: -* Dr. Kristin Lewis (Volpe) -* Matthew Pearlson (Volpe) -* Alexander Oberg (Volpe) -* Olivia Gillham (Volpe) -* Gary Baker (Volpe) -* Dr. Scott B. Smith (Volpe) -* Amy Vogel (Volpe) -* Amro El-Adle (Volpe) -* Kirby Ledvina (Volpe) -* Kevin Zhang (Volpe) -* Michelle Gilmore (Volpe) -* Mark Mockett (iBiz) - -## Project Sponsors: -The development of FTOT that contributed to this public version was funded by the U.S. Federal Aviation Administration (FAA) Office of Environment and Energy and the Department of Defense (DOD) Office of Naval Research through Interagency Agreements (IAA) FA4SCJ and FB48CS under the supervision of FAA’s Nathan Brown and by the U.S. Department of Energy (DOE) Office of Policy under IAA VXS3A2 under the supervision of Zachary Clement. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the FAA nor of DOE. - -## Acknowledgements: -The FTOT team thanks our Beta testers and collaborators for valuable input during the FTOT Public Release beta testing, including Dane Camenzind, Kristin Brandt, and Mike Wolcott (Washington State University), Mik Dale (Clemson University), Emily Newes and Ling Tao (National Renewable Energy Laboratory), Seckin Ozkul, Robert Hooker, and George Philippides (Univ. of South Florida), and Chris Ringo (Oregon State University). - -## License: -This project is licensed under the terms of the FTOT End User License Agreement. Please read it carefully. +# Forest Residuals-to-Jet Fuel Supply Chain Documentation (2021.1 Version) + + +Created by WSU, +Jie Zhao (Graduate Research Assistant, Washington State University, Pullman, WA 99164, USA, E-mail: jie.zhao2@wsu.edu); +Ji Yun Lee (Assistant Professor, Washington State University, Pullman, WA 99164, USA, E-mail: jiyun.lee@wsu.edu) + +## Overview: +Supply chain resilience assessment includes two parts: integrated risk assessment to capture the combined effects of multiple risk factors on supply chain performance, and resilience assessment to calculate the long-term supply chain resilience in planning horizon. +The purpose of this documentation is to illustrate how to run the Forest Residuals-to-Jet Fuel supply chain resilience assessment based on FTOT optimization and several outputs from risk assessment. + +## Getting Started: +(a) Forest Residuals-to-Jet Fuel supply chain scenario is stored C:\FTOT_SCR\scenarios\ForestResiduals_SCR folder. +(b) Download the common_data folder from the FTOT dataset and save it into C:\FTOT_SCR/scenarios folder. +(c) ArcGIS 64‐bit background geoprocessing is required to run this scenario due to the large computational burden. +(d) The batch script file (run_v5_ 1.bat) is included in the ForestResiduals_SCR folder to automate each step required in FTOT optimization. + +## ForestResiduals_SCR folder includes: +(a) scenario configuration file (scenario.XML): to define the locations of the files and parameter values used in the run. +(b) the batch script file (run_v5_ 1.bat): to execute FTOT run in a sequence of steps. +(c) input_data folder: to record facility-related data and each facility type has a separate CSV file. +(d) input_GISdata folder: to contain a facilities.gdb file. +(e) 1- to 6- Python files: to simulate the effects of risk factors on supply chain performance (risk assessment). +(f) CSV and TXT files: to present all inputs data into 1- to 6- Python files for risk assessment. + +## Run Step: +(a) Go to FTOT_SCR\scenarios\ForestResiduals_SCR folder. +(b) Risk assessment: run 1- to 6- Python files in sequence, and then the outputs can be used into modified FTOT optimization run. +(c) Resilience assessment: run the batch script by double-clicking it or manually executing it in the Command Prompt. + +## Run Scenario: +(a) Informational logging is available in the ForestResiduals_SCR\logs folder. +(b) Optimization steps would be iterated for each scenario and time period. +(c) Each optimization should run take about 10 minutes, the total simulation will take about 20 days. + +## Results: +(a) Final result (supply chain resilience for every scenario) is save into a NPY file, called Resilience.npy. +(b) Related results, such as resilience components, weight factors for each component, are saved in associated NPY files. diff --git a/program/ftot.py b/program/ftot.py index 4c46a83..2fb9477 100644 --- a/program/ftot.py +++ b/program/ftot.py @@ -12,6 +12,7 @@ import ftot_supporting import traceback import datetime +import numpy as np import pint from pint import UnitRegistry @@ -20,13 +21,48 @@ Q_ = ureg.Quantity ureg.define('usd = [currency]') # add the US Dollar, "USD" to the unit registry # solves issue in pint 0.9 -if float(pint.__version__) < 1: +if pint.__version__ == 0.9: ureg.define('short_hundredweight = short_hunderdweight') ureg.define('long_hundredweight = long_hunderdweight') ureg.define('us_ton = US_ton') -VERSION_NUMBER = "5.0.6" -VERSION_DATE = "09/30/2020" +VERSION_NUMBER = "5.0.9" +VERSION_DATE = "3/30/2021" +#=============load parameters from newly developed Python files====================== +# Load earthquake scenario matrix +earthquake_scenario = np.load("earthquake_events.npy") +total_repair_time = np.load("total_repair_time.npy") + +total_fuel_amount = 2941633.8 +# planning horizon in this example: 20 years +plan_horizon = 20 # unit: year +# total scenario amount: here we consider 50 scenarios +N = 30 +# resilience array +Resilience = np.zeros((N)) +R1 = np.zeros((N)) +R2 = np.zeros((N)) +R3 = np.zeros((N)) +weight1 = np.zeros((N)) +weight2 = np.zeros((N)) +weight3 = np.zeros((N)) + +# simulation weeks/days in one year +WeekInYear = 52 +DayInYear = 52*7 +# Unmet demand threshold +UD_level = 4190600 +# Total final demand +total_demand_fuel = 136497 # total demand for this supply chain layout +total_final_demand = 4073312.9 + 254401.1 # total demand before optimization +# Variables for output from optimization +#UnmetDemandAmount = np.zeros((int(WeekInYear*7),plan_horizon, N)) +#DailyCost = np.zeros((int(WeekInYear*7),plan_horizon, N)) +#np.save("UnmetDemandAmount.npy", UnmetDemandAmount) +#np.save("DailyCost.npy", DailyCost) +reward_rate = 5000 +# =================================================================================================== + # =================================================================================================== @@ -54,7 +90,8 @@ f = facilities; process GIS feature classes and commodity input CSV files c = connectivity; connects facilities to the network using artificial links and exports geodatabase FCs as shapefiles - g = graph; reads in shapefiles using networkX and prepares, cleans, and stores the network digraph in the database + g = graph; reads in shapefiles using networkX and prepares, cleans, and stores the network digraph in the + database # optimization options # -------------------- @@ -87,9 +124,11 @@ m = create map documents with simple basemap mb = optionally create map documents with a light gray basemap mc = optionally create map documents with a topographic basemap + md = optionally create map documents with a streets basemap m2 = time and commodity mapping with simple basemap m2b = optionally create time and commodity mapping with a light gray basemap m2c = optionally create time and commodity mapping with a topographic basemap + m2d = optionally create time and commodity mapping with a streets basemap # utilities, tools, and advanced options # --------------------------------------- @@ -104,7 +143,7 @@ parser.add_argument("task", choices=("s", "f", "f2", "c", "c2", "g", "g2", "o", "oc", "o1", "o2", "o2b", "oc1", "oc2", "oc2b", "oc3", "os", "p", - "d", "m", "mb", "mc", "m2", "m2b", "m2c", + "d", "m", "mb", "mc", "md", "m2", "m2b", "m2c", "m2d" "test" ), type=str) parser.add_argument('-skip_arcpy_check', action='store_true', @@ -123,7 +162,7 @@ if os.path.exists(args.config_file): ftot_program_directory = os.path.dirname(os.path.realpath(__file__)) else: - print "{} doesn't exist".format(args.config_file) + print ("{} doesn't exist".format(args.config_file)) sys.exit() # set up logging and report run start time @@ -165,14 +204,13 @@ if not args.skip_arcpy_check: try: import arcpy - arcmap_version = arcpy.GetInstallInfo()['Version'] - if not arcmap_version in ['10.1', '10.2', '10.2.1', '10.2.2', '10.3', '10.3.1', '10.4', '10.4.1', - '10.5', '10.5.1', '10.6', '10.6.1', '10.7', '10.7.1', '10.8', '10.8.1']: - logger.error("Version {} of ArcGIS is not currently supported. Exiting.".format(arcmap_version)) + arcgis_pro_version = arcpy.GetInstallInfo()['Version'] + if float(arcgis_pro_version[0:3]) < 2.6: + logger.error("Version {} of ArcGIS Pro is not supported. Exiting.".format(arcgis_pro_version)) sys.exit() except RuntimeError: - logger.error("You will need ArcGIS 10.1 or later to run this script. If you do have ArcGIS installed, " + logger.error("ArcGIS Pro 2.6 or later is required to run this script. If you do have ArcGIS Pro installed, " "confirm that it is properly licensed and/or that the license manager is accessible. Exiting.") sys.exit() @@ -212,10 +250,132 @@ graph(the_scenario, logger) # optimization - elif args.task in ['o']: - from ftot_pulp import o1, o2 - o1(the_scenario, logger) - o2(the_scenario, logger) + elif args.task in ['o','o1','o2']: + #=================================================Modification================================================================ + # Purpose of following part: (a)iterate optimization per week/year; (b)divide yearly and weekly simulation, based on earthquake occurrence; + # (c)incorporate resilience calculation for each scenario; (d)save related outputs. + + # for each scenario and time: + for i in range(N): + UnmetDemandAmount = np.load("UnmetDemandAmount.npy") + DailyCost = np.load("DailyCost.npy") + cost1 = 0 + cost2 = 0 + cost3 = 0 + w1 = np.zeros((plan_horizon)) + w2 = np.zeros((plan_horizon)) + w3 = np.zeros((plan_horizon)) + scenario_num = i + np.save("scenario_num.npy", scenario_num) + for t in range(plan_horizon): + time_horizon = t + np.save("time_horizon.npy", time_horizon) + # for earthquake occurrence scenario: weekly basis interval + if earthquake_scenario[i][t] != 0: + temp_repair_week = float(total_repair_time[i][t])/float(7) # one week = 7 days + for j in range(WeekInYear): + earthquake_week = j + #temp_day = int(7*j) + np.save("earthquake_week.npy", earthquake_week) + + + from ftot_pulp_weekly import o1, o2 + o1(the_scenario, logger) + o2(the_scenario, logger) + + + for day_1 in range (int(j*7),int((j+1)*7)): + UnmetDemandAmount[day_1][t][i] = np.load("unmet_demand_daily.npy") + DailyCost[day_1][t][i] = np.load("costs_daily.npy") + + np.save("UnmetDemandAmount.npy", UnmetDemandAmount) + np.save("DailyCost.npy", DailyCost) + + + # Calculate initial daily costs for each scenario + temp_day = int(7*j) + daily_cost = DailyCost[temp_day][t][i] - UnmetDemandAmount[0][0][0] * 5000 + initial_cost = DailyCost[0][0][0] - UnmetDemandAmount[0][0][0] * 5000 + UD = UnmetDemandAmount[temp_day][t][i] - float(UD_level)/float(DayInYear) + UDR = float(UD*DayInYear)/float(total_demand_fuel) + production = float(total_final_demand)/float(DayInYear) - UnmetDemandAmount[temp_day][t][i] + + + if j <= temp_repair_week: + UDP = abs(UD * 5000) + R2[i] = R2[i] + (float(daily_cost-initial_cost + UDP)/float(production))*7 + w2[t] = w2[t] + (float(daily_cost-initial_cost + UDP)/float(production)) + + else: + if UDR > 0: + UDP = UD * 5000 + R1[i] = R1[i] + (float(daily_cost-initial_cost + UDP)/float(production))*7 + w1[t] = w1[t] + (float(daily_cost-initial_cost + UDP)/float(production)) + + else: + R3[i] = R3[i] + (float(UD*reward_rate)/float(production))*7 + w3[t] = w3[t] + (float(UD*reward_rate)/float(production)) + + # weight factor for R2 + weight2[i] = weight2[i] + float(sum(w2[:]))/float(temp_repair_week + 0.01) + + + # for non earthquake occurrence scenario: yearly basis interval + else: + from ftot_pulp_yearly import o1, o2 + o1(the_scenario, logger) + o2(the_scenario, logger) + + temp_demand = np.load("unmet_demand_yearly.npy") + UnmetDemandAmount[:,t,i] = float(temp_demand)/float(DayInYear) + temp_dailycost= np.load("costs_yearly.npy") + DailyCost[:,t,i] = float(temp_dailycost)/float(DayInYear) + np.save("UnmetDemandAmount.npy", UnmetDemandAmount) + np.save("DailyCost.npy", DailyCost) + + # Calculate initial daily costs for each scenario + temp_day = 2 # either day is okay, because daily cost is the same for every day + daily_cost = DailyCost[temp_day][t][i] - UnmetDemandAmount[0][0][0] * 5000 + initial_cost = DailyCost[0][0][0]- UnmetDemandAmount[0][0][0] * 5000 + UD = UnmetDemandAmount[temp_day][t][i] - float(UD_level)/float(DayInYear) + UDR = float(UD*DayInYear)/float(total_demand_fuel) + production = float(total_final_demand)/float(DayInYear) - UnmetDemandAmount[temp_day][t][i] + + if UDR > 0: + UDP = UD * 5000 + R1[i] = R1[i] + (float(daily_cost-initial_cost + UDP)/float(production))*DayInYear + w1[t] = w1[t] + (float(daily_cost-initial_cost + UDP)/float(production)) + else: + R3[i] = R3[i] + (float(UD*reward_rate)/float(production))*DayInYear + w3[t] = w3[t] + (float(UD*reward_rate)/float(production)) + + # weight factor for each resilience componenet equals to average daily cost + weight1[i] = float(sum(w1[:]))/float(np.count_nonzero(w1) + 0.01) + weight2[i] = float(weight2[i])/float(np.count_nonzero(earthquake_scenario[i,:])+0.01) + weight3[i] = float(sum(w3[:]))/float(np.count_nonzero(w3)+0.01) + # assume the same weight for resilience cost and initial cost + weight_factor = 1 + + Resilience[i] = weight_factor*(-abs(weight1[i]*R1[i])-abs(weight2[i]*R2[i]) + abs(weight3[i]*R3[i])) + initial_cost*DayInYear*plan_horizon + + #Resilience[i] = weight_factor*(weight1[i]*R1[i] + weight2[i]*R2[i] + weight3[i]*R3[i])+initial_cost*year*plan_horizon + + + logger.info("Weight for hazard-induced CLF for scenario {}: {}".format(i,weight2[i])) + logger.info("Weight for non-hazard-induced CLF for scenario {}: {}".format(i,weight1[i])) + logger.info("Weight for opportunity-induced CGF for scenario {}: {}".format(i,weight3[i])) + logger.info("Resilience for scenario {} : {}".format(i,Resilience[i])) + + + np.save("R1.npy",R1) + np.save("R2.npy",R2) + np.save("R3.npy",R3) + np.save("weight1.npy",weight1) + np.save("weight2.npy",weight2) + np.save("weight3.npy",weight3) + np.save("Resilience.npy",Resilience) + + # candidate optimization elif args.task in ['oc']: @@ -224,20 +384,11 @@ oc2(the_scenario, logger) oc3(the_scenario, logger) - # optimization setup - elif args.task in ['o1']: - from ftot_pulp import o1 - o1(the_scenario, logger) - - # optimization solve - elif args.task in ['o2']: - from ftot_pulp import o2 - o2(the_scenario, logger) # optional step to solve and save pulp problem from pickle elif args.task == 'o2b': from ftot_pulp import o2b - o2b(the_scenario,logger) + o2b(the_scenario, logger) # optimization option - processor candidates generation elif args.task == 'oc1': @@ -274,17 +425,19 @@ generate_reports(the_scenario, logger) # currently m step has three basemap alternatives-- see key above - elif args.task in ["m", "mb", "mc"]: + elif args.task in ["m", "mb", "mc", "md"]: from ftot_maps import new_map_creation new_map_creation(the_scenario, logger, args.task) # currently m2 step has three basemap alternatives-- see key above - elif args.task in ["m2", "m2b", "m2c"]: + elif args.task in ["m2", "m2b", "m2c", "m2d"]: from ftot_maps import prepare_time_commodity_subsets_for_mapping prepare_time_commodity_subsets_for_mapping(the_scenario, logger, args.task) elif args.task == "test": logger.info("in the test case") + import pdb + pdb.set_trace() except: diff --git a/program/ftot_facilities.py b/program/ftot_facilities.py index cd68a7b..e412685 100644 --- a/program/ftot_facilities.py +++ b/program/ftot_facilities.py @@ -16,6 +16,7 @@ import os import sqlite3 from ftot import ureg, Q_ +from six import iteritems LCC_PROJ = arcpy.SpatialReference('USA Contiguous Lambert Conformal Conic') # =============================================================================== @@ -50,7 +51,7 @@ def db_cleanup_tables(the_scenario, logger): # DB CLEAN UP # ------------ - logger.info("start: db_cleanup_tables" ) + logger.info("start: db_cleanup_tables") # a new run is a new scenario in the main.db # so drop and create the following tables if they exists @@ -78,7 +79,7 @@ def db_cleanup_tables(the_scenario, logger): main_db_con.execute("drop table if exists facilities;") logger.debug("create the facilities table") main_db_con.executescript( - "create table facilities(facility_ID INTEGER PRIMARY KEY, location_id integer, facility_name text, facility_type_id integer, ignore_facility text, candidate binary, schedule_id integer);") + "create table facilities(facility_ID INTEGER PRIMARY KEY, location_id integer, facility_name text, facility_type_id integer, ignore_facility text, candidate binary, schedule_id integer, max_capacity float);") # facility_type_id table logger.debug("drop the facility_type_id table") @@ -109,8 +110,8 @@ def db_cleanup_tables(the_scenario, logger): """create table commodities(commodity_ID INTEGER PRIMARY KEY, commodity_name text, supertype text, subtype text, units text, phase_of_matter text, max_transport_distance numeric, proportion_of_supertype numeric, share_max_transport_distance text, CONSTRAINT unique_name UNIQUE(commodity_name) );""") - # proportion_of_supertype specifies how much demand is satisfied by this subtype relative to the "pure" fuel/commodity - # this will depend on the process + # proportion_of_supertype specifies how much demand is satisfied by this subtype relative to the "pure" + # fuel/commodity. this will depend on the process # schedule_names table logger.debug("drop the schedule names table") @@ -126,6 +127,13 @@ def db_cleanup_tables(the_scenario, logger): main_db_con.executescript( """create table schedules(schedule_id integer, day integer, availability numeric);""") + # coprocessing reference table + logger.debug("drop the coprocessing table") + main_db_con.execute("drop table if exists coprocessing;") + logger.debug("create the coprocessing table") + main_db_con.executescript( + """create table coprocessing(coproc_id integer, label text, description text);""") + logger.debug("finished: main.db cleanup") # =============================================================================== @@ -140,6 +148,9 @@ def db_populate_tables(the_scenario, logger): # populate locations table populate_locations_table(the_scenario, logger) + # populate coprocessing table + populate_coprocessing_table(the_scenario, logger) + # populate the facilities, commodities, and facility_commodities table # with the input CSVs. # Note: processor_candidate_commodity_data is generated for FTOT generated candidate @@ -150,10 +161,9 @@ def db_populate_tables(the_scenario, logger): the_scenario.processors_commodity_data, the_scenario.processor_candidates_commodity_data]: # this should just catch processors not specified. - if str(commodity_input_file).lower() == "null" or \ - str(commodity_input_file).lower() == "none": - logger.debug("Commodity Input Data specified in the XML: {}".format(commodity_input_file)) - continue + if str(commodity_input_file).lower() == "null" or str(commodity_input_file).lower() == "none": + logger.debug("Commodity Input Data specified in the XML: {}".format(commodity_input_file)) + continue else: populate_facility_commodities_table(the_scenario, commodity_input_file, logger) @@ -246,12 +256,12 @@ def db_report_commodity_potentials(the_scenario, logger): logger.result("---------------|---------------|----|---------------|----------") for row in db_data: logger.result("{:15.15} {:15.15} {:4.1} {:15,.1f} {:15.10}".format(row[0], row[1], row[2], row[3], - row[4])) + row[4])) logger.result("-------------------------------------------------------------------") - # =================================================================================================== + def load_schedules_input_data(schedule_input_file, logger): logger.debug("start: load_schedules_input_data") @@ -266,7 +276,7 @@ def load_schedules_input_data(schedule_input_file, logger): # read through facility_commodities input CSV import csv - with open(schedule_input_file, 'rb') as f: + with open(schedule_input_file, 'rt') as f: reader = csv.DictReader(f) # adding row index for issue #220 to alert user on which row their error is in @@ -276,20 +286,20 @@ def load_schedules_input_data(schedule_input_file, logger): day = int(row['day']) # cast day to an int availability = float(row['availability']) # cast availability to float - if schedule_name in schedules.keys(): + if schedule_name in list(schedules.keys()): schedules[schedule_name][day] = availability else: schedules[schedule_name] = {day: availability} # initialize sub-dict - # Enforce default schedule req. and default availaility req. for all schedules. + # Enforce default schedule req. and default availability req. for all schedules. # if user has not defined 'default' schedule if 'default' not in schedules: logger.debug("Default schedule not found. Adding 'default' with default availability of 1.") schedules['default'] = {0: 1} # if schedule does not have a default value (value assigned to day 0), then add as 1. - for schedule_name in schedules.keys(): - if 0 not in schedules[schedule_name].keys(): - logger.debug("Schedule {} missing default value. Adding default availaility of 1.".format(schedule_name)) + for schedule_name in list(schedules.keys()): + if 0 not in list(schedules[schedule_name].keys()): + logger.debug("Schedule {} missing default value. Adding default availability of 1.".format(schedule_name)) schedules[schedule_name][0] = 1 return schedules @@ -307,7 +317,7 @@ def populate_schedules_table(the_scenario, logger): with sqlite3.connect(the_scenario.main_db) as db_con: id_num = 0 - for schedule_name, schedule_data in schedules_dict.iteritems(): + for schedule_name, schedule_data in iteritems(schedules_dict): id_num += 1 # 1-index # add schedule name into schedule_names table @@ -317,15 +327,17 @@ def populate_schedules_table(the_scenario, logger): db_con.execute(sql) # add each day into schedules table - for day, availability in schedule_data.iteritems(): + for day, availability in iteritems(schedule_data): sql = "insert into schedules " \ "(schedule_id, day, availability) " \ "values ({},{},{});".format(id_num, day, availability) db_con.execute(sql) logger.debug("finished: populate_locations_table") + # ============================================================================== + def check_for_input_error(input_type, input_val, filename, index, units=None): """ :param input_type: a string with the type of input (e.g. 'io', 'facility_name', etc. @@ -395,6 +407,7 @@ def check_for_input_error(input_type, input_val, filename, index, units=None): return error_message +# ============================================================================== def load_facility_commodities_input_data(the_scenario, commodity_input_file, logger): @@ -411,24 +424,24 @@ def load_facility_commodities_input_data(the_scenario, commodity_input_file, log # read through facility_commodities input CSV import csv - with open(commodity_input_file, 'rb') as f: + with open(commodity_input_file, 'rt') as f: reader = csv.DictReader(f) # adding row index for issue #220 to alert user on which row their error is in for index, row in enumerate(reader): # re: issue #149 -- if the line is empty, just skip it - if row.values()[0] == '': + if list(row.values())[0] == '': logger.debug('the CSV file has a blank in the first column. Skipping this line: {}'.format( - row.values())) + list(row.values()))) continue - # {'units': 'kgal', 'facility_name': 'd:01053', 'phase_of_matter': 'liquid', 'value': '9181.521484', 'commodity': 'diesel', 'io': 'o', - # 'share_max_transport_distance'; 'Y'} + # {'units': 'kgal', 'facility_name': 'd:01053', 'phase_of_matter': 'liquid', 'value': '9181.521484', + # 'commodity': 'diesel', 'io': 'o', 'share_max_transport_distance'; 'Y'} io = row["io"] facility_name = str(row["facility_name"]) facility_type = row["facility_type"] commodity_name = row["commodity"].lower() # re: issue #131 - make all commodities lower case commodity_quantity = row["value"] - commodity_unit = str(row["units"]).replace(' ', '_').lower() # remove spaces and make units lower case + commodity_unit = str(row["units"]).replace(' ', '_').lower() # remove spaces and make units lower case commodity_phase = row["phase_of_matter"] # check for proc_cand-specific "non-commodities" to ignore validation (issue #254) @@ -458,18 +471,23 @@ def load_facility_commodities_input_data(the_scenario, commodity_input_file, log logger.debug("Skipping input validation on special candidate processor commodity: {}" .format(commodity_name)) - if "max_transport_distance" in row.keys(): + if "max_processor_input" in list(row.keys()): + max_processor_input = row["max_processor_input"] + else: + max_processor_input = "Null" + + if "max_transport_distance" in list(row.keys()): commodity_max_transport_distance = row["max_transport_distance"] else: commodity_max_transport_distance = "Null" - if "share_max_transport_distance" in row.keys(): + if "share_max_transport_distance" in list(row.keys()): share_max_transport_distance = row["share_max_transport_distance"] else: share_max_transport_distance = 'N' # add schedule_id, if available - if "schedule" in row.keys(): + if "schedule" in list(row.keys()): schedule_name = str(row["schedule"]).lower() # blank schedule name should be cast to default @@ -500,19 +518,21 @@ def load_facility_commodities_input_data(the_scenario, commodity_input_file, log commodity_quantity = commodity_quantity_and_units.to(commodity_unit).magnitude # add to the dictionary of facility_commodities mapping - if not facility_name in temp_facility_commodities_dict.keys(): + if facility_name not in list(temp_facility_commodities_dict.keys()): temp_facility_commodities_dict[facility_name] = [] temp_facility_commodities_dict[facility_name].append([facility_type, commodity_name, commodity_quantity, commodity_unit, commodity_phase, commodity_max_transport_distance, io, - share_max_transport_distance, schedule_name]) + share_max_transport_distance, max_processor_input, + schedule_name]) logger.debug("finished: load_facility_commodities_input_data") return temp_facility_commodities_dict # ============================================================================== + def populate_facility_commodities_table(the_scenario, commodity_input_file, logger): logger.debug("start: populate_facility_commodities_table {}".format(commodity_input_file)) @@ -530,7 +550,7 @@ def populate_facility_commodities_table(the_scenario, commodity_input_file, logg # connect to main.db and add values to table # --------------------------------------------------------- with sqlite3.connect(the_scenario.main_db) as db_con: - for facility_name, facility_data in facility_commodities_dict.iteritems(): + for facility_name, facility_data in iteritems(facility_commodities_dict): # unpack the facility_type (should be the same for all entries) facility_type = facility_data[0][0] @@ -542,9 +562,11 @@ def populate_facility_commodities_table(the_scenario, commodity_input_file, logg schedule_name = facility_data[0][-1] schedule_id = get_schedule_id(the_scenario, db_con, schedule_name, logger) + max_processor_input = facility_data[0][-2] + # get the facility_id from the db (add the facility if it doesn't exists) # and set up entry in facility_id table - facility_id = get_facility_id(the_scenario, db_con, location_id, facility_name, facility_type_id, candidate, schedule_id, logger) + facility_id = get_facility_id(the_scenario, db_con, location_id, facility_name, facility_type_id, candidate, schedule_id, max_processor_input, logger) # iterate through each commodity for commodity_data in facility_data: @@ -552,7 +574,7 @@ def populate_facility_commodities_table(the_scenario, commodity_input_file, logg # get commodity_id. (adds commodity if it doesn't exist) commodity_id = get_commodity_id(the_scenario, db_con, commodity_data, logger) - [facility_type, commodity_name, commodity_quantity, commodity_units, commodity_phase, commodity_max_transport_distance, io, share_max_transport_distance, schedule_id] = commodity_data + [facility_type, commodity_name, commodity_quantity, commodity_units, commodity_phase, commodity_max_transport_distance, io, share_max_transport_distance, unused_var_max_processor_input, schedule_id] = commodity_data if not commodity_quantity == "0.0": # skip anything with no material sql = "insert into facility_commodities " \ @@ -605,6 +627,36 @@ def db_check_multiple_input_commodities_for_processor(the_scenario, logger): # ============================================================================== +def populate_coprocessing_table(the_scenario, logger): + + logger.info("start: populate_coprocessing_table") + + # connect to db + with sqlite3.connect(the_scenario.main_db) as db_con: + + # should be filled from the file coprocessing.csv, which needs to be added to xml still + # I would place this file in the common data folder probably, since it is a fixed reference table + # for now, filling the db table here manually with the data from the csv file + sql = """INSERT INTO coprocessing (coproc_id, label, description) + VALUES + (1, 'single', 'code should throw error if processor has more than one input commodity'), + (2, 'fixed combination', 'every input commodity listed for the processor is required, in the ratio ' || + 'specified by their quantities. Output requires all inputs to be present; '), + (3, 'substitutes allowed', 'any one of the input commodities listed for the processor can be used ' || + 'to generate the output, with the ratios specified by quantities. A ' || + 'combination of inputs is allowed. '), + (4, 'external input', 'assumes all specified inputs are required, in that ratio, with the addition ' || + 'of an External or Non-Transported input commodity. This is included in the ' || + 'ratio and as part of the input capacity, but is available in unlimited ' || + 'quantity at the processor location. ') + ; """ + db_con.execute(sql) + + logger.debug("not yet implemented: populate_coprocessing_table") + +# ============================================================================= + + def populate_locations_table(the_scenario, logger): logger.info("start: populate_locations_table") @@ -688,12 +740,12 @@ def get_facility_location_id(the_scenario, db_con, facility_name, logger): # ============================================================================= -def get_facility_id(the_scenario, db_con, location_id, facility_name, facility_type_id, candidate, schedule_id, logger): +def get_facility_id(the_scenario, db_con, location_id, facility_name, facility_type_id, candidate, schedule_id, max_processor_input, logger): # if it doesn't exist, add to facilities table and generate a facility id. db_con.execute("insert or ignore into facilities " - "(location_id, facility_name, facility_type_id, candidate, schedule_id) " - "values ('{}', '{}', {}, {}, {});".format(location_id, facility_name, facility_type_id, candidate, schedule_id)) + "(location_id, facility_name, facility_type_id, candidate, schedule_id, max_capacity) " + "values ('{}', '{}', {}, {}, {}, {});".format(location_id, facility_name, facility_type_id, candidate, schedule_id, max_processor_input)) # get facility_id db_cur = db_con.execute("select facility_id " @@ -733,16 +785,15 @@ def get_facility_id_type(the_scenario, db_con, facility_type, logger): else: return facility_type_id - # =================================================================================================== def get_commodity_id(the_scenario, db_con, commodity_data, logger): [facility_type, commodity_name, commodity_quantity, commodity_unit, commodity_phase, - commodity_max_transport_distance, io, share_max_transport_distance, schedule_id] = commodity_data + commodity_max_transport_distance, io, share_max_transport_distance, max_processor_input, schedule_id] = commodity_data - # get the commodiy_id. + # get the commodity_id. db_cur = db_con.execute("select commodity_id " "from commodities c " "where c.commodity_name = '{}';".format(commodity_name)) @@ -752,18 +803,19 @@ def get_commodity_id(the_scenario, db_con, commodity_data, logger): if not commodity_id: # if it doesn't exist, add the commodity to the commodities table and generate a commodity id if commodity_max_transport_distance in ['Null', '', 'None']: - sql = "insert into commodities " \ - "(commodity_name, units, phase_of_matter, share_max_transport_distance) " \ - "values ('{}', '{}', '{}','{}');".format(commodity_name, commodity_unit, commodity_phase, share_max_transport_distance) + sql = "insert into commodities " \ + "(commodity_name, units, phase_of_matter, share_max_transport_distance) " \ + "values ('{}', '{}', '{}','{}');".format(commodity_name, commodity_unit, commodity_phase, + share_max_transport_distance) else: - sql = "insert into commodities " \ - "(commodity_name, units, phase_of_matter, max_transport_distance,share_max_transport_distance) " \ - "values ('{}', '{}', '{}', {}, '{}');".format(commodity_name, commodity_unit, commodity_phase, - commodity_max_transport_distance, - share_max_transport_distance) + sql = "insert into commodities " \ + "(commodity_name, units, phase_of_matter, max_transport_distance,share_max_transport_distance) " \ + "values ('{}', '{}', '{}', {}, '{}');".format(commodity_name, commodity_unit, commodity_phase, + commodity_max_transport_distance, + share_max_transport_distance) db_con.execute(sql) - # get the commodiy_id. + # get the commodity_id. db_cur = db_con.execute("select commodity_id " "from commodities c " "where c.commodity_name = '{}';".format(commodity_name)) @@ -777,7 +829,6 @@ def get_commodity_id(the_scenario, db_con, commodity_data, logger): else: return commodity_id - # =================================================================================================== @@ -796,9 +847,9 @@ def get_schedule_id(the_scenario, db_con, schedule_name, logger): return schedule_id[0] - # =================================================================================================== + def gis_clean_fc(the_scenario, logger): logger.info("start: gis_clean_fc") @@ -817,7 +868,8 @@ def gis_clean_fc(the_scenario, logger): # clear the processors gis_clear_feature_class(the_scenario.locations_fc, logger) - logger.debug("finished: gis_clean_fc: Runtime (HMS): \t{}".format(ftot_supporting.get_total_runtime_string(start_time))) + logger.debug("finished: gis_clean_fc: Runtime (HMS): \t{}".format + (ftot_supporting.get_total_runtime_string(start_time))) # ============================================================================== @@ -855,7 +907,8 @@ def gis_populate_fc(the_scenario, logger): # populate the processors fc in main.gdb gis_processors_setup_fc(the_scenario, logger) - logger.debug("finished: gis_populate_fc: Runtime (HMS): \t{}".format(ftot_supporting.get_total_runtime_string(start_time))) + logger.debug("finished: gis_populate_fc: Runtime (HMS): \t{}".format + (ftot_supporting.get_total_runtime_string(start_time))) # ------------------------------------------------------------ @@ -883,13 +936,13 @@ def gis_ultimate_destinations_setup_fc(the_scenario, logger): # read through facility_commodities input CSV import csv - with open(the_scenario.destinations_commodity_data, 'rb') as f: + with open(the_scenario.destinations_commodity_data, 'rt') as f: reader = csv.DictReader(f) for row in reader: facility_name = str(row["facility_name"]) - commodity_quantity = row["value"] + commodity_quantity = float(row["value"]) - if not facility_name in temp_facility_commodities_dict.keys(): + if facility_name not in list(temp_facility_commodities_dict.keys()): if commodity_quantity > 0: temp_facility_commodities_dict[facility_name] = True @@ -899,7 +952,7 @@ def gis_ultimate_destinations_setup_fc(the_scenario, logger): pass else: cursor.deleteRow() - counter +=1 + counter += 1 del cursor logger.config("Number of Destinations removed due to lack of commodity data: \t{}".format(counter)) @@ -919,7 +972,8 @@ def gis_ultimate_destinations_setup_fc(the_scenario, logger): result = gis_get_feature_count(destinations_fc, logger) logger.config("Number of Destinations: \t{}".format(result)) - logger.debug("finish: gis_ultimate_destinations_setup_fc: Runtime (HMS): \t{}".format(ftot_supporting.get_total_runtime_string(start_time))) + logger.debug("finish: gis_ultimate_destinations_setup_fc: Runtime (HMS): \t{}".format + (ftot_supporting.get_total_runtime_string(start_time))) # ============================================================================= @@ -946,14 +1000,14 @@ def gis_rmp_setup_fc(the_scenario, logger): # read through facility_commodities input CSV import csv - with open(the_scenario.rmp_commodity_data, 'rb') as f: + with open(the_scenario.rmp_commodity_data, 'rt') as f: reader = csv.DictReader(f) for row in reader: facility_name = str(row["facility_name"]) - commodity_quantity = row["value"] + commodity_quantity = float(row["value"]) - if not facility_name in temp_facility_commodities_dict.keys(): + if facility_name not in list(temp_facility_commodities_dict.keys()): if commodity_quantity > 0: temp_facility_commodities_dict[facility_name] = True @@ -1005,9 +1059,8 @@ def gis_processors_setup_fc(the_scenario, logger): arcpy.Delete_management(processors_fc) logger.debug("deleted existing {} layer".format(processors_fc)) - arcpy.CreateFeatureclass_management(the_scenario.main_gdb, "processors", \ - "POINT", "#", "DISABLED", "DISABLED", ftot_supporting_gis.LCC_PROJ, "#", - "0", "0", "0") + arcpy.CreateFeatureclass_management(the_scenario.main_gdb, "processors", "POINT", "#", "DISABLED", "DISABLED", + ftot_supporting_gis.LCC_PROJ, "#", "0", "0", "0") arcpy.AddField_management(processors_fc, "Facility_Name", "TEXT", "#", "#", "25", "#", "NULLABLE", "NON_REQUIRED", "#") @@ -1033,14 +1086,14 @@ def gis_processors_setup_fc(the_scenario, logger): # read through facility_commodities input CSV import csv - with open(the_scenario.processors_commodity_data, 'rb') as f: + with open(the_scenario.processors_commodity_data, 'rt') as f: reader = csv.DictReader(f) for row in reader: facility_name = str(row["facility_name"]) - commodity_quantity = row["value"] + commodity_quantity = float(row["value"]) - if facility_name not in temp_facility_commodities_dict.keys(): + if facility_name not in list(temp_facility_commodities_dict.keys()): if commodity_quantity > 0: temp_facility_commodities_dict[facility_name] = True @@ -1055,7 +1108,6 @@ def gis_processors_setup_fc(the_scenario, logger): del cursor logger.config("Number of processors removed due to lack of commodity data: \t{}".format(counter)) - with arcpy.da.SearchCursor(processors_fc, ['Facility_Name', 'SHAPE@X', 'SHAPE@Y']) as scursor: for row in scursor: # Check if coordinates of facility are roughly within North America @@ -1071,7 +1123,6 @@ def gis_processors_setup_fc(the_scenario, logger): del scursor - # check for candidates or other processors specified in either XML or layers_to_merge = [] @@ -1121,4 +1172,4 @@ def gis_merge_processor_fc(the_scenario, layers_to_merge, logger): logger.debug("Total count: {} records in {}".format(total_processor_count, os.path.split(processors_fc)[0])) return -# ---------------------------------------------------------------------------- \ No newline at end of file +# ---------------------------------------------------------------------------- diff --git a/program/ftot_maps.py b/program/ftot_maps.py index af7cbb6..48b5560 100644 --- a/program/ftot_maps.py +++ b/program/ftot_maps.py @@ -8,6 +8,7 @@ import imageio import sqlite3 import datetime +from six import iteritems # =================================================================================================== @@ -20,61 +21,79 @@ def new_map_creation(the_scenario, logger, task): if task == "m": basemap = "default_basemap" - if task == "mb": + elif task == "mb": basemap = "gray_basemap" - if task == "mc": + elif task == "mc": basemap = "topo_basemap" + elif task == "md": + basemap = "street_basemap" + else: + basemap = "default_basemap" the_scenario.mapping_directory = os.path.join(the_scenario.scenario_run_directory, "Maps", basemap + "_" + timestamp_folder_name) - scenario_mxd_location = os.path.join(the_scenario.scenario_run_directory, "Maps", "ftot_maps.mxd") + scenario_gdb = the_scenario.main_gdb + + scenario_aprx_location = os.path.join(the_scenario.scenario_run_directory, "Maps", "ftot_maps.aprx") if not os.path.exists(the_scenario.mapping_directory): logger.debug("creating maps directory.") os.makedirs(the_scenario.mapping_directory) - # Must delete existing scenario mxd because there can only be one per scenario (everything in the mxd references + # Must delete existing scenario aprx because there can only be one per scenario (everything in the aprx references # the existing scenario main.gdb) - if os.path.exists(scenario_mxd_location): - os.remove(scenario_mxd_location) + if os.path.exists(scenario_aprx_location): + os.remove(scenario_aprx_location) - # set mxd locations: root is in the common data folder, and the scenario is in the local maps folder - root_ftot_mxd_location = os.path.join(the_scenario.common_data_folder, "ftot_maps.mxd") + # set aprx locations: root is in the common data folder, and the scenario is in the local maps folder + root_ftot_aprx_location = os.path.join(the_scenario.common_data_folder, "ftot_maps.aprx") base_layers_location = os.path.join(the_scenario.common_data_folder, "Base_Layers") - # copy the mxd from the common data director to the scenario maps directory - logger.debug("copying the root mxd from common data to the scenario maps directory.") - copy(root_ftot_mxd_location, scenario_mxd_location) + # copy the aprx from the common data director to the scenario maps directory + logger.debug("copying the root aprx from common data to the scenario maps directory.") + copy(root_ftot_aprx_location, scenario_aprx_location) # load the map document into memory - mxd = arcpy.mapping.MapDocument(scenario_mxd_location) + aprx = arcpy.mp.ArcGISProject(scenario_aprx_location) + + # Fix the non-base layers here + broken_list = aprx.listBrokenDataSources() + for broken_item in broken_list: + if broken_item.supports("DATASOURCE"): + if not broken_item.longName.find("Base") == 0: + conprop = broken_item.connectionProperties + conprop['connection_info']['database'] = scenario_gdb + broken_item.updateConnectionProperties(broken_item.connectionProperties, conprop) - list_broken_data_sources(mxd, base_layers_location, logger) + list_broken_data_sources(aprx, base_layers_location, logger) - reset_map_base_layers(mxd, logger, basemap) + reset_map_base_layers(aprx, logger, basemap) - export_map_steps(mxd, the_scenario, logger, basemap) + export_map_steps(aprx, the_scenario, logger, basemap) logger.info("maps located here: {}".format(the_scenario.mapping_directory)) # =================================================================================================== -def list_broken_data_sources(mxd, base_layers_location, logger): - brkn_list = arcpy.mapping.ListBrokenDataSources(mxd) - for brkn_item in brkn_list: - if brkn_item.supports("DATASOURCE"): - if brkn_item.longName.find("Base") == 0: - brkn_item.replaceDataSource(base_layers_location) - logger.debug("\t" + "broken base layer path fixed: " + brkn_item.name) +def list_broken_data_sources(aprx, base_layers_location, logger): + broken_list = aprx.listBrokenDataSources() + for broken_item in broken_list: + if broken_item.supports("DATASOURCE"): + if broken_item.longName.find("Base") == 0: + conprop = broken_item.connectionProperties + conprop['connection_info']['database'] = base_layers_location + broken_item.updateConnectionProperties(broken_item.connectionProperties, conprop) + logger.debug("\t" + "broken base layer path fixed: " + broken_item.name) else: - logger.debug("\t" + "broken mxd data source path: " + brkn_item.name) + logger.debug("\t" + "broken aprx data source path: " + broken_item.name) # =================================================================================================== -def reset_map_base_layers(mxd, logger, basemap): +def reset_map_base_layers(aprx, logger, basemap): logger.debug("start: reset_map_base_layers") # turn on base layers, turn off everything else. - for lyr in arcpy.mapping.ListLayers(mxd): + map = aprx.listMaps("ftot_map")[0] + for lyr in map.listLayers(): if lyr.longName.find("Base") == 0: lyr.visible = True @@ -88,6 +107,10 @@ def reset_map_base_layers(mxd, logger, basemap): if basemap not in ["topo_basemap"] and "USGSTopo" in lyr.longName and not lyr.longName.endswith("Layers"): lyr.visible = False + # This group layer should be off unless md step is being run + if basemap not in ["street_basemap"] and "World Street Map" in lyr.longName and not lyr.longName.endswith("Layers"): + lyr.visible = False + # These layers can't be turned off. So leave them on. elif lyr.longName.endswith("Light Gray Canvas Base"): lyr.visible = True @@ -97,17 +120,19 @@ def reset_map_base_layers(mxd, logger, basemap): else: lyr.visible = False - elm = arcpy.mapping.ListLayoutElements(mxd, "TEXT_ELEMENT")[0] + layout = aprx.listLayouts("ftot_layout")[0] + elm = layout.listElements("TEXT_ELEMENT")[0] elm.text = " " return # =================================================================================================== -def get_layer_dictionary(mxd, logger): +def get_layer_dictionary(aprx, logger): logger.debug("start: get_layer_dictionary") layer_dictionary = {} - for lyr in arcpy.mapping.ListLayers(mxd): + map = aprx.listMaps("ftot_map")[0] + for lyr in map.listLayers(): layer_dictionary[lyr.longName.replace(" ", "_").replace("\\", "_")] = lyr @@ -115,29 +140,31 @@ def get_layer_dictionary(mxd, logger): # =================================================================================================== -def debug_layer_status(mxd, logger): +def debug_layer_status(aprx, logger): - layer_dictionary = get_layer_dictionary(mxd, logger) + layer_dictionary = get_layer_dictionary(aprx, logger) - for layer_name, lyr in sorted(layer_dictionary.iteritems()): + for layer_name, lyr in sorted(iteritems(layer_dictionary)): logger.info("layer: {}, visible: {}".format(layer_name, lyr.visible)) # ===================================================================================================- -def export_to_png(map_name, mxd, the_scenario, logger): +def export_to_png(map_name, aprx, the_scenario, logger): file_name = str(map_name + ".png").replace(" ", "_").replace("\\", "_") file_location = os.path.join(the_scenario.mapping_directory, file_name) - arcpy.mapping.ExportToPNG(mxd, file_location) + layout = aprx.listLayouts("ftot_layout")[0] + + layout.exportToPNG(file_location) logger.info("exporting: {}".format(map_name)) # =================================================================================================== -def export_map_steps(mxd, the_scenario, logger, basemap): +def export_map_steps(aprx, the_scenario, logger, basemap): # ------------------------------------------------------------------------------------ @@ -172,23 +199,23 @@ def export_map_steps(mxd, the_scenario, logger, basemap): new_sr = create_custom_spatial_ref(ll, ur) - set_extent(mxd, extent, sr, new_sr) + set_extent(aprx, extent, sr, new_sr) # Clean up arcpy.Delete_management(r"in_memory\temp") del ext, desc, array, extent_list - # Save mxd so that after step is run user can open the mxd at the right zoom/ extent to continue examining the data. - mxd.save() + # Save aprx so that after step is run user can open the aprx at the right zoom/ extent to continue examining the data. + aprx.save() # reset the map so we are working from a clean and known starting point. - reset_map_base_layers(mxd, logger, basemap) + reset_map_base_layers(aprx, logger, basemap) - # get a dictionary of all the layers in the mxd + # get a dictionary of all the layers in the aprx # might want to get a list of groups - layer_dictionary = get_layer_dictionary(mxd, logger) + layer_dictionary = get_layer_dictionary(aprx, logger) - logger.debug("layer_dictionary.keys(): \t {}".format(layer_dictionary.keys())) + logger.debug("layer_dictionary.keys(): \t {}".format(list(layer_dictionary.keys()))) # create a variable for each layer so we can access each layer easily @@ -207,15 +234,8 @@ def export_map_steps(mxd, the_scenario, logger, basemap): o_step_rmp_opt_vs_non_opt_lyr = layer_dictionary["O_STEP_RMP_OPTIMAL_VS_NON_OPTIMAL"] o_step_proc_opt_vs_non_opt_lyr = layer_dictionary["O_STEP_PROC_OPTIMAL_VS_NON_OPTIMAL"] o_step_dest_opt_vs_non_opt_lyr = layer_dictionary["O_STEP_DEST_OPTIMAL_VS_NON_OPTIMAL"] - # Following if/then statements for ftot_maps.mxd backward compatibility - if "O_STEP_RMP_TO_PROC" in layer_dictionary: - o_step_raw_producer_to_processor_lyr = layer_dictionary["O_STEP_RMP_TO_PROC"] - else: - o_step_raw_producer_to_processor_lyr = layer_dictionary["O_STEP_RAW_MATERIAL_PRODUCER_TO_PROCESSOR"] - if "O_STEP_PROC_TO_DEST" in layer_dictionary: - o_step_processor_to_destination_lyr = layer_dictionary["O_STEP_PROC_TO_DEST"] - else: - o_step_processor_to_destination_lyr = layer_dictionary["O_STEP_PROCESSOR_TO_ULTIMATE_DESTINATION"] + o_step_raw_producer_to_processor_lyr = layer_dictionary["O_STEP_RMP_TO_PROC"] + o_step_processor_to_destination_lyr = layer_dictionary["O_STEP_PROC_TO_DEST"] f2_step_candidates_lyr = layer_dictionary["F2_STEP_CANDIDATES"] f2_step_merged_lyr = layer_dictionary["F2_STEP_MERGE"] f2_step_candidates_labels_lyr = layer_dictionary["F2_STEP_CANDIDATES_W_LABELS"] @@ -225,6 +245,8 @@ def export_map_steps(mxd, the_scenario, logger, basemap): # features they would like to map. Code will look for this group layer name, which should not change. if "CUSTOM_USER_CREATED_MAPS" in layer_dictionary: custom_maps_parent_lyr = layer_dictionary["CUSTOM_USER_CREATED_MAPS"] + else: + custom_maps_parent_lyr = None # START MAKING THE MAPS! @@ -236,128 +258,143 @@ def export_map_steps(mxd, the_scenario, logger, basemap): # map_name = "" # caption = "" # call generate_map() -# generate_map(caption, map_name, mxd, the_scenario, logger, basemap) +# generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # S_STEP s_step_lyr.visible = True - for subLayer in s_step_lyr: - subLayer.visible = True + sublayers = s_step_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "01_S_Step_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # F_STEP_RMP f_step_rmp_lyr.visible = True - for subLayer in f_step_rmp_lyr: - subLayer.visible = True + sublayers = f_step_rmp_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "02a_F_Step_RMP_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # F_STEP_PROC f_step_proc_lyr.visible = True - for subLayer in f_step_proc_lyr: - subLayer.visible = True + sublayers = f_step_proc_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "02b_F_Step_User_Defined_PROC_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # F_STEP_DEST f_step_dest_lyr.visible = True - for subLayer in f_step_dest_lyr: - subLayer.visible = True + sublayers = f_step_dest_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "02c_F_Step_DEST_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # F_STEP f_step_lyr.visible = True - for subLayer in f_step_lyr: - subLayer.visible = True + sublayers = f_step_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "02d_F_Step_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # F_STEP_RMP_LABELS f_step_rmp_labels_lyr.visible = True - for subLayer in f_step_rmp_labels_lyr: - subLayer.visible = True + sublayers = f_step_rmp_labels_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "02a_F_Step_RMP_With_Labels_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # F_STEP_PROC_LABELS f_step_proc_labels_lyr.visible = True - for subLayer in f_step_proc_labels_lyr: - subLayer.visible = True + sublayers = f_step_proc_labels_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "02b_F_Step_User_Defined_PROC_With_Labels_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # F_STEP_DEST_LABELS f_step_dest_labels_lyr.visible = True - for subLayer in f_step_dest_labels_lyr: - subLayer.visible = True + sublayers = f_step_dest_labels_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "02c_F_Step_DEST_With_Labels_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # F_STEP_LABELS f_step_labels_lyr.visible = True - for subLayer in f_step_labels_lyr: - subLayer.visible = True + sublayers = f_step_labels_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "02d_F_Step_With_Labels_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # O_STEP_FINAL_OPTIMAL_ROUTES_WITH_COMMODITY_FLOW # O_STEP - A - o_step_opt_flow_lyr.visible = True - for subLayer in o_step_opt_flow_lyr: - subLayer.visible = True + sublayers = o_step_opt_flow_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "04a_O_Step_Final_Optimal_Routes_With_Commodity_Flow_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # O_STEP - B - same as above but with no labels to make it easier to see routes. o_step_opt_flow_no_labels_lyr.visible = True - for subLayer in o_step_opt_flow_no_labels_lyr: - subLayer.visible = True + sublayers = o_step_opt_flow_no_labels_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "04b_O_Step_Final_Optimal_Routes_With_Commodity_Flow_NO_LABELS_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # O_STEP - C - just flows. no Origins or Destinations o_step_opt_flow_just_flow_lyr.visible = True - for subLayer in o_step_opt_flow_just_flow_lyr: - subLayer.visible = True + sublayers = o_step_opt_flow_just_flow_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "04c_O_Step_Final_Optimal_Routes_With_Commodity_Flow_JUST_FLOW_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # O_STEP_OPTIMAL_AND_NON_OPTIMAL_RMP o_step_rmp_opt_vs_non_opt_lyr.visible = True - for subLayer in o_step_rmp_opt_vs_non_opt_lyr: - subLayer.visible = True + sublayers = o_step_rmp_opt_vs_non_opt_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "04d_O_Step_Optimal_and_Non_Optimal_RMP_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # O_STEP_OPTIMAL_AND_NON_OPTIMAL_DEST o_step_proc_opt_vs_non_opt_lyr.visible = True - for subLayer in o_step_proc_opt_vs_non_opt_lyr: - subLayer.visible = True + sublayers = o_step_proc_opt_vs_non_opt_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "04e_O_Step_Optimal_and_Non_Optimal_PROC_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # O_STEP_OPTIMAL_AND_NON_OPTIMAL_DEST o_step_dest_opt_vs_non_opt_lyr.visible = True - for subLayer in o_step_dest_opt_vs_non_opt_lyr: - subLayer.visible = True + sublayers = o_step_dest_opt_vs_non_opt_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "04f_O_Step_Optimal_and_Non_Optimal_DEST_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # check if processor fc exists. if it doesn't stop here since there won't be any more relevant maps to make @@ -368,51 +405,57 @@ def export_map_steps(mxd, the_scenario, logger, basemap): logger.debug("Processor Feature Class has {} features....".format(processor_fc_feature_count)) # F2_STEP_CANDIDATES f2_step_candidates_lyr.visible = True - for subLayer in f2_step_candidates_lyr: - subLayer.visible = True + sublayers = f2_step_candidates_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "03a_F2_Step_Processor_Candidates_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # F2_STEP_MERGE f2_step_merged_lyr.visible = True - for subLayer in f2_step_merged_lyr: - subLayer.visible = True + sublayers = f2_step_merged_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "03b_F2_Step_Processors_All_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # F2_STEP_CANDIDATES_W_LABELS f2_step_candidates_labels_lyr.visible = True - for subLayer in f2_step_candidates_labels_lyr: - subLayer.visible = True + sublayers = f2_step_candidates_labels_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "03a_F2_Step_Processor_Candidates_With_Labels_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # F2_STEP_MERGE_W_LABELS f2_step_merged_labels_lyr.visible = True - for subLayer in f2_step_merged_labels_lyr: - subLayer.visible = True + sublayers = f2_step_merged_labels_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "03b_F2_Step_Processors_All_With_Labels_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # O_STEP_RAW_MATERIAL_PRODUCER_TO_PROCESSOR o_step_raw_producer_to_processor_lyr.visible = True - for subLayer in o_step_raw_producer_to_processor_lyr: - subLayer.visible = True + sublayers = o_step_raw_producer_to_processor_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "05a_O_Step_Raw_Material_Producer_To_Processor_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # O_STEP_PROCESSOR_TO_ULTIMATE_DESTINATION o_step_processor_to_destination_lyr.visible = True - for subLayer in o_step_processor_to_destination_lyr: - subLayer.visible = True + sublayers = o_step_processor_to_destination_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True map_name = "05b_O_Step_Processor_To_Ultimate_Destination_" + basemap caption = "" - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) else: logger.info("processor layer has {} features. skipping the rest of the mapping".format @@ -420,32 +463,33 @@ def export_map_steps(mxd, the_scenario, logger, basemap): # CUSTOM_MAPS if "CUSTOM_USER_CREATED_MAPS" in layer_dictionary: - custom_map_layer_dictionary = get_layer_dictionary(custom_maps_parent_lyr, logger) - for layer in custom_map_layer_dictionary: - custom_lyr = custom_map_layer_dictionary[layer] + sub_groups_custom_maps = custom_maps_parent_lyr.listLayers() + for sub_group in sub_groups_custom_maps: # Need to limit maps to just the group layers which are NOT the parent group layer. - if custom_lyr.isGroupLayer and custom_lyr != custom_maps_parent_lyr: + if sub_group.isGroupLayer and sub_group != custom_maps_parent_lyr: # First ensure that the group layer is on as this as all layers are turned off at beginning by default custom_maps_parent_lyr.visible = True # Then turn on the individual custom map - custom_lyr.visible = True + sub_group.visible = True count = 0 - for subLayer in custom_lyr: - subLayer.visible = True - count +=1 - map_name = layer + "_" + basemap + sublayers = sub_group.listLayers() + for sublayer in sublayers: + sublayer.visible = True + count += 1 + map_name = str(sub_group) + "_" + basemap caption = "" # Only generate map if there are actual features inside the group layer if count > 0: - generate_map(caption, map_name, mxd, the_scenario, logger, basemap) + generate_map(caption, map_name, aprx, the_scenario, logger, basemap) # =================================================================================================== -def generate_map(caption, map_name, mxd, the_scenario, logger, basemap): +def generate_map(caption, map_name, aprx, the_scenario, logger, basemap): import datetime - # create a text element on the mxd for the caption - elm = arcpy.mapping.ListLayoutElements(mxd, "TEXT_ELEMENT")[0] + # create a text element on the aprx for the caption + layout = aprx.listLayouts("ftot_layout")[0] + elm = layout.listElements("TEXT_ELEMENT")[0] # set the caption text dt = datetime.datetime.now() @@ -461,10 +505,10 @@ def generate_map(caption, map_name, mxd, the_scenario, logger, basemap): elm.elementHeight = 0.0017*len(caption) + 0.0 # export that map to png - export_to_png(map_name, mxd, the_scenario, logger) + export_to_png(map_name, aprx, the_scenario, logger) # reset the map layers - reset_map_base_layers(mxd, logger, basemap) + reset_map_base_layers(aprx, logger, basemap) # =================================================================================================== @@ -474,15 +518,19 @@ def prepare_time_commodity_subsets_for_mapping(the_scenario, logger, task): if task == "m2": basemap = "default_basemap" - if task == "m2b": + elif task == "m2b": basemap = "gray_basemap" - if task == "m2c": + elif task == "m2c": basemap = "topo_basemap" + elif task == "m2d": + basemap = "street_basemap" + else: + basemap = "default_basemap" timestamp_folder_name = 'maps_' + datetime.datetime.now().strftime("%Y_%m_%d_%H-%M-%S") the_scenario.mapping_directory = os.path.join(the_scenario.scenario_run_directory, "Maps_Time_Commodity", basemap + "_" + timestamp_folder_name) - scenario_mxd_location = os.path.join(the_scenario.scenario_run_directory, "Maps_Time_Commodity", "ftot_maps.mxd") + scenario_aprx_location = os.path.join(the_scenario.scenario_run_directory, "Maps_Time_Commodity", "ftot_maps.aprx") scenario_gdb = the_scenario.main_gdb arcpy.env.workspace = scenario_gdb @@ -490,21 +538,30 @@ def prepare_time_commodity_subsets_for_mapping(the_scenario, logger, task): logger.debug("creating maps directory.") os.makedirs(the_scenario.mapping_directory) - if os.path.exists(scenario_mxd_location): - os.remove(scenario_mxd_location) + if os.path.exists(scenario_aprx_location): + os.remove(scenario_aprx_location) - # set mxd locations: root is in the common data folder, and the scenario is in the local maps folder - root_ftot_mxd_location = os.path.join(the_scenario.common_data_folder, "ftot_maps.mxd") + # set aprx locations: root is in the common data folder, and the scenario is in the local maps folder + root_ftot_aprx_location = os.path.join(the_scenario.common_data_folder, "ftot_maps.aprx") base_layers_location = os.path.join(the_scenario.common_data_folder, "Base_Layers") - # copy the mxd from the common data director to the scenario maps directory - logger.debug("copying the root mxd from common data to the scenario maps directory.") - copy(root_ftot_mxd_location, scenario_mxd_location) + # copy the aprx from the common data director to the scenario maps directory + logger.debug("copying the root aprx from common data to the scenario maps directory.") + copy(root_ftot_aprx_location, scenario_aprx_location) # load the map document into memory - mxd = arcpy.mapping.MapDocument(scenario_mxd_location) + aprx = arcpy.mp.ArcGISProject(scenario_aprx_location) + + # Fix the non-base layers here + broken_list = aprx.listBrokenDataSources() + for broken_item in broken_list: + if broken_item.supports("DATASOURCE"): + if not broken_item.longName.find("Base") == 0: + conprop = broken_item.connectionProperties + conprop['connection_info']['database'] = scenario_gdb + broken_item.updateConnectionProperties(broken_item.connectionProperties, conprop) - list_broken_data_sources(mxd, base_layers_location, logger) + list_broken_data_sources(aprx, base_layers_location, logger) # Project and zoom to extent of features # Get extent of the locations and optimized_route_segments FCs and zoom to buffered extent @@ -537,14 +594,14 @@ def prepare_time_commodity_subsets_for_mapping(the_scenario, logger, task): new_sr = create_custom_spatial_ref(ll, ur) - set_extent(mxd, extent, sr, new_sr) + set_extent(aprx, extent, sr, new_sr) # Clean up arcpy.Delete_management(r"in_memory\temp") del ext, desc, array, extent_list - # Save mxd so that after step is run user can open the mxd at the right zoom/ extent to continue examining data. - mxd.save() + # Save aprx so that after step is run user can open the aprx at the right zoom/ extent to continue examining data. + aprx.save() logger.info("start: building dictionaries of time steps and commodities occurring within the scenario") commodity_dict = {} @@ -590,7 +647,7 @@ def prepare_time_commodity_subsets_for_mapping(the_scenario, logger, task): the_scenario, logger) # Make map - make_time_commodity_maps(mxd, image_name, the_scenario, logger, basemap) + make_time_commodity_maps(aprx, image_name, the_scenario, logger, basemap) # Clear flag fields clear_flag_fields(the_scenario) @@ -610,7 +667,7 @@ def prepare_time_commodity_subsets_for_mapping(the_scenario, logger, task): the_scenario, logger) # Make map - make_time_commodity_maps(mxd, image_name, the_scenario, logger, basemap) + make_time_commodity_maps(aprx, image_name, the_scenario, logger, basemap) # Clear flag fields clear_flag_fields(the_scenario) @@ -631,7 +688,7 @@ def prepare_time_commodity_subsets_for_mapping(the_scenario, logger, task): the_scenario, logger) # Make map - make_time_commodity_maps(mxd, image_name, the_scenario, logger, basemap) + make_time_commodity_maps(aprx, image_name, the_scenario, logger, basemap) # Clear flag fields clear_flag_fields(the_scenario) @@ -694,7 +751,7 @@ def link_subset_to_route_segments_and_facilities(sql_where_clause, the_scenario) # =================================================================================================== -def make_time_commodity_maps(mxd, image_name, the_scenario, logger, basemap): +def make_time_commodity_maps(aprx, image_name, the_scenario, logger, basemap): scenario_gdb = the_scenario.main_gdb @@ -710,11 +767,11 @@ def make_time_commodity_maps(mxd, image_name, the_scenario, logger, basemap): if count > 0: # reset the map so we are working from a clean and known starting point. - reset_map_base_layers(mxd, logger, basemap) + reset_map_base_layers(aprx, logger, basemap) - # get a dictionary of all the layers in the mxd + # get a dictionary of all the layers in the aprx # might want to get a list of groups - layer_dictionary = get_layer_dictionary(mxd, logger) + layer_dictionary = get_layer_dictionary(aprx, logger) # create a variable for for each layer of interest for the time and commodity mapping # so we can access each layer easily @@ -725,32 +782,34 @@ def make_time_commodity_maps(mxd, image_name, the_scenario, logger, basemap): # Establish definition queries to define the subset for each layer, # turn off if there are no features for that particular subset. for groupLayer in [time_commodity_segments_lyr]: - for subLayer in groupLayer: - if subLayer.supports("DATASOURCE"): - if subLayer.dataSource == os.path.join(scenario_gdb, 'optimized_route_segments'): - subLayer.definitionQuery = "Include_Map = 1" + sublayers = groupLayer.listLayers() + for sublayer in sublayers: + if sublayer.supports("DATASOURCE"): + if sublayer.dataSource == os.path.join(scenario_gdb, 'optimized_route_segments'): + sublayer.definitionQuery = "Include_Map = 1" - if subLayer.dataSource == os.path.join(scenario_gdb, 'raw_material_producers'): - subLayer.definitionQuery = "Include_Map = 1" - rmp_count = get_feature_count(subLayer, logger) + if sublayer.dataSource == os.path.join(scenario_gdb, 'raw_material_producers'): + sublayer.definitionQuery = "Include_Map = 1" + rmp_count = get_feature_count(sublayer, logger) - if subLayer.dataSource == os.path.join(scenario_gdb, 'processors'): - subLayer.definitionQuery = "Include_Map = 1" - proc_count = get_feature_count(subLayer, logger) + if sublayer.dataSource == os.path.join(scenario_gdb, 'processors'): + sublayer.definitionQuery = "Include_Map = 1" + proc_count = get_feature_count(sublayer, logger) - if subLayer.dataSource == os.path.join(scenario_gdb, 'ultimate_destinations'): - subLayer.definitionQuery = "Include_Map = 1" - dest_count = get_feature_count(subLayer, logger) + if sublayer.dataSource == os.path.join(scenario_gdb, 'ultimate_destinations'): + sublayer.definitionQuery = "Include_Map = 1" + dest_count = get_feature_count(sublayer, logger) # Actually export map to file time_commodity_segments_lyr.visible = True - for subLayer in time_commodity_segments_lyr: - subLayer.visible = True + sublayers = time_commodity_segments_lyr.listLayers() + for sublayer in sublayers: + sublayer.visible = True caption = "" - generate_map(caption, image_name, mxd, the_scenario, logger, basemap) + generate_map(caption, image_name, aprx, the_scenario, logger, basemap) - # Clean up mxd - del mxd + # Clean up aprx + del aprx # No mapping if there are no routes else: @@ -775,6 +834,11 @@ def clear_flag_fields(the_scenario): # =================================================================================================== def map_animation(the_scenario, logger): + # Below animation is currently only set up to animate scenario time steps + # NOT commodities or a combination of commodity and time steps. + + # Clean Up-- delete existing gif if it exists already + try: os.remove(os.path.join(the_scenario.mapping_directory, 'optimal_flows_time.gif')) logger.debug("deleted existing time animation gif") @@ -791,10 +855,7 @@ def map_animation(the_scenario, logger): if len(images) > 0: imageio.mimsave(os.path.join(the_scenario.mapping_directory, 'optimal_flows_time.gif'), images, duration=2) - # Below animation is currently only set up to animate scenario time steps - # NOT commodities or a combination of commodity and time steps. - # Clean Up-- delete existing gif if it exists already # =================================================================================================== def get_feature_count(fc, logger): result = arcpy.GetCount_management(fc) @@ -844,11 +905,13 @@ def create_custom_spatial_ref(ll, ur): # =================================================================================================== -def set_extent(mxd, extent, sr, new_sr): +def set_extent(aprx, extent, sr, new_sr): - data_frame = arcpy.mapping.ListDataFrames(mxd)[0] + map = aprx.listMaps("ftot_map")[0] + map_layout = aprx.listLayouts("ftot_layout")[0] + map_frame = map_layout.listElements("MAPFRAME_ELEMENT", "FTOT")[0] - data_frame.spatialReference = new_sr + map.spatialReference = new_sr ll_geom, lr_geom, ur_geom, ul_geom = [arcpy.PointGeometry(extent.lowerLeft, sr).projectAs(new_sr), arcpy.PointGeometry(extent.lowerRight, sr).projectAs(new_sr), @@ -873,9 +936,7 @@ def set_extent(mxd, extent, sr, new_sr): buff_poly = polygon_tmp_1.buffer(ext_buff_dist) new_extent = buff_poly.extent - data_frame.extent = new_extent - - arcpy.RefreshActiveView() + map_frame.camera.setExtent(new_extent) # =================================================================================================== diff --git a/program/ftot_networkx.py b/program/ftot_networkx.py index a19fba5..719402a 100644 --- a/program/ftot_networkx.py +++ b/program/ftot_networkx.py @@ -12,6 +12,9 @@ import ftot_supporting import arcpy import os +import multiprocessing +import math +from ftot_pulp import commodity_mode_setup # ----------------------------------------------------------------------------- @@ -19,6 +22,9 @@ def graph(the_scenario, logger): + #Check for permitted modes before creating nX graph + check_permitted_modes(the_scenario, logger) + # export the assets from GIS export_fcs_from_main_gdb export_fcs_from_main_gdb(the_scenario, logger) @@ -34,8 +40,45 @@ def graph(the_scenario, logger): # cost the network in the db set_network_costs_in_db(the_scenario, logger) + # generate shortest paths through the network + presolve_network(G, the_scenario, logger) + + # Eliminate the graph and related shape files before moving on + delete_shape_files(the_scenario, logger) + G = nx.null_graph() + # ----------------------------------------------------------------------------- +def delete_shape_files(the_scenario, logger): + # delete temporary files + logger.debug("start: delete the temp_networkx_shp_files dir") + input_path = the_scenario.networkx_files_dir + rmtree(input_path) + logger.debug("finish: delete the temp_networkx_shp_files dir") + +# ----------------------------------------------------------------------------- + + +#Scan the XML and input_data to ensure that pipelines are permitted and relevant +def check_permitted_modes(the_scenario, logger): + logger.debug("start: check permitted modes") + commodity_mode_dict = commodity_mode_setup(the_scenario, logger) + with sqlite3.connect(the_scenario.main_db) as db_cur: + # get pipeline records with an allow_yn == y + sql = "select * from commodity_mode where mode like 'pipeline%' and allowed_yn like 'y';" + pipeline_allowed = db_cur.execute(sql).fetchall() + if not pipeline_allowed: + logger.info("pipelines are not allowed") + new_permitted_mode_list = [] + for mode in the_scenario.permittedModes: + if 'pipeline' not in mode: + new_permitted_mode_list.append(mode) + elif 'pipeline' in mode: + continue # we don't want to include pipeline fcs if no csv exists to specify product flow + the_scenario.permittedModes = new_permitted_mode_list + logger.debug("finish: check permitted modes") + +# ----------------------------------------------------------------------------- def make_networkx_graph(the_scenario, logger): # High level work flow: @@ -44,13 +87,12 @@ def make_networkx_graph(the_scenario, logger): # create the multidigraph # convert the node labels to integers # reverse the graph and compose with self - # delete temporary files logger.info("start: make_networkx_graph") start_time = datetime.datetime.now() # read the shapefiles in the customized read_shp method - input_path = the_scenario.lyr_files_dir + input_path = the_scenario.networkx_files_dir logger.debug("start: read_shp") G = read_shp(input_path, logger, simplify=True, @@ -73,10 +115,6 @@ def make_networkx_graph(the_scenario, logger): logger.debug("start: compose G and H") G = nx.compose(G, H) - # delete temporary files - logger.debug("start: delete the temp_networkx_shp_files dir") - rmtree(input_path) - # print out some stats on the Graph logger.info("Number of nodes in the raw graph: {}".format(G.order())) logger.info("Number of edges in the raw graph: {}".format(G.size())) @@ -87,6 +125,316 @@ def make_networkx_graph(the_scenario, logger): return G + +# ----------------------------------------------------------------------------- + +def presolve_network(G, the_scenario, logger): + logger.debug("start: presolve_network") + + # Create a table to hold the shortest edges + with sqlite3.connect(the_scenario.main_db) as db_cur: + # clean up the db + sql = "drop table if exists shortest_edges;" + db_cur.execute(sql) + sql = "create table if not exists shortest_edges (from_node_id INT, to_node_id INT, edge_id INT, " \ + "CONSTRAINT unique_from_to_edge_id_tuple UNIQUE (from_node_id, to_node_id, edge_id));" + db_cur.execute(sql) + + # insert all pipeline edges iff pipelines are permitted and explicitly allowed in the commodity mode CSV + if 'pipeline_prod_trf_rts' or 'pipeline_crude_trf_rts' in the_scenario.permittedModes: + sql = """ + INSERT OR IGNORE INTO shortest_edges SELECT networkx_edges.from_node_id,networkx_edges.to_node_id, + networkx_edges.edge_id from networkx_edges where networkx_edges.mode_source like "pipeline%"; + """ + logger.debug("start: insert pipeline edges into shortest_edges table") + db_cur.execute(sql) + db_cur.commit() + logger.debug("finish: update shortest_edges table with pipelines") + db_cur.commit() + + # if NDR_On = False (default case) in XML scenario, then skip NDR + if not the_scenario.ndrOn: + with sqlite3.connect(the_scenario.main_db) as db_cur: + sql = """ + INSERT or IGNORE into shortest_edges + SELECT from_node_id, to_node_id, edge_id + FROM networkx_edges; + """ + logger.debug("NDR skipped as specified in XML scenario") + db_cur.execute(sql) + db_cur.commit() + logger.debug("finish: presolve_network") + return + + # if capacity or candidate generation is active, then skip NDR + if the_scenario.capacityOn or the_scenario.processors_candidate_slate_data != 'None': + with sqlite3.connect(the_scenario.main_db) as db_cur: + sql = """ + INSERT or IGNORE into shortest_edges + SELECT from_node_id, to_node_id, edge_id + FROM networkx_edges; + """ + logger.debug("NDR de-activated due to capacity enforcement or candidate generation") + db_cur.execute(sql) + db_cur.commit() + logger.debug("finish: presolve_network") + return + + # Otherwise, determine the weights associated with the edges in the nX graph + nx_graph_weighting(G, the_scenario, logger) + + # Create a dictionary of edge_ids from the database which is used later to uniquely identify edges + edge_id_dict = find_edge_ids(the_scenario, logger) + + # Make origin-destination pairs where + # od_pairs is a dictionary keyed off [target, value is a list of sources for that target] + od_pairs = make_od_pairs(the_scenario, logger) + + # Use multi-processing to determine shortest_paths for each target in the od_pairs dictionary + manager = multiprocessing.Manager() + all_shortest_edges = manager.list() + logger.debug("multiprocessing.cpu_count() = {}".format(multiprocessing.cpu_count())) + + # To parallelize computations, assign a set of targets to be passed to each processor + stuff_to_pass = [] + logger.debug("start: identify shortest_path between each o-d pair") + # by destination + for a_target in od_pairs: + stuff_to_pass.append([G, od_pairs[a_target], a_target, all_shortest_edges, edge_id_dict]) + + # Allow multiprocessing, with no more than 75% of cores to be used, rounding down if necessary + logger.info("start: the multiprocessing route solve.") + processors_to_save = int(math.ceil(multiprocessing.cpu_count() * 0.25)) + processors_to_use = multiprocessing.cpu_count() - processors_to_save + logger.info("number of CPUs to use = {}".format(processors_to_use)) + + pool = multiprocessing.Pool(processes=processors_to_use) + pool.map(multi_shortest_paths, stuff_to_pass) + pool.close() + pool.join() + + logger.info("end: identify shortest_path between each o-d pair") + + with sqlite3.connect(the_scenario.main_db) as db_cur: + sql = """ + INSERT or IGNORE into shortest_edges + (from_node_id, to_node_id, edge_id) + values (?,?,?); + """ + + logger.debug("start: update shortest_edges table") + db_cur.executemany(sql, all_shortest_edges) + db_cur.commit() + logger.debug("end: update shortest_edges table") + + logger.debug("finish: presolve_network") + +# ----------------------------------------------------------------------------- +# This method uses a shortest_path algorithm from the nx library to flag edges in the +# network that are a part of the shortest path connecting an origin to a destination + + +def multi_shortest_paths(stuff_to_pass): + global all_shortest_edges + G, sources, target, all_shortest_edges, edge_id_dict = stuff_to_pass + t = target + + shortest_paths_to_t = nx.shortest_path(G, target=t, weight='weight') + for a_source in sources: + # This accounts for when a_source may not be connected to t, + # as is the case when certain modes may not be permitted + if a_source not in shortest_paths_to_t: + # TODO: create a managed list that tracks which source/destination pairs + # are skipped/ignored here. + continue + s = a_source + for index, from_node in enumerate(shortest_paths_to_t[s]): + if index < (len(shortest_paths_to_t[s]) - 1): + to_node = shortest_paths_to_t[s][index + 1] + for edge_id in edge_id_dict[to_node][from_node]: + all_shortest_edges.append((from_node, to_node, edge_id)) + +# ----------------------------------------------------------------------------- +# Assigns for each link in the networkX graph a weight which mirrors the cost +# of each edge for the optimizer + + +def nx_graph_weighting(G, the_scenario, logger): + + # pull the route cost for all edges in the graph + logger.debug("start: assign edge weights to networkX graph") + nx.set_edge_attributes(G, 0, name='weight') + for (u, v, c, d) in G.edges(keys=True, data='route_cost_scaling', default=False): + from_node_id = u + to_node_id = v + route_cost_scaling = G.edges[(u, v, c)]['route_cost_scaling'] + mileage = G.edges[(u, v, c)]['MILES'] + source = G.edges[(u, v, c)]['source'] + artificial = G.edges[(u, v, c)]['Artificial'] + phase = "unspecified" + weight = get_network_link_cost(the_scenario, phase, source, artificial, logger) + if 'pipeline' not in source: + G.edges[(u, v, c)]['weight'] = mileage * route_cost_scaling * weight + else: + # when pipelines are a permitted mode, all pipeline edges are included in + # the shortest_edges table. We inflate the cost of pipelines here to avoid + # keep them from forming a shortest path with the remaining edges + G.edges[(u, v, c)]['weight'] = 1000000 + + logger.debug("end: assign edge weights to networkX graph") + +# ----------------------------------------------------------------------------- + + +# Returns a dictionary of edge_ids keyed off (to_node_id,from_node_id) +def find_edge_ids(the_scenario, logger): + logger.debug("start: create edge_id dictionary") + logger.debug("start: pull edge_ids from SQL") + with sqlite3.connect(the_scenario.main_db) as db_cur: + sql = "SELECT from_node_id, to_node_id, edge_id FROM networkx_edges " \ + "ORDER BY to_node_id desc;" + sql_list = db_cur.execute(sql).fetchall() + logger.debug("end: pull edge_ids from SQL") + + edge_id_dict = {} + + for row in sql_list: + from_node = row[0] + to_node = row[1] + edge_id = row[2] + if to_node not in edge_id_dict: + edge_id_dict[to_node] = {} + if from_node not in edge_id_dict[to_node].keys(): + edge_id_dict[to_node][from_node] = [] + edge_id_dict[to_node][from_node].append(edge_id) + + logger.debug("end: create edge_id dictionary") + + return edge_id_dict + +# ----------------------------------------------------------------------------- + + +# Creates a dictionary of all feasible origin-destination pairs, including: +# RMP-DEST, RMP-PROC, PROC-DEST, etc. +def make_od_pairs(the_scenario, logger): + with sqlite3.connect(the_scenario.main_db) as db_cur: + # Create a table for od_pairs in the database + logger.info("start: create o-d pairs table") + sql = "drop table if exists od_pairs;" + db_cur.execute(sql) + sql = ''' + create table od_pairs(scenario_rt_id INTEGER PRIMARY KEY, from_location_id integer, to_location_id integer, + from_facility_id integer, to_facility_id integer, commodity_id integer, phase_of_matter text, route_status text, + from_node_id INTEGER, to_node_id INTEGER, from_location_1 INTEGER, to_location_1 INTEGER); + ''' + db_cur.execute(sql) + + # Populate the od_pairs table from a temporary table that collects both origins and destinations + sql = "drop table if exists tmp_connected_facilities_with_commodities;" + db_cur.execute(sql) + sql = ''' + create table tmp_connected_facilities_with_commodities as + select + facilities.facility_id, + facilities.location_id, + facilities.facility_name, + facility_type_id.facility_type, + ignore_facility, + facility_commodities.commodity_id, + facility_commodities.io, + commodities.phase_of_matter, + networkx_nodes.node_id, + networkx_nodes.location_1 + from facilities + join facility_commodities on + facility_commodities.facility_id = facilities.facility_ID + join commodities on + facility_commodities.commodity_id = commodities.commodity_id + join facility_type_id on + facility_type_id.facility_type_id = facilities.facility_type_id + join networkx_nodes on + networkx_nodes.location_id = facilities.location_id + where ignore_facility = 'false'; + ''' + db_cur.execute(sql) + + sql = ''' + insert into od_pairs (from_location_id, to_location_id, from_facility_id, to_facility_id, commodity_id, + phase_of_matter, from_node_id, to_node_id, from_location_1, to_location_1) + select distinct + origin.location_id AS from_location_id, + destination.location_id AS to_location_id, + origin.facility_id AS from_facility_id, + destination.facility_id AS to_facility_id, + origin.commodity_id AS commodity_id, + origin.phase_of_matter AS phase_of_matter, + origin.node_id AS from_node_id, + destination.node_id AS to_node_id, + origin.location_1 AS from_location_1, + destination.location_1 AS to_location_1 + from + tmp_connected_facilities_with_commodities as origin + inner join + tmp_connected_facilities_with_commodities destination ON + CASE + WHEN origin.facility_type <> 'processor' and destination.facility_type <> 'processor' -- THE NORMAL CASE, RMP->PROC, RMP->DEST, or PROC->DEST + THEN + origin.facility_type <> destination.facility_type -- not the same facility_type + and + origin.commodity_id = destination.commodity_id -- match on commodity + and + origin.facility_id <> destination.facility_id -- not the same facility + and + origin.facility_type <> "ultimate_destination" -- restrict origin + and + destination.facility_type <> "raw_material_producer_as_processor" -- restrict other origin types todo - MNP - 12/6/17 MAY NOT NEED THIS IF WE MAKE TEMP CANDIDATES AS THE RMP + and + destination.facility_type <> "raw_material_producer" -- restrict destination types + and + origin.location_1 like '%_OUT' -- restrict to the correct out/in node_id's + AND destination.location_1 like '%_IN' + ELSE -- THE CASE WHEN PROCESSORS AND SENDING STUFF TO OTHER PROCESSORS + origin.io = 'o' -- make sure processors origins send outputs + and + destination.io = 'i' -- make sure processors origins receive inputs + and + origin.commodity_id = destination.commodity_id -- match on commodity + and + origin.facility_id <> destination.facility_id -- not the same facility + and + origin.facility_type <> "ultimate_destination" -- restrict origin + and + destination.facility_type <> "raw_material_producer_as_processor" -- restrict other origin types todo - MNP - 12/6/17 MAY NOT NEED THIS IF WE MAKE TEMP CANDIDATES AS THE RMP + and + destination.facility_type <> "raw_material_producer" -- restrict destination types + and + origin.location_1 like '%_OUT' -- restrict to the correct out/in node_id's + AND destination.location_1 like '%_IN' + END; + ''' + db_cur.execute(sql) + + logger.debug("drop the tmp_connected_facilities_with_commodities table") + db_cur.execute("drop table if exists tmp_connected_facilities_with_commodities;") + + logger.info("end: create o-d pairs table") + + # Fetch all od-pairs, ordered by target + sql = "SELECT to_node_id, from_node_id FROM od_pairs ORDER BY to_node_id DESC;" + sql_list = db_cur.execute(sql).fetchall() + + # Loop through the od_pairs + od_pairs = {} + for row in sql_list: + target = row[0] + source = row[1] + if not target in od_pairs: + od_pairs[target] = [] + od_pairs[target].append(source) + + return od_pairs + # ----------------------------------------------------------------------------- @@ -97,8 +445,8 @@ def export_fcs_from_main_gdb(the_scenario, logger): # export network and locations fc's to shapefiles main_gdb = the_scenario.main_gdb - output_path = the_scenario.lyr_files_dir - input_features = "\"" + output_path = the_scenario.networkx_files_dir + input_features = [] logger.debug("start: create temp_networkx_shp_files dir") if os.path.exists(output_path): @@ -109,11 +457,15 @@ def export_fcs_from_main_gdb(the_scenario, logger): logger.debug("creating new temp_networkx_shp_files dir") os.makedirs(output_path) + location_list = ['\\locations', '\\network\\intermodal', '\\network\\locks'] + + # only add shape files associated with modes that are permitted in the scenario file + for mode in the_scenario.permittedModes: + location_list.append('\\network\\{}'.format(mode)) + # get the locations and network feature layers - for fc in ['\\locations;', '\\network\\intermodal;', '\\network\\locks;', '\\network\\pipeline_prod_trf_rts;', - '\\network\\pipeline_crude_trf_rts;', '\\network\\water;', '\\network\\rail;', '\\network\\road']: - input_features += main_gdb + fc - input_features += "\"" + for fc in location_list: + input_features.append(main_gdb + fc) arcpy.FeatureClassToShapefile_conversion(Input_Features=input_features, Output_Folder=output_path) logger.debug("finished: export_fcs_from_main_gdb: Runtime (HMS): \t{}".format( @@ -141,7 +493,7 @@ def clean_networkx_graph(the_scenario, G, logger): edge_attrs = {} # for storing the edge attributes which are set all at once deleted_edge_count = 0 - for u, v, keys, artificial in G.edges(data='Artificial', keys=True): + for u, v, keys, artificial in list(G.edges(data='Artificial', keys=True)): # initialize the route_cost_scaling variable to something # absurd so we know if its getting set properly in the loop: @@ -236,29 +588,29 @@ def clean_networkx_graph(the_scenario, G, logger): elif artificial == 1: # delete edges we dont want - try: - if G.edges[u, v, keys]['LOCATION_1'].find("_OUT") > -1 and reversed_link == 1: - G.remove_edge(u, v, keys) - deleted_edge_count += 1 - continue # move on to the next edge - elif G.edges[u, v, keys]['LOCATION_1'].find("_IN") > -1 and reversed_link == 0: - G.remove_edge(u, v, keys) - deleted_edge_count += 1 - continue # move on to the next edge - - # there is no scaling of artificial links. - # the cost_penalty is calculated in get_network_link_cost() - else: - route_cost_scaling = 1 - except: - logger.warning("the following keys didn't work:u - {}, v- {}".format(u, v)) + try: + if G.edges[u, v, keys]['LOCATION_1'].find("_OUT") > -1 and reversed_link == 1: + G.remove_edge(u, v, keys) + deleted_edge_count += 1 + continue # move on to the next edge + elif G.edges[u, v, keys]['LOCATION_1'].find("_IN") > -1 and reversed_link == 0: + G.remove_edge(u, v, keys) + deleted_edge_count += 1 + continue # move on to the next edge + + # there is no scaling of artificial links. + # the cost_penalty is calculated in get_network_link_cost() + else: + route_cost_scaling = 1 + except: + logger.warning("the following keys didn't work:u - {}, v- {}".format(u, v)) else: logger.warning("found an edge without artificial attribute: {} ") continue edge_attrs[u, v, keys] = { 'route_cost_scaling': route_cost_scaling - } + } nx.set_edge_attributes(G, edge_attrs) @@ -282,17 +634,27 @@ def get_network_link_cost(the_scenario, phase_of_matter, mode, artificial, logge if phase_of_matter == "solid": # set the mode costs - truck_base_cost = the_scenario.solid_truck_base_cost - railroad_class_1_cost = the_scenario.solid_railroad_class_1_cost - barge_cost = the_scenario.solid_barge_cost - transloading_cost = the_scenario.transloading_dollars_per_ton + truck_base_cost = the_scenario.solid_truck_base_cost.magnitude + railroad_class_1_cost = the_scenario.solid_railroad_class_1_cost.magnitude + barge_cost = the_scenario.solid_barge_cost.magnitude + transloading_cost = the_scenario.transloading_dollars_per_ton.magnitude elif phase_of_matter == "liquid": # set the mode costs - truck_base_cost = the_scenario.liquid_truck_base_cost - railroad_class_1_cost = the_scenario.liquid_railroad_class_1_cost - barge_cost = the_scenario.liquid_barge_cost - transloading_cost = the_scenario.transloading_dollars_per_thousand_gallons + truck_base_cost = the_scenario.liquid_truck_base_cost.magnitude + railroad_class_1_cost = the_scenario.liquid_railroad_class_1_cost.magnitude + barge_cost = the_scenario.liquid_barge_cost.magnitude + transloading_cost = the_scenario.transloading_dollars_per_thousand_gallons.magnitude + + # This accounts for the networkX shortest_path method, in which phase_of_matter is unknown + elif phase_of_matter == "unspecified": + # set the mode costs + # TODO KZ-- 3/19/2021-- using liquid base costs (in usd/kgal/mile), make shortest_path compatible with both solid/liquid-- changes to nx_graph_weighting will be needed. + truck_base_cost = 0.54 + railroad_class_1_cost = 0.09 + barge_cost = 0.07 + transloading_cost = 40.00 + else: logger.error("the phase of matter: -- {} -- is not supported. returning") raise NotImplementedError @@ -323,36 +685,16 @@ def get_network_link_cost(the_scenario, phase_of_matter, mode, artificial, logge link_cost = transloading_cost elif artificial == 0: - - if phase_of_matter == "solid": - if mode == "road": - link_cost = the_scenario.solid_truck_base_cost - elif mode == "rail": - link_cost = the_scenario.solid_railroad_class_1_cost - elif mode == "water": - link_cost = the_scenario.solid_barge_cost - else: - logger.warning("MODE: {} IS NOT SUPPORTED! for PHASE_OF_MATTER: {} ".format(mode, phase_of_matter)) - link_cost = 9999 - - elif phase_of_matter == "liquid": - if mode == "road": - link_cost = the_scenario.liquid_truck_base_cost - elif mode == "rail": - link_cost = the_scenario.liquid_railroad_class_1_cost - elif mode == "water": - link_cost = the_scenario.liquid_barge_cost - elif mode == "pipeline_crude_trf_rts": - link_cost = 1 # so we multiply by the base_rate - elif mode == "pipeline_prod_trf_rts": - link_cost = 1 # so we multiply by base_rate - else: - logger.warning("MODE: {} IS NOT SUPPORTED!".format(mode)) - else: - logger.warning("PHASE OF MATTER {} IS NOT SUPPORTED!".format(phase_of_matter)) - - else: - logger.warning("the commodity phase is not supported: {}".format(phase_of_matter)) + if mode == "road": + link_cost = truck_base_cost + elif mode == "rail": + link_cost = railroad_class_1_cost + elif mode == "water": + link_cost = barge_cost + elif mode == "pipeline_crude_trf_rts": + link_cost = 1 # so we multiply by the base_rate + elif mode == "pipeline_prod_trf_rts": + link_cost = 1 # so we multiply by base_rate return link_cost @@ -395,13 +737,9 @@ def get_phases_of_matter_in_scenario(the_scenario, logger): # ----------------------------------------------------------------------------- - +# set the network costs in the db by phase_of_matter def set_network_costs_in_db(the_scenario, logger): - # set the network costs in the db by phase_of_matter - # dollar_cost = mileage * phase_of_matter_cost - # routing_cost = dollar_cost * route_cost_scaling (factor in the networkx_edge table) - logger.info("start: set_network_costs_in_db") with sqlite3.connect(the_scenario.main_db) as db_con: # clean up the db @@ -495,7 +833,6 @@ def set_network_costs_in_db(the_scenario, logger): def digraph_to_db(the_scenario, G, logger): - # moves the networkX digraph into the database for the pulp handshake logger.info("start: digraph_to_db") @@ -563,7 +900,6 @@ def digraph_to_db(the_scenario, G, logger): db_con.execute(sql) for (u, v, c, d) in G.edges(keys=True, data='route_cost_scaling', default=False): - from_node_id = u to_node_id = v miles = G.edges[(u, v, c)]['MILES'] @@ -601,16 +937,14 @@ def digraph_to_db(the_scenario, G, logger): INSERT into networkx_edges values (null,?,?,?,?,?,?,?,?,?,?) ;""" - + # Add one more question mark here db_con.executemany(update_sql, edge_list) db_con.commit() logger.debug("finished network_x edges commit") - # ---------------------------------------------------------------------------- def read_shp(path, logger, simplify=True, geom_attrs=True, strict=True): - # the modified read_shp() multidigraph code logger.debug("start: read_shp -- simplify: {}, geom_attrs: {}, strict: {}".format(simplify, geom_attrs, strict)) @@ -655,7 +989,7 @@ def read_shp(path, logger, simplify=True, geom_attrs=True, strict=True): else: continue fld_data = [f.GetField(f.GetFieldIndex(x)) for x in fields] - attributes = dict(zip(fields, fld_data)) + attributes = dict(list(zip(fields, fld_data))) attributes["ShpName"] = lyr.GetName() # Note: Using layer level geometry type if g.GetGeometryType() == ogr.wkbPoint: @@ -664,10 +998,9 @@ def read_shp(path, logger, simplify=True, geom_attrs=True, strict=True): ogr.wkbMultiLineString): for edge in edges_from_line(g, attributes, simplify, geom_attrs): - e1, e2, attr = edge net.add_edge(e1, e2) - key = len(net[e1][e2].keys()) - 1 + key = len(list(net[e1][e2].keys())) - 1 net[e1][e2][key].update(attr) else: if strict: diff --git a/program/ftot_postprocess.py b/program/ftot_postprocess.py index 4c4abe9..0ea15f9 100644 --- a/program/ftot_postprocess.py +++ b/program/ftot_postprocess.py @@ -68,10 +68,9 @@ def route_post_optimization_db(the_scenario, logger): # -- Processors fc and reporting make_optimal_processors_featureclass(the_scenario, logger) - # -- Ultimate Destinations fc and reportings + # -- Ultimate Destinations fc and reporting make_optimal_destinations_featureclass(the_scenario, logger) - # ====================================================================================================================== @@ -108,7 +107,6 @@ def make_optimal_facilities_db(the_scenario, logger): db_con.execute(sql) - # ====================================================================================================================== @@ -142,9 +140,6 @@ def make_optimal_intermodal_db(the_scenario, logger): db_con.execute(sql) - - - # ====================================================================================================================== @@ -184,7 +179,6 @@ def make_optimal_intermodal_featureclass(the_scenario, logger): edit.stopOperation() edit.stopEditing(True) - # ====================================================================================================================== @@ -198,8 +192,8 @@ def make_optimal_raw_material_producer_featureclass(the_scenario, logger): for field in arcpy.ListFields(rmp_fc ): - if field.name.lower() =="optimal": - arcpy.DeleteField_management(rmp_fc , "optimal") + if field.name.lower() == "optimal": + arcpy.DeleteField_management(rmp_fc, "optimal") arcpy.AddField_management(rmp_fc, "optimal", "DOUBLE") @@ -215,21 +209,19 @@ def make_optimal_raw_material_producer_featureclass(the_scenario, logger): # loop through the GIS feature class and see if any of the facility_names match the list of optimal facilities. with arcpy.da.UpdateCursor(rmp_fc, ["facility_name", "optimal"]) as cursor: - for row in cursor: - facility_name = row[0] - row[1] = 0 # assume to start, it is not an optimal facility - for opt_fac in rmp_db_data: + for row in cursor: + facility_name = row[0] + row[1] = 0 # assume to start, it is not an optimal facility + for opt_fac in rmp_db_data: - if facility_name in opt_fac: # if the name matches and optimal facility + if facility_name in opt_fac: # if the name matches and optimal facility - row[1] = 1 # give it a positive value since we're not keep track of flows. - cursor.updateRow(row) + row[1] = 1 # give it a positive value since we're not keep track of flows. + cursor.updateRow(row) edit.stopOperation() edit.stopEditing(True) - - # ====================================================================================================================== @@ -242,10 +234,10 @@ def make_optimal_processors_featureclass(the_scenario, logger): processor_fc = the_scenario.processors_fc - for field in arcpy.ListFields(processor_fc ): + for field in arcpy.ListFields(processor_fc): - if field.name.lower() =="optimal": - arcpy.DeleteField_management(processor_fc , "optimal") + if field.name.lower() == "optimal": + arcpy.DeleteField_management(processor_fc, "optimal") arcpy.AddField_management(processor_fc, "optimal", "DOUBLE") @@ -261,13 +253,13 @@ def make_optimal_processors_featureclass(the_scenario, logger): # loop through the GIS feature class and see if any of the facility_names match the list of optimal facilities. with arcpy.da.UpdateCursor(processor_fc, ["facility_name", "optimal"]) as cursor: - for row in cursor: - facility_name = row[0] - row[1] = 0 # assume to start, it is not an optimal facility - for opt_fac in opt_fac_db_data: - if facility_name in opt_fac: # if the name matches and optimal facility - row[1] = 1 # give it a positive value since we're not keep track of flows. - cursor.updateRow(row) + for row in cursor: + facility_name = row[0] + row[1] = 0 # assume to start, it is not an optimal facility + for opt_fac in opt_fac_db_data: + if facility_name in opt_fac: # if the name matches and optimal facility + row[1] = 1 # give it a positive value since we're not keep track of flows. + cursor.updateRow(row) edit.stopOperation() edit.stopEditing(True) @@ -284,9 +276,9 @@ def make_optimal_destinations_featureclass(the_scenario, logger): destinations_fc = the_scenario.destinations_fc - for field in arcpy.ListFields(destinations_fc ): + for field in arcpy.ListFields(destinations_fc): - if field.name.lower() =="optimal": + if field.name.lower() == "optimal": arcpy.DeleteField_management(destinations_fc , "optimal") arcpy.AddField_management(destinations_fc, "optimal", "DOUBLE") @@ -303,13 +295,13 @@ def make_optimal_destinations_featureclass(the_scenario, logger): # loop through the GIS feature class and see if any of the facility_names match the list of optimal facilities. with arcpy.da.UpdateCursor(destinations_fc, ["facility_name", "optimal"]) as cursor: - for row in cursor: - facility_name = row[0] - row[1] = 0 # assume to start, it is not an optimal facility - for opt_fac in opt_fac_db_data: - if facility_name in opt_fac: # if the name matches and optimal facility - row[1] = 1 # give it a positive value since we're not keep track of flows. - cursor.updateRow(row) + for row in cursor: + facility_name = row[0] + row[1] = 0 # assume to start, it is not an optimal facility + for opt_fac in opt_fac_db_data: + if facility_name in opt_fac: # if the name matches and optimal facility + row[1] = 1 # give it a positive value since we're not keep track of flows. + cursor.updateRow(row) edit.stopOperation() edit.stopEditing(True) @@ -317,6 +309,8 @@ def make_optimal_destinations_featureclass(the_scenario, logger): scenario_gdb = the_scenario.main_gdb # ===================================================================================================================== + + def make_optimal_route_segments_db(the_scenario, logger): # iterate through the db to create a dictionary of dictionaries (DOD) # then generate the graph using the method @@ -327,8 +321,6 @@ def make_optimal_route_segments_db(the_scenario, logger): logger.info("START: make_optimal_route_segments_db") - - with sqlite3.connect(the_scenario.main_db) as db_con: # drop the table @@ -351,7 +343,6 @@ def make_optimal_route_segments_db(the_scenario, logger): );""" db_con.execute(sql) - # intiialize dod optimal_segments_list = [] with sqlite3.connect(the_scenario.main_db) as db_con: @@ -391,7 +382,6 @@ def make_optimal_route_segments_db(the_scenario, logger): row_count = len(rows) logger.info("number of rows in the optimal segments select to process: {}".format(row_count)) - logger.info("starting to iterate through the results and build up the dod") for row in rows: mode = row[0] @@ -400,7 +390,7 @@ def make_optimal_route_segments_db(the_scenario, logger): opt_flow = row[3] volume = row[4] capacity = row[5] - capacity_minus_volume= row[6] + capacity_minus_volume = row[6] units = row[7] time_period = row[8] miles = row[9] @@ -410,8 +400,8 @@ def make_optimal_route_segments_db(the_scenario, logger): artificial = row[13] # format for the optimal route segments table - optimal_segments_list.append([1, #rt id - None, # rt variant id + optimal_segments_list.append([1, # rt id + None, # rt variant id mode, mode_oid, None, # segment order @@ -425,7 +415,7 @@ def make_optimal_route_segments_db(the_scenario, logger): units, phase_of_matter, miles, - None, # route type + None, # route type link_dollar_cost, link_routing_cost, artificial, @@ -446,9 +436,9 @@ def make_optimal_route_segments_db(the_scenario, logger): return - # =================================================================================================== + def make_optimal_scenario_results_db(the_scenario, logger): logger.info("starting make_optimal_scenario_results_db") @@ -491,8 +481,6 @@ def make_optimal_scenario_results_db(the_scenario, logger): ;""" db_con.execute(sql_total_flow) - - # miles by mode # total network miles, not route miles sql_total_miles = """ -- total scenario miles (not route miles) @@ -585,7 +573,6 @@ def make_optimal_scenario_results_db(the_scenario, logger): db_con.execute(sql_routing_costs) - # loads by mode # use artificial links =1 and = 2 to calculate loads per loading event # (e.g. leaving a facility or switching modes at intermodal facility) @@ -596,7 +583,6 @@ def make_optimal_scenario_results_db(the_scenario, logger): loads_per_vehicle_dict = ftot_supporting_gis.get_loads_per_vehicle_dict(the_scenario) - for row in mode_and_phase_of_matter_list: mode = row[0] if 'pipeline_crude' in mode: @@ -683,7 +669,6 @@ def make_optimal_scenario_results_db(the_scenario, logger): fclass_and_urban_code["CO2ruralRestricted"] = {'co2_val': the_scenario.CO2ruralRestricted, 'where': "urban_rural_code = 99999 and fclass = 1"} fclass_and_urban_code["CO2ruralUnrestricted"] = {'co2_val': the_scenario.CO2ruralUnrestricted, 'where': "urban_rural_code = 99999 and fclass <> 1"} - for row in mode_and_phase_of_matter_list: mode = row[0] phase_of_matter = row[1] @@ -757,7 +742,6 @@ def make_optimal_scenario_results_db(the_scenario, logger): db_con.execute(sql_co2) - # Fuel burn # covert VMT to fuel burn # truck, rail, barge. no pipeline @@ -800,7 +784,6 @@ def make_optimal_scenario_results_db(the_scenario, logger): db_con.execute(sql_fuel_burn) - # # Destination Deliveries # # look at artificial = 1 and join on facility_name or d_location # # compare optimal flow at the facility (flow on all links) @@ -846,7 +829,6 @@ def make_optimal_scenario_results_db(the_scenario, logger): group by d_facility, mode;""" db_con.execute(sql_destination_demand_met) - sql_destination_demand_met = """insert into optimal_scenario_results select "facility_summary", @@ -861,12 +843,11 @@ def make_optimal_scenario_results_db(the_scenario, logger): join facilities f on f.facility_name = ov.d_facility join facility_commodities fc on fc.facility_id = f.facility_id join commodities c on c.commodity_id = fc.commodity_id - join facility_type_id fti on fti.facility_type_id = f.facility_type_id + join facility_type_id fti on fti.facility_type_id = f.facility_type_id where d_facility > 0 and edge_type = 'transport' and fti.facility_type = 'ultimate_destination' group by d_facility, mode;""" db_con.execute(sql_destination_demand_met) - # RMP supplier report sql_rmp_utilization = """insert into optimal_scenario_results select @@ -884,7 +865,7 @@ def make_optimal_scenario_results_db(the_scenario, logger): join facilities f on f.facility_name = ov.o_facility join facility_commodities fc on fc.facility_id = f.facility_id join commodities c on c.commodity_id = fc.commodity_id - join facility_type_id fti on fti.facility_type_id = f.facility_type_id + join facility_type_id fti on fti.facility_type_id = f.facility_type_id where o_facility > 0 and edge_type = 'transport' and fti.facility_type = 'raw_material_producer' group by o_facility, mode;""" db_con.execute(sql_rmp_utilization) @@ -903,9 +884,9 @@ def make_optimal_scenario_results_db(the_scenario, logger): from facility_commodities fc join facilities f on f.facility_id = fc.facility_id join commodities c on c.commodity_id = fc.commodity_id - join facility_type_id fti on fti.facility_type_id = f.facility_type_id + join facility_type_id fti on fti.facility_type_id = f.facility_type_id where fti.facility_type = 'raw_material_producer' - order by facility_name""" + order by facility_name""" db_con.execute(sql_rmp_utilization) # RMP supplier report @@ -923,12 +904,11 @@ def make_optimal_scenario_results_db(the_scenario, logger): join facilities f on f.facility_name = ov.o_facility join facility_commodities fc on fc.facility_id = f.facility_id join commodities c on c.commodity_id = fc.commodity_id - join facility_type_id fti on fti.facility_type_id = f.facility_type_id + join facility_type_id fti on fti.facility_type_id = f.facility_type_id where o_facility > 0 and edge_type = 'transport' and fti.facility_type = 'raw_material_producer' group by o_facility, mode;""" db_con.execute(sql_rmp_utilization) - # Processor report sql_processor_output = """insert into optimal_scenario_results select @@ -944,12 +924,11 @@ def make_optimal_scenario_results_db(the_scenario, logger): join facilities f on f.facility_name = ov.o_facility join facility_commodities fc on fc.facility_id = f.facility_id join commodities c on c.commodity_id = fc.commodity_id - join facility_type_id fti on fti.facility_type_id = f.facility_type_id + join facility_type_id fti on fti.facility_type_id = f.facility_type_id where o_facility > 0 and edge_type = 'transport' and fti.facility_type = 'processor' and fc.commodity_id = ov.commodity_id group by o_facility, mode, ov.commodity_name;""" db_con.execute(sql_processor_output) - sql_processor_input = """insert into optimal_scenario_results select "facility_summary", @@ -964,12 +943,11 @@ def make_optimal_scenario_results_db(the_scenario, logger): join facilities f on f.facility_name = ov.d_facility join facility_commodities fc on fc.facility_id = f.facility_id join commodities c on c.commodity_id = fc.commodity_id - join facility_type_id fti on fti.facility_type_id = f.facility_type_id + join facility_type_id fti on fti.facility_type_id = f.facility_type_id where d_facility > 0 and edge_type = 'transport' and fti.facility_type = 'processor' and fc.commodity_id = ov.commodity_id group by d_facility, mode, ov.commodity_name;""" db_con.execute(sql_processor_input) - # measure totals sql_total = """insert into optimal_scenario_results select table_name, commodity, facility_name, measure, "_total" as mode, sum(value), units, notes @@ -992,9 +970,9 @@ def generate_scenario_summary(the_scenario, logger): data = db_cur.fetchall() for row in data: - if row[2] == None: # these are the commodity summaries with no facility name + if row[2] == None: # these are the commodity summaries with no facility name logger.result('{}_{}_{}_{}: \t {:,.2f} : \t {}'.format(row[0].upper(), row[3].upper(), row[1].upper(), row[4].upper(), row[5], row[6])) - else: # these are the commodity summaries with no facility name + else: # these are the commodity summaries with no facility name logger.result('{}_{}_{}_{}_{}: \t {:,.2f} : \t {}'.format(row[0].upper(), row[2].upper(), row[3].upper(), row[1].upper(), row[4].upper(), row[5], row[6])) logger.debug("finish: generate_scenario_summary()") @@ -1106,8 +1084,8 @@ def make_optimal_route_segments_featureclass_from_db(the_scenario, # if from pos = 0 then traveresed in direction of underlying arc, otherwise traversed against the flow of the arc. # used to determine if from_to_dollar cost should be used or to_from_dollar cost should be used. - arcpy.AddField_management(optimized_route_segments_fc, "FromPosition", "DOUBLE") # mnp 8/30/18 -- deprecated - arcpy.AddField_management(optimized_route_segments_fc, "FromJunctionID", "DOUBLE") # mnp - 060116 - added this for processor siting step + arcpy.AddField_management(optimized_route_segments_fc, "FromPosition", "DOUBLE") # mnp 8/30/18 -- deprecated + arcpy.AddField_management(optimized_route_segments_fc, "FromJunctionID", "DOUBLE") # mnp - 060116 - added this for processor siting step arcpy.AddField_management(optimized_route_segments_fc, "TIME_PERIOD", "TEXT") arcpy.AddField_management(optimized_route_segments_fc, "COMMODITY", "TEXT") arcpy.AddField_management(optimized_route_segments_fc, "COMMODITY_FLOW", "FLOAT") @@ -1123,18 +1101,18 @@ def make_optimal_route_segments_featureclass_from_db(the_scenario, # get a list of the modes used in the optimal route_segments stored in the db. with sqlite3.connect(the_scenario.main_db) as db_con: - db_cur = db_con.cursor() + db_cur = db_con.cursor() - # get a list of the source_ids and convert to modes: - sql = "select distinct network_source_id from optimal_route_segments;" - db_cur = db_con.execute(sql) - rows = db_cur.fetchall() + # get a list of the source_ids and convert to modes: + sql = "select distinct network_source_id from optimal_route_segments;" + db_cur = db_con.execute(sql) + rows = db_cur.fetchall() - mode_source_list = [] - for row in rows: + mode_source_list = [] + for row in rows: mode_source_list.append(row[0]) - logger.debug("List of modes used in the optimal solution: {}".format(mode_source_list)) + logger.debug("List of modes used in the optimal solution: {}".format(mode_source_list)) # iterate through the segment_dict_by_source_name # and get the segments to insert into the optimal segments layer @@ -1145,109 +1123,108 @@ def make_optimal_route_segments_featureclass_from_db(the_scenario, "VOLUME", "CAPACITY", "CAPACITY_MINUS_VOLUME", "UNITS", "PHASE_OF_MATTER", "ARTIFICIAL", "LINK_ROUTING_COST", "LINK_DOLLAR_COST") for network_source in mode_source_list: - if network_source == 'intermodal': + if network_source == 'intermodal': logger.debug("network_source is: {}. can't flatten this, so skipping.".format(network_source)) continue - network_link_counter = 0 - network_link_coverage_counter = 0 - network_source_fc = os.path.join(the_scenario.main_gdb, "network", network_source) - - sql = """select DISTINCT - scenario_rt_id, - NULL, - network_source_id, - network_source_oid, - NULL, - NULL, - time_period, - commodity_name, - commodity_flow, - volume, - capacity, - capacity_minus_volume, - units, - phase_of_matter, - miles, - route_type, - artificial, - link_dollar_cost, - link_routing_cost - from optimal_route_segments - where network_source_id = '{}' - ; - """.format(network_source) - - logger.debug("starting the execute for the {} mode".format(network_source)) - db_cur = db_con.execute(sql) - logger.debug("done with the execute") - logger.debug("starting fetch all for the {} mode".format(network_source)) - rows = db_cur.fetchall() - logger.debug("done with the fetchall") - - optimal_route_segments_dict = {} - - logger.debug("starting to build the dict for {}".format(network_source)) - for row in rows: - if not optimal_route_segments_dict.has_key(row[3]): - optimal_route_segments_dict[row[3]] = [] - network_link_coverage_counter += 1 - - - route_id = row[0] - route_id_variant = row[1] - network_source_id = row[2] - network_object_id = row[3] - from_position = row[4] - from_junction_id = row[5] - time_period = row[6] - commodity = row[7] - commodity_flow = row[8] - volume = row[9] - capacity = row[10] - capacity_minus_volume= row[11] - units = row[12] - phase_of_matter = row[13] - miles = row[14] - route_type = row[15] - artificial = row[16] - link_dollar_cost = row[17] - link_routing_cost = row[18] - - optimal_route_segments_dict[row[3]].append([route_id, route_id_variant, network_source_id, \ - network_object_id, from_position, from_junction_id, time_period, \ - commodity, commodity_flow, volume, capacity, capacity_minus_volume, \ - units, phase_of_matter, miles, \ - route_type, artificial, link_dollar_cost, link_routing_cost]) - - logger.debug("done building the dict for {}".format(network_source)) - - logger.debug("starting the search cursor") - with arcpy.da.SearchCursor(network_source_fc, ["OBJECTID", "SHAPE@"]) as search_cursor: - logger.info("start: looping through the {} mode".format(network_source)) - for row in search_cursor: - network_link_counter += 1 - object_id = row[0] - geom = row[1] - - if not optimal_route_segments_dict.has_key(object_id): - continue - else: + network_link_counter = 0 + network_link_coverage_counter = 0 + network_source_fc = os.path.join(the_scenario.main_gdb, "network", network_source) + + sql = """select DISTINCT + scenario_rt_id, + NULL, + network_source_id, + network_source_oid, + NULL, + NULL, + time_period, + commodity_name, + commodity_flow, + volume, + capacity, + capacity_minus_volume, + units, + phase_of_matter, + miles, + route_type, + artificial, + link_dollar_cost, + link_routing_cost + from optimal_route_segments + where network_source_id = '{}' + ; + """.format(network_source) + + logger.debug("starting the execute for the {} mode".format(network_source)) + db_cur = db_con.execute(sql) + logger.debug("done with the execute") + logger.debug("starting fetch all for the {} mode".format(network_source)) + rows = db_cur.fetchall() + logger.debug("done with the fetchall") + + optimal_route_segments_dict = {} + + logger.debug("starting to build the dict for {}".format(network_source)) + for row in rows: + if row[3] not in optimal_route_segments_dict: + optimal_route_segments_dict[row[3]] = [] + network_link_coverage_counter += 1 + + route_id = row[0] + route_id_variant = row[1] + network_source_id = row[2] + network_object_id = row[3] + from_position = row[4] + from_junction_id = row[5] + time_period = row[6] + commodity = row[7] + commodity_flow = row[8] + volume = row[9] + capacity = row[10] + capacity_minus_volume= row[11] + units = row[12] + phase_of_matter = row[13] + miles = row[14] + route_type = row[15] + artificial = row[16] + link_dollar_cost = row[17] + link_routing_cost = row[18] + + optimal_route_segments_dict[row[3]].append([route_id, route_id_variant, network_source_id, + network_object_id, from_position, from_junction_id, time_period, + commodity, commodity_flow, volume, capacity, + capacity_minus_volume, units, phase_of_matter, miles, + route_type, artificial, link_dollar_cost, link_routing_cost]) + + logger.debug("done building the dict for {}".format(network_source)) + + logger.debug("starting the search cursor") + with arcpy.da.SearchCursor(network_source_fc, ["OBJECTID", "SHAPE@"]) as search_cursor: + logger.info("start: looping through the {} mode".format(network_source)) + for row in search_cursor: + network_link_counter += 1 + object_id = row[0] + geom = row[1] + + if object_id not in optimal_route_segments_dict: + continue + else: for segment_info in optimal_route_segments_dict[object_id]: - (route_id, route_id_variant, network_source_id, network_object_id, from_position, \ - from_junction_id, time_period, commodity, commodity_flow, volume, capacity, capacity_minus_volume, \ + (route_id, route_id_variant, network_source_id, network_object_id, from_position, + from_junction_id, time_period, commodity, commodity_flow, volume, capacity, capacity_minus_volume, units, phase_of_matter, miles, route_type, artificial, link_dollar_cost, link_routing_cost) = segment_info with arcpy.da.InsertCursor(optimized_route_segments_fc, optimized_route_seg_flds) as insert_cursor: - insert_cursor.insertRow([geom, route_id, route_id_variant, route_type, network_source_id, - network_object_id, from_position, from_junction_id, - miles, time_period, commodity, commodity_flow, volume, capacity, capacity_minus_volume, - units, phase_of_matter, artificial, link_routing_cost, link_dollar_cost]) - - logger.debug("finish: looping through the {} mode".format(network_source)) - logger.info("mode: {} coverage: {:,.1f} - total links: {} , total links used: {}".format(network_source, + insert_cursor.insertRow([geom, route_id, route_id_variant, route_type, network_source_id, + network_object_id, from_position, from_junction_id, + miles, time_period, commodity, commodity_flow, volume, capacity, + capacity_minus_volume, units, phase_of_matter, artificial, + link_routing_cost, link_dollar_cost]) + + logger.debug("finish: looping through the {} mode".format(network_source)) + logger.info("mode: {} coverage: {:,.1f} - total links: {} , total links used: {}".format(network_source, 100.0*(int(network_link_coverage_counter)/float(network_link_counter)), network_link_counter, network_link_coverage_counter )) - # ====================================================================================================================== @@ -1257,15 +1234,15 @@ def add_fclass_and_urban_code(the_scenario, logger): scenario_gdb = the_scenario.main_gdb optimized_route_segments_fc = os.path.join(scenario_gdb, "optimized_route_segments") - # build up dictiony network links we are interested in + # build up dictionary network links we are interested in # ---------------------------------------------------- network_source_oid_dict = {} network_source_oid_dict['road'] = {} with arcpy.da.SearchCursor(optimized_route_segments_fc, ("NET_SOURCE_NAME", "NET_SOURCE_OID")) as search_cursor: - for row in search_cursor: - if row[0] == 'road': + for row in search_cursor: + if row[0] == 'road': network_source_oid_dict['road'][row[1]] = True # iterate different network layers (i.e. rail, road, water, pipeline) @@ -1275,8 +1252,8 @@ def add_fclass_and_urban_code(the_scenario, logger): road_fc = os.path.join(scenario_gdb, "network", "road") flds = ("OID@", "FCLASS", "URBAN_CODE") with arcpy.da.SearchCursor(road_fc, flds) as search_cursor: - for row in search_cursor: - if row[0] in network_source_oid_dict['road']: + for row in search_cursor: + if row[0] in network_source_oid_dict['road']: network_source_oid_dict['road'][row[0]] = [row[1], row[2]] # add fields to hold data from network links @@ -1297,7 +1274,7 @@ def add_fclass_and_urban_code(the_scenario, logger): with arcpy.da.UpdateCursor(optimized_route_segments_fc, flds) as update_cursor: - for row in update_cursor: + for row in update_cursor: net_source = row[0] net_oid = row[1] @@ -1321,6 +1298,7 @@ def add_fclass_and_urban_code(the_scenario, logger): db_con.executemany(update_sql, update_list) db_con.commit() + # ====================================================================================================================== diff --git a/program/ftot_processor.py b/program/ftot_processor.py index 76f034d..0c86b55 100644 --- a/program/ftot_processor.py +++ b/program/ftot_processor.py @@ -18,6 +18,7 @@ from ftot_facilities import get_schedule_id from ftot_pulp import parse_optimal_solution_db from ftot import Q_ +from six import iteritems # ============================================================================== @@ -127,8 +128,9 @@ def populate_candidate_process_commodities(the_scenario, candidate_process_commo commodity_max_transport_dist = 'Null' io = commodity[6] shared_max_transport_distance = 'N' + processor_max_input = commodity[3] # empty string for facility type and schedule id because fields are not used - commodity_data = ['', commodity_name, commodity_quantity, commodity_unit, commodity_phase, commodity_max_transport_dist, io, shared_max_transport_distance, ''] + commodity_data = ['', commodity_name, commodity_quantity, commodity_unit, commodity_phase, commodity_max_transport_dist, io, shared_max_transport_distance, processor_max_input, ''] # get commodity_id. (adds commodity if it doesn't exist) commodity_id = get_commodity_id(the_scenario, db_con, commodity_data, logger) @@ -234,7 +236,7 @@ def populate_candidate_process_list_table(the_scenario, candidate_process_list, for row in db_data: process_id_name_dict[row[0]] = row[1] logger.info("note: added {} candidate processes to the candidate_process_list table".format( - len(process_id_name_dict.keys()))) + len(list(process_id_name_dict.keys())))) logger.debug("ID || Process Name:") logger.debug("-----------------------") for process_name in process_id_name_dict: @@ -255,7 +257,7 @@ def get_candidate_process_data(the_scenario, logger): candidate_process_commodities = [] candidate_process_list = {} - for facility_name, facility_commodity_list in candidate_process_data.iteritems(): + for facility_name, facility_commodity_list in iteritems(candidate_process_data): for row in facility_commodity_list: commodity_name = row[1] commodity_id = None @@ -274,7 +276,7 @@ def get_candidate_process_data(the_scenario, logger): # store the facility size and cost information in the # candidate_process_list (CPL). else: - if facility_name not in candidate_process_list.keys(): + if facility_name not in list(candidate_process_list.keys()): candidate_process_list[facility_name] = [] # add schedule name to the candidate_process_list array for the facility candidate_process_list[facility_name].append(["schedule_name", schedule_name]) @@ -336,7 +338,7 @@ def get_candidate_processor_slate_output_ratios(the_scenario, logger): quantity = float(row[5]) units = row[6] - if process_name not in output_dict.keys(): + if process_name not in list(output_dict.keys()): output_dict[process_name] = {} output_dict[process_name]['i'] = [] output_dict[process_name]['o'] = [] # initialize the output dict at the same time @@ -455,14 +457,16 @@ def processor_candidates(the_scenario, logger): with open(the_scenario.processor_candidates_commodity_data, 'w') as wf: # write the header line - header_line = "facility_name,facility_type,commodity,value,units,phase_of_matter,io,schedule" + header_line = "facility_name,facility_type,commodity,value,units,phase_of_matter,io,schedule," \ + "max_processor_input" wf.write(str(header_line + "\n")) - ## WRITE THE CSV FILE OF THE PROCESSOR CANDIDATES PRODUCT SLATE + # WRITE THE CSV FILE OF THE PROCESSOR CANDIDATES PRODUCT SLATE sql = """ select - facility_name, 'processor', commodity_name, quantity, units, phase_of_matter, io, schedule_name, cpl.process_name + facility_name, 'processor', commodity_name, quantity, units, phase_of_matter, io, schedule_name, + cpl.process_name from candidate_processors cp join candidate_process_list cpl on cpl.process_id = cp.process_id ;""" @@ -479,8 +483,10 @@ def processor_candidates(the_scenario, logger): io = row[6] schedule_name = row[7] process_name = row[8] + max_processor_input = input_quantity - wf.write("{},{},{},{},{},{},{},{}\n".format(row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7])) + wf.write("{},{},{},{},{},{},{},{},{}\n".format(row[0], row[1], row[2], row[3], row[4], row[5], row[6], + row[7], max_processor_input)) # write the scaled output commodities too # first get the input for the denomenator @@ -492,8 +498,9 @@ def processor_candidates(the_scenario, logger): output_phase_of_matter = output_scaler[2] output_quantity = Q_(input_quantity, input_units) * output_scaler_quantity / input_scaler_quantity wf.write( - "{},{},{},{},{},{},{}\n".format(row[0], row[1], output_commodity_name, output_quantity.magnitude, - output_quantity.units, output_phase_of_matter, 'o')) + "{},{},{},{},{},{},{},{},{}\n".format(row[0], row[1], output_commodity_name, output_quantity.magnitude, + output_quantity.units, output_phase_of_matter, 'o', + schedule_name, max_processor_input)) # MAKE THE FIRST PROCESSOR POINT LAYER # this layer consists of candidate nodes where flow exceeds the min facility size at a RMP, @@ -642,7 +649,7 @@ def generate_bulk_processor_candidates(the_scenario, logger): location_id = row[4] quantity = row[5] units = row[6] - if not location_id in fuel_ratios.keys(): + if not location_id in list(fuel_ratios.keys()): fuel_ratios[location_id] = [] fuel_ratios[location_id].append(0) # one feedstock fuel_ratios[location_id].append([]) # many feedstocks-as-fuel @@ -810,7 +817,7 @@ def generate_bulk_processor_candidates(the_scenario, logger): max_conversion_process, the_scenario, logger) # write the outputs from the fuel_dict. - for ouput_commodity, values in output_dict.iteritems(): + for ouput_commodity, values in iteritems(output_dict): logger.info("processing commodity: {} quantity: {}".format(ouput_commodity, values[0])) commodity = ouput_commodity quantity = values[0] @@ -824,11 +831,11 @@ def generate_bulk_processor_candidates(the_scenario, logger): phase_of_matter, io)) # add the mode to the dictionary and initialize as empty dict - if network_source not in candidate_location_oids_dict.keys(): + if network_source not in list(candidate_location_oids_dict.keys()): candidate_location_oids_dict[network_source] = {} # add the network link to the dict and initialize as empty list - if not net_source_oid in candidate_location_oids_dict[network_source].keys(): + if not net_source_oid in list(candidate_location_oids_dict[network_source].keys()): candidate_location_oids_dict[network_source][net_source_oid] = [] # prepare the candidate_location_oids_dict with all the information @@ -941,7 +948,7 @@ def make_flat_locationxy_flow_dict(the_scenario, logger): # flatten the optimal_route_flows by time period # ------------------------------------------------- - for route_id, optimal_data_list in optimal_route_flows.iteritems(): + for route_id, optimal_data_list in iteritems(optimal_route_flows): for data in optimal_data_list: # break out the data: # e.g optimal_route_flows[5633] : ['275, 189', 1, u'2', 63.791322], ['275, 189', 2, u'2', 63.791322]] @@ -983,9 +990,9 @@ def get_commodity_max_transport_dist_list(the_scenario, logger): commodity_name = row[1] max_trans_dist = row[2] - print "found optimal commodity: {} (id: {}) has max raw trans distance: {}".format(commodity_name, + print ("found optimal commodity: {} (id: {}) has max raw trans distance: {}".format(commodity_name, commodity_id, - max_trans_dist) + max_trans_dist)) optimal_commodity_max_transport_dist_list[commodity_id] = max_trans_dist return optimal_commodity_max_transport_dist_list @@ -1146,7 +1153,7 @@ def get_processor_fc_summary_statistics(the_scenario, candidates_fc, logger): table_short_name = table.replace("candidate_processor_summary_by_", "") - if table_short_name not in summary_dict.keys(): + if table_short_name not in list(summary_dict.keys()): summary_dict[table_short_name] = {} summary_field = row[1].upper() diff --git a/program/ftot_pulp.py b/program/ftot_pulp.py index 969cdd1..fb431e9 100644 --- a/program/ftot_pulp.py +++ b/program/ftot_pulp.py @@ -14,11 +14,11 @@ import re import sqlite3 from collections import defaultdict +from six import iteritems from pulp import * import ftot_supporting -import test_aftot_pulp from ftot_supporting import get_total_runtime_string # =================== constants============= @@ -29,8 +29,6 @@ THOUSAND_GALLONS_PER_THOUSAND_BARRELS = 42 -default_sched = test_aftot_pulp.schedule_full_availability() -last_day_sched = test_aftot_pulp.schedule_last_day_only() storage_cost_1 = 0.01 storage_cost_2 = 0.05 facility_onsite_storage_max = 10000000000 @@ -179,7 +177,7 @@ def make_commodity_mode_dict(the_scenario, logger): # then, fill in for all other unspecified commodities and modes, pipeline liquid only commodity_mode_dict = make_commodity_mode_dict(the_scenario, logger) for mode in the_scenario.permittedModes: - for k, v in commodities.iteritems(): + for k, v in iteritems(commodities): commodity_name = k commodity_id = v[0] @@ -447,9 +445,6 @@ def generate_all_vertices(the_scenario, schedule_dict, schedule_length, logger): source_facility_id = row_b[7] new_source_facility_id = facility_id - day = default_sched.first_day - stop_day = default_sched.last_day - availability = default_sched.availability # vertices for generic demand type, or any subtype specified by the destination for day_before, availability in enumerate(schedule_dict[schedule_id]): if io == 'i': @@ -608,7 +603,7 @@ def generate_all_vertices(the_scenario, schedule_dict, schedule_length, logger): storage_vertex, demand, udp, source_facility_id, iob) values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, {}, {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, - day, new_commodity_id, availability[day], primary, + day_before+1, new_commodity_id, availability, primary, quantity, the_scenario.unMetDemandPenalty, zero_source_facility_id, iob)) @@ -1496,7 +1491,14 @@ def generate_all_edges_without_max_commodity_constraint(the_scenario, schedule_l # for row in db_cur.execute( # "select commodity_id from commodities where commodity_name = '{}';""".format(multi_commodity_name)): # id_for_mult_commodities = row[0] - for row in db_cur.execute("select count(*) from networkx_edges;"): + logger.info("COUNTING TOTAL TRANSPORT ROUTES") + for row in db_cur.execute(""" + select count(*) from networkx_edges, shortest_edges + WHERE networkx_edges.from_node_id = shortest_edges.from_node_id + AND networkx_edges.to_node_id = shortest_edges.to_node_id + AND networkx_edges.edge_id = shortest_edges.edge_id + ; + """): total_transport_routes = row[0] # for all commodities with no max transport distance @@ -1524,11 +1526,13 @@ def generate_all_edges_without_max_commodity_constraint(the_scenario, schedule_l ne.capacity, ne.artificial, ne.mode_source_oid - from networkx_edges ne, networkx_nodes fn, networkx_nodes tn, networkx_edge_costs nec + from networkx_edges ne, networkx_nodes fn, networkx_nodes tn, networkx_edge_costs nec, shortest_edges se where ne.from_node_id = fn.node_id and ne.to_node_id = tn.node_id and ne.edge_id = nec.edge_id + and ne.from_node_id = se.from_node_id -- match to the shortest_edges table + and ne.to_node_id = se.to_node_id and ifnull(ne.capacity, 1) > 0 ;""" nx_edge_data = main_db_con.execute(sql) @@ -2125,7 +2129,7 @@ def create_constraint_unmet_demand(logger, the_scenario, prob, flow_var, unmet_d def create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, flow_var): logger.debug("STARTING: create_constraint_max_flow_out_of_supply_vertex") - logger.debug("Length of flow_var: {}".format(len(flow_var.items()))) + logger.debug("Length of flow_var: {}".format(len(list(flow_var.items())))) # create_constraint_max_flow_out_of_supply_vertex # primary vertices only @@ -2166,16 +2170,20 @@ def create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, def create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_var, processor_build_vars, processor_daily_flow_vars): logger.debug("STARTING: create_constraint_daily_processor_capacity") + import pdb + # pdb.set_trace() # primary vertices only - # flow out of a vertex <= nameplate capcity / 365, true for every day and totaled output commodities + # flow into vertex is capped at facility max_capacity per day + # sum over all input commodities, grouped by day and facility + # conservation of flow and ratios are handled in other methods - ### get primary processor vertex and its output quantity + ### get primary processor vertex and its input quantityi total_scenario_min_capacity = 0 with sqlite3.connect(the_scenario.main_db) as main_db_con: db_cur = main_db_con.cursor() - sql = """select fc.commodity_id, f.facility_id, - ifnull(f.candidate, 0), fc.quantity, v.schedule_day, v.activity_level + sql = """select f.facility_id, + ifnull(f.candidate, 0), ifnull(f.max_capacity, -1), v.schedule_day, v.activity_level from facility_commodities fc, facility_type_id ft, facilities f, vertices v where ft.facility_type = 'processor' and ft.facility_type_id = f.facility_type_id @@ -2183,12 +2191,11 @@ def create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_ and fc.io = 'i' and v.facility_id = f.facility_id and v.storage_vertex = 0 - group by fc.commodity_id, f.facility_id, - ifnull(f.candidate, 0), fc.quantity, v.schedule_day, v.activity_level + group by f.facility_id, ifnull(f.candidate, 0), f.max_capacity, v.schedule_day, v.activity_level ; """ # iterate through processor facilities, one constraint per facility per day - # different copies by subcommodity, summed for constraint as long as same day + # no handling of subcommodities processor_facilities = db_cur.execute(sql) @@ -2197,36 +2204,42 @@ def create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_ for row_a in processor_facilities: # input_commodity_id = row_a[0] - facility_id = row_a[1] - is_candidate = row_a[2] - max_capacity = row_a[3] - day = row_a[4] - daily_activity_level = row_a[5] - - daily_inflow_max_capacity = float(max_capacity) * float(daily_activity_level) - daily_inflow_min_capacity = daily_inflow_max_capacity / 2 - logger.debug( - "processor {}, day {}, capacity min: {} max: {}".format(facility_id, day, daily_inflow_min_capacity, - daily_inflow_max_capacity)) - total_scenario_min_capacity = total_scenario_min_capacity + daily_inflow_min_capacity - flow_in = [] - - # all edges that start in that processor facility, any primary vertex, on that day - so all subcommodities - db_cur2 = main_db_con.cursor() - for row_b in db_cur2.execute("""select edge_id from edges e, vertices v - where e.start_day = {} - and e.d_vertex_id = v.vertex_id - and v.facility_id = {} - and v.storage_vertex = 0 - group by edge_id""".format(day, facility_id)): - input_edge_id = row_b[0] - flow_in.append(flow_var[input_edge_id]) - - logger.debug( - "flow in for capacity constraint on processor facility {} day {}: {}".format(facility_id, day, flow_in)) - prob += lpSum(flow_in) <= daily_inflow_max_capacity * processor_daily_flow_vars[(facility_id, day)], \ - "constraint max flow out of processor facility {}, day {}, flow var {}".format( - facility_id, day, processor_daily_flow_vars[facility_id, day]) + facility_id = row_a[0] + is_candidate = row_a[1] + max_capacity = row_a[2] + day = row_a[3] + daily_activity_level = row_a[4] + + if max_capacity >= 0: + daily_inflow_max_capacity = float(max_capacity) * float(daily_activity_level) + daily_inflow_min_capacity = daily_inflow_max_capacity / 2 + logger.debug( + "processor {}, day {}, input capacity min: {} max: {}".format(facility_id, day, daily_inflow_min_capacity, + daily_inflow_max_capacity)) + total_scenario_min_capacity = total_scenario_min_capacity + daily_inflow_min_capacity + flow_in = [] + + # all edges that end in that processor facility primary vertex, on that day + db_cur2 = main_db_con.cursor() + for row_b in db_cur2.execute("""select edge_id from edges e, vertices v + where e.start_day = {} + and e.d_vertex_id = v.vertex_id + and v.facility_id = {} + and v.storage_vertex = 0 + group by edge_id""".format(day, facility_id)): + input_edge_id = row_b[0] + flow_in.append(flow_var[input_edge_id]) + + logger.debug( + "flow in for capacity constraint on processor facility {} day {}: {}".format(facility_id, day, flow_in)) + prob += lpSum(flow_in) <= daily_inflow_max_capacity * processor_daily_flow_vars[(facility_id, day)], \ + "constraint max flow into processor facility {}, day {}, flow var {}".format( + facility_id, day, processor_daily_flow_vars[facility_id, day]) + + prob += lpSum(flow_in) >= daily_inflow_min_capacity * processor_daily_flow_vars[ + (facility_id, day)], "constraint min flow into processor {}, day {}".format(facility_id, day) + # else: + # pdb.set_trace() if is_candidate == 1: # forces processor build var to be correct @@ -2235,9 +2248,6 @@ def create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_ (facility_id, day)], "constraint forces processor build var to be correct {}, {}".format( facility_id, processor_build_vars[facility_id]) - prob += lpSum(flow_in) >= daily_inflow_min_capacity * processor_daily_flow_vars[ - (facility_id, day)], "constraint min flow into processor {}, day {}".format(facility_id, day) - logger.debug("FINISHED: create_constraint_daily_processor_capacity") return prob @@ -2258,7 +2268,7 @@ def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow # dividing by "required quantity" functionally converts all commodities to the same "processor-specific units" # processor primary vertices with input commodity and quantity needed to produce specified output quantities - # 2 sets of contraints; one for the primary processor vertex to cover total flow in and out + # 2 sets of constraints; one for the primary processor vertex to cover total flow in and out # one for each input and output commodity (sum over sources) to ensure its ratio matches facility_commodities # the current construction of this method is dependent on having only one input commodity type per processor @@ -2386,7 +2396,7 @@ def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow constrained_input_flow_vars = set([]) # pdb.set_trace() - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): #value is a dictionary with commodity & source as keys # set up a dictionary that will be filled with input lists to check ratio against compare_input_dict = {} @@ -2398,7 +2408,7 @@ def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow in_quantity = 0 in_commodity_id = 0 in_source_facility_id = -1 - for ikey, ivalue in flow_in_lists[vertex_id].iteritems(): + for ikey, ivalue in iteritems(flow_in_lists[vertex_id]): in_commodity_id = ikey[0] in_quantity = ikey[1] in_source = ikey[2] @@ -2415,7 +2425,7 @@ def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow # value is a dict - we loop once here for each output commodity and source at the vertex - for key2, value2 in value.iteritems(): + for key2, value2 in iteritems(value): out_commodity_id = key2[0] out_quantity = key2[1] out_source = key2[2] @@ -2457,7 +2467,7 @@ def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow for flow_var in compare_input_list: constrained_input_flow_vars.add(flow_var) else: - for k, v in compare_input_dict_commod.iteritems(): + for k, v in iteritems(compare_input_dict_commod): # pdb.set_trace() # as long as the input source doesn't match an output that needs to inherit compare_input_list = list(v) @@ -2473,9 +2483,9 @@ def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow for flow_var in compare_input_list: constrained_input_flow_vars.add(flow_var) - for key, value in flow_in_lists.iteritems(): + for key, value in iteritems(flow_in_lists): vertex_id = key - for key2, value2 in value.iteritems(): + for key2, value2 in iteritems(value): commodity_id = key2[0] # out_quantity = key2[1] source = key2[2] @@ -2576,7 +2586,7 @@ def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, flow_var[edge_id]) logger.info("adding processor excess variabless to conservation of flow") - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): vertex_id = key[0] # commodity_id = key[1] # day = key[2] @@ -2584,7 +2594,7 @@ def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, if facility_type == 'processor': flow_out_lists.setdefault(key, []).append(processor_excess_vars[vertex_id]) - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): if key in flow_in_lists: prob += lpSum(flow_out_lists[key]) == lpSum( @@ -2597,7 +2607,7 @@ def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, key[2]) storage_vertex_constraint_counter = storage_vertex_constraint_counter + 1 - for key, value in flow_in_lists.iteritems(): + for key, value in iteritems(flow_in_lists): if key not in flow_out_lists: prob += lpSum(flow_in_lists[key]) == lpSum( @@ -2686,7 +2696,7 @@ def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, flow_out_lists.setdefault((node_id, intermodal, source_facility_id, constraint_day, commodity_id), []).append(flow_var[edge_id]) - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): node_id = key[0] # intermodal_flag = key[1] source_facility_id = key[2] @@ -2709,7 +2719,7 @@ def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, " mode {}".format(node_id, source_facility_id, commodity_id, day, node_mode) node_constraint_counter = node_constraint_counter + 1 - for key, value in flow_in_lists.iteritems(): + for key, value in iteritems(flow_in_lists): node_id = key[0] # intermodal_flag = key[1] source_facility_id = key[2] @@ -2786,7 +2796,7 @@ def create_constraint_max_route_capacity(logger, the_scenario, prob, flow_var): flow_lists.setdefault((route_id, aggregate_storage_capac, storage_route_name, start_day), []).append( flow_var[edge_id]) - for key, flow in flow_lists.iteritems(): + for key, flow in iteritems(flow_lists): prob += lpSum(flow) <= key[1], "constraint max flow on storage route {} named {} for day {}".format(key[0], key[2], key[3]) @@ -2861,7 +2871,7 @@ def create_constraint_max_route_capacity(logger, the_scenario, prob, flow_var): flow_lists.setdefault((nx_edge_id, converted_capacity, start_day), []).append(flow_var[edge_id]) - for key, flow in flow_lists.iteritems(): + for key, flow in iteritems(flow_lists): prob += lpSum(flow) <= key[1], "constraint max flow on nx edge {} for day {}".format(key[0], key[2]) logger.debug("route_capacity constraints created for all non-pipeline transport routes") @@ -2875,7 +2885,7 @@ def create_constraint_max_route_capacity(logger, the_scenario, prob, flow_var): def create_constraint_pipeline_capacity(logger, the_scenario, prob, flow_var): logger.debug("STARTING: create_constraint_pipeline_capacity") - logger.debug("Length of flow_var: {}".format(len(flow_var.items()))) + logger.debug("Length of flow_var: {}".format(len(list(flow_var.items())))) logger.info("modes with background flow turned on: {}".format(the_scenario.backgroundFlowModes)) logger.info("minimum available capacity floor set at: {}".format(the_scenario.minCapacityLevel)) @@ -2942,7 +2952,7 @@ def create_constraint_pipeline_capacity(logger, the_scenario, prob, flow_var): # add flow from all relevant edges, for one start; may be multiple tariffs flow_lists.setdefault((link_id, link_use_capacity, start_day, edge_mode), []).append(flow_var[edge_id]) - for key, flow in flow_lists.iteritems(): + for key, flow in iteritems(flow_lists): prob += lpSum(flow) <= key[1], "constraint max flow on pipeline link {} for mode {} for day {}".format( key[0], key[3], key[2]) @@ -2986,8 +2996,9 @@ def setup_pulp_problem(the_scenario, logger): # This constraint is being excluded because 1) it is not used in current scenarios and 2) it is not supported by # this version - it conflicts with the change permitting multiple inputs - # prob = create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_vars, processor_build_vars, - # processor_vertex_flow_vars) + # adding back 12/2020 + prob = create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_vars, processor_build_vars, + processor_vertex_flow_vars) prob = create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow_vars) @@ -3025,7 +3036,7 @@ def solve_pulp_problem(prob_final, the_scenario, logger): # status = prob_final.solve (PULP_CBC_CMD(maxSeconds = i_max_sec, fracGap = d_opt_gap, msg=1)) # CBC time limit and relative optimality gap tolerance status = prob_final.solve(PULP_CBC_CMD(msg=1)) # CBC time limit and relative optimality gap tolerance - print('Completion code: %d; Solution status: %s; Best obj value found: %s' % ( + logger.info('Completion code: %d; Solution status: %s; Best obj value found: %s' % ( status, LpStatus[prob_final.status], value(prob_final.objective))) dup2(orig_std_out, 1) @@ -3083,11 +3094,14 @@ def save_pulp_solution(the_scenario, prob, logger, zero_threshold=0.00001): # insert the optimal data into the DB # ------------------------------------- for v in prob.variables(): - if v.varValue > zero_threshold: # eliminates values too close to zero - sql = """insert into optimal_solution (variable_name, variable_value) values ("{}", {});""".format( - v.name, float(v.varValue)) - db_con.execute(sql) - non_zero_variable_count = non_zero_variable_count + 1 + if v.varValue is None: + logger.debug("Variable value is none: " + str(v.name)) + else: + if v.varValue > zero_threshold: # eliminates values too close to zero + sql = """insert into optimal_solution (variable_name, variable_value) values ("{}", {});""".format( + v.name, float(v.varValue)) + db_con.execute(sql) + non_zero_variable_count = non_zero_variable_count + 1 # query the optimal_solution table in the DB for each variable we care about # ---------------------------------------------------------------------------- diff --git a/program/ftot_pulp_candidate_generation.py b/program/ftot_pulp_candidate_generation.py index e8e2ec5..1826187 100644 --- a/program/ftot_pulp_candidate_generation.py +++ b/program/ftot_pulp_candidate_generation.py @@ -11,11 +11,11 @@ import re import sqlite3 from collections import defaultdict +from six import iteritems from pulp import * import ftot_supporting -import test_aftot_pulp from ftot_supporting import get_total_runtime_string # =================== constants============= @@ -27,8 +27,6 @@ candidate_processing_facilities = [] -default_sched = test_aftot_pulp.schedule_full_availability() -last_day_sched = test_aftot_pulp.schedule_last_day_only() storage_cost_1 = 0.01 storage_cost_2 = 0.05 facility_onsite_storage_max = 10000000000 @@ -68,7 +66,7 @@ def check_max_transport_distance_for_OC_step(the_scenario, logger): sql = "SELECT COUNT(*) FROM commodities WHERE max_transport_distance IS NOT NULL;" db_cur = main_db_con.execute(sql) count_data = db_cur.fetchone()[0] - print count_data + print (count_data) if count_data == 0: logger.error("running the OC step requires that at least commodity from the RMPs have a max transport " "distance specified in the input CSV file.") @@ -1349,7 +1347,7 @@ def create_constraint_unmet_demand(logger, the_scenario, prob, flow_var, unmet_d def create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, flow_var): logger.debug("STARTING: create_constraint_max_flow_out_of_supply_vertex") - logger.debug("Length of flow_var: {}".format(len(flow_var.items()))) + logger.debug("Length of flow_var: {}".format(len(list(flow_var.items())))) # force flow out of origins to be <= supply # for each primary (non-storage) supply vertex @@ -1386,88 +1384,6 @@ def create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, # =============================================================================== -def create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_var, processor_build_vars, - processor_daily_flow_vars): - logger.debug("STARTING: create_constraint_daily_processor_capacity") - # primary vertices only - # flow out of a vertex <= nameplate capacity / 365, true for every day and totaled output commodities - - # get primary processor vertex and its output quantity - total_scenario_min_capacity = 0 - - with sqlite3.connect(the_scenario.main_db) as main_db_con: - db_cur = main_db_con.cursor() - sql = """select fc.commodity_id, f.facility_id, - ifnull(f.candidate, 0), fc.quantity, v.schedule_day, v.activity_level - from facility_commodities fc, facility_type_id ft, facilities f, vertices v - where ft.facility_type = 'processor' - and ft.facility_type_id = f.facility_type_id - and f.facility_id = fc.facility_id - and fc.io = 'i' - and v.facility_id = f.facility_id - and v.storage_vertex = 0 - group by fc.commodity_id, f.facility_id, - ifnull(f.candidate, 0), fc.quantity, v.schedule_day, v.activity_level - ; - """ - # iterate through processor facilities, one constraint per facility per day - # different copies by source facility, summed for constraint as long as same day - - processor_facilities = db_cur.execute(sql) - - processor_facilities = processor_facilities.fetchall() - - for row_a in processor_facilities: - - # output_commodity_id = row_a[0] - facility_id = row_a[1] - is_candidate = row_a[2] - max_capacity = row_a[3] - day = row_a[4] - daily_activity_level = row_a[5] - - daily_outflow_max_capacity = float(max_capacity) * float(daily_activity_level) - daily_outflow_min_capacity = daily_outflow_max_capacity / 10 - logger.debug( - "processor {}, day {}, capacity min: {} max: {}".format(facility_id, day, daily_outflow_min_capacity, - daily_outflow_max_capacity)) - total_scenario_min_capacity = total_scenario_min_capacity + daily_outflow_min_capacity - flow_out = [] - - # all edges that start in that processor facility, any primary vertex, on that day - so all subcommodities - db_cur2 = main_db_con.cursor() - for row_b in db_cur2.execute("""select edge_id from edges e, vertices v - where e.start_day = {} - and e.o_vertex_id = v.vertex_id - and v.facility_id = {} - and v.storage_vertex = 0 - group by edge_id""".format(day, facility_id)): - output_edge_id = row_b[0] - flow_out.append(flow_var[output_edge_id]) - - logger.debug("flow out for capacity constraint on processor facility {} day {}: {}".format(facility_id, day, - flow_out)) - prob += lpSum(flow_out) <= daily_outflow_max_capacity * processor_daily_flow_vars[ - (facility_id, day)], "constraint max flow out of processor facility {}, day {}, flow var {}".format( - facility_id, day, processor_daily_flow_vars[facility_id, day]) - - if is_candidate == 1: - # forces processor build var to be correct - if there is flow througha candidate processor then is - # has to be built - prob += processor_build_vars[facility_id] >= processor_daily_flow_vars[ - (facility_id, day)], "constraint forces processor build var to be correct {}, {}".format( - facility_id, processor_build_vars[facility_id]) - - prob += lpSum(flow_out) >= daily_outflow_min_capacity * processor_daily_flow_vars[ - (facility_id, day)], "constraint min flow out of processor {}, day {}".format(facility_id, day) - - logger.debug("FINISHED: create_constraint_daily_processor_capacity") - return prob - - -# =============================================================================== - - def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow_var): logger.info("STARTING: create_primary_processor_vertex_constraints - capacity and conservation of flow") # for all of these vertices, flow in always == flow out @@ -1579,14 +1495,14 @@ def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow # 1---------------------------------------------------------------------- - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): vertex_id = key zero_in = False if vertex_id in flow_in_lists: compare_input_list = [] in_quantity = 0 in_commodity_id = 0 - for ikey, ivalue in flow_in_lists[vertex_id].iteritems(): + for ikey, ivalue in iteritems(flow_in_lists[vertex_id]): in_commodity_id = ikey[0] in_quantity = ikey[1] # edge_list = value2 @@ -1595,7 +1511,7 @@ def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow zero_in = True # value is a dict - we loop once here for each output commodity at the vertex - for key2, value2 in value.iteritems(): + for key2, value2 in iteritems(value): out_commodity_id = key2[0] out_quantity = key2[1] # edge_list = value2 @@ -1614,13 +1530,13 @@ def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow vertex_id, out_commodity_id, in_commodity_id) # 2---------------------------------------------------------------------- - for key, value in flow_in_lists.iteritems(): + for key, value in iteritems(flow_in_lists): vertex_id = key zero_out = False if vertex_id in flow_out_lists: compare_output_list = [] out_quantity = 0 - for okey, ovalue in flow_out_lists[vertex_id].iteritems(): + for okey, ovalue in iteritems(flow_out_lists[vertex_id]): out_commodity_id = okey[0] out_quantity = okey[1] # edge_list = value2 @@ -1629,7 +1545,7 @@ def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow zero_out = True # value is a dict - we loop once here for each input commodity at the vertex - for key2, value2 in value.iteritems(): + for key2, value2 in iteritems(value): in_commodity_id = key2[0] in_quantity = key2[1] # edge_list = value2 @@ -1730,13 +1646,13 @@ def create_constraint_conservation_of_flow_storage_vertices(logger, the_scenario flow_var[edge_id]) logger.info("adding processor excess variables to conservation of flow") - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): vertex_id = key[0] facility_type = key[3] if facility_type == 'processor': flow_out_lists.setdefault(key, []).append(processor_excess_vars[vertex_id]) - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): if key in flow_in_lists: prob += lpSum(flow_out_lists[key]) == lpSum( @@ -1749,7 +1665,7 @@ def create_constraint_conservation_of_flow_storage_vertices(logger, the_scenario key[2]) storage_vertex_constraint_counter = storage_vertex_constraint_counter + 1 - for key, value in flow_in_lists.iteritems(): + for key, value in iteritems(flow_in_lists): if key not in flow_out_lists: prob += lpSum(flow_in_lists[key]) == lpSum( @@ -1916,7 +1832,7 @@ def create_constraint_conservation_of_flow_endcap_nodes(logger, the_scenario, pr endcap_dict = {} - for node, value in endcap_ref.iteritems(): + for node, value in iteritems(endcap_ref): # add output commodities to endcap_ref[node][3] # endcap_ref[node][2] is a list of source facilities this endcap matches for the input commodity process_id = value[0] @@ -1924,7 +1840,7 @@ def create_constraint_conservation_of_flow_endcap_nodes(logger, the_scenario, pr endcap_ref[node][3] = process_outputs_dict[process_id] # if this node has at least one edge flowing out - for key, value in flow_in_lists.iteritems(): + for key, value in iteritems(flow_in_lists): if node_constraint_counter < 50000: if node_constraint_counter % 5000 == 0: @@ -2009,7 +1925,7 @@ def create_constraint_conservation_of_flow_endcap_nodes(logger, the_scenario, pr # now we have a dict of allowed outputs for this input; construct keys for matching flows # allow cross-modal flow?no, these are not candidates yet, want to use exists routes - for o_commodity_id, quant in outputs_dict.iteritems(): + for o_commodity_id, quant in iteritems(outputs_dict): # creating one constraint per output commodity, per endcap node node_constraint_counter = node_constraint_counter + 1 output_source_facility_id = 0 @@ -2090,7 +2006,7 @@ def create_constraint_conservation_of_flow_endcap_nodes(logger, the_scenario, pr node_id, source_facility_id, commodity_id, day, node_mode) node_constraint_counter = node_constraint_counter + 1 - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): node_id = key[0] source_facility_id = key[2] day = key[3] @@ -2125,7 +2041,7 @@ def create_constraint_conservation_of_flow_endcap_nodes(logger, the_scenario, pr def create_constraint_pipeline_capacity(logger, the_scenario, prob, flow_var): logger.debug("STARTING: create_constraint_pipeline_capacity") - logger.debug("Length of flow_var: {}".format(len(flow_var.items()))) + logger.debug("Length of flow_var: {}".format(len(list(flow_var.items())))) logger.info("modes with background flow turned on: {}".format(the_scenario.backgroundFlowModes)) logger.info("minimum available capacity floor set at: {}".format(the_scenario.minCapacityLevel)) @@ -2192,7 +2108,7 @@ def create_constraint_pipeline_capacity(logger, the_scenario, prob, flow_var): # add flow from all relevant edges, for one start; may be multiple tariffs flow_lists.setdefault((link_id, link_use_capacity, start_day, edge_mode), []).append(flow_var[edge_id]) - for key, flow in flow_lists.iteritems(): + for key, flow in iteritems(flow_lists): prob += lpSum(flow) <= key[1], "constraint max flow on pipeline link {} for mode {} for day {}".format( key[0], key[3], key[2]) @@ -2237,6 +2153,8 @@ def setup_pulp_problem_candidate_generation(the_scenario, logger): prob = create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, flow_vars) + from ftot_pulp import create_constraint_daily_processor_capacity + logger.debug("----- Using create_constraint_daily_processor_capacity method imported from ftot_pulp ------") prob = create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_vars, processor_build_vars, processor_vertex_flow_vars) diff --git a/program/ftot_pulp_sourcing.py b/program/ftot_pulp_sourcing.py index 4b3e6f9..649be5a 100644 --- a/program/ftot_pulp_sourcing.py +++ b/program/ftot_pulp_sourcing.py @@ -10,9 +10,7 @@ import sys import ftot_supporting -import test_aftot_pulp import ftot_pulp -import xtot_objects import sqlite3 import re import pdb @@ -27,6 +25,7 @@ from pulp import * from ftot_supporting import get_total_runtime_string #import ftot_supporting_gis +from six import iteritems #=================== constants============= storage = 1 @@ -40,8 +39,6 @@ candidate_processing_facilities = [] -default_sched = test_aftot_pulp.schedule_full_availability() -last_day_sched = test_aftot_pulp.schedule_last_day_only() storage_cost_1 = 0.01 storage_cost_2 = 0.05 facility_onsite_storage_max = 10000000000 @@ -181,7 +178,7 @@ def source_as_subcommodity_setup(the_scenario, logger): return #=============================================================================== -def generate_all_vertices_from_optimal_for_sasc(the_scenario, logger): +def generate_all_vertices_from_optimal_for_sasc(the_scenario, schedule_dict, schedule_length, logger): logger.info("START: generate_all_vertices_from_optimal_for_sasc") #this should be run instead of generate_all_vertices_table, not in addition @@ -191,7 +188,6 @@ def generate_all_vertices_from_optimal_for_sasc(the_scenario, logger): storage_availability = 1 - logger.debug("TODO - facilities do not yet have schedule strings, but we need this for vertex creation - dummy data from test_aftot_pulp.py currently") with sqlite3.connect(the_scenario.main_db) as main_db_con: main_db_con.executescript( """ @@ -231,7 +227,7 @@ def generate_all_vertices_from_optimal_for_sasc(the_scenario, logger): # facility_type can be "raw_material_producer", "ultimate_destination","processor"; get id from original vertices table for sasc # use UNION (? check for sqlite) to get a non-repeat list of facilities - for row_a in db_cur.execute("""select f.facility_id, facility_type, facility_name, location_id, f.facility_type_id + for row_a in db_cur.execute("""select f.facility_id, facility_type, facility_name, location_id, f.facility_type_id, schedule_id from facilities f, facility_type_id ft, optimal_variables_prior ov where ignore_facility = '{}' and f.facility_type_id = ft.facility_type_id @@ -249,6 +245,7 @@ def generate_all_vertices_from_optimal_for_sasc(the_scenario, logger): facility_name = row_a[2] facility_location_id = row_a[3] facility_type_id = row_a[4] + schedule_id = row_a[5] if counter % 10000 == 0: logger.info("vertices created for {} facilities of {}".format(counter, total_facilities)) for row_d in db_cur4.execute("select count(distinct vertex_id) from vertices;"): @@ -285,33 +282,29 @@ def generate_all_vertices_from_optimal_for_sasc(the_scenario, logger): commodity_name = row_b[6] subcommodity_id = row_b[7] source_facility_id = row_b[8] - day = default_sched.first_day - stop_day = default_sched.last_day - availability = default_sched.availability #vertices for generic demand type, or any subtype specified by the destination - while day <= stop_day: + for day_before, availability in enumerate(schedule_dict[schedule_id]): if io == 'i': main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, activity_level, storage_vertex, subcommodity_id, source_facility_id) values ({}, {}, {}, '{}', {}, {}, {}, {}, {},{} - );""".format(facility_location_id, facility_id, facility_type_id, facility_name, day, id_for_mult_commodities, availability[day], primary, + );""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, id_for_mult_commodities, availability, primary, subcommodity_id, source_facility_id)) main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, activity_level, storage_vertex, demand, subcommodity_id, source_facility_id) values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, - {}, {});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day, commodity_id, storage_availability, storage, quantity, + {}, {});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, commodity_id, storage_availability, storage, quantity, subcommodity_id, source_facility_id)) else: if commodity_name != 'total_fuel': main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, activity_level, storage_vertex, supply, subcommodity_id, source_facility_id) - values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, {}, {});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day, commodity_id, storage_availability, storage, quantity, subcommodity_id, source_facility_id)) - day = day + 1 + values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, {}, {});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, commodity_id, storage_availability, storage, quantity, subcommodity_id, source_facility_id)) elif facility_type == "raw_material_producer": #raw material producer @@ -331,23 +324,19 @@ def generate_all_vertices_from_optimal_for_sasc(the_scenario, logger): total_potential_production[commodity_id] = total_potential_production[commodity_id] + quantity else: total_potential_production[commodity_id] = quantity - day = default_sched.first_day - stop_day = default_sched.last_day - availability = default_sched.availability - while day <= stop_day: + for day_before, availability in enumerate(schedule_dict[schedule_id]): main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, activity_level, storage_vertex, supply, subcommodity_id, source_facility_id) values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, - {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day, commodity_id, availability[day], primary, quantity, + {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, commodity_id, availability, primary, quantity, subcommodity_id, source_facility_id)) main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, activity_level, storage_vertex, supply, subcommodity_id, source_facility_id) values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, - {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day, commodity_id, storage_availability, storage, quantity, + {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, commodity_id, storage_availability, storage, quantity, subcommodity_id, source_facility_id)) - day = day + 1 elif facility_type == "storage": #storage facility # handle as such, exploding by time and commodity @@ -360,17 +349,13 @@ def generate_all_vertices_from_optimal_for_sasc(the_scenario, logger): units = row_b[2] subcommodity_id = row_b[3] source_facility_id = row_b[4] - day = default_sched.first_day - stop_day = default_sched.last_day - availability = default_sched.availability - while day <= stop_day: + for day_before, availability in enumerate(schedule_dict[schedule_id]): main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, storage_vertex, subcommodity_id, source_facility_id) values ({}, {}, {}, '{}', {}, {}, {}, - {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day, commodity_id, storage, + {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, commodity_id, storage, subcommodity_id, source_facility_id)) - day = day + 1 elif facility_type == "ultimate_destination": #ultimate_destination # handle as such, exploding by time and commodity @@ -388,27 +373,23 @@ def generate_all_vertices_from_optimal_for_sasc(the_scenario, logger): subcommodity_id = row_b[5] source_facility_id = row_b[6] - #this is where the alternate schedule for destination demand can be set - day = default_sched.first_day - stop_day = default_sched.last_day - availability = default_sched.availability #vertices for generic demand type, or any subtype specified by the destination - while day <= stop_day: + for day_before, availability in enumerate(schedule_dict[schedule_id]): main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, activity_level, storage_vertex, demand, udp, subcommodity_id, source_facility_id) values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, {}, - {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day, - commodity_id, availability[day], primary, quantity, the_scenario.unMetDemandPenalty, + {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, availability, primary, quantity, the_scenario.unMetDemandPenalty, subcommodity_id, source_facility_id)) main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, activity_level, storage_vertex, demand, subcommodity_id, source_facility_id) values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, - {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day, commodity_id, storage_availability, storage, quantity, + {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, commodity_id, storage_availability, storage, quantity, subcommodity_id, source_facility_id)) - day = day + 1 + #vertices for other fuel subtypes that match the destination's supertype #if the subtype is in the commodity table, it is produced by some facility (not ignored) in the scenario db_cur3 = main_db_con.cursor() @@ -416,20 +397,19 @@ def generate_all_vertices_from_optimal_for_sasc(the_scenario, logger): where supertype = '{}';""".format(commodity_supertype)): new_commodity_id = row_c[0] new_units= row_c[1] - while day <= stop_day: + for day_before, availability in enumerate(schedule_dict[schedule_id]): main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, activity_level, storage_vertex, demand, udp, subcommodity_id, source_facility_id) values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, {}, - {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day, new_commodity_id, availability[day], primary, quantity, the_scenario.unMetDemandPenalty, + {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, new_commodity_id, availability, primary, quantity, the_scenario.unMetDemandPenalty, subcommodity_id, source_facility_id)) main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, activity_level, storage_vertex, demand, subcommodity_id, source_facility_id) values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, - {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day, new_commodity_id, storage_availability, storage, quantity, + {},{});""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, new_commodity_id, storage_availability, storage, quantity, subcommodity_id, source_facility_id)) - day = day + 1 else: @@ -487,7 +467,7 @@ def add_storage_routes(the_scenario, logger): return #=============================================================================== -def generate_all_edges_from_optimal_for_sasc(the_scenario, logger): +def generate_all_edges_from_optimal_for_sasc(the_scenario, schedule_length, logger): logger.info("START: generate_all_edges_from_optimal_for_sasc") #create edges table #plan to generate start and end days based on nx edge time to traverse and schedule @@ -621,8 +601,8 @@ def generate_all_edges_from_optimal_for_sasc(the_scenario, logger): # for all allowed commodities, as currently defined by link phase of matter - for day in range(default_sched.first_day, default_sched.last_day+1): - if (day+fixed_route_duration <= default_sched.last_day): #if link is traversable in the timeframe + for day in range(1, schedule_length+1): + if (day+fixed_route_duration <= schedule_length): #if link is traversable in the timeframe if (simple_mode != 'pipeline' or tariff_id >= 0): #for allowed commodities for row_c in db_cur3.execute("select commodity_id, sub_id, source_facility_id from subcommodity where phase_of_matter = '{}' group by commodity_id, sub_id, source_facility_id".format(phase_of_matter)): @@ -977,10 +957,12 @@ def pre_setup_pulp_from_optimal(logger, the_scenario): source_as_subcommodity_setup(the_scenario, logger) - generate_all_vertices_from_optimal_for_sasc(the_scenario, logger) + schedule_dict, schedule_length = ftot_pulp.generate_schedules(the_scenario, logger) + + generate_all_vertices_from_optimal_for_sasc(the_scenario, schedule_dict, schedule_length, logger) add_storage_routes(the_scenario, logger) - generate_all_edges_from_optimal_for_sasc(the_scenario, logger) + generate_all_edges_from_optimal_for_sasc(the_scenario, schedule_length, logger) set_edge_capacity_and_volume(the_scenario, logger) logger.info("Edges generated for modes: {}".format(the_scenario.permittedModes)) @@ -1187,7 +1169,7 @@ def create_constraint_unmet_demand(logger, the_scenario, prob, flow_var, unmet_d # flow out of a vertex <= supply of the vertex, true for every day and commodity def create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, flow_var): logger.debug("STARTING: create_constraint_max_flow_out_of_supply_vertex") - logger.debug("Length of flow_var: {}".format(len(flow_var.items()))) + logger.debug("Length of flow_var: {}".format(len(list(flow_var.items())))) #for each primary (non-storage) supply vertex #Assumption - each RMP produces a single commodity - @@ -1221,82 +1203,6 @@ def create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, #=============================================================================== -# primary vertices only -# flow out of a vertex <= nameplate capcity / 365, true for every day and totaled output commodities -#currently this constraint also controls the proportions for processing -def create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_var, processor_build_vars, processor_daily_flow_vars): - logger.debug("STARTING: create_constraint_daily_processor_capacity") - #daily constraint means checking availability daily; total all outflow from primary vertices (multiple processes) that day - #need to get availability/activity level from primary vertex, max capacity from output vertex or facility_commodity - - -### get primary processor vertex and it's output quantity -##new for SASC - need to sum over multiple vertices for this constraint -##key on facility id, sum over subcommodities - total_scenario_min_capacity = 0 - - - with sqlite3.connect(the_scenario.main_db) as main_db_con: - db_cur = main_db_con.cursor() - sql = """select fc.commodity_id, f.facility_id, - ifnull(f.candidate, 0), fc.quantity, v.schedule_day, v.activity_level - from facility_commodities fc, facility_type_id ft, facilities f, vertices v - where ft.facility_type = 'processor' - and ft.facility_type_id = f.facility_type_id - and f.facility_id = fc.facility_id - and fc.io = 'o' - and v.facility_id = f.facility_id - and v.storage_vertex = 0 - group by fc.commodity_id, f.facility_id, - ifnull(f.candidate, 0), fc.quantity, v.schedule_day, v.activity_level - ; - """ - processor_facilities = main_db_con.execute(sql) - #iterate through processor facilities, one constraint per facility per day - #different copies by subcommodity, summed for constraint as long as same day - - processor_facilities = db_cur.execute(sql) - - processor_facilities = processor_facilities.fetchall() - - for row_a in processor_facilities: - - output_commodity_id = row_a[0] - facility_id = row_a[1] - is_candidate = row_a[2] - max_capacity = row_a[3] - day = row_a[4] - daily_activity_level = row_a[5] - - - daily_outflow_max_capacity = float(max_capacity) *float(daily_activity_level) - daily_outflow_min_capacity = daily_outflow_max_capacity/(10) - logger.debug("processor {}, day {}, capacity min: {} max: {}".format(facility_id, day, daily_outflow_min_capacity, daily_outflow_max_capacity)) - total_scenario_min_capacity = total_scenario_min_capacity + daily_outflow_min_capacity - flow_out = [] - - #all edges that start in that processor facility, any primary vertex, on that day - so all subcommodities - db_cur2 = main_db_con.cursor() - for row_b in db_cur2.execute("""select edge_id from edges e, vertices v - where e.start_day = {} - and e.o_vertex_id = v.vertex_id - and v.facility_id = {} - and v.storage_vertex = 0 - group by edge_id""".format(day, facility_id)): - output_edge_id = row_b[0] - flow_out.append(flow_var[output_edge_id]) - - logger.debug("flow out for capacity constraint on processor facility {} day {}: {}".format(facility_id, day, flow_out)) - prob += lpSum(flow_out) <= daily_outflow_max_capacity*processor_daily_flow_vars[(facility_id, day)], "constraint max flow out of processor facility {}, day {}, flow var {}".format(facility_id, day, processor_daily_flow_vars[facility_id, day]) - - if is_candidate == 1: - #forces processor build var to be correct - if there is flow througha candidate processor then is has to be built - prob += processor_build_vars[facility_id] >= processor_daily_flow_vars[(facility_id, day)], "constraint forces processor build var to be correct {}, {}".format(facility_id, processor_build_vars[facility_id]) - - prob += lpSum(flow_out) >= daily_outflow_min_capacity*processor_daily_flow_vars[(facility_id, day)], "constraint min flow out of processor {}, day {}".format(facility_id, day) - - logger.debug("FINISHED: create_constraint_daily_processor_capacity") - return prob # for all of these vertices, flow in always == flow out def create_processor_constraints(logger, the_scenario, prob, flow_var, processor_excess_vars, processor_build_vars, processor_daily_flow_vars): @@ -1398,7 +1304,7 @@ def create_processor_constraints(logger, the_scenario, prob, flow_var, processor #require ratio to match for each output commodityalso do processor capacity constraint here - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): flow_out = value[0] out_quantity = value[1] vertex_id = key[0] @@ -1496,7 +1402,7 @@ def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, flow_out_lists.setdefault((vertex_id, commodity_name, constraint_day, facility_type), []).append(flow_var[edge_id]) logger.info("adding processor excess variabless to conservation of flow") - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): vertex_id = key[0] commodity_name = key[1] day = key[2] @@ -1505,7 +1411,7 @@ def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, flow_out_lists.setdefault(key, []).append(processor_excess_vars[vertex_id]) - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): if key in flow_in_lists: prob += lpSum(flow_out_lists[key]) == lpSum(flow_in_lists[key]), "conservation of flow, vertex {}, commodity {}, day {}".format(key[0], key[1], key[2]) @@ -1514,7 +1420,7 @@ def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, prob += lpSum(flow_out_lists[key]) == lpSum(0), "conservation of flow (zero out), vertex {}, commodity {}, day {}".format(key[0], key[1], key[2]) storage_vertex_constraint_counter = storage_vertex_constraint_counter + 1 - for key, value in flow_in_lists.iteritems(): + for key, value in iteritems(flow_in_lists): if key not in flow_out_lists: prob += lpSum(flow_in_lists[key]) == lpSum(0), "conservation of flow (zero in), vertex {}, commodity {}, day {}".format(key[0], key[1], key[2]) @@ -1588,7 +1494,7 @@ def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, flow_out_lists.setdefault((node_id, intermodal, subcommodity_id, constraint_day), []).append(flow_var[edge_id]) - for key, value in flow_out_lists.iteritems(): + for key, value in iteritems(flow_out_lists): node_id = key[0] intermodal_flag = key[1] subcommodity_id = key[2] @@ -1603,7 +1509,7 @@ def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, prob += lpSum(flow_out_lists[key]) == lpSum(0), "conservation of flow (zero out), nx node {}, subcommodity {}, day {}, mode {}".format(node_id, subcommodity_id, day, node_mode) node_constraint_counter = node_constraint_counter + 1 - for key, value in flow_in_lists.iteritems(): + for key, value in iteritems(flow_in_lists): node_id = key[0] intermodal_flag = key[1] subcommodity_id = key[2] @@ -1628,7 +1534,7 @@ def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, #do storage capacity separately, even though it is also routes def create_constraint_max_route_capacity(logger, the_scenario, prob, flow_var): logger.debug("STARTING: create_constraint_max_route_capacity") - logger.debug("Length of flow_var: {}".format(len(flow_var.items()))) + logger.debug("Length of flow_var: {}".format(len(list(flow_var.items())))) logger.info("modes with background flow turned on: {}".format(the_scenario.backgroundFlowModes)) #min_capacity_level must be a number from 0 to 1, inclusive #min_capacity_level is only relevant when background flows are turned on @@ -1673,7 +1579,7 @@ def create_constraint_max_route_capacity(logger, the_scenario, prob, flow_var): - for key, flow in flow_lists.iteritems(): + for key, flow in iteritems(flow_lists): prob += lpSum(flow) <= key[1], "constraint max flow on storage route {} named {} for day {}".format(key[0], key[2], key[3]) logger.debug("route_capacity constraints created for all storage routes") @@ -1743,7 +1649,7 @@ def create_constraint_max_route_capacity(logger, the_scenario, prob, flow_var): flow_lists.setdefault((nx_edge_id, converted_capacity, start_day), []).append(flow_var[edge_id]) - for key, flow in flow_lists.iteritems(): + for key, flow in iteritems(flow_lists): prob += lpSum(flow) <= key[1], "constraint max flow on nx edge {} for day {}".format(key[0], key[2]) logger.debug("route_capacity constraints created for all non-pipeline transport routes") @@ -1754,7 +1660,7 @@ def create_constraint_max_route_capacity(logger, the_scenario, prob, flow_var): def create_constraint_pipeline_capacity(logger, the_scenario, prob, flow_var): logger.debug("STARTING: create_constraint_pipeline_capacity") - logger.debug("Length of flow_var: {}".format(len(flow_var.items()))) + logger.debug("Length of flow_var: {}".format(len(list(flow_var.items())))) logger.info("modes with background flow turned on: {}".format(the_scenario.backgroundFlowModes)) logger.info("minimum available capacity floor set at: {}".format(the_scenario.minCapacityLevel)) @@ -1817,7 +1723,7 @@ def create_constraint_pipeline_capacity(logger, the_scenario, prob, flow_var): #add flow from all relevant edges, for one start; may be multiple tariffs flow_lists.setdefault((link_id, link_use_capacity, start_day, edge_mode), []).append(flow_var[edge_id]) - for key, flow in flow_lists.iteritems(): + for key, flow in iteritems(flow_lists): prob += lpSum(flow) <= key[1], "constraint max flow on pipeline link {} for mode {} for day {}".format(key[0], key[3], key[2]) logger.debug("pipeline capacity constraints created for all transport routes") @@ -1872,8 +1778,8 @@ def setup_pulp_problem(the_scenario, logger): prob = create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, flow_vars) - logger.info("calling create_constraint_daily_processor_capacity") -#replaced by create_processor_constraints in b2, used again in bsc at least for development as of 9-19 + logger.info("calling create_constraint_daily_processor_capacity from ftot_pulp") + from ftot_pulp import create_constraint_daily_processor_capacity prob = create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_vars, processor_build_vars, processor_vertex_flow_vars) logger.info("calling create_processor_constraints") diff --git a/program/ftot_pulp_weekly.py b/program/ftot_pulp_weekly.py new file mode 100644 index 0000000..3df07ac --- /dev/null +++ b/program/ftot_pulp_weekly.py @@ -0,0 +1,3474 @@ +# --------------------------------------------------------------------------------------------------- +# Name: ftot_pulp +# +# Purpose: PulP optimization - create and run a modified facility location problem. +# Take NetworkX and GIS scenario data as recorded in main.db and convert to a structure of edges, nodes, vertices. +# Create variables for flow over edges, unmet demand, processor use, and candidate processors to build if present +# Solve cost minimization for unmet demand, transportation, and facility build costs +# Constraints ensure compliance with scenario requirements (e.g. max_route_capacity) +# as well as general problem structure (e.g. conservation_of_flow) +# --------------------------------------------------------------------------------------------------- + +import datetime +import pdb +import re +import sqlite3 +import numpy as np +from collections import defaultdict +from six import iteritems + +from pulp import * + +import ftot_supporting +from ftot_supporting import get_total_runtime_string + +# =================== constants============= +storage = 1 +primary = 0 +fixed_schedule_id = 2 +fixed_route_duration = 0 + +THOUSAND_GALLONS_PER_THOUSAND_BARRELS = 42 + +storage_cost_1 = 0.01 +storage_cost_2 = 0.05 +facility_onsite_storage_max = 10000000000 +facility_onsite_storage_min = 0 + +default_max_capacity = 10000000000 +default_min_capacity = 0 + +#==================load output from newly developed Python files===== +# load facility capacity index +facility_cap_noEarthquake = np.load("facility_cap_noEarthquake.npy") +facility_cap = np.load("facility_cap.npy") +# load facility damage state index +facility_DS = np.load("facility_DS.npy") + +# load repair cost array: +repair_costs = np.load("repair_costs.npy") + +# Load edge capacity index +edge_cap = np.load("edge_cap.npy") +# load bridge damage state index +bridge_DS = np.load("bridge_DS.npy") + +# load reair time array: +repair_time_facility = np.load("repair_time_facility.npy") +repair_time_edge = np.load("repair_time_edge.npy") + +# load catalyst replament cost +CatalystReplace_cost = np.load("CatalystReplace_cost.npy") +# GFT facility operation cost +Operation_unitcost = 0.00579 # unit:$/kgal (2021 dollars) +total_final_demand = 4073312.9 + 254401.1 +Inflation_rate = 0.0279 +# daily index +daily_index = float(1)/float(365) +#unmet_demand_yearly = np.zeros((N,plan_horizon)) +#costs_yearly = np.zeros((N,plan_horizon)) +number = np.zeros(len(facility_cap_noEarthquake[:,0,0])) +for i in range(len(number)): + number[i] = i +#============================================ +def o1(the_scenario, logger): + # create vertices, then edges for permitted modes, then set volume & capacity on edges + pre_setup_pulp(logger, the_scenario) + + +def o2(the_scenario, logger): + # create variables, problem to optimize, and constraints + prob = setup_pulp_problem(the_scenario, logger) + prob = solve_pulp_problem(prob, the_scenario, logger) + save_pulp_solution(the_scenario, prob, logger) + record_pulp_solution(the_scenario, logger) + from ftot_supporting import post_optimization + post_optimization(the_scenario, 'o2', logger) + + +# helper function that reads in schedule data and returns dict of Schedule objects +def generate_schedules(the_scenario, logger): + logger.debug("start: generate_schedules") + default_availabilities = {} + day_availabilities = {} + last_day = 1 + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + + schedule_data = db_cur.execute(""" + select schedule_id, + day, + availability + + from schedules""") + + schedule_data = schedule_data.fetchall() + for row_a in schedule_data: + schedule_id = row_a[0] + day = row_a[1] + availability = float(row_a[2]) + + if day == 0: # denotes the default value + default_availabilities[schedule_id] = availability + elif schedule_id in day_availabilities.keys(): + day_availabilities[schedule_id][day] = availability + else: + day_availabilities[schedule_id] = {day: availability} # initialize sub-dic + # last_day is the greatest day specified across ALL schedules to avoid mismatch of length + if day > last_day: + last_day = day + + # make dictionary to store schedule objects + schedule_dict = {} + + # after reading in csv, parse data into dictionary object + for schedule_id in default_availabilities.keys(): + # initialize list of length + schedule_dict[schedule_id] = [default_availabilities[schedule_id] for i in range(last_day)] + # change different days to actual availability instead of the default values + # schedule_id may not be in day_availabilities if schedule has default value on all days + if schedule_id in day_availabilities.keys(): + for day in day_availabilities[schedule_id].keys(): + # schedule_dict[schedule + schedule_dict[schedule_id][day-1] = day_availabilities[schedule_id][day] + + for key in schedule_dict.keys(): + logger.debug("schedule name: " + str(key)) + logger.debug("availability: ") + logger.debug(schedule_dict[key]) + logger.debug("finished: generate_schedules") + + return schedule_dict, last_day + + +def commodity_mode_setup(the_scenario, logger): + + # helper method to read in the csv and make a dict + def make_commodity_mode_dict(the_scenario, logger): + logger.info("START: make_commodity_mode_dict") + # check if path to table exists + if not os.path.exists(the_scenario.commodity_mode_data): + logger.warning("warning: cannot find commodity_mode_data file: {}".format(the_scenario.commodity_mode_data)) + return {} # return empty dict + # otherwise, initialize dict and read through commodity_mode CSV + commodity_mode_dict = {} + with open(the_scenario.commodity_mode_data, 'r') as rf: + line_num = 1 + modes = None # will assign within for loop + for line in rf: + if line_num == 1: + modes = line.rstrip('\n').split(',') + else: + flds = line.rstrip('\n').split(',') + commodity_name = flds[0] + allowed = flds[1:] + commodity_mode_dict[commodity_name] = dict(zip(modes[1:], allowed)) + # now do a check + for mode in modes[1:]: + if commodity_mode_dict[commodity_name][mode] in ['Y', 'N']: + logger.info("Commodity: {}, Mode: {}, Allowed: {}".format(commodity_name, mode, + commodity_mode_dict[commodity_name][mode])) + else: + # if val isn't Y or N, remove the key from the dict + del commodity_mode_dict[commodity_name][mode] + logger.info( + "improper or no value in Commodity_Mode_Data csv for commodity: {} and mode: {}".\ + format(commodity_name, mode)) + logger.info("default value will be used for commodity: {} and mode: {}".\ + format(commodity_name, mode)) + + line_num += 1 + logger.info("FINISHED: make_commodity_mode_dict") + return commodity_mode_dict + + logger.info("START: commodity_mode_setup") + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + main_db_con.executescript(""" + drop table if exists commodity_mode; + + create table commodity_mode( + mode text, + commodity_id text, + commodity_phase text, + allowed_yn text, + CONSTRAINT unique_commodity_and_mode UNIQUE(commodity_id, mode)) + ;""") + + commod = main_db_con.execute("select commodity_name, commodity_id, phase_of_matter from commodities;") + commod = commod.fetchall() + commodities = {} + for row in commod: + commodity_name = row[0] + commodity_id = row[1] + phase_of_matter = row[2] + commodities[commodity_name] = (commodity_id, phase_of_matter) + + # read in from csv file and insert those entries first, using commodity id + # then, fill in for all other unspecified commodities and modes, pipeline liquid only + commodity_mode_dict = make_commodity_mode_dict(the_scenario, logger) + for mode in the_scenario.permittedModes: + for k, v in iteritems(commodities): + + commodity_name = k + commodity_id = v[0] + phase_of_matter = v[1] + # check commodity mode permissions for all modes. Apply resiction is specified, otherwise default to + # allowed if not specified. + if commodity_name in commodity_mode_dict and mode in commodity_mode_dict[commodity_name]: + allowed = commodity_mode_dict[commodity_name][mode] + else: + allowed = 'Y' + # pipeline is a special case. so double check if it is explicitly allowed, otherwise override to 'N'. + # restrict solids on pipeline no matter what. + if mode.partition('_')[0] == 'pipeline': + # 8/26/19 -- MNP -- note: that mode is the long name, and the dict is the short name + if mode == 'pipeline_crude_trf_rts': + short_name_mode = 'pipeline_crude' + elif mode == 'pipeline_prod_trf_rts': + short_name_mode = 'pipeline_prod' + else: logger.warning("a pipeline was specified that is not supported") + # restrict pipeline if its not explicitly allowed or if solid + if commodity_name in commodity_mode_dict and short_name_mode in commodity_mode_dict[commodity_name]: + allowed = commodity_mode_dict[commodity_name][short_name_mode] + else: + allowed = 'N' + if phase_of_matter != 'liquid': + allowed = 'N' + + main_db_con.execute(""" + insert or ignore into commodity_mode + (mode, commodity_id, commodity_phase, allowed_yn) + VALUES + ('{}',{},'{}','{}') + ; + """.format(mode, commodity_id, phase_of_matter, allowed)) + + return +# =============================================================================== + + +def source_tracking_setup(the_scenario, logger): + logger.info("START: source_tracking_setup") + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + max_inputs = main_db_con.execute("""select max(inputs) from + (select count(fc.commodity_id) inputs + from facility_commodities fc, facility_type_id ft + where ft.facility_type = 'processor' + and fc.io = 'i' + group by fc.facility_id) + ;""") + + for row in max_inputs: + if row[0] == None: + max_inputs_to_a_processor = 0 + else: + max_inputs_to_a_processor = int(row[0]) + if max_inputs_to_a_processor > 1: + logger.warning("error, this version of the optimization step only functions correctly with a single input" + " commodity type per processor") + + main_db_con.executescript(""" + + insert or ignore into commodities(commodity_name) values ('multicommodity'); + + drop table if exists source_commodity_ref + ; + + create table source_commodity_ref(id INTEGER PRIMARY KEY, + source_facility_id integer, + source_facility_name text, + source_facility_type_id integer, --lets us differentiate spiderweb from processor + commodity_id integer, + commodity_name text, + units text, + phase_of_matter text, + max_transport_distance numeric, + max_transport_distance_flag text, + share_max_transport_distance text, + CONSTRAINT unique_source_and_name UNIQUE(commodity_id, source_facility_id)) + ; + + insert or ignore into source_commodity_ref ( + source_facility_id, + source_facility_name, + source_facility_type_id, + commodity_id, + commodity_name, + units, + phase_of_matter, + max_transport_distance, + max_transport_distance_flag, + share_max_transport_distance) + select + f.facility_id, + f.facility_name, + f.facility_type_id, + c.commodity_id, + c.commodity_name, + c.units, + c.phase_of_matter, + (case when c.max_transport_distance is not null then + c.max_transport_distance else Null end) max_transport_distance, + (case when c.max_transport_distance is not null then 'Y' else 'N' end) max_transport_distance_flag, + (case when ifnull(c.share_max_transport_distance, 'N') = 'Y' then 'Y' else 'N' end) share_max_transport_distance + + from commodities c, facilities f, facility_commodities fc + where f.facility_id = fc.facility_id + and f.ignore_facility = 'false' + and fc.commodity_id = c.commodity_id + and fc.io = 'o' + and ifnull(c.share_max_transport_distance, 'N') != 'Y' + ; + + insert or ignore into source_commodity_ref ( + source_facility_id, + source_facility_name, + source_facility_type_id, + commodity_id, + commodity_name, + units, + phase_of_matter, + max_transport_distance, + max_transport_distance_flag, + share_max_transport_distance) + select + sc.source_facility_id, + sc.source_facility_name, + sc.source_facility_type_id, + o.commodity_id, + c.commodity_name, + c.units, + c.phase_of_matter, + sc.max_transport_distance, + sc.max_transport_distance_flag, + o.share_max_transport_distance + from source_commodity_ref sc, facility_commodities i, facility_commodities o, commodities c + where o.share_max_transport_distance = 'Y' + and sc.commodity_id = i.commodity_id + and o.facility_id = i.facility_id + and o.io = 'o' + and i.io = 'i' + and o.commodity_id = c.commodity_id + ; + """ + ) + + return + + +# =============================================================================== + + +def generate_all_vertices(the_scenario, schedule_dict, schedule_length, logger): + logger.info("START: generate_all_vertices table") + i = np.load("scenario_num.npy") + t = np.load("time_horizon.npy") + j = np.load("earthquake_week.npy") + logger.info("scenario {}".format(i)) + logger.info("time horizon {}".format(t)) + logger.info("earthquake week {}".format(j)) + total_potential_production = {} + multi_commodity_name = "multicommodity" + + storage_availability = 1 + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + logger.debug("create the vertices table") + # create the vertices table + main_db_con.executescript(""" + drop table if exists vertices + ; + + create table if not exists vertices ( + vertex_id INTEGER PRIMARY KEY, location_id, + facility_id integer, facility_name text, facility_type_id integer, schedule_day integer, + commodity_id integer, activity_level numeric, storage_vertex binary, + udp numeric, supply numeric, demand numeric, + source_facility_id integer, + iob text, --allows input, output, or both + CONSTRAINT unique_vertex UNIQUE(facility_id, schedule_day, commodity_id, source_facility_id, storage_vertex)) + ;""") + + # create indexes for the networkx nodes and links tables + logger.info("create an index for the networkx nodes and links tables") + main_db_con.executescript(""" + CREATE INDEX IF NOT EXISTS node_index ON networkx_nodes (node_id, location_id) + ; + + create index if not exists nx_edge_index on + networkx_edges(from_node_id, to_node_id, + artificial, mode_source, mode_source_OID, + miles, route_cost_scaling, capacity) + ; + """) + + # -------------------------------- + + db_cur = main_db_con.cursor() + # nested cursor + db_cur4 = main_db_con.cursor() + counter = 0 + total_facilities = 0 + + for row in db_cur.execute("select count(distinct facility_id) from facilities;"): + total_facilities = row[0] + + # create vertices for each non-ignored facility facility + # facility_type can be "raw_material_producer", "ultimate_destination","processor"; + # get id from facility_type_id table + # any other facility types are not currently handled + + facility_data = db_cur.execute(""" + select facility_id, + facility_type, + facility_name, + location_id, + f.facility_type_id, + schedule_id + + from facilities f, facility_type_id ft + where ignore_facility = '{}' + and f.facility_type_id = ft.facility_type_id; + """.format('false')) + facility_data = facility_data.fetchall() + for row_a in facility_data: + + db_cur2 = main_db_con.cursor() + facility_id = row_a[0] + facility_type = row_a[1] + facility_name = row_a[2] + facility_location_id = row_a[3] + facility_type_id = row_a[4] + schedule_id = row_a[5] + if counter % 10000 == 1: + logger.info("vertices created for {} facilities of {}".format(counter, total_facilities)) + for row_d in db_cur4.execute("select count(distinct vertex_id) from vertices;"): + logger.info('{} vertices created'.format(row_d[0])) + counter = counter + 1 + + if facility_type == "processor": + # actual processors - will deal with endcaps in edges section + + # create processor vertices for any commodities that do not inherit max transport distance + proc_data = db_cur2.execute("""select fc.commodity_id, + ifnull(fc.quantity, 0), + fc.units, + ifnull(c.supertype, c.commodity_name), + fc.io, + mc.commodity_id, + c.commodity_name, + ifnull(s.source_facility_id, 0) + from facility_commodities fc, commodities c, commodities mc + left outer join source_commodity_ref s + on (fc.commodity_id = s.commodity_id and s.max_transport_distance_flag = 'Y') + where fc.facility_id = {} + and fc.commodity_id = c.commodity_id + and mc.commodity_name = '{}';""".format(facility_id, multi_commodity_name)) + + proc_data = proc_data.fetchall() + # entry for each incoming commodity and its potential sources + # each outgoing commodity with this processor as their source IF there is a max commod distance + for row_b in proc_data: + + commodity_id = row_b[0] + #=========================================Modification==================================================================== + # Following part is modified to include time-varying facility capacity in the aftermath of earthquake + temp_day = int(7*j) + if temp_day < repair_time_facility[float(facility_id)-1][t+1][i]: + quantity = facility_cap[float(facility_id)-1][t+1][i]*row_b[1]*daily_index + else: + quantity = facility_cap_noEarthquake[float(facility_id)-1][t+1][i]*row_b[1]*daily_index + #============================================================================================= + io = row_b[4] + id_for_mult_commodities = row_b[5] + commodity_name = row_b[6] + source_facility_id = row_b[7] + new_source_facility_id = facility_id + + # vertices for generic demand type, or any subtype specified by the destination + for day_before, availability in enumerate(schedule_dict[schedule_id]): + if io == 'i': + main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, + facility_type_id, facility_name, schedule_day, commodity_id, activity_level, + storage_vertex, source_facility_id, iob) values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, + '{}' );""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + id_for_mult_commodities, availability, primary, + new_source_facility_id, 'b')) + + main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, + facility_type_id, facility_name, schedule_day, commodity_id, activity_level, + storage_vertex, demand, source_facility_id, iob) values ({}, {}, {}, '{}', {}, {}, {}, + {}, {}, {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, + facility_name, + day_before+1, commodity_id, storage_availability, storage, quantity, + source_facility_id, io)) + + else: + if commodity_name != 'total_fuel': + main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, + facility_type_id, facility_name, schedule_day, commodity_id, activity_level, + storage_vertex, supply, source_facility_id, iob) values ({}, {}, {}, '{}', {}, {}, + {}, {}, {}, {}, '{}');""".format( + facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, storage_availability, storage, quantity, source_facility_id, io)) + + + elif facility_type == "raw_material_producer": + rmp_data = db_cur.execute("""select fc.commodity_id, fc.quantity, fc.units, + ifnull(s.source_facility_id, 0), io + from facility_commodities fc + left outer join source_commodity_ref s + on (fc.commodity_id = s.commodity_id + and s.max_transport_distance_flag = 'Y' + and s.source_facility_id = {}) + where fc.facility_id = {};""".format(facility_id, facility_id)) + + rmp_data = rmp_data.fetchall() + + for row_b in rmp_data: + commodity_id = row_b[0] + quantity = facility_cap_noEarthquake[float(facility_id)-1][t+1][i]*row_b[1]*daily_index + # units = row_b[2] + source_facility_id = row_b[3] + iob = row_b[4] + + if commodity_id in total_potential_production: + total_potential_production[commodity_id] = total_potential_production[commodity_id] + quantity + else: + total_potential_production[commodity_id] = quantity + + for day_before, availability in enumerate(schedule_dict[schedule_id]): + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + activity_level, storage_vertex, supply, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, availability, primary, quantity, + source_facility_id, iob)) + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + activity_level, storage_vertex, supply, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, storage_availability, storage, quantity, + source_facility_id, iob)) + + elif facility_type == "storage": # storage facility + storage_fac_data = db_cur.execute("""select + fc.commodity_id, + fc.quantity, + fc.units, + ifnull(s.source_facility_id, 0), + io + from facility_commodities fc + left outer join source_commodity_ref s + on (fc.commodity_id = s.commodity_id and s.max_transport_distance_flag = 'Y') + where fc.facility_id = {} ;""".format(facility_id)) + + storage_fac_data = storage_fac_data.fetchall() + + for row_b in storage_fac_data: + commodity_id = row_b[0] + source_facility_id = row_b[3] # 0 if not source-tracked + iob = row_b[4] + + for day_before in range(schedule_length): + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + storage_vertex, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, storage, + source_facility_id, iob)) + + elif facility_type == "ultimate_destination": + + dest_data = db_cur2.execute("""select + fc.commodity_id, + ifnull(fc.quantity, 0), + fc.units, + fc.commodity_id, + ifnull(c.supertype, c.commodity_name), + ifnull(s.source_facility_id, 0), + io + from facility_commodities fc, commodities c + left outer join source_commodity_ref s + on (fc.commodity_id = s.commodity_id and s.max_transport_distance_flag = 'Y') + where fc.facility_id = {} + and fc.commodity_id = c.commodity_id;""".format(facility_id)) + + dest_data = dest_data.fetchall() + + for row_b in dest_data: + commodity_id = row_b[0] + quantity = row_b[1] + commodity_supertype = row_b[4] + source_facility_id = row_b[5] + iob = row_b[6] + zero_source_facility_id = 0 # material merges at primary vertex + + # vertices for generic demand type, or any subtype specified by the destination + for day_before, availability in enumerate(schedule_dict[schedule_id]): + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + activity_level, storage_vertex, demand, udp, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, + {}, {}, {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, availability, primary, quantity, + the_scenario.unMetDemandPenalty, + zero_source_facility_id, iob)) + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + activity_level, storage_vertex, demand, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, storage_availability, storage, quantity, + source_facility_id, iob)) + # vertices for other fuel subtypes that match the destination's supertype + # if the subtype is in the commodity table, it is produced by some facility in the scenario + db_cur3 = main_db_con.cursor() + for row_c in db_cur3.execute("""select commodity_id, units from commodities + where supertype = '{}';""".format(commodity_supertype)): + new_commodity_id = row_c[0] + # new_units = row_c[1] + for day_before, availability in schedule_dict[schedule_id]: + main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, + facility_type_id, facility_name, schedule_day, commodity_id, activity_level, + storage_vertex, demand, udp, source_facility_id, iob) values ({}, {}, {}, '{}', {}, {}, + {}, {}, {}, {}, {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, + facility_name, + day_before+1, new_commodity_id, availability, primary, + quantity, + the_scenario.unMetDemandPenalty, + zero_source_facility_id, iob)) + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + activity_level, storage_vertex, demand, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, + day_before+1, new_commodity_id, storage_availability, storage, quantity, + source_facility_id, iob)) + + else: + logger.warning( + "error, unexpected facility_type: {}, facility_type_id: {}".format(facility_type, facility_type_id)) + + for row_d in db_cur4.execute("select count(distinct vertex_id) from vertices;"): + logger.info('{} vertices created'.format(row_d[0])) + + logger.debug("total possible production in scenario: {}".format(total_potential_production)) + + +# =============================================================================== + + +def add_storage_routes(the_scenario, logger): + logger.info("start: add_storage_routes") + # these are loops to and from the same facility; when multiplied to edges, + # they will connect primary to storage vertices, and storage vertices day to day + # will always create edge for this route from storage to storage vertex + # IF a primary vertex exists, will also create an edge connecting the storage vertex to the primary + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + logger.debug("create the storage_routes table") + + main_db_con.execute("drop table if exists storage_routes;") + main_db_con.execute("""create table if not exists storage_routes as + select facility_name || '_storage' as route_name, + location_id, + facility_id, + facility_name as o_name, + facility_name as d_name, + {} as cost_1, + {} as cost_2, + 1 as travel_time, + {} as storage_max, + 0 as storage_min + from facilities + where ignore_facility = 'false' + ;""".format(storage_cost_1, storage_cost_2, facility_onsite_storage_max)) + main_db_con.execute("""create table if not exists route_reference( + route_id INTEGER PRIMARY KEY, route_type text, route_name text, scenario_rt_id integer, + CONSTRAINT unique_routes UNIQUE(route_type, route_name, scenario_rt_id));""") + main_db_con.execute( + "insert or ignore into route_reference select null,'storage', route_name, 0 from storage_routes;") + + return + + +# =============================================================================== + + +def generate_connector_and_storage_edges(the_scenario, logger): + logger.info("START: generate_connector_and_storage_edges") + + multi_commodity_name = "multicommodity" + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + db_cur = main_db_con.cursor() + if ('pipeline_crude_trf_rts' in the_scenario.permittedModes) or ( + 'pipeline_prod_trf_rts' in the_scenario.permittedModes): + logger.info("create indices for the capacity_nodes and pipeline_mapping tables") + main_db_con.executescript( + """ + CREATE INDEX IF NOT EXISTS pm_index + ON pipeline_mapping (id, id_field_name, mapping_id_field_name, mapping_id); + CREATE INDEX IF NOT EXISTS cn_index ON capacity_nodes (source, id_field_name, source_OID); + """) + + for row in db_cur.execute( + "select commodity_id from commodities where commodity_name = '{}';""".format(multi_commodity_name)): + id_for_mult_commodities = row[0] + + # create storage & connector edges + main_db_con.execute("drop table if exists edges;") + main_db_con.executescript(""" + create table edges (edge_id INTEGER PRIMARY KEY, + route_id integer, + from_node_id integer, + to_node_id integer, + start_day integer, + end_day integer, + commodity_id integer, + o_vertex_id integer, + d_vertex_id integer, + max_edge_capacity numeric, + volume numeric, + capac_minus_volume_zero_floor numeric, + min_edge_capacity numeric, + capacity_units text, + units_conversion_multiplier numeric, + edge_flow_cost numeric, + edge_flow_cost2 numeric, + edge_type text, + nx_edge_id integer, + mode text, + mode_oid integer, + miles numeric, + simple_mode text, + tariff_id numeric, + phase_of_matter text, + source_facility_id integer, + miles_travelled numeric, + children_created text, + edge_count_from_source integer, + total_route_cost numeric, + CONSTRAINT unique_nx_subc_day UNIQUE(nx_edge_id, commodity_id, source_facility_id, start_day)) + ; + + insert or ignore into edges (route_id, + start_day, end_day, + commodity_id, o_vertex_id, d_vertex_id, + max_edge_capacity, min_edge_capacity, + edge_flow_cost, edge_flow_cost2,edge_type, + source_facility_id) + select o.route_id, o.schedule_day, d.schedule_day, + o.commodity_id, o.vertex_id, d.vertex_id, + o.storage_max, o.storage_min, + o.cost_1, o.cost_2, 'storage', + o.source_facility_id + from vertices d, + (select v.vertex_id, v.schedule_day, + v.commodity_id, v.storage_vertex, v.source_facility_id, + t.* + from vertices v, + (select sr.route_name, o_name, d_name, cost_1,cost_2, travel_time, + storage_max, storage_min, rr.route_id, location_id, facility_id + from storage_routes sr, route_reference rr where sr.route_name = rr.route_name) t + where v.facility_id = t.facility_id + and v.storage_vertex = 1) o + where d.facility_id = o.facility_id + and d.schedule_day = o.schedule_day+o.travel_time + and d.commodity_id = o.commodity_id + and o.vertex_id != d.vertex_id + and d.storage_vertex = 1 + and d.source_facility_id = o.source_facility_id + ; + + insert or ignore into edges (route_id, start_day, end_day, + commodity_id, o_vertex_id, d_vertex_id, + edge_flow_cost, edge_type, + source_facility_id) + select s.route_id, s.schedule_day, p.schedule_day, + (case when s.commodity_id = {} then p.commodity_id else s.commodity_id end) commodity_id, + --inbound commodies start at storage and go into primary + --outbound starts at primary and goes into storage + --anything else is an error for a connector edge + (case when fc.io = 'i' then s.vertex_id + when fc.io = 'o' then p.vertex_id + else 0 end) as o_vertex, + (case when fc.io = 'i' then p.vertex_id + when fc.io = 'o' then s.vertex_id + else 0 end) as d_vertex, + 0, 'connector', + s.source_facility_id + from vertices p, facility_commodities fc, + --s for storage vertex info, p for primary vertex info + (select v.vertex_id, v.schedule_day, + v.commodity_id, v.storage_vertex, v.source_facility_id, + t.* + from vertices v, + (select sr.route_name, o_name, d_name, cost_1,cost_2, travel_time, + storage_max, storage_min, rr.route_id, location_id, facility_id + from storage_routes sr, route_reference rr where sr.route_name = rr.route_name) t --t is route data + where v.facility_id = t.facility_id + and v.storage_vertex = 1) s + --from storage into primary, same day = inbound connectors + where p.facility_id = s.facility_id + and p.schedule_day = s.schedule_day + and (p.commodity_id = s.commodity_id or p.commodity_id = {} ) + and p.facility_id = fc.facility_id + and fc.commodity_id = s.commodity_id + and p.storage_vertex = 0 + --either edge is inbound and aggregating, or kept separate by source, or primary vertex is not source tracked + and + (p.source_facility_id = 0 or p.source_facility_id = p.facility_id or p.source_facility_id = s.source_facility_id) + ;""".format(id_for_mult_commodities, id_for_mult_commodities)) + + for row_d in db_cur.execute("select count(distinct edge_id) from edges where edge_type = 'connector';"): + logger.info('{} connector edges created'.format(row_d[0])) + # clear any transport edges from table + db_cur.execute("delete from edges where edge_type = 'transport';") + + return + + +# =============================================================================== + + +def generate_first_edges_from_source_facilities(the_scenario, schedule_length, logger): + + logger.info("START: generate_first_edges_from_source_facilities") + # create edges table + # plan to generate start and end days based on nx edge time to traverse and schedule + # can still have route_id, but only for storage routes now; nullable + + # multi_commodity_name = "multicommodity" + transport_edges_created = 0 + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + db_cur = main_db_con.cursor() + edges_requiring_children = 0 + + counter = 0 + + # create transport edges, only between storage vertices and nodes, based on networkx graph + + commodity_mode_data = main_db_con.execute("select * from commodity_mode;") + commodity_mode_data = commodity_mode_data.fetchall() + commodity_mode_dict = {} + for row in commodity_mode_data: + mode = row[0] + commodity_id = int(row[1]) + commodity_phase = row[2] + allowed_yn = row[3] + commodity_mode_dict[mode, commodity_id] = allowed_yn + + + source_edge_data = main_db_con.execute("""select + ne.edge_id, + ifnull(CAST(fn.location_id as integer), 'NULL'), + ifnull(CAST(tn.location_id as integer), 'NULL'), + ne.mode_source, + ifnull(nec.phase_of_matter_id, 'NULL'), + nec.route_cost, + ne.from_node_id, + ne.to_node_id, + nec.dollar_cost, + ne.miles, + ne.capacity, + ne.artificial, + ne.mode_source_oid, + v.commodity_id, + v.schedule_day, + v.vertex_id, + v.source_facility_id, + tv.vertex_id, + ifnull(t.miles_travelled, 0), + ifnull(t.edge_count_from_source, 0), + t.mode + from networkx_edges ne, networkx_nodes fn, networkx_nodes tn, networkx_edge_costs nec, vertices v, + facility_type_id ft, facility_commodities fc + left outer join (--get facilities with incoming transport edges with tracked mileage + select vi.facility_id, min(ei.miles_travelled) miles_travelled, ei.source_facility_id, + ei.edge_count_from_source, ei.mode + from edges ei, vertices vi + where vi.vertex_id = ei.d_vertex_id + and edge_type = 'transport' + and ifnull(miles_travelled, 0) > 0 + group by vi.facility_id, ei.source_facility_id, ei.mode) t + on t.facility_id = v.facility_id and t.source_facility_id = v.source_facility_id and ne.mode_source = t.mode + left outer join vertices tv + on (CAST(tn.location_id as integer) = tv.location_id and v.source_facility_id = tv.source_facility_id) + where v.location_id = CAST(fn.location_id as integer) + and fc.facility_id = v.facility_id + and fc.commodity_id = v.commodity_id + and fc.io = 'o' + and ft.facility_type_id = v.facility_type_id + and (ft.facility_type = 'raw_material_producer' or t.facility_id = v.facility_id) + and ne.from_node_id = fn.node_id + and ne.to_node_id = tn.node_id + and ne.edge_id = nec.edge_id + and ifnull(ne.capacity, 1) > 0 + and v.storage_vertex = 1 + and v.source_facility_id != 0 --max commodity distance applies + ;""") + source_edge_data = source_edge_data.fetchall() + for row_a in source_edge_data: + + nx_edge_id = row_a[0] + from_location = row_a[1] + to_location = row_a[2] + mode = row_a[3] + phase_of_matter = row_a[4] + route_cost = row_a[5] + from_node = row_a[6] + to_node = row_a[7] + dollar_cost = row_a[8] + miles = row_a[9] + # max_daily_capacity = row_a[10] + # artificial = row_a[11] + mode_oid = row_a[12] + commodity_id = row_a[13] + origin_day = row_a[14] + vertex_id = row_a[15] + source_facility_id = row_a[16] + to_vertex = row_a[17] + previous_miles_travelled = row_a[18] + previous_edge_count = row_a[19] + previous_mode = row_a[20] + + simple_mode = row_a[3].partition('_')[0] + edge_count_from_source = 1 + previous_edge_count + total_route_cost = route_cost + miles_travelled = previous_miles_travelled + miles + + if counter % 10000 == 0: + for row_d in db_cur.execute("select count(distinct edge_id) from edges;"): + logger.info('{} edges created'.format(row_d[0])) + counter = counter + 1 + + tariff_id = 0 + if simple_mode == 'pipeline': + + # find tariff_ids + + sql = "select mapping_id from pipeline_mapping " \ + "where id = {} and id_field_name = 'source_OID' " \ + "and source = '{}' and mapping_id is not null;".format( + mode_oid, mode) + for tariff_row in db_cur.execute(sql): + tariff_id = tariff_row[0] + + + if mode in the_scenario.permittedModes and (mode, commodity_id) in commodity_mode_dict.keys() \ + and commodity_mode_dict[mode, commodity_id] == 'Y': + + # Edges are placeholders for flow variables + # 4-17: if both ends have no location, iterate through viable commodities and days, create edge + # for all days (restrict by link schedule if called for) + # for all allowed commodities, as currently defined by link phase of matter + + # days range from 1 to schedule_length + if origin_day in range(1, schedule_length+1): + if origin_day + fixed_route_duration <= schedule_length: + # if link is traversable in the timeframe + if simple_mode != 'pipeline' or tariff_id >= 0: + # for allowed commodities + # step 1 from source is from non-Null location to (probably) null location + + if from_location != 'NULL' and to_location == 'NULL': + # for each day and commodity, + # get the corresponding origin vertex id to include with the edge info + # origin vertex must not be "ultimate_destination + # transport link outgoing from facility - checking fc.io is more thorough + # than checking if facility type is 'ultimate destination' + # only connect to vertices with matching source_facility_id + # source_facility_id is zero for commodities without source tracking + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + o_vertex_id, + min_edge_capacity,edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, + tariff_id,phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, + {}, {}, {}, + '{}',{},'{}',{}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + vertex_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + miles_travelled, 'N', edge_count_from_source, total_route_cost)) + + + elif from_location != 'NULL' and to_location != 'NULL': + # for each day and commodity, get the corresponding origin and destination vertex ids + # to include with the edge info + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + o_vertex_id, d_vertex_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, {}, + {}, {}, {}, + '{}',{},'{}', {}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + vertex_id, to_vertex, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + miles_travelled, 'N', edge_count_from_source, total_route_cost)) + + for row in db_cur.execute("select count(distinct edge_id) from edges where edge_type = 'transport';"): + transport_edges_created = row[0] + logger.info('{} transport edges created'.format(transport_edges_created)) + for row in db_cur.execute("""select count(distinct edge_id), children_created from edges + where edge_type = 'transport' + group by children_created;"""): + if row[1] == 'N': + edges_requiring_children = row[0] + + elif row[1] == 'Y': + logger.info('{} transport edges that have already been checked for children'.format(row[0])) + edges_requiring_children = transport_edges_created - row[0] + # edges_requiring_children is updated under either condition here, since one may not be triggered + logger.info('{} transport edges that need children'.format(edges_requiring_children)) + + return + + +# =============================================================================== + + +def generate_all_edges_from_source_facilities(the_scenario, schedule_length, logger): + # method only runs for commodities with a max commodity constraint + + logger.info("START: generate_all_edges_from_source_facilities") + + multi_commodity_name = "multicommodity" + # initializations - all of these get updated if >0 edges exist + edges_requiring_children = 0 + endcap_edges = 0 + edges_resolved = 0 + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + db_cur = main_db_con.cursor() + transport_edges_created = 0 + nx_edge_count = 0 + source_based_edges_created = 0 + + for row_d in db_cur.execute("""select count(distinct e.edge_id), count(distinct e.nx_edge_id) + from edges e where e.edge_type = 'transport' + and children_created in ('N', 'Y', 'E');"""): + source_based_edges_created = row_d[0] + + for row_d in db_cur.execute("""select count(distinct e.edge_id), count(distinct e.nx_edge_id) + from edges e where e.edge_type = 'transport';"""): + transport_edges_created = row_d[0] + nx_edge_count = row_d[1] + + + commodity_mode_data = main_db_con.execute("select * from commodity_mode;") + commodity_mode_data = commodity_mode_data.fetchall() + commodity_mode_dict = {} + for row in commodity_mode_data: + mode = row[0] + commodity_id = int(row[1]) + commodity_phase = row[2] + allowed_yn = row[3] + commodity_mode_dict[mode, commodity_id] = allowed_yn + + current_edge_data = db_cur.execute("""select count(distinct edge_id), children_created + from edges + where edge_type = 'transport' + group by children_created + order by children_created asc;""") + current_edge_data = current_edge_data.fetchall() + for row in current_edge_data: + if row[1] == 'N': + edges_requiring_children = row[0] + elif row[1] == 'Y': + edges_resolved = row[0] + elif row[1] == 'E': + endcap_edges = row[0] + if source_based_edges_created == edges_resolved + endcap_edges: + edges_requiring_children = 0 + if source_based_edges_created == edges_requiring_children + endcap_edges: + edges_resolved = 0 + if source_based_edges_created == edges_requiring_children + edges_resolved: + endcap_edges = 0 + + logger.info( + '{} transport edges created; {} require children'.format(transport_edges_created, edges_requiring_children)) + + # set up a table to keep track of endcap nodes + sql = """ + drop table if exists endcap_nodes; + create table if not exists endcap_nodes( + node_id integer NOT NULL, + + location_id integer, + + --mode of edges that it's an endcap for + mode_source text NOT NULL, + + --facility it's an endcap for + source_facility_id integer NOT NULL, + + --commodities it's an endcap for + commodity_id integer NOT NULL, + + CONSTRAINT endcap_key PRIMARY KEY (node_id, mode_source, source_facility_id, commodity_id)) + --the combination of these four (excluding location_id) should be unique, + --and all fields except location_id should be filled + ;""" + db_cur.executescript(sql) + + # create transport edges, only between storage vertices and nodes, based on networkx graph + + edge_into_facility_counter = 0 + while_count = 0 + + while edges_requiring_children > 0: + while_count = while_count+1 + + # --get nx edges that align with the existing "in edges" - data from nx to create new edges --for each of + # those nx_edges, if they connect to more than one "in edge" in this batch, only consider connecting to + # the shortest -- if there is a valid nx_edge to build, the crossing node is not an endcap if total miles + # of the new route is over max transport distance, then endcap what if one child goes over max transport + # and another doesn't then the node will get flagged as an endcap, and another path may continue off it, + # allow both for now --check that day and commodity are permitted by nx + + potential_edge_data = main_db_con.execute(""" + select + ch.edge_id as ch_nx_edge_id, + ifnull(CAST(chfn.location_id as integer), 'NULL') fn_location_id, + ifnull(CAST(chtn.location_id as integer), 'NULL') tn_location_id, + ch.mode_source, + p.phase_of_matter, + nec.route_cost, + ch.from_node_id, + ch.to_node_id, + nec.dollar_cost, + ch.miles, + ch.capacity, + ch.artificial, + ch.mode_source_oid, + --parent edge into + p.commodity_id, + p.end_day, + --parent's dest. vertex if exists + ifnull(p.d_vertex_id,0) o_vertex, + p.source_facility_id, + p.leadin_miles_travelled, + + (p.edge_count_from_source +1) as new_edge_count, + (p.total_route_cost + nec.route_cost) new_total_route_cost, + p.edge_id leadin_edge, + p.nx_edge_id leadin_nx_edge, + + --new destination vertex if exists + ifnull(chtv.vertex_id,0) d_vertex, + + sc.max_transport_distance + + from + (select count(edge_id) parents, + min(miles_travelled) leadin_miles_travelled, + * + from edges + where children_created = 'N' + -----------------do not mess with this "group by" + group by to_node_id, source_facility_id, commodity_id, end_day + ------------------it affects which columns we're checking over for min miles travelled + --------------so that we only get the parent edges we want + order by parents desc + ) p, --parent edges to use in this batch + networkx_edges ch, + networkx_edge_costs nec, + source_commodity_ref sc, + networkx_nodes chfn, + networkx_nodes chtn + left outer join vertices chtv + on (CAST(chtn.location_id as integer) = chtv.location_id + and p.source_facility_id = chtv.source_facility_id + and chtv.commodity_id = p.commodity_id + and p.end_day = chtv.schedule_day + and chtv.iob = 'i') + + where p.to_node_id = ch.from_node_id + --and p.mode = ch.mode_source --build across modes, control at conservation of flow + and ch.to_node_id = chtn.node_id + and ch.from_node_id = chfn.node_id + and p.phase_of_matter = nec.phase_of_matter_id + and ch.edge_id = nec.edge_id + and ifnull(ch.capacity, 1) > 0 + and p.commodity_id = sc.commodity_id + ;""") + + # --should only get a single leadin edge per networkx/source/commodity/day combination + # leadin edge should be in route_data, current set of min. identifiers + # if we're trying to add an edge that has an entry in route_data, new miles travelled must be less + + potential_edge_data = potential_edge_data.fetchall() + + main_db_con.execute("update edges set children_created = 'Y' where children_created = 'N';") + + for row_a in potential_edge_data: + nx_edge_id = row_a[0] + from_location = row_a[1] + to_location = row_a[2] + mode = row_a[3] + phase_of_matter = row_a[4] + route_cost = row_a[5] + from_node = row_a[6] + to_node = row_a[7] + dollar_cost = row_a[8] + miles = row_a[9] + mode_oid = row_a[12] + commodity_id = row_a[13] + origin_day = row_a[14] + vertex_id = row_a[15] + source_facility_id = row_a[16] + leadin_edge_miles_travelled = row_a[17] + new_edge_count = row_a[18] + total_route_cost = row_a[19] + leadin_edge_id = row_a[20] + # leadin_nx_edge_id = row_a[21] + to_vertex = row_a[22] + max_commodity_travel_distance = row_a[23] + + # end_day = origin_day + fixed_route_duration + new_miles_travelled = miles + leadin_edge_miles_travelled + + if mode in the_scenario.permittedModes and (mode, commodity_id) in commodity_mode_dict.keys() \ + and commodity_mode_dict[mode, commodity_id] == 'Y': + + if new_miles_travelled > max_commodity_travel_distance: + # designate leadin edge as endcap + children_created = 'E' + # update the incoming edge to indicate it's an endcap + db_cur.execute( + "update edges set children_created = '{}' where edge_id = {}".format(children_created, + leadin_edge_id)) + if from_location != 'NULL': + db_cur.execute("""insert or ignore into endcap_nodes( + node_id, location_id, mode_source, source_facility_id, commodity_id) + VALUES ({}, {}, '{}', {}, {}); + """.format(from_node, from_location, mode, source_facility_id, commodity_id)) + else: + db_cur.execute("""insert or ignore into endcap_nodes( + node_id, mode_source, source_facility_id, commodity_id) + VALUES ({}, '{}', {}, {}); + """.format(from_node, mode, source_facility_id, commodity_id)) + + # create new edge + elif new_miles_travelled <= max_commodity_travel_distance: + + simple_mode = row_a[3].partition('_')[0] + tariff_id = 0 + if simple_mode == 'pipeline': + + # find tariff_ids + + sql = """select mapping_id + from pipeline_mapping + where id = {} + and id_field_name = 'source_OID' + and source = '{}' + and mapping_id is not null;""".format(mode_oid, mode) + for tariff_row in db_cur.execute(sql): + tariff_id = tariff_row[0] + + # if there are no edges yet for this day, nx, subc combination, + # AND this is the shortest existing leadin option for this day, nx, subc combination + # we'd be creating an edge for (otherwise wait for the shortest option) + # at this step, some leadin edge should always exist + + if origin_day in range(1, schedule_length+1): + if origin_day + fixed_route_duration <= schedule_length: + # if link is traversable in the timeframe + if simple_mode != 'pipeline' or tariff_id >= 0: + # for allowed commodities + + if from_location == 'NULL' and to_location == 'NULL': + # for each day and commodity, + # get the corresponding origin vertex id to include with the edge info + # origin vertex must not be "ultimate_destination + # transport link outgoing from facility - checking fc.io is more thorough than + # checking if facility type is 'ultimate destination' + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + min_edge_capacity,edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, + tariff_id,phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, {}, {}, + '{}',{},'{}',{}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + new_miles_travelled, 'N', new_edge_count, total_route_cost)) + # only create edge going into a location if an appropriate vertex exists + elif from_location == 'NULL' and to_location != 'NULL' and to_vertex > 0: + edge_into_facility_counter = edge_into_facility_counter + 1 + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + d_vertex_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, + {}, {}, {}, + '{}',{},'{}', {}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + to_vertex, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + new_miles_travelled, 'N', new_edge_count, total_route_cost)) + + # designate leadin edge as endcap + # this does, deliberately, allow endcap status to be + # overwritten if we've found a shorter path to a previous endcap + elif from_location != 'NULL' and to_location == 'NULL': + # for each day and commodity, get the corresponding origin vertex id + # to include with the edge info + # origin vertex must not be "ultimate_destination + # transport link outgoing from facility - checking fc.io is more thorough than + # checking if facility type is 'ultimate destination' + # new for bsc, only connect to vertices with matching source facility id + # (only limited for RMP vertices) + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + o_vertex_id, + min_edge_capacity,edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, + {}, {}, {}, + '{}',{},'{}',{}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + vertex_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + new_miles_travelled, 'N', new_edge_count, total_route_cost)) + + + elif from_location != 'NULL' and to_location != 'NULL': + # for each day and commodity, get the corresponding origin and + # destination vertex ids to include with the edge info + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + o_vertex_id, d_vertex_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, + total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, {}, + {}, {}, {}, + '{}',{},'{}', {}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + vertex_id, to_vertex, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + new_miles_travelled, 'N', new_edge_count, total_route_cost)) + + for row_d in db_cur.execute("""select count(distinct e.edge_id), count(distinct e.nx_edge_id) + from edges e where e.edge_type = 'transport' + and children_created in ('N', 'Y', 'E');"""): + source_based_edges_created = row_d[0] + + for row_d in db_cur.execute("""select count(distinct e.edge_id), count(distinct e.nx_edge_id) + from edges e where e.edge_type = 'transport';"""): + transport_edges_created = row_d[0] + nx_edge_count = row_d[1] + + current_edge_data = db_cur.execute("""select count(distinct edge_id), children_created + from edges + where edge_type = 'transport' + group by children_created + order by children_created asc;""") + current_edge_data = current_edge_data.fetchall() + for row in current_edge_data: + if row[1] == 'N': + edges_requiring_children = row[0] + elif row[1] == 'Y': + edges_resolved = row[0] + elif row[1] == 'E': + endcap_edges = row[0] + logger.debug('{} endcap edges designated for candidate generation step'.format(endcap_edges)) + if source_based_edges_created == edges_resolved + endcap_edges: + edges_requiring_children = 0 + if source_based_edges_created == edges_requiring_children + endcap_edges: + edges_resolved = 0 + if source_based_edges_created == edges_requiring_children + edges_resolved: + endcap_edges = 0 + + if while_count % 1000 == 0 or edges_requiring_children == 0: + logger.info( + '{} transport edges on {} nx edges, created in {} loops, {} edges_requiring_children'.format( + transport_edges_created, nx_edge_count, while_count, edges_requiring_children)) + + # edges going in to the facility by re-running "generate first edges + # then re-run this method + + logger.info('{} transport edges on {} nx edges, created in {} loops, {} edges_requiring_children'.format( + transport_edges_created, nx_edge_count, while_count, edges_requiring_children)) + logger.info("all source-based transport edges created") + + logger.info("create an index for the edges table by nodes") + + sql = ("""CREATE INDEX IF NOT EXISTS edge_index ON edges ( + edge_id, route_id, from_node_id, to_node_id, commodity_id, + start_day, end_day, commodity_id, o_vertex_id, d_vertex_id, + max_edge_capacity, min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, source_facility_id);""") + db_cur.execute(sql) + + return + + +# =============================================================================== + + +def generate_all_edges_without_max_commodity_constraint(the_scenario, schedule_length, logger): + + global total_transport_routes + logger.info("START: generate_all_edges_without_max_commodity_constraint") + + multi_commodity_name = "multicommodity" + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + db_cur = main_db_con.cursor() + + commodity_mode_data = main_db_con.execute("select * from commodity_mode;") + commodity_mode_data = commodity_mode_data.fetchall() + commodity_mode_dict = {} + for row in commodity_mode_data: + mode = row[0] + commodity_id = int(row[1]) + commodity_phase = row[2] + allowed_yn = row[3] + commodity_mode_dict[mode, commodity_id] = allowed_yn + + counter = 0 + # for row in db_cur.execute( + # "select commodity_id from commodities where commodity_name = '{}';""".format(multi_commodity_name)): + # id_for_mult_commodities = row[0] + logger.info("COUNTING TOTAL TRANSPORT ROUTES") + for row in db_cur.execute(""" + select count(*) from networkx_edges, shortest_edges + WHERE networkx_edges.from_node_id = shortest_edges.from_node_id + AND networkx_edges.to_node_id = shortest_edges.to_node_id + AND networkx_edges.edge_id = shortest_edges.edge_id + ; + """): + total_transport_routes = row[0] + + # for all commodities with no max transport distance + source_facility_id = 0 + + # create transport edges, only between storage vertices and nodes, based on networkx graph + # never touch primary vertices; either or both vertices can be null (or node-type) if it's a mid-route link + # iterate through nx edges: if neither node has a location, create 1 edge per viable commodity + # should also be per day, subject to nx edge schedule + # before creating an edge, check: commodity allowed by nx and max transport distance if not null + # will need nodes per day and commodity? or can I just check that with constraints? + # select data for transport edges + + sql = """select + ne.edge_id, + ifnull(fn.location_id, 'NULL'), + ifnull(tn.location_id, 'NULL'), + ne.mode_source, + ifnull(nec.phase_of_matter_id, 'NULL'), + nec.route_cost, + ne.from_node_id, + ne.to_node_id, + nec.dollar_cost, + ne.miles, + ne.capacity, + ne.artificial, + ne.mode_source_oid + from networkx_edges ne, networkx_nodes fn, networkx_nodes tn, networkx_edge_costs nec, shortest_edges se + + where ne.from_node_id = fn.node_id + and ne.to_node_id = tn.node_id + and ne.edge_id = nec.edge_id + and ne.from_node_id = se.from_node_id -- match to the shortest_edges table + and ne.to_node_id = se.to_node_id + and ifnull(ne.capacity, 1) > 0 + ;""" + nx_edge_data = main_db_con.execute(sql) + nx_edge_data = nx_edge_data.fetchall() + for row_a in nx_edge_data: + + nx_edge_id = row_a[0] + from_location = row_a[1] + to_location = row_a[2] + mode = row_a[3] + phase_of_matter = row_a[4] + route_cost = row_a[5] + from_node = row_a[6] + to_node = row_a[7] + dollar_cost = row_a[8] + miles = row_a[9] + # max_daily_capacity = row_a[10] + mode_oid = row_a[12] + simple_mode = row_a[3].partition('_')[0] + + counter = counter + 1 + + tariff_id = 0 + if simple_mode == 'pipeline': + + # find tariff_ids + + sql = "select mapping_id from pipeline_mapping " \ + "where id = {} and id_field_name = 'source_OID' and source = '{}' " \ + "and mapping_id is not null;".format( + mode_oid, mode) + for tariff_row in db_cur.execute(sql): + tariff_id = tariff_row[0] + + + if mode in the_scenario.permittedModes: + + # Edges are placeholders for flow variables + # for all days (restrict by link schedule if called for) + # for all allowed commodities, as currently defined by link phase of matter + + for day in range(1, schedule_length+1): + if day + fixed_route_duration <= schedule_length: + # if link is traversable in the timeframe + if simple_mode != 'pipeline' or tariff_id >= 0: + # for allowed commodities that can be output by some facility in the scenario + for row_c in db_cur.execute("""select commodity_id + from source_commodity_ref s + where phase_of_matter = '{}' + and max_transport_distance_flag = 'N' + and share_max_transport_distance = 'N' + group by commodity_id, source_facility_id""".format(phase_of_matter)): + db_cur4 = main_db_con.cursor() + commodity_id = row_c[0] + # source_facility_id = row_c[1] # fixed to 0 for all edges created by this method + if (mode, commodity_id) in commodity_mode_dict.keys() \ + and commodity_mode_dict[mode, commodity_id] == 'Y': + if from_location == 'NULL' and to_location == 'NULL': + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, source_facility_id) VALUES ({}, {}, + {}, {}, {}, + {}, {}, {}, + '{}',{},'{}',{},{},'{}',{},'{}',{}); + """.format(from_node, to_node, + day, day + fixed_route_duration, commodity_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, miles, simple_mode, + tariff_id, phase_of_matter, source_facility_id)) + + elif from_location != 'NULL' and to_location == 'NULL': + # for each day and commodity, get the corresponding origin vertex id + # to include with the edge info + # origin vertex must not be "ultimate_destination + # transport link outgoing from facility - checking fc.io is more thorough + # than checking if facility type is 'ultimate destination' + # new for bsc, only connect to vertices with matching source_facility_id + for row_d in db_cur4.execute("""select vertex_id + from vertices v, facility_commodities fc + where v.location_id = {} and v.schedule_day = {} + and v.commodity_id = {} and v.source_facility_id = {} + and v.storage_vertex = 1 + and v.facility_id = fc.facility_id + and v.commodity_id = fc.commodity_id + and fc.io = 'o'""".format(from_location, day, commodity_id, source_facility_id)): + from_vertex_id = row_d[0] + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + o_vertex_id, + min_edge_capacity,edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, source_facility_id) VALUES ({}, {}, + {}, {}, {}, + {}, + {}, {}, {}, + '{}',{},'{}',{},{},'{}',{},'{}',{}); + """.format(from_node, to_node, + day, day + fixed_route_duration, commodity_id, + from_vertex_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, miles, simple_mode, + tariff_id, phase_of_matter, source_facility_id)) + elif from_location == 'NULL' and to_location != 'NULL': + # for each day and commodity, get the corresponding destination vertex id + # to include with the edge info + for row_d in db_cur4.execute("""select vertex_id + from vertices v, facility_commodities fc + where v.location_id = {} and v.schedule_day = {} + and v.commodity_id = {} and v.source_facility_id = {} + and v.storage_vertex = 1 + and v.facility_id = fc.facility_id + and v.commodity_id = fc.commodity_id + and fc.io = 'i'""".format(to_location, day, commodity_id, source_facility_id)): + to_vertex_id = row_d[0] + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + d_vertex_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, source_facility_id) VALUES ({}, {}, + {}, {}, {}, + {}, + {}, {}, {}, + '{}',{},'{}',{},{},'{}',{},'{}',{}); + """.format(from_node, to_node, + day, day + fixed_route_duration, commodity_id, + to_vertex_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, miles, simple_mode, + tariff_id, phase_of_matter, source_facility_id)) + elif from_location != 'NULL' and to_location != 'NULL': + # for each day and commodity, get the corresponding origin and destination vertex + # ids to include with the edge info + for row_d in db_cur4.execute("""select vertex_id + from vertices v, facility_commodities fc + where v.location_id = {} and v.schedule_day = {} + and v.commodity_id = {} and v.source_facility_id = {} + and v.storage_vertex = 1 + and v.facility_id = fc.facility_id + and v.commodity_id = fc.commodity_id + and fc.io = 'o'""".format(from_location, day, commodity_id, source_facility_id)): + from_vertex_id = row_d[0] + db_cur5 = main_db_con.cursor() + for row_e in db_cur5.execute("""select vertex_id + from vertices v, facility_commodities fc + where v.location_id = {} and v.schedule_day = {} + and v.commodity_id = {} and v.source_facility_id = {} + and v.storage_vertex = 1 + and v.facility_id = fc.facility_id + and v.commodity_id = fc.commodity_id + and fc.io = 'i'""".format(to_location, day, commodity_id, source_facility_id)): + to_vertex_id = row_e[0] + main_db_con.execute("""insert or ignore into edges (from_node_id, + to_node_id, start_day, end_day, commodity_id, o_vertex_id, d_vertex_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, edge_type, + nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, source_facility_id) VALUES ({}, {}, {}, {}, {}, {}, {}, + {}, {}, {}, '{}',{},'{}', {},{},'{}',{},'{}',{} + )""".format(from_node, + to_node, day, + day + fixed_route_duration, + commodity_id, + from_vertex_id, + to_vertex_id, + default_min_capacity, + route_cost, + dollar_cost, + 'transport', + nx_edge_id, mode, + mode_oid, miles, + simple_mode, + tariff_id, + phase_of_matter, + source_facility_id)) + + logger.debug("all transport edges created") + + logger.info("all edges created") + logger.info("create an index for the edges table by nodes") + index_start_time = datetime.datetime.now() + sql = ("""CREATE INDEX IF NOT EXISTS edge_index ON edges ( + edge_id, route_id, from_node_id, to_node_id, commodity_id, + start_day, end_day, commodity_id, o_vertex_id, d_vertex_id, + max_edge_capacity, min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, source_facility_id);""") + db_cur.execute(sql) + logger.info("edge_index Total Runtime (HMS): \t{} \t ".format(get_total_runtime_string(index_start_time))) + return + + +# =============================================================================== + + +def set_edges_volume_capacity(the_scenario, logger): + logger.info("starting set_edges_volume_capacity") + with sqlite3.connect(the_scenario.main_db) as main_db_con: + logger.debug("starting to record volume and capacity for non-pipeline edges") + + main_db_con.execute( + "update edges set volume = (select ifnull(ne.volume,0) from networkx_edges ne " + "where ne.edge_id = edges.nx_edge_id ) where simple_mode in ('rail','road','water');") + main_db_con.execute( + "update edges set max_edge_capacity = (select ne.capacity from networkx_edges ne " + "where ne.edge_id = edges.nx_edge_id) where simple_mode in ('rail','road','water');") + logger.debug("volume and capacity recorded for non-pipeline edges") + + logger.debug("starting to record volume and capacity for pipeline edges") + ## + main_db_con.executescript("""update edges set volume = + (select l.background_flow + from pipeline_mapping pm, + (select id_field_name, cn.source_OID as link_id, min(cn.capacity) capac, + max(cn.volume) background_flow, source + from capacity_nodes cn + where cn.id_field_name = 'MASTER_OID' + and ifnull(cn.capacity,0)>0 + group by link_id) l + + where edges.tariff_id = pm.id + and pm.id_field_name = 'tariff_ID' + and pm.mapping_id_field_name = 'MASTER_OID' + and l.id_field_name = 'MASTER_OID' + and pm.mapping_id = l.link_id + and instr(edges.mode, l.source)>0) + where simple_mode = 'pipeline' + ; + + update edges set max_edge_capacity = + (select l.capac + from pipeline_mapping pm, + (select id_field_name, cn.source_OID as link_id, min(cn.capacity) capac, + max(cn.volume) background_flow, source + from capacity_nodes cn + where cn.id_field_name = 'MASTER_OID' + and ifnull(cn.capacity,0)>0 + group by link_id) l + + where edges.tariff_id = pm.id + and pm.id_field_name = 'tariff_ID' + and pm.mapping_id_field_name = 'MASTER_OID' + and l.id_field_name = 'MASTER_OID' + and pm.mapping_id = l.link_id + and instr(edges.mode, l.source)>0) + where simple_mode = 'pipeline' + ;""") + logger.debug("volume and capacity recorded for pipeline edges") + logger.debug("starting to record units and conversion multiplier") + main_db_con.execute("""update edges + set capacity_units = + (case when simple_mode = 'pipeline' then 'kbarrels' + when simple_mode = 'road' then 'truckload' + when simple_mode = 'rail' then 'railcar' + when simple_mode = 'water' then 'barge' + else 'unexpected mode' end) + ;""") + main_db_con.execute("""update edges + set units_conversion_multiplier = + (case when simple_mode = 'pipeline' and phase_of_matter = 'liquid' then {} + when simple_mode = 'road' and phase_of_matter = 'liquid' then {} + when simple_mode = 'road' and phase_of_matter = 'solid' then {} + when simple_mode = 'rail' and phase_of_matter = 'liquid' then {} + when simple_mode = 'rail' and phase_of_matter = 'solid' then {} + when simple_mode = 'water' and phase_of_matter = 'liquid' then {} + when simple_mode = 'water' and phase_of_matter = 'solid' then {} + else 1 end) + ;""".format(THOUSAND_GALLONS_PER_THOUSAND_BARRELS, + the_scenario.truck_load_liquid.magnitude, + the_scenario.truck_load_solid.magnitude, + the_scenario.railcar_load_liquid.magnitude, + the_scenario.railcar_load_solid.magnitude, + the_scenario.barge_load_liquid.magnitude, + the_scenario.barge_load_solid.magnitude, + )) + logger.debug("units and conversion multiplier recorded for all edges; starting capacity minus volume") + main_db_con.execute("""update edges + set capac_minus_volume_zero_floor = + max((select (max_edge_capacity - ifnull(volume,0)) where max_edge_capacity is not null),0) + where max_edge_capacity is not null + ;""") + logger.debug("capacity minus volume (minimum set to zero) recorded for all edges") + return + + +# =============================================================================== + + +def pre_setup_pulp(logger, the_scenario): + logger.info("START: pre_setup_pulp") + + commodity_mode_setup(the_scenario, logger) + + # create table to track source facility of commodities with a max transport distance set + source_tracking_setup(the_scenario, logger) + + schedule_dict, schedule_length = generate_schedules(the_scenario, logger) + + generate_all_vertices(the_scenario, schedule_dict, schedule_length, logger) + + add_storage_routes(the_scenario, logger) + generate_connector_and_storage_edges(the_scenario, logger) + + # start edges for commodities that inherit max transport distance + generate_first_edges_from_source_facilities(the_scenario, schedule_length, logger) + + # replicate all_routes by commodity and time into all_edges dictionary + generate_all_edges_from_source_facilities(the_scenario, schedule_length, logger) + + # replicate all_routes by commodity and time into all_edges dictionary + generate_all_edges_without_max_commodity_constraint(the_scenario, schedule_length, logger) + logger.info("Edges generated for modes: {}".format(the_scenario.permittedModes)) + + set_edges_volume_capacity(the_scenario, logger) + + return + + +# =============================================================================== + + +def create_flow_vars(the_scenario, logger): + logger.info("START: create_flow_vars") + + # we have a table called edges. + # call helper method to get list of unique IDs from the Edges table. + # use the rowid as a simple unique integer index + edge_list = [] + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + edge_list_cur = db_cur.execute("""select edge_id--, commodity_id, start_day, source_facility_id + from edges;""") + edge_list_data = edge_list_cur.fetchall() + counter = 0 + for row in edge_list_data: + if counter % 500000 == 0: + logger.info( + "processed {:,.0f} records. size of edge_list {:,.0f}".format(counter, sys.getsizeof(edge_list))) + counter += 1 + # create an edge for each commodity allowed on this link - this construction may change + # as specific commodity restrictions are added + # TODO4-18 add days, but have no scheduel for links currently + # running just with nodes for now, will add proper facility info and storage back soon + edge_list.append((row[0])) + + + flow_var = LpVariable.dicts("Edge", edge_list, 0, None) + return flow_var + + +# =============================================================================== + + +def create_unmet_demand_vars(the_scenario, logger): + logger.info("START: create_unmet_demand_vars") + demand_var_list = [] + # may create vertices with zero demand, but only for commodities that the facility has demand for at some point + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + for row in db_cur.execute("""select v.facility_id, v.schedule_day, + ifnull(c.supertype, c.commodity_name) top_level_commodity_name, v.udp + from vertices v, commodities c, facility_type_id ft, facilities f + where v.commodity_id = c.commodity_id + and ft.facility_type = "ultimate_destination" + and v.storage_vertex = 0 + and v.facility_type_id = ft.facility_type_id + and v.facility_id = f.facility_id + and f.ignore_facility = 'false' + group by v.facility_id, v.schedule_day, ifnull(c.supertype, c.commodity_name) + ;""".format('')): + # facility_id, day, and simplified commodity name + demand_var_list.append((row[0], row[1], row[2], row[3])) + + unmet_demand_var = LpVariable.dicts("UnmetDemand", demand_var_list, None, None) + + return unmet_demand_var + + +# =============================================================================== + + +def create_candidate_processor_build_vars(the_scenario, logger): + logger.info("START: create_candidate_processor_build_vars") + processors_build_list = [] + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + for row in db_cur.execute( + """select f.facility_id from facilities f, facility_type_id ft + where f.facility_type_id = ft.facility_type_id and facility_type = 'processor' + and candidate = 1 and ignore_facility = 'false' group by facility_id;"""): + # grab all candidate processor facility IDs + processors_build_list.append(row[0]) + + processor_build_var = LpVariable.dicts("BuildProcessor", processors_build_list, 0, None, 'Binary') + + return processor_build_var + + +# =============================================================================== + + +def create_binary_processor_vertex_flow_vars(the_scenario, logger): + logger.info("START: create_binary_processor_vertex_flow_vars") + processors_flow_var_list = [] + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + for row in db_cur.execute("""select v.facility_id, v.schedule_day + from vertices v, facility_type_id ft + where v.facility_type_id = ft.facility_type_id + and facility_type = 'processor' + and storage_vertex = 0 + group by v.facility_id, v.schedule_day;"""): + # facility_id, day + processors_flow_var_list.append((row[0], row[1])) + + processor_flow_var = LpVariable.dicts("ProcessorDailyFlow", processors_flow_var_list, 0, None, 'Binary') + + return processor_flow_var + + +# =============================================================================== + + +def create_processor_excess_output_vars(the_scenario, logger): + logger.info("START: create_processor_excess_output_vars") + + excess_var_list = [] + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + xs_cur = db_cur.execute(""" + select vertex_id, commodity_id + from vertices v, facility_type_id ft + where v.facility_type_id = ft.facility_type_id + and facility_type = 'processor' + and storage_vertex = 1;""") + # facility_id, day, and simplified commodity name + xs_data = xs_cur.fetchall() + for row in xs_data: + excess_var_list.append(row[0]) + + excess_var = LpVariable.dicts("XS", excess_var_list, 0, None) + + return excess_var + + +# =============================================================================== + + +def create_opt_problem(logger, the_scenario, unmet_demand_vars, flow_vars, processor_build_vars): + logger.debug("START: create_opt_problem") + prob = LpProblem("Flow assignment", LpMinimize) + + unmet_demand_costs = [] + operation_costs = [] + flow_costs = {} + processor_build_costs = [] + for u in unmet_demand_vars: + # facility_id = u[0] + # schedule_day = u[1] + # demand_commodity_name = u[2] + udp = u[3] + unmet_demand_costs.append(udp * unmet_demand_vars[u]) + operation_costs.append(Operation_unitcost * (total_final_demand - unmet_demand_vars[u])) + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + # Flow cost memory improvements: only get needed data; dict instead of list; narrow in lpsum + flow_cost_var = db_cur.execute("select edge_id, edge_flow_cost from edges e group by edge_id;") + flow_cost_data = flow_cost_var.fetchall() + counter = 0 + for row in flow_cost_data: + edge_id = row[0] + edge_flow_cost = row[1] + counter += 1 + + # flow costs cover transportation and storage + flow_costs[edge_id] = edge_flow_cost + # flow_costs.append(edge_flow_cost * flow_vars[(edge_id)]) + + logger.info("check if candidate tables exist") + sql = "SELECT name FROM sqlite_master WHERE type='table' " \ + "AND name in ('candidate_processors', 'candidate_process_list');" + count = len(db_cur.execute(sql).fetchall()) + + if count == 2: + + processor_build_cost = db_cur.execute(""" + select f.facility_id, (p.cost_formula*c.quantity) build_cost + from facilities f, facility_type_id ft, candidate_processors c, candidate_process_list p + where f.facility_type_id = ft.facility_type_id + and facility_type = 'processor' + and candidate = 1 + and ignore_facility = 'false' + and f.facility_name = c.facility_name + and c.process_id = p.process_id + group by f.facility_id, build_cost;""") + processor_build_cost_data = processor_build_cost.fetchall() + for row in processor_build_cost_data: + candidate_proc_facility_id = row[0] + proc_facility_build_cost = row[1] + processor_build_costs.append( + proc_facility_build_cost * processor_build_vars[candidate_proc_facility_id]) + + prob += (lpSum(unmet_demand_costs) + lpSum(operation_costs)+ lpSum(flow_costs[k] * flow_vars[k] for k in flow_costs) + lpSum( + processor_build_costs)), "Total Cost of Transport, storage, facility building, and penalties" + + logger.debug("FINISHED: create_opt_problem") + return prob + + +# =============================================================================== + + +def create_constraint_unmet_demand(logger, the_scenario, prob, flow_var, unmet_demand_var): + logger.debug("START: create_constraint_unmet_demand") + + # apply activity_level to get corresponding actual demand for var + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + # var has form(facility_name, day, simple_fuel) + # unmet demand commodity should be simple_fuel = supertype + + demand_met_dict = defaultdict(list) + actual_demand_dict = {} + + # demand_met = [] + # want to specify that all edges leading into this vertex + unmet demand = total demand + # demand primary (non-storage) vertices + + db_cur = main_db_con.cursor() + # each row_a is a primary vertex whose edges in contributes to the met demand of var + # will have one row for each fuel subtype in the scenario + unmet_data = db_cur.execute("""select v.vertex_id, v.commodity_id, + v.demand, ifnull(c.proportion_of_supertype, 1), ifnull(v.activity_level, 1), v.source_facility_id, + v.facility_id, v.schedule_day, ifnull(c.supertype, c.commodity_name), v.udp, e.edge_id + from vertices v, commodities c, facility_type_id ft, facilities f, edges e + where v.facility_id = f.facility_id + and ft.facility_type = 'ultimate_destination' + and f.facility_type_id = ft.facility_type_id + and f.ignore_facility = 'false' + and v.facility_type_id = ft.facility_type_id + and v.storage_vertex = 0 + and c.commodity_id = v.commodity_id + and e.d_vertex_id = v.vertex_id + group by v.vertex_id, v.commodity_id, + v.demand, ifnull(c.proportion_of_supertype, 1), ifnull(v.activity_level, 1), v.source_facility_id, + v.facility_id, v.schedule_day, ifnull(c.supertype, c.commodity_name), v.udp, e.edge_id + ;""") + + unmet_data = unmet_data.fetchall() + for row_a in unmet_data: + # primary_vertex_id = row_a[0] + # commodity_id = row_a[1] + var_full_demand = row_a[2] + proportion_of_supertype = row_a[3] + var_activity_level = row_a[4] + # source_facility_id = row_a[5] + facility_id = row_a[6] + day = row_a[7] + top_level_commodity = row_a[8] + udp = row_a[9] + edge_id = row_a[10] + var_actual_demand = var_full_demand * var_activity_level + + # next get inbound edges, apply appropriate modifier proportion to get how much of var's demand they satisfy + demand_met_dict[(facility_id, day, top_level_commodity, udp)].append( + flow_var[edge_id] * proportion_of_supertype) + actual_demand_dict[(facility_id, day, top_level_commodity, udp)] = var_actual_demand + + for key in unmet_demand_var: + if key in demand_met_dict: + # then there are some edges in + prob += lpSum(demand_met_dict[key]) == actual_demand_dict[key] - unmet_demand_var[ + key], "constraint set unmet demand variable for facility {}, day {}, commodity {}".format(key[0], + key[1], + key[2]) + else: + if key not in actual_demand_dict: + pdb.set_trace() + # no edges in, so unmet demand equals full demand + prob += actual_demand_dict[key] == unmet_demand_var[ + key], "constraint set unmet demand variable for facility {}, day {}, " \ + "commodity {} - no edges able to meet demand".format( + key[0], key[1], key[2]) + + logger.debug("FINISHED: create_constraint_unmet_demand and return the prob ") + return prob + + +# =============================================================================== + + +def create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, flow_var): + logger.debug("STARTING: create_constraint_max_flow_out_of_supply_vertex") + logger.debug("Length of flow_var: {}".format(len(list(flow_var.items())))) + + # create_constraint_max_flow_out_of_supply_vertex + # primary vertices only + # flow out of a vertex <= supply of the vertex, true for every day and commodity + + # for each primary (non-storage) supply vertex + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + for row_a in db_cur.execute("""select vertex_id, activity_level, supply + from vertices v, facility_type_id ft + where v.facility_type_id = ft.facility_type_id + and ft.facility_type = 'raw_material_producer' + and storage_vertex = 0;"""): + supply_vertex_id = row_a[0] + activity_level = row_a[1] + max_daily_supply = row_a[2] + actual_vertex_supply = activity_level * max_daily_supply + + flow_out = [] + db_cur2 = main_db_con.cursor() + # select all edges leaving that vertex and sum their flows + # should be a single connector edge + for row_b in db_cur2.execute("select edge_id from edges where o_vertex_id = {};".format(supply_vertex_id)): + edge_id = row_b[0] + flow_out.append(flow_var[edge_id]) + + prob += lpSum(flow_out) <= actual_vertex_supply, "constraint max flow of {} out of origin vertex {}".format( + actual_vertex_supply, supply_vertex_id) + # could easily add human-readable vertex info to this if desirable + + logger.debug("FINISHED: create_constraint_max_flow_out_of_supply_vertex") + return prob + + +# =============================================================================== + + +def create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_var, processor_build_vars, + processor_daily_flow_vars): + logger.debug("STARTING: create_constraint_daily_processor_capacity") + import pdb + # pdb.set_trace() + # primary vertices only + # flow into vertex is capped at facility max_capacity per day + # sum over all input commodities, grouped by day and facility + # conservation of flow and ratios are handled in other methods + + ### get primary processor vertex and its input quantityi + total_scenario_min_capacity = 0 + + i = np.load("scenario_num.npy") + t = np.load("time_horizon.npy") + j = np.load("earthquake_week.npy") + logger.info("scenario {}".format(i)) + logger.info("time horizon {}".format(t)) + logger.info("earthquake week {}".format(j)) + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + sql = """select f.facility_id, + ifnull(f.candidate, 0), ifnull(f.max_capacity, -1), v.schedule_day, v.activity_level + from facility_commodities fc, facility_type_id ft, facilities f, vertices v + where ft.facility_type = 'processor' + and ft.facility_type_id = f.facility_type_id + and f.facility_id = fc.facility_id + and fc.io = 'i' + and v.facility_id = f.facility_id + and v.storage_vertex = 0 + group by f.facility_id, ifnull(f.candidate, 0), f.max_capacity, v.schedule_day, v.activity_level + ; + """ + # iterate through processor facilities, one constraint per facility per day + # no handling of subcommodities + + processor_facilities = db_cur.execute(sql) + + processor_facilities = processor_facilities.fetchall() + + for row_a in processor_facilities: + + # input_commodity_id = row_a[0] + facility_id = row_a[0] + is_candidate = row_a[1] + #========================================Modification========================================================================== + # In order to incorporate time-varying facility capacity into FTOT + temp_day = int(7*j) + if temp_day < repair_time_facility[float(facility_id)-1][t+1][i]: + max_capacity = facility_cap[float(facility_id)-1][t+1][i]*row_a[2]*daily_index + else: + max_capacity = facility_cap_noEarthquake[float(facility_id)-1][t+1][i]*row_a[2]*daily_index + #----------------------------------------------------------------------------------------------- + + day = row_a[3] + daily_activity_level = row_a[4] + + if max_capacity >= 0: + daily_inflow_max_capacity = float(max_capacity) * float(daily_activity_level) + daily_inflow_min_capacity = daily_inflow_max_capacity / 2 + logger.debug( + "processor {}, day {}, input capacity min: {} max: {}".format(facility_id, day, daily_inflow_min_capacity, + daily_inflow_max_capacity)) + total_scenario_min_capacity = total_scenario_min_capacity + daily_inflow_min_capacity + flow_in = [] + + # all edges that end in that processor facility primary vertex, on that day + db_cur2 = main_db_con.cursor() + for row_b in db_cur2.execute("""select edge_id from edges e, vertices v + where e.start_day = {} + and e.d_vertex_id = v.vertex_id + and v.facility_id = {} + and v.storage_vertex = 0 + group by edge_id""".format(day, facility_id)): + input_edge_id = row_b[0] + flow_in.append(flow_var[input_edge_id]) + + logger.debug( + "flow in for capacity constraint on processor facility {} day {}: {}".format(facility_id, day, flow_in)) + prob += lpSum(flow_in) <= daily_inflow_max_capacity * processor_daily_flow_vars[(facility_id, day)], \ + "constraint max flow into processor facility {}, day {}, flow var {}".format( + facility_id, day, processor_daily_flow_vars[facility_id, day]) + + prob += lpSum(flow_in) >= daily_inflow_min_capacity * processor_daily_flow_vars[ + (facility_id, day)], "constraint min flow into processor {}, day {}".format(facility_id, day) + # else: + # pdb.set_trace() + + if is_candidate == 1: + # forces processor build var to be correct + # if there is flow through a candidate processor then it has to be built + prob += processor_build_vars[facility_id] >= processor_daily_flow_vars[ + (facility_id, day)], "constraint forces processor build var to be correct {}, {}".format( + facility_id, processor_build_vars[facility_id]) + + logger.debug("FINISHED: create_constraint_daily_processor_capacity") + return prob + + +# =============================================================================== + + +def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow_var): + logger.debug("STARTING: create_primary_processor_vertex_constraints - conservation of flow") + # for all of these vertices, flow in always == flow out + # node_counter = 0 + # node_constraint_counter = 0 + i = np.load("scenario_num.npy") + t = np.load("time_horizon.npy") + logger.info("scenario {}".format(i)) + logger.info("time horizon {}".format(t)) + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + + # total flow in == total flow out, subject to conversion; + # dividing by "required quantity" functionally converts all commodities to the same "processor-specific units" + + # processor primary vertices with input commodity and quantity needed to produce specified output quantities + # 2 sets of constraints; one for the primary processor vertex to cover total flow in and out + # one for each input and output commodity (sum over sources) to ensure its ratio matches facility_commodities + + # the current construction of this method is dependent on having only one input commodity type per processor + # this limitation makes sharing max transport distance from the input to an output commodity feasible + + logger.debug("conservation of flow and commodity ratios, primary processor vertices:") + sql = """select v.vertex_id, + (case when e.o_vertex_id = v.vertex_id then 'out' + when e.d_vertex_id = v.vertex_id then 'in' else 'error' end) in_or_out_edge, + (case when e.o_vertex_id = v.vertex_id then start_day + when e.d_vertex_id = v.vertex_id then end_day else 0 end) constraint_day, + e.commodity_id, + e.mode, + e.edge_id, + nx_edge_id, fc.quantity, v.facility_id, c.commodity_name, + fc.io, + v.activity_level, + ifnull(f.candidate, 0) candidate_check, + e.source_facility_id, + v.source_facility_id, + v.commodity_id, + c.share_max_transport_distance + from vertices v, facility_commodities fc, facility_type_id ft, commodities c, facilities f + join edges e on (v.vertex_id = e.o_vertex_id or v.vertex_id = e.d_vertex_id) + where ft.facility_type = 'processor' + and v.facility_id = f.facility_id + and ft.facility_type_id = v.facility_type_id + and storage_vertex = 0 + and v.facility_id = fc.facility_id + and fc.commodity_id = c.commodity_id + and fc.commodity_id = e.commodity_id + group by v.vertex_id, + in_or_out_edge, + constraint_day, + e.commodity_id, + e.mode, + e.edge_id, + nx_edge_id, fc.quantity, v.facility_id, c.commodity_name, + fc.io, + v.activity_level, + candidate_check, + e.source_facility_id, + v.commodity_id, + v.source_facility_id, + ifnull(c.share_max_transport_distance, 'N') + order by v.facility_id, e.source_facility_id, v.vertex_id, fc.io, e.edge_id + ;""" + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + sql_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for :") + logger.info( + "execute for processor primary vertices, with their in and out edges - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + sql_data = sql_data.fetchall() + logger.info( + "fetchall processor primary vertices, with their in and out edges - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + # Nested dictionaries + # flow_in_lists[primary_processor_vertex_id] = dict of commodities handled by that processor vertex + + # flow_in_lists[primary_processor_vertex_id][commodity1] = + # list of edge ids that flow that commodity into that vertex + + # flow_in_lists[vertex_id].values() to get all flow_in edges for all commodities, a list of lists + # if edge out commodity inherits transport distance, then source_facility id must match. if not, aggregate + + flow_in_lists = {} + flow_out_lists = {} + inherit_max_transport = {} + # inherit_max_transport[commodity_id] = 'Y' or 'N' + + for row_a in sql_data: + + vertex_id = row_a[0] + in_or_out_edge = row_a[1] + # constraint_day = row_a[2] + commodity_id = row_a[3] + # mode = row_a[4] + edge_id = row_a[5] + # nx_edge_id = row_a[6] + facility_id = row_a[8] + quantity = facility_cap_noEarthquake[float(facility_id)-1][t+1][i]*float(row_a[7]) + + # facility_id = row_a[8] + # commodity_name = row_a[9] + # fc_io_commodity = row_a[10] + # activity_level = row_a[11] + # is_candidate = row_a[12] + edge_source_facility_id = row_a[13] + vertex_source_facility_id = row_a[14] + # v_commodity_id = row_a[15] + inherit_max_transport_distance = row_a[16] + if commodity_id not in inherit_max_transport.keys(): + if inherit_max_transport_distance == 'Y': + inherit_max_transport[commodity_id] = 'Y' + else: + inherit_max_transport[commodity_id] = 'N' + + if in_or_out_edge == 'in': + # if the vertex isn't in the main dict yet, add it + # could have multiple source facilities + # could also have more than one input commodity now + flow_in_lists.setdefault(vertex_id, {}) + flow_in_lists[vertex_id].setdefault((commodity_id, quantity, edge_source_facility_id), []).append(flow_var[edge_id]) + # flow_in_lists[vertex_id] is itself a dict keyed on commodity, quantity (ratio) and edge_source_facility; + # value is a list of edge ids into that vertex of that commodity and edge source + + elif in_or_out_edge == 'out': + # for out-lists, could have multiple commodities as well as multiple sources + # some may have a max transport distance, inherited or independent, some may not + flow_out_lists.setdefault(vertex_id, {}) # if the vertex isn't in the main dict yet, add it + flow_out_lists[vertex_id].setdefault((commodity_id, quantity, edge_source_facility_id), []).append(flow_var[edge_id]) + + # Because we keyed on commodity, source facility tracking is merged as we pass through the processor vertex + + # 1) for each output commodity, check against an input to ensure correct ratio - only need one input + # 2) for each input commodity, check against an output to ensure correct ratio - only need one output; + # 2a) first sum sub-flows over input commodity + + # 1---------------------------------------------------------------------- + constrained_input_flow_vars = set([]) + # pdb.set_trace() + + for key, value in iteritems(flow_out_lists): + #value is a dictionary with commodity & source as keys + # set up a dictionary that will be filled with input lists to check ratio against + compare_input_dict = {} + compare_input_dict_commod = {} + vertex_id = key + zero_in = False + #value is a dictionary keyed on output commodity, quantity required, edge source + if vertex_id in flow_in_lists: + in_quantity = 0 + in_commodity_id = 0 + in_source_facility_id = -1 + for ikey, ivalue in iteritems(flow_in_lists[vertex_id]): + in_commodity_id = ikey[0] + in_quantity = ikey[1] + in_source = ikey[2] + # list of edges + compare_input_dict[in_source] = ivalue + # to accommodate and track multiple input commodities; does not keep sources separate + # aggregate lists over sources, by commodity + if in_commodity_id not in compare_input_dict_commod.keys(): + compare_input_dict_commod[in_commodity_id] = set([]) + for edge in ivalue: + compare_input_dict_commod[in_commodity_id].add(edge) + else: + zero_in = True + + + # value is a dict - we loop once here for each output commodity and source at the vertex + for key2, value2 in iteritems(value): + out_commodity_id = key2[0] + out_quantity = key2[1] + out_source = key2[2] + # edge_list = value2 + flow_var_list = value2 + # if we need to match source facility, there is only one set of input lists + # otherwise, use all input lists - this aggregates sources + # need to keep commodities separate, units may be different + # known issue - we could have double-counting problems if only some outputs have to inherit max + # transport distance through this facility + match_source = inherit_max_transport[out_commodity_id] + compare_input_list = [] + if match_source == 'Y': + if len(compare_input_dict_commod.keys()) >1: + error = "Multiple input commodities for processors and shared max transport distance are" \ + " not supported within the same scenario." + logger.error(error) + raise Exception(error) + + if out_source in compare_input_dict.keys(): + compare_input_list = compare_input_dict[out_source] + # if no valid input edges - none for vertex, or if output needs to match source and there are no + # matching source + if zero_in or (match_source == 'Y' and len(compare_input_list) == 0): + prob += lpSum( + flow_var_list) == 0, "processor flow, vertex {} has zero in so zero out of commodity {} " \ + "with source {} if applicable".format( + vertex_id, out_commodity_id, out_source) + else: + if match_source == 'Y': + # ratio constraint for this output commodity relative to total input of each commodity + required_flow_out = lpSum(flow_var_list) / out_quantity + # check against an input dict + prob += required_flow_out == lpSum( + compare_input_list) / in_quantity, "processor flow, vertex {}, source_facility {}," \ + " commodity {} output quantity" \ + " checked against single input commodity quantity".format( + vertex_id, out_source, out_commodity_id, in_commodity_id) + for flow_var in compare_input_list: + constrained_input_flow_vars.add(flow_var) + else: + for k, v in iteritems(compare_input_dict_commod): + # pdb.set_trace() + # as long as the input source doesn't match an output that needs to inherit + compare_input_list = list(v) + in_commodity_id = k + # ratio constraint for this output commodity relative to total input of each commodity + required_flow_out = lpSum(flow_var_list) / out_quantity + # check against an input dict + prob += required_flow_out == lpSum( + compare_input_list) / in_quantity, "processor flow, vertex {}, source_facility {}," \ + " commodity {} output quantity" \ + " checked against commodity {} input quantity".format( + vertex_id, out_source, out_commodity_id, in_commodity_id) + for flow_var in compare_input_list: + constrained_input_flow_vars.add(flow_var) + + for key, value in iteritems(flow_in_lists): + vertex_id = key + for key2, value2 in iteritems(value): + commodity_id = key2[0] + # out_quantity = key2[1] + source = key2[2] + # edge_list = value2 + flow_var_list = value2 + for flow_var in flow_var_list: + if flow_var not in constrained_input_flow_vars: + prob += flow_var == 0, "processor flow, vertex {} has no matching out edges so zero in of " \ + "commodity {} with source {}".format( + vertex_id, commodity_id, source) + + logger.debug("FINISHED: create_primary_processor_conservation_of_flow_constraints") + return prob + + +# =============================================================================== + + +def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, processor_excess_vars): + logger.debug("STARTING: create_constraint_conservation_of_flow") + # node_counter = 0 + node_constraint_counter = 0 + storage_vertex_constraint_counter = 0 + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + + logger.info("conservation of flow, storage vertices:") + # storage vertices, any facility type + # these have at most one direction of transport edges, so no need to track mode + sql = """select v.vertex_id, + (case when e.o_vertex_id = v.vertex_id then 'out' + when e.d_vertex_id = v.vertex_id then 'in' else 'error' end) in_or_out_edge, + (case when e.o_vertex_id = v.vertex_id then start_day + when e.d_vertex_id = v.vertex_id then end_day else 0 end) constraint_day, + v.commodity_id, + e.edge_id, + nx_edge_id, v.facility_id, c.commodity_name, + v.activity_level, + ft.facility_type + + from vertices v, facility_type_id ft, commodities c, facilities f + join edges e on ((v.vertex_id = e.o_vertex_id or v.vertex_id = e.d_vertex_id) + and (e.o_vertex_id = v.vertex_id or e.d_vertex_id = v.vertex_id) and v.commodity_id = e.commodity_id) + + where v.facility_id = f.facility_id + and ft.facility_type_id = v.facility_type_id + and storage_vertex = 1 + and v.commodity_id = c.commodity_id + + group by v.vertex_id, + in_or_out_edge, + constraint_day, + v.commodity_id, + e.edge_id, + nx_edge_id,v.facility_id, c.commodity_name, + v.activity_level + + order by v.facility_id, v.vertex_id, e.edge_id + ;""" + + # get the data from sql and see how long it takes. + logger.info("Starting the long step:") + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + vertexid_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for :") + logger.info("execute for storage vertices, with their in and out edges - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + vertexid_data = vertexid_data.fetchall() + logger.info( + "fetchall nodes with no location id, with their in and out edges - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + flow_in_lists = {} + flow_out_lists = {} + for row_v in vertexid_data: + vertex_id = row_v[0] + in_or_out_edge = row_v[1] + constraint_day = row_v[2] + commodity_id = row_v[3] + edge_id = row_v[4] + # nx_edge_id = row_v[5] + # facility_id = row_v[6] + # commodity_name = row_v[7] + # activity_level = row_v[8] + facility_type = row_v[9] + + if in_or_out_edge == 'in': + flow_in_lists.setdefault((vertex_id, commodity_id, constraint_day, facility_type), []).append( + flow_var[edge_id]) + elif in_or_out_edge == 'out': + flow_out_lists.setdefault((vertex_id, commodity_id, constraint_day, facility_type), []).append( + flow_var[edge_id]) + + logger.info("adding processor excess variabless to conservation of flow") + for key, value in iteritems(flow_out_lists): + vertex_id = key[0] + # commodity_id = key[1] + # day = key[2] + facility_type = key[3] + if facility_type == 'processor': + flow_out_lists.setdefault(key, []).append(processor_excess_vars[vertex_id]) + + for key, value in iteritems(flow_out_lists): + + if key in flow_in_lists: + prob += lpSum(flow_out_lists[key]) == lpSum( + flow_in_lists[key]), "conservation of flow, vertex {}, commodity {}, day {}".format(key[0], key[1], + key[2]) + storage_vertex_constraint_counter = storage_vertex_constraint_counter + 1 + else: + prob += lpSum(flow_out_lists[key]) == lpSum( + 0), "conservation of flow (zero out), vertex {}, commodity {}, day {}".format(key[0], key[1], + key[2]) + storage_vertex_constraint_counter = storage_vertex_constraint_counter + 1 + + for key, value in iteritems(flow_in_lists): + + if key not in flow_out_lists: + prob += lpSum(flow_in_lists[key]) == lpSum( + 0), "conservation of flow (zero in), vertex {}, commodity {}, day {}".format(key[0], key[1], + key[2]) + storage_vertex_constraint_counter = storage_vertex_constraint_counter + 1 + + logger.info( + "total conservation of flow constraints created on nodes: {}".format(storage_vertex_constraint_counter)) + + logger.info("conservation of flow, nx_nodes:") + # for each day, get all edges in and out of the node. + # Sort edges by commodity and whether they're going in or out of the node + sql = """select nn.node_id, + (case when e.from_node_id = nn.node_id then 'out' + when e.to_node_id = nn.node_id then 'in' else 'error' end) in_or_out_edge, + (case when e.from_node_id = nn.node_id then start_day + when e.to_node_id = nn.node_id then end_day else 0 end) constraint_day, + e.commodity_id, + ifnull(mode, 'NULL'), + e.edge_id, nx_edge_id, + miles, + (case when ifnull(nn.source, 'N') == 'intermodal' then 'Y' else 'N' end) intermodal_flag, + e.source_facility_id, + e.commodity_id + from networkx_nodes nn + join edges e on (nn.node_id = e.from_node_id or nn.node_id = e.to_node_id) + where nn.location_id is null + order by nn.node_id, e.commodity_id, + (case when e.from_node_id = nn.node_id then start_day + when e.to_node_id = nn.node_id then end_day else 0 end), + in_or_out_edge, e.source_facility_id, e.commodity_id + ;""" + + logger.info("Starting the long step:") + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + nodeid_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for :") + logger.info( + "execute for nodes with no location id, with their in and out edges - Total Runtime (HMS): \t{} \t " + "".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + nodeid_data = nodeid_data.fetchall() + logger.info( + "fetchall nodes with no location id, with their in and out edges - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + flow_in_lists = {} + flow_out_lists = {} + + for row_a in nodeid_data: + node_id = row_a[0] + in_or_out_edge = row_a[1] + constraint_day = row_a[2] + # commodity_id = row_a[3] + mode = row_a[4] + edge_id = row_a[5] + # nx_edge_id = row_a[6] + # miles = row_a[7] + intermodal = row_a[8] + source_facility_id = row_a[9] + commodity_id = row_a[10] + + # node_counter = node_counter +1 + # if node is not intermodal, conservation of flow holds per mode; + # if intermodal, then across modes + if intermodal == 'N': + if in_or_out_edge == 'in': + flow_in_lists.setdefault( + (node_id, intermodal, source_facility_id, constraint_day, commodity_id, mode), []).append( + flow_var[edge_id]) + elif in_or_out_edge == 'out': + flow_out_lists.setdefault( + (node_id, intermodal, source_facility_id, constraint_day, commodity_id, mode), []).append( + flow_var[edge_id]) + else: + if in_or_out_edge == 'in': + flow_in_lists.setdefault((node_id, intermodal, source_facility_id, constraint_day, commodity_id), + []).append(flow_var[edge_id]) + elif in_or_out_edge == 'out': + flow_out_lists.setdefault((node_id, intermodal, source_facility_id, constraint_day, commodity_id), + []).append(flow_var[edge_id]) + + for key, value in iteritems(flow_out_lists): + node_id = key[0] + # intermodal_flag = key[1] + source_facility_id = key[2] + day = key[3] + commodity_id = key[4] + if len(key) == 6: + node_mode = key[5] + else: + node_mode = 'intermodal' + if key in flow_in_lists: + prob += lpSum(flow_out_lists[key]) == lpSum(flow_in_lists[ + key]), "conservation of flow, nx node {}, " \ + "source facility {}, commodity {}, " \ + "day {}, mode {}".format( + node_id, source_facility_id, commodity_id, day, node_mode) + node_constraint_counter = node_constraint_counter + 1 + else: + prob += lpSum(flow_out_lists[key]) == lpSum( + 0), "conservation of flow (zero out), nx node {}, source facility {}, commodity {}, day {}," \ + " mode {}".format(node_id, source_facility_id, commodity_id, day, node_mode) + node_constraint_counter = node_constraint_counter + 1 + + for key, value in iteritems(flow_in_lists): + node_id = key[0] + # intermodal_flag = key[1] + source_facility_id = key[2] + day = key[3] + commodity_id = key[4] + if len(key) == 6: + node_mode = key[5] + else: + node_mode = 'intermodal' + + if key not in flow_out_lists: + prob += lpSum(flow_in_lists[key]) == lpSum( + 0), "conservation of flow (zero in), nx node {}, source facility {}, commodity {}, day {}," \ + " mode {}".format(node_id, source_facility_id, commodity_id, day, node_mode) + node_constraint_counter = node_constraint_counter + 1 + + logger.info("total conservation of flow constraints created on nodes: {}".format(node_constraint_counter)) + + # Note: no consesrvation of flow for primary vertices for supply & demand - they have unique constraints + + logger.debug("FINISHED: create_constraint_conservation_of_flow") + + return prob + + +# =============================================================================== + + +def create_constraint_max_route_capacity(logger, the_scenario, prob, flow_var): + logger.info("STARTING: create_constraint_max_route_capacity") + logger.info("modes with background flow turned on: {}".format(the_scenario.backgroundFlowModes)) + # min_capacity_level must be a number from 0 to 1, inclusive + # min_capacity_level is only relevant when background flows are turned on + # it sets a floor to how much capacity can be reduced by volume. + # min_capacity_level = .25 means route capacity will never be less than 25% of full capacity, + # even if "volume" would otherwise restrict it further + # min_capacity_level = 0 allows a route to be made unavailable for FTOT flow if base volume is too high + # this currently applies to all modes + logger.info("minimum available capacity floor set at: {}".format(the_scenario.minCapacityLevel)) + i = np.load("scenario_num.npy") + t = np.load("time_horizon.npy") + j = np.load("earthquake_week.npy") + logger.info("scenario {}".format(i)) + logger.info("time horizon {}".format(t)) + logger.info("earthquake week {}".format(j)) + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + # capacity for storage routes + sql = """select + rr.route_id, sr.storage_max, sr.route_name, e.edge_id, e.start_day + from route_reference rr + join storage_routes sr on sr.route_name = rr.route_name + join edges e on rr.route_id = e.route_id + ;""" + # get the data from sql and see how long it takes. + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + storage_edge_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for storage edges:") + logger.info("execute for edges for storage - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + storage_edge_data = storage_edge_data.fetchall() + logger.info("fetchall edges for storage - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + flow_lists = {} + + for row_a in storage_edge_data: + route_id = row_a[0] + aggregate_storage_capac = row_a[1] + storage_route_name = row_a[2] + edge_id = row_a[3] + start_day = row_a[4] + + flow_lists.setdefault((route_id, aggregate_storage_capac, storage_route_name, start_day), []).append( + flow_var[edge_id]) + + for key, flow in iteritems(flow_lists): + prob += lpSum(flow) <= key[1], "constraint max flow on storage route {} named {} for day {}".format(key[0], + key[2], + key[3]) + + logger.debug("route_capacity constraints created for all storage routes") + + # capacity for transport routes + # Assumption - all flowing material is in kgal, all flow is summed on a single non-pipeline nx edge + sql = """select e.edge_id, e.commodity_id, e.nx_edge_id, e.max_edge_capacity, e.start_day, e.simple_mode, e.phase_of_matter, + e.capac_minus_volume_zero_floor + from edges e + where e.max_edge_capacity is not null + and e.simple_mode != 'pipeline' + ;""" + # get the data from sql and see how long it takes. + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + route_capac_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for transport edges:") + logger.info("execute for non-pipeline edges for transport edge capacity - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + route_capac_data = route_capac_data.fetchall() + logger.info("fetchall non-pipeline edges for transport edge capacity - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + flow_lists = {} + + for row_a in route_capac_data: + edge_id = row_a[0] + commodity_id = row_a[1] + nx_edge_id = row_a[2] + nx_edge_capacity = row_a[3] + start_day = row_a[4] + simple_mode = row_a[5] + phase_of_matter = row_a[6] + capac_minus_background_flow = max(row_a[7], 0) + min_restricted_capacity = max(capac_minus_background_flow, nx_edge_capacity * the_scenario.minCapacityLevel) + + if simple_mode in the_scenario.backgroundFlowModes: + use_capacity = min_restricted_capacity + else: + use_capacity = nx_edge_capacity + + # flow is in thousand gallons (kgal), for liquid, or metric tons, for solid + # capacity is in truckload, rail car, barge, or pipeline movement per day + # if mode is road and phase is liquid, capacity is in truckloads per day, we want it in kgal + # ftot_supporting_gis tells us that there are 8 kgal per truckload, + # so capacity * 8 gives us correct units or kgal per day + # => use capacity * ftot_supporting_gis multiplier to get capacity in correct flow units + + multiplier = 1 # if units match, otherwise specified here + if simple_mode == 'road': + if phase_of_matter == 'liquid': + multiplier = the_scenario.truck_load_liquid.magnitude + elif phase_of_matter == 'solid': + multiplier = the_scenario.truck_load_solid.magnitude + elif simple_mode == 'water': + if phase_of_matter == 'liquid': + multiplier = the_scenario.barge_load_liquid.magnitude + elif phase_of_matter == 'solid': + multiplier = the_scenario.barge_load_solid.magnitude + elif simple_mode == 'rail': + if phase_of_matter == 'liquid': + multiplier = the_scenario.railcar_load_liquid.magnitude + elif phase_of_matter == 'solid': + multiplier = the_scenario.railcar_load_solid.magnitude + + #=========================================Modification==================================================================== + # Following part is modified to include time-varying edge capacity in FTOT + temp_day = int(7*j) + if float(edge_id) in repair_time_edge[:,1,1]: + if temp_day < repair_time_edge[float(edge_id)-1][t+1][i]: + capacity_reduction_index = edge_cap[float(edge_id)-1][t+1][i] + else: + capacity_reduction_index = 1 + else: + capacity_reduction_index = 1 + #logger.info("capacity_reduction_index {}".format(capacity_reduction_index)) + converted_capacity = use_capacity * multiplier * capacity_reduction_index + + flow_lists.setdefault((nx_edge_id, converted_capacity, commodity_id, start_day), []).append(flow_var[edge_id]) + + for key, flow in iteritems(flow_lists): + prob += lpSum(flow) <= key[1], "constraint max flow on nx edge {} commodity_id {} for day {}".format(key[0], key[2], key[3]) + + logger.debug("route_capacity constraints created for all non-pipeline transport routes") + + logger.debug("FINISHED: create_constraint_max_route_capacity") + return prob + + +# =============================================================================== + + +def create_constraint_pipeline_capacity(logger, the_scenario, prob, flow_var): + logger.debug("STARTING: create_constraint_pipeline_capacity") + logger.debug("Length of flow_var: {}".format(len(list(flow_var.items())))) + logger.info("modes with background flow turned on: {}".format(the_scenario.backgroundFlowModes)) + logger.info("minimum available capacity floor set at: {}".format(the_scenario.minCapacityLevel)) + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + + # capacity for pipeline tariff routes + # with sasc, may have multiple flows per segment, slightly diff commodities + sql = """select e.edge_id, e.tariff_id, l.link_id, l.capac, e.start_day, l.capac-l.background_flow allowed_flow, + l.source, e.mode, instr(e.mode, l.source) + from edges e, pipeline_mapping pm, + (select id_field_name, cn.source_OID as link_id, min(cn.capacity) capac, + max(cn.volume) background_flow, source + from capacity_nodes cn + where cn.id_field_name = 'MASTER_OID' + and ifnull(cn.capacity,0)>0 + group by link_id) l + + where e.tariff_id = pm.id + and pm.id_field_name = 'tariff_ID' + and pm.mapping_id_field_name = 'MASTER_OID' + and l.id_field_name = 'MASTER_OID' + and pm.mapping_id = l.link_id + and instr(e.mode, l.source)>0 + group by e.edge_id, e.tariff_id, l.link_id, l.capac, e.start_day, allowed_flow, l.source + ;""" + # capacity needs to be shared over link_id for any edge_id associated with that link + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + pipeline_capac_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for transport edges:") + logger.info("execute for edges for transport edge capacity - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + pipeline_capac_data = pipeline_capac_data.fetchall() + logger.info("fetchall edges for transport edge capacity - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + flow_lists = {} + + for row_a in pipeline_capac_data: + edge_id = row_a[0] + # tariff_id = row_a[1] + link_id = row_a[2] + # Link capacity is recorded in "thousand barrels per day"; 1 barrel = 42 gall + # Link capacity * 42 is now in kgal per day, to match flow in kgal + link_capacity_kgal_per_day = THOUSAND_GALLONS_PER_THOUSAND_BARRELS * row_a[3] + start_day = row_a[4] + capac_minus_background_flow_kgal = max(THOUSAND_GALLONS_PER_THOUSAND_BARRELS * row_a[5], 0) + min_restricted_capacity = max(capac_minus_background_flow_kgal, + link_capacity_kgal_per_day * the_scenario.minCapacityLevel) + + # capacity_nodes_mode_source = row_a[6] + edge_mode = row_a[7] + # mode_match_check = row_a[8] + if 'pipeline' in the_scenario.backgroundFlowModes: + link_use_capacity = min_restricted_capacity + else: + link_use_capacity = link_capacity_kgal_per_day + + # add flow from all relevant edges, for one start; may be multiple tariffs + flow_lists.setdefault((link_id, link_use_capacity, start_day, edge_mode), []).append(flow_var[edge_id]) + + for key, flow in iteritems(flow_lists): + prob += lpSum(flow) <= key[1], "constraint max flow on pipeline link {} for mode {} for day {}".format( + key[0], key[3], key[2]) + + logger.debug("pipeline capacity constraints created for all transport routes") + + logger.debug("FINISHED: create_constraint_pipeline_capacity") + return prob + + +# =============================================================================== + + +def setup_pulp_problem(the_scenario, logger): + logger.info("START: setup PuLP problem") + + # flow_var is the flow on each edge by commodity and day. + # the optimal value of flow_var will be solved by PuLP + flow_vars = create_flow_vars(the_scenario, logger) + + # unmet_demand_var is the unmet demand at each destination, being determined + unmet_demand_vars = create_unmet_demand_vars(the_scenario, logger) + + # processor_build_vars is the binary variable indicating whether a candidate processor is used + # and thus whether its build cost is charged + processor_build_vars = create_candidate_processor_build_vars(the_scenario, logger) + + # binary tracker variables + processor_vertex_flow_vars = create_binary_processor_vertex_flow_vars(the_scenario, logger) + + # tracking unused production + processor_excess_vars = create_processor_excess_output_vars(the_scenario, logger) + + # THIS IS THE OBJECTIVE FUCTION FOR THE OPTIMIZATION + # ================================================== + + prob = create_opt_problem(logger, the_scenario, unmet_demand_vars, flow_vars, processor_build_vars) + + prob = create_constraint_unmet_demand(logger, the_scenario, prob, flow_vars, unmet_demand_vars) + + prob = create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, flow_vars) + + # This constraint is being excluded because 1) it is not used in current scenarios and 2) it is not supported by + # this version - it conflicts with the change permitting multiple inputs + # adding back 12/2020 + prob = create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_vars, processor_build_vars, + processor_vertex_flow_vars) + + prob = create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow_vars) + + prob = create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_vars, processor_excess_vars) + + if the_scenario.capacityOn: + prob = create_constraint_max_route_capacity(logger, the_scenario, prob, flow_vars) + + prob = create_constraint_pipeline_capacity(logger, the_scenario, prob, flow_vars) + + del unmet_demand_vars + + del flow_vars + + # The problem data is written to an .lp file + prob.writeLP(os.path.join(the_scenario.scenario_run_directory, "debug", "LP_output_c2.lp")) + + logger.info("FINISHED: setup PuLP problem") + return prob + + +# =============================================================================== + + +def solve_pulp_problem(prob_final, the_scenario, logger): + import datetime + + logger.info("START: solve_pulp_problem") + start_time = datetime.datetime.now() + from os import dup, dup2, close + f = open(os.path.join(the_scenario.scenario_run_directory, "debug", 'probsolve_capture.txt'), 'w') + orig_std_out = dup(1) + dup2(f.fileno(), 1) + + # status = prob_final.solve (PULP_CBC_CMD(maxSeconds = i_max_sec, fracGap = d_opt_gap, msg=1)) + # CBC time limit and relative optimality gap tolerance + status = prob_final.solve(PULP_CBC_CMD(msg=1)) # CBC time limit and relative optimality gap tolerance + logger.info('Completion code: %d; Solution status: %s; Best obj value found: %s' % ( + status, LpStatus[prob_final.status], value(prob_final.objective))) + + dup2(orig_std_out, 1) + close(orig_std_out) + f.close() + # The problem is solved using PuLP's choice of Solver + + logger.info("completed calling prob.solve()") + logger.info( + "FINISH: prob.solve(): Runtime (HMS): \t{}".format(ftot_supporting.get_total_runtime_string(start_time))) + + # THIS IS THE SOLUTION + + # The status of the solution is printed to the screen + ##LpStatus key string value numerical value + ##LpStatusOptimal ?Optimal? 1 + ##LpStatusNotSolved ?Not Solved? 0 + ##LpStatusInfeasible ?Infeasible? -1 + ##LpStatusUnbounded ?Unbounded? -2 + ##LpStatusUndefined ?Undefined? -3 + logger.result("prob.Status: \t {}".format(LpStatus[prob_final.status])) + + logger.result( + "Total Scenario Cost = (transportation + unmet demand penalty + processor construction): \t ${0:,.0f}".format( + float(value(prob_final.objective)))) + + return prob_final + + +# =============================================================================== + +def save_pulp_solution(the_scenario, prob, logger, zero_threshold=0.00001): + i = np.load("scenario_num.npy") + t = np.load("time_horizon.npy") + j = np.load("earthquake_week.npy") + logger.info("scenario {}".format(i)) + logger.info("time horizon {}".format(t)) + logger.info("earthquake week {}".format(j)) + + import datetime + start_time = datetime.datetime.now() + logger.info("START: save_pulp_solution") + non_zero_variable_count = 0 + + with sqlite3.connect(the_scenario.main_db) as db_con: + + db_cur = db_con.cursor() + # drop the optimal_solution table + # ----------------------------- + db_cur.executescript("drop table if exists optimal_solution;") + + # create the optimal_solution table + # ----------------------------- + db_cur.executescript(""" + create table optimal_solution + ( + variable_name string, + variable_value real + ); + """) + + # insert the optimal data into the DB + # ------------------------------------- + for v in prob.variables(): + if v.varValue is None: + logger.debug("Variable value is none: " + str(v.name)) + else: + if v.varValue > zero_threshold: # eliminates values too close to zero + sql = """insert into optimal_solution (variable_name, variable_value) values ("{}", {});""".format( + v.name, float(v.varValue)) + db_con.execute(sql) + non_zero_variable_count = non_zero_variable_count + 1 + + # query the optimal_solution table in the DB for each variable we care about + # ---------------------------------------------------------------------------- + sql = "select count(variable_name) from optimal_solution where variable_name like 'BuildProcessor%';" + data = db_con.execute(sql) + optimal_processors_count = data.fetchone()[0] + logger.info("number of optimal_processors: {}".format(optimal_processors_count)) + + sql = "select count(variable_name) from optimal_solution where variable_name like 'UnmetDemand%';" + data = db_con.execute(sql) + optimal_unmet_demand_count = data.fetchone()[0] + logger.info("number facilities with optimal_unmet_demand : {}".format(optimal_unmet_demand_count)) + sql = "select ifnull(sum(variable_value),0) from optimal_solution where variable_name like 'UnmetDemand%';" + data = db_con.execute(sql) + optimal_unmet_demand_sum = data.fetchone()[0] + + unmet_demand_daily = optimal_unmet_demand_sum + np.save("unmet_demand_daily.npy", unmet_demand_daily) + + + + logger.info("Total Unmet Demand : {}".format(optimal_unmet_demand_sum)) + logger.info("Penalty per unit of Unmet Demand : ${0:,.0f}".format(the_scenario.unMetDemandPenalty)) + logger.info("Total Cost of Unmet Demand : \t ${0:,.0f}".format( + optimal_unmet_demand_sum * the_scenario.unMetDemandPenalty)) + + #=================================non consider the UDP when UD < 0====================================== + if optimal_unmet_demand_sum < 0: + costs_daily = (abs(float(value(prob.objective))- optimal_unmet_demand_sum * the_scenario.unMetDemandPenalty))*((1+Inflation_rate)**t) + sum(CatalystReplace_cost[:,t+1,i])*daily_index + repair_costs[int(j*7)][t][i] + + logger.result( + "Total Scenario Cost = (transportation + processor construction + operation + catalyst replacement): \t ${0:,.0f}" + "".format(costs_daily)) + else: + costs_daily = (abs(float(value(prob.objective))-optimal_unmet_demand_sum * the_scenario.unMetDemandPenalty))*((1+Inflation_rate)**t)+sum(CatalystReplace_cost[:,t+1,i])*daily_index + repair_costs[int(j*7)][t][i] + (optimal_unmet_demand_sum*the_scenario.unMetDemandPenalty)*((1+Inflation_rate)**t) + logger.result( + "Total Scenario Cost = (transportation + unmet demand penalty + processor construction + operation+ catalyst replacement): \t ${0:,.0f}" + "".format(costs_daily)) + + np.save("costs_daily.npy", costs_daily) + logger.info("costs_daily: {}".format(costs_daily)) + #============================================================ + + sql = "select count(variable_name) from optimal_solution where variable_name like 'Edge%';" + data = db_con.execute(sql) + optimal_edges_count = data.fetchone()[0] + logger.info("number of optimal edges: {}".format(optimal_edges_count)) + + + logger.info( + "FINISH: save_pulp_solution: Runtime (HMS): \t{}".format(ftot_supporting.get_total_runtime_string(start_time))) + +# =============================================================================== + +def record_pulp_solution(the_scenario, logger): + logger.info("START: record_pulp_solution") + non_zero_variable_count = 0 + + with sqlite3.connect(the_scenario.main_db) as db_con: + + logger.info("number of solution variables greater than zero: {}".format(non_zero_variable_count)) + sql = """ + create table optimal_variables as + select + 'UnmetDemand' as variable_type, + cast(substr(variable_name, 13) as int) var_id, + variable_value, + null as converted_capacity, + null as converted_volume, + null as converted_capac_minus_volume, + null as edge_type, + null as commodity_name, + null as o_facility, + 'placeholder' as d_facility, + null as o_vertex_id, + null as d_vertex_id, + null as from_node_id, + null as to_node_id, + null as time_period, + null as commodity_id, + null as source_facility_id, + null as source_facility_name, + null as units, + variable_name, + null as nx_edge_id, + null as mode, + null as mode_oid, + null as miles, + null as original_facility, + null as final_facility, + null as prior_edge, + null as miles_travelled + from optimal_solution + where variable_name like 'UnmetDemand%' + union + select + 'Edge' as variable_type, + cast(substr(variable_name, 6) as int) var_id, + variable_value, + edges.max_edge_capacity*edges.units_conversion_multiplier as converted_capacity, + edges.volume*edges.units_conversion_multiplier as converted_volume, + edges.capac_minus_volume_zero_floor*edges.units_conversion_multiplier as converted_capac_minus_volume, + edges.edge_type, + commodities.commodity_name, + ov.facility_name as o_facility, + dv.facility_name as d_facility, + o_vertex_id, + d_vertex_id, + from_node_id, + to_node_id, + start_day time_period, + edges.commodity_id, + edges.source_facility_id, + s.source_facility_name, + commodities.units, + variable_name, + edges.nx_edge_id, + edges.mode, + edges.mode_oid, + edges.miles, + null as original_facility, + null as final_facility, + null as prior_edge, + edges.miles_travelled as miles_travelled + from optimal_solution + join edges on edges.edge_id = cast(substr(variable_name, 6) as int) + join commodities on edges.commodity_id = commodities.commodity_ID + left outer join vertices as ov on edges.o_vertex_id = ov.vertex_id + left outer join vertices as dv on edges.d_vertex_id = dv.vertex_id + left outer join source_commodity_ref as s on edges.source_facility_id = s.source_facility_id + where variable_name like 'Edge%' + union + select + 'BuildProcessor' as variable_type, + cast(substr(variable_name, 16) as int) var_id, + variable_value, + null as converted_capacity, + null as converted_volume, + null as converted_capac_minus_volume, + null as edge_type, + null as commodity_name, + 'placeholder' as o_facility, + 'placeholder' as d_facility, + null as o_vertex_id, + null as d_vertex_id, + null as from_node_id, + null as to_node_id, + null as time_period, + null as commodity_id, + null as source_facility_id, + null as source_facility_name, + null as units, + variable_name, + null as nx_edge_id, + null as mode, + null as mode_oid, + null as miles, + null as original_facility, + null as final_facility, + null as prior_edge, + null as miles_travelled + from optimal_solution + where variable_name like 'Build%'; + """ + db_con.execute("drop table if exists optimal_variables;") + db_con.execute(sql) + + logger.info("FINISH: record_pulp_solution") + +# =============================================================================== + + +def parse_optimal_solution_db(the_scenario, logger): + logger.info("starting parse_optimal_solution") + + optimal_processors = [] + optimal_processor_flows = [] + optimal_route_flows = {} + optimal_unmet_demand = {} + optimal_storage_flows = {} + optimal_excess_material = {} + + with sqlite3.connect(the_scenario.main_db) as db_con: + + # do the Storage Edges + sql = "select variable_name, variable_value from optimal_solution where variable_name like 'Edge%_storage';" + data = db_con.execute(sql) + optimal_storage_edges = data.fetchall() + for edge in optimal_storage_edges: + optimal_storage_flows[edge] = optimal_storage_edges[edge] + + # do the Route Edges + sql = """select + variable_name, variable_value, + cast(substr(variable_name, 6) as int) edge_id, + route_id, start_day time_period, edges.commodity_id, + o_vertex_id, d_vertex_id, + v1.facility_id o_facility_id, + v2.facility_id d_facility_id + from optimal_solution + join edges on edges.edge_id = cast(substr(variable_name, 6) as int) + join vertices v1 on edges.o_vertex_id = v1.vertex_id + join vertices v2 on edges.d_vertex_id = v2.vertex_id + where variable_name like 'Edge%_' and variable_name not like 'Edge%_storage'; + """ + data = db_con.execute(sql) + optimal_route_edges = data.fetchall() + for edge in optimal_route_edges: + + variable_name = edge[0] + + variable_value = edge[1] + + edge_id = edge[2] + + route_id = edge[3] + + time_period = edge[4] + + commodity_flowed = edge[5] + + od_pair_name = "{}, {}".format(edge[8], edge[9]) + + # first time route_id is used on a day or commodity + if route_id not in optimal_route_flows: + optimal_route_flows[route_id] = [[od_pair_name, time_period, commodity_flowed, variable_value]] + + else: # subsequent times route is used on different day or for other commodity + optimal_route_flows[route_id].append([od_pair_name, time_period, commodity_flowed, variable_value]) + + # do the processors + sql = "select variable_name, variable_value from optimal_solution where variable_name like 'BuildProcessor%';" + data = db_con.execute(sql) + optimal_candidates_processors = data.fetchall() + for proc in optimal_candidates_processors: + optimal_processors.append(proc) + + # do the processor vertex flows + sql = "select variable_name, variable_value from optimal_solution where variable_name like 'ProcessorVertexFlow%';" + data = db_con.execute(sql) + optimal_processor_flows_sql = data.fetchall() + for proc in optimal_processor_flows_sql: + optimal_processor_flows.append(proc) + + # do the UnmetDemand + sql = "select variable_name, variable_value from optimal_solution where variable_name like 'UnmetDemand%';" + data = db_con.execute(sql) + optimal_unmetdemand = data.fetchall() + for ultimate_destination in optimal_unmetdemand: + v_name = ultimate_destination[0] + v_value = ultimate_destination[1] + + search = re.search('\(.*\)', v_name.replace("'", "")) + + if search: + parts = search.group(0).replace("(", "").replace(")", "").split(",_") + + dest_name = parts[0] + commodity_flowed = parts[2] + if not dest_name in optimal_unmet_demand: + optimal_unmet_demand[dest_name] = {} + + if not commodity_flowed in optimal_unmet_demand[dest_name]: + optimal_unmet_demand[dest_name][commodity_flowed] = int(v_value) + else: + optimal_unmet_demand[dest_name][commodity_flowed] += int(v_value) + + + logger.info("length of optimal_processors list: {}".format(len(optimal_processors))) # a list of optimal processors + logger.info("length of optimal_processor_flows list: {}".format( + len(optimal_processor_flows))) # a list of optimal processor flows + logger.info("length of optimal_route_flows dict: {}".format( + len(optimal_route_flows))) # a dictionary of routes keys and commodity flow values + logger.info("length of optimal_unmet_demand dict: {}".format( + len(optimal_unmet_demand))) # a dictionary of route keys and unmet demand values + + return optimal_processors, optimal_route_flows, optimal_unmet_demand, optimal_storage_flows, optimal_excess_material + + diff --git a/program/ftot_pulp_yearly.py b/program/ftot_pulp_yearly.py new file mode 100644 index 0000000..d065a37 --- /dev/null +++ b/program/ftot_pulp_yearly.py @@ -0,0 +1,3423 @@ +# --------------------------------------------------------------------------------------------------- +# Name: ftot_pulp +# +# Purpose: PulP optimization - create and run a modified facility location problem. +# Take NetworkX and GIS scenario data as recorded in main.db and convert to a structure of edges, nodes, vertices. +# Create variables for flow over edges, unmet demand, processor use, and candidate processors to build if present +# Solve cost minimization for unmet demand, transportation, and facility build costs +# Constraints ensure compliance with scenario requirements (e.g. max_route_capacity) +# as well as general problem structure (e.g. conservation_of_flow) +# --------------------------------------------------------------------------------------------------- + +import datetime +import pdb +import re +import sqlite3 +import numpy as np +from collections import defaultdict +from six import iteritems + +from pulp import * + +import ftot_supporting +from ftot_supporting import get_total_runtime_string + +# =================== constants============= +storage = 1 +primary = 0 +fixed_schedule_id = 2 +fixed_route_duration = 0 + +THOUSAND_GALLONS_PER_THOUSAND_BARRELS = 42 + +storage_cost_1 = 0.01 +storage_cost_2 = 0.05 +facility_onsite_storage_max = 10000000000 +facility_onsite_storage_min = 0 + +default_max_capacity = 10000000000 +default_min_capacity = 0 +#==================load output from newly developed Python files===== +# load facility capacity index +facility_cap_noEarthquake = np.load("facility_cap_noEarthquake.npy") +# load catalyst replament cost +CatalystReplace_cost = np.load("CatalystReplace_cost.npy") +# GFT facility operation cost +Operation_unitcost = 0.00579 # unit:$/kgal (2021 dollars) +total_final_demand = 4073312.9 + 254401.1 +Inflation_rate = 0.0279 +# daily index +daily_index = float(1)/float(365) +#unmet_demand_yearly = np.zeros((N,plan_horizon)) +#costs_yearly = np.zeros((N,plan_horizon)) +number = np.zeros(len(facility_cap_noEarthquake[:,0,0])) +for i in range(len(number)): + number[i] = i +#==================================================================== + +def o1(the_scenario, logger): + # create vertices, then edges for permitted modes, then set volume & capacity on edges + pre_setup_pulp(logger, the_scenario) + + +def o2(the_scenario, logger): + # create variables, problem to optimize, and constraints + prob = setup_pulp_problem(the_scenario, logger) + prob = solve_pulp_problem(prob, the_scenario, logger) + save_pulp_solution(the_scenario, prob, logger) + record_pulp_solution(the_scenario, logger) + from ftot_supporting import post_optimization + post_optimization(the_scenario, 'o2', logger) + + +# helper function that reads in schedule data and returns dict of Schedule objects +def generate_schedules(the_scenario, logger): + logger.debug("start: generate_schedules") + default_availabilities = {} + day_availabilities = {} + last_day = 1 + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + + schedule_data = db_cur.execute(""" + select schedule_id, + day, + availability + + from schedules""") + + schedule_data = schedule_data.fetchall() + for row_a in schedule_data: + schedule_id = row_a[0] + day = row_a[1] + availability = float(row_a[2]) + + if day == 0: # denotes the default value + default_availabilities[schedule_id] = availability + elif schedule_id in day_availabilities.keys(): + day_availabilities[schedule_id][day] = availability + else: + day_availabilities[schedule_id] = {day: availability} # initialize sub-dic + # last_day is the greatest day specified across ALL schedules to avoid mismatch of length + if day > last_day: + last_day = day + + # make dictionary to store schedule objects + schedule_dict = {} + + # after reading in csv, parse data into dictionary object + for schedule_id in default_availabilities.keys(): + # initialize list of length + schedule_dict[schedule_id] = [default_availabilities[schedule_id] for i in range(last_day)] + # change different days to actual availability instead of the default values + # schedule_id may not be in day_availabilities if schedule has default value on all days + if schedule_id in day_availabilities.keys(): + for day in day_availabilities[schedule_id].keys(): + # schedule_dict[schedule + schedule_dict[schedule_id][day-1] = day_availabilities[schedule_id][day] + + for key in schedule_dict.keys(): + logger.debug("schedule name: " + str(key)) + logger.debug("availability: ") + logger.debug(schedule_dict[key]) + logger.debug("finished: generate_schedules") + + return schedule_dict, last_day + + +def commodity_mode_setup(the_scenario, logger): + + # helper method to read in the csv and make a dict + def make_commodity_mode_dict(the_scenario, logger): + logger.info("START: make_commodity_mode_dict") + # check if path to table exists + if not os.path.exists(the_scenario.commodity_mode_data): + logger.warning("warning: cannot find commodity_mode_data file: {}".format(the_scenario.commodity_mode_data)) + return {} # return empty dict + # otherwise, initialize dict and read through commodity_mode CSV + commodity_mode_dict = {} + with open(the_scenario.commodity_mode_data, 'r') as rf: + line_num = 1 + modes = None # will assign within for loop + for line in rf: + if line_num == 1: + modes = line.rstrip('\n').split(',') + else: + flds = line.rstrip('\n').split(',') + commodity_name = flds[0] + allowed = flds[1:] + commodity_mode_dict[commodity_name] = dict(zip(modes[1:], allowed)) + # now do a check + for mode in modes[1:]: + if commodity_mode_dict[commodity_name][mode] in ['Y', 'N']: + logger.info("Commodity: {}, Mode: {}, Allowed: {}".format(commodity_name, mode, + commodity_mode_dict[commodity_name][mode])) + else: + # if val isn't Y or N, remove the key from the dict + del commodity_mode_dict[commodity_name][mode] + logger.info( + "improper or no value in Commodity_Mode_Data csv for commodity: {} and mode: {}".\ + format(commodity_name, mode)) + logger.info("default value will be used for commodity: {} and mode: {}".\ + format(commodity_name, mode)) + + line_num += 1 + logger.info("FINISHED: make_commodity_mode_dict") + return commodity_mode_dict + + logger.info("START: commodity_mode_setup") + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + main_db_con.executescript(""" + drop table if exists commodity_mode; + + create table commodity_mode( + mode text, + commodity_id text, + commodity_phase text, + allowed_yn text, + CONSTRAINT unique_commodity_and_mode UNIQUE(commodity_id, mode)) + ;""") + + commod = main_db_con.execute("select commodity_name, commodity_id, phase_of_matter from commodities;") + commod = commod.fetchall() + commodities = {} + for row in commod: + commodity_name = row[0] + commodity_id = row[1] + phase_of_matter = row[2] + commodities[commodity_name] = (commodity_id, phase_of_matter) + + # read in from csv file and insert those entries first, using commodity id + # then, fill in for all other unspecified commodities and modes, pipeline liquid only + commodity_mode_dict = make_commodity_mode_dict(the_scenario, logger) + for mode in the_scenario.permittedModes: + for k, v in iteritems(commodities): + + commodity_name = k + commodity_id = v[0] + phase_of_matter = v[1] + # check commodity mode permissions for all modes. Apply resiction is specified, otherwise default to + # allowed if not specified. + if commodity_name in commodity_mode_dict and mode in commodity_mode_dict[commodity_name]: + allowed = commodity_mode_dict[commodity_name][mode] + else: + allowed = 'Y' + # pipeline is a special case. so double check if it is explicitly allowed, otherwise override to 'N'. + # restrict solids on pipeline no matter what. + if mode.partition('_')[0] == 'pipeline': + # 8/26/19 -- MNP -- note: that mode is the long name, and the dict is the short name + if mode == 'pipeline_crude_trf_rts': + short_name_mode = 'pipeline_crude' + elif mode == 'pipeline_prod_trf_rts': + short_name_mode = 'pipeline_prod' + else: logger.warning("a pipeline was specified that is not supported") + # restrict pipeline if its not explicitly allowed or if solid + if commodity_name in commodity_mode_dict and short_name_mode in commodity_mode_dict[commodity_name]: + allowed = commodity_mode_dict[commodity_name][short_name_mode] + else: + allowed = 'N' + if phase_of_matter != 'liquid': + allowed = 'N' + + main_db_con.execute(""" + insert or ignore into commodity_mode + (mode, commodity_id, commodity_phase, allowed_yn) + VALUES + ('{}',{},'{}','{}') + ; + """.format(mode, commodity_id, phase_of_matter, allowed)) + + return +# =============================================================================== + + +def source_tracking_setup(the_scenario, logger): + logger.info("START: source_tracking_setup") + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + max_inputs = main_db_con.execute("""select max(inputs) from + (select count(fc.commodity_id) inputs + from facility_commodities fc, facility_type_id ft + where ft.facility_type = 'processor' + and fc.io = 'i' + group by fc.facility_id) + ;""") + + for row in max_inputs: + if row[0] == None: + max_inputs_to_a_processor = 0 + else: + max_inputs_to_a_processor = int(row[0]) + if max_inputs_to_a_processor > 1: + logger.warning("error, this version of the optimization step only functions correctly with a single input" + " commodity type per processor") + + main_db_con.executescript(""" + + insert or ignore into commodities(commodity_name) values ('multicommodity'); + + drop table if exists source_commodity_ref + ; + + create table source_commodity_ref(id INTEGER PRIMARY KEY, + source_facility_id integer, + source_facility_name text, + source_facility_type_id integer, --lets us differentiate spiderweb from processor + commodity_id integer, + commodity_name text, + units text, + phase_of_matter text, + max_transport_distance numeric, + max_transport_distance_flag text, + share_max_transport_distance text, + CONSTRAINT unique_source_and_name UNIQUE(commodity_id, source_facility_id)) + ; + + insert or ignore into source_commodity_ref ( + source_facility_id, + source_facility_name, + source_facility_type_id, + commodity_id, + commodity_name, + units, + phase_of_matter, + max_transport_distance, + max_transport_distance_flag, + share_max_transport_distance) + select + f.facility_id, + f.facility_name, + f.facility_type_id, + c.commodity_id, + c.commodity_name, + c.units, + c.phase_of_matter, + (case when c.max_transport_distance is not null then + c.max_transport_distance else Null end) max_transport_distance, + (case when c.max_transport_distance is not null then 'Y' else 'N' end) max_transport_distance_flag, + (case when ifnull(c.share_max_transport_distance, 'N') = 'Y' then 'Y' else 'N' end) share_max_transport_distance + + from commodities c, facilities f, facility_commodities fc + where f.facility_id = fc.facility_id + and f.ignore_facility = 'false' + and fc.commodity_id = c.commodity_id + and fc.io = 'o' + and ifnull(c.share_max_transport_distance, 'N') != 'Y' + ; + + insert or ignore into source_commodity_ref ( + source_facility_id, + source_facility_name, + source_facility_type_id, + commodity_id, + commodity_name, + units, + phase_of_matter, + max_transport_distance, + max_transport_distance_flag, + share_max_transport_distance) + select + sc.source_facility_id, + sc.source_facility_name, + sc.source_facility_type_id, + o.commodity_id, + c.commodity_name, + c.units, + c.phase_of_matter, + sc.max_transport_distance, + sc.max_transport_distance_flag, + o.share_max_transport_distance + from source_commodity_ref sc, facility_commodities i, facility_commodities o, commodities c + where o.share_max_transport_distance = 'Y' + and sc.commodity_id = i.commodity_id + and o.facility_id = i.facility_id + and o.io = 'o' + and i.io = 'i' + and o.commodity_id = c.commodity_id + ; + """ + ) + + return + + +# =============================================================================== + + +def generate_all_vertices(the_scenario, schedule_dict, schedule_length, logger): + logger.info("START: generate_all_vertices table") + + i = np.load("scenario_num.npy") + t = np.load("time_horizon.npy") + logger.info("scenario {}".format(i)) + logger.info("time horizon {}".format(t)) + + + total_potential_production = {} + multi_commodity_name = "multicommodity" + + storage_availability = 1 + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + logger.debug("create the vertices table") + # create the vertices table + main_db_con.executescript(""" + drop table if exists vertices + ; + + create table if not exists vertices ( + vertex_id INTEGER PRIMARY KEY, location_id, + facility_id integer, facility_name text, facility_type_id integer, schedule_day integer, + commodity_id integer, activity_level numeric, storage_vertex binary, + udp numeric, supply numeric, demand numeric, + source_facility_id integer, + iob text, --allows input, output, or both + CONSTRAINT unique_vertex UNIQUE(facility_id, schedule_day, commodity_id, source_facility_id, storage_vertex)) + ;""") + + # create indexes for the networkx nodes and links tables + logger.info("create an index for the networkx nodes and links tables") + main_db_con.executescript(""" + CREATE INDEX IF NOT EXISTS node_index ON networkx_nodes (node_id, location_id) + ; + + create index if not exists nx_edge_index on + networkx_edges(from_node_id, to_node_id, + artificial, mode_source, mode_source_OID, + miles, route_cost_scaling, capacity) + ; + """) + + # -------------------------------- + + db_cur = main_db_con.cursor() + # nested cursor + db_cur4 = main_db_con.cursor() + counter = 0 + total_facilities = 0 + + for row in db_cur.execute("select count(distinct facility_id) from facilities;"): + total_facilities = row[0] + + # create vertices for each non-ignored facility facility + # facility_type can be "raw_material_producer", "ultimate_destination","processor"; + # get id from facility_type_id table + # any other facility types are not currently handled + + facility_data = db_cur.execute(""" + select facility_id, + facility_type, + facility_name, + location_id, + f.facility_type_id, + schedule_id + + from facilities f, facility_type_id ft + where ignore_facility = '{}' + and f.facility_type_id = ft.facility_type_id; + """.format('false')) + facility_data = facility_data.fetchall() + for row_a in facility_data: + + db_cur2 = main_db_con.cursor() + facility_id = row_a[0] + facility_type = row_a[1] + facility_name = row_a[2] + facility_location_id = row_a[3] + facility_type_id = row_a[4] + schedule_id = row_a[5] + if counter % 10000 == 1: + logger.info("vertices created for {} facilities of {}".format(counter, total_facilities)) + for row_d in db_cur4.execute("select count(distinct vertex_id) from vertices;"): + logger.info('{} vertices created'.format(row_d[0])) + counter = counter + 1 + + if facility_type == "processor": + # actual processors - will deal with endcaps in edges section + + # create processor vertices for any commodities that do not inherit max transport distance + proc_data = db_cur2.execute("""select fc.commodity_id, + ifnull(fc.quantity, 0), + fc.units, + ifnull(c.supertype, c.commodity_name), + fc.io, + mc.commodity_id, + c.commodity_name, + ifnull(s.source_facility_id, 0) + from facility_commodities fc, commodities c, commodities mc + left outer join source_commodity_ref s + on (fc.commodity_id = s.commodity_id and s.max_transport_distance_flag = 'Y') + where fc.facility_id = {} + and fc.commodity_id = c.commodity_id + and mc.commodity_name = '{}';""".format(facility_id, multi_commodity_name)) + + proc_data = proc_data.fetchall() + # entry for each incoming commodity and its potential sources + # each outgoing commodity with this processor as their source IF there is a max commod distance + for row_b in proc_data: + + commodity_id = row_b[0] + quantity = facility_cap_noEarthquake[float(facility_id)-1][t+1][i]*row_b[1] + io = row_b[4] + id_for_mult_commodities = row_b[5] + commodity_name = row_b[6] + source_facility_id = row_b[7] + new_source_facility_id = facility_id + + # vertices for generic demand type, or any subtype specified by the destination + for day_before, availability in enumerate(schedule_dict[schedule_id]): + if io == 'i': + main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, + facility_type_id, facility_name, schedule_day, commodity_id, activity_level, + storage_vertex, source_facility_id, iob) values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, + '{}' );""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + id_for_mult_commodities, availability, primary, + new_source_facility_id, 'b')) + + main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, + facility_type_id, facility_name, schedule_day, commodity_id, activity_level, + storage_vertex, demand, source_facility_id, iob) values ({}, {}, {}, '{}', {}, {}, {}, + {}, {}, {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, + facility_name, + day_before+1, commodity_id, storage_availability, storage, quantity, + source_facility_id, io)) + + else: + if commodity_name != 'total_fuel': + main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, + facility_type_id, facility_name, schedule_day, commodity_id, activity_level, + storage_vertex, supply, source_facility_id, iob) values ({}, {}, {}, '{}', {}, {}, + {}, {}, {}, {}, '{}');""".format( + facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, storage_availability, storage, quantity, source_facility_id, io)) + + + elif facility_type == "raw_material_producer": + rmp_data = db_cur.execute("""select fc.commodity_id, fc.quantity, fc.units, + ifnull(s.source_facility_id, 0), io + from facility_commodities fc + left outer join source_commodity_ref s + on (fc.commodity_id = s.commodity_id + and s.max_transport_distance_flag = 'Y' + and s.source_facility_id = {}) + where fc.facility_id = {};""".format(facility_id, facility_id)) + + rmp_data = rmp_data.fetchall() + + for row_b in rmp_data: + commodity_id = row_b[0] + quantity = facility_cap_noEarthquake[float(facility_id)-1][t+1][i]*row_b[1] + # units = row_b[2] + source_facility_id = row_b[3] + iob = row_b[4] + + if commodity_id in total_potential_production: + total_potential_production[commodity_id] = total_potential_production[commodity_id] + quantity + else: + total_potential_production[commodity_id] = quantity + + for day_before, availability in enumerate(schedule_dict[schedule_id]): + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + activity_level, storage_vertex, supply, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, availability, primary, quantity, + source_facility_id, iob)) + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + activity_level, storage_vertex, supply, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, storage_availability, storage, quantity, + source_facility_id, iob)) + + elif facility_type == "storage": # storage facility + storage_fac_data = db_cur.execute("""select + fc.commodity_id, + fc.quantity, + fc.units, + ifnull(s.source_facility_id, 0), + io + from facility_commodities fc + left outer join source_commodity_ref s + on (fc.commodity_id = s.commodity_id and s.max_transport_distance_flag = 'Y') + where fc.facility_id = {} ;""".format(facility_id)) + + storage_fac_data = storage_fac_data.fetchall() + + for row_b in storage_fac_data: + commodity_id = row_b[0] + source_facility_id = row_b[3] # 0 if not source-tracked + iob = row_b[4] + + for day_before in range(schedule_length): + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + storage_vertex, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, storage, + source_facility_id, iob)) + + elif facility_type == "ultimate_destination": + + dest_data = db_cur2.execute("""select + fc.commodity_id, + ifnull(fc.quantity, 0), + fc.units, + fc.commodity_id, + ifnull(c.supertype, c.commodity_name), + ifnull(s.source_facility_id, 0), + io + from facility_commodities fc, commodities c + left outer join source_commodity_ref s + on (fc.commodity_id = s.commodity_id and s.max_transport_distance_flag = 'Y') + where fc.facility_id = {} + and fc.commodity_id = c.commodity_id;""".format(facility_id)) + + dest_data = dest_data.fetchall() + + for row_b in dest_data: + commodity_id = row_b[0] + quantity = row_b[1] + commodity_supertype = row_b[4] + source_facility_id = row_b[5] + iob = row_b[6] + zero_source_facility_id = 0 # material merges at primary vertex + + # vertices for generic demand type, or any subtype specified by the destination + for day_before, availability in enumerate(schedule_dict[schedule_id]): + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + activity_level, storage_vertex, demand, udp, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, + {}, {}, {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, availability, primary, quantity, + the_scenario.unMetDemandPenalty, + zero_source_facility_id, iob)) + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + activity_level, storage_vertex, demand, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, day_before+1, + commodity_id, storage_availability, storage, quantity, + source_facility_id, iob)) + # vertices for other fuel subtypes that match the destination's supertype + # if the subtype is in the commodity table, it is produced by some facility in the scenario + db_cur3 = main_db_con.cursor() + for row_c in db_cur3.execute("""select commodity_id, units from commodities + where supertype = '{}';""".format(commodity_supertype)): + new_commodity_id = row_c[0] + # new_units = row_c[1] + for day_before, availability in schedule_dict[schedule_id]: + main_db_con.execute("""insert or ignore into vertices ( location_id, facility_id, + facility_type_id, facility_name, schedule_day, commodity_id, activity_level, + storage_vertex, demand, udp, source_facility_id, iob) values ({}, {}, {}, '{}', {}, {}, + {}, {}, {}, {}, {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, + facility_name, + day_before+1, new_commodity_id, availability, primary, + quantity, + the_scenario.unMetDemandPenalty, + zero_source_facility_id, iob)) + main_db_con.execute("""insert or ignore into vertices ( + location_id, facility_id, facility_type_id, facility_name, schedule_day, commodity_id, + activity_level, storage_vertex, demand, + source_facility_id, iob) + values ({}, {}, {}, '{}', {}, {}, {}, {}, {}, + {}, '{}');""".format(facility_location_id, facility_id, facility_type_id, facility_name, + day_before+1, new_commodity_id, storage_availability, storage, quantity, + source_facility_id, iob)) + + else: + logger.warning( + "error, unexpected facility_type: {}, facility_type_id: {}".format(facility_type, facility_type_id)) + + for row_d in db_cur4.execute("select count(distinct vertex_id) from vertices;"): + logger.info('{} vertices created'.format(row_d[0])) + + logger.debug("total possible production in scenario: {}".format(total_potential_production)) + + +# =============================================================================== + + +def add_storage_routes(the_scenario, logger): + logger.info("start: add_storage_routes") + # these are loops to and from the same facility; when multiplied to edges, + # they will connect primary to storage vertices, and storage vertices day to day + # will always create edge for this route from storage to storage vertex + # IF a primary vertex exists, will also create an edge connecting the storage vertex to the primary + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + logger.debug("create the storage_routes table") + + main_db_con.execute("drop table if exists storage_routes;") + main_db_con.execute("""create table if not exists storage_routes as + select facility_name || '_storage' as route_name, + location_id, + facility_id, + facility_name as o_name, + facility_name as d_name, + {} as cost_1, + {} as cost_2, + 1 as travel_time, + {} as storage_max, + 0 as storage_min + from facilities + where ignore_facility = 'false' + ;""".format(storage_cost_1, storage_cost_2, facility_onsite_storage_max)) + main_db_con.execute("""create table if not exists route_reference( + route_id INTEGER PRIMARY KEY, route_type text, route_name text, scenario_rt_id integer, + CONSTRAINT unique_routes UNIQUE(route_type, route_name, scenario_rt_id));""") + main_db_con.execute( + "insert or ignore into route_reference select null,'storage', route_name, 0 from storage_routes;") + + return + + +# =============================================================================== + + +def generate_connector_and_storage_edges(the_scenario, logger): + logger.info("START: generate_connector_and_storage_edges") + + multi_commodity_name = "multicommodity" + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + db_cur = main_db_con.cursor() + if ('pipeline_crude_trf_rts' in the_scenario.permittedModes) or ( + 'pipeline_prod_trf_rts' in the_scenario.permittedModes): + logger.info("create indices for the capacity_nodes and pipeline_mapping tables") + main_db_con.executescript( + """ + CREATE INDEX IF NOT EXISTS pm_index + ON pipeline_mapping (id, id_field_name, mapping_id_field_name, mapping_id); + CREATE INDEX IF NOT EXISTS cn_index ON capacity_nodes (source, id_field_name, source_OID); + """) + + for row in db_cur.execute( + "select commodity_id from commodities where commodity_name = '{}';""".format(multi_commodity_name)): + id_for_mult_commodities = row[0] + + # create storage & connector edges + main_db_con.execute("drop table if exists edges;") + main_db_con.executescript(""" + create table edges (edge_id INTEGER PRIMARY KEY, + route_id integer, + from_node_id integer, + to_node_id integer, + start_day integer, + end_day integer, + commodity_id integer, + o_vertex_id integer, + d_vertex_id integer, + max_edge_capacity numeric, + volume numeric, + capac_minus_volume_zero_floor numeric, + min_edge_capacity numeric, + capacity_units text, + units_conversion_multiplier numeric, + edge_flow_cost numeric, + edge_flow_cost2 numeric, + edge_type text, + nx_edge_id integer, + mode text, + mode_oid integer, + miles numeric, + simple_mode text, + tariff_id numeric, + phase_of_matter text, + source_facility_id integer, + miles_travelled numeric, + children_created text, + edge_count_from_source integer, + total_route_cost numeric, + CONSTRAINT unique_nx_subc_day UNIQUE(nx_edge_id, commodity_id, source_facility_id, start_day)) + ; + + insert or ignore into edges (route_id, + start_day, end_day, + commodity_id, o_vertex_id, d_vertex_id, + max_edge_capacity, min_edge_capacity, + edge_flow_cost, edge_flow_cost2,edge_type, + source_facility_id) + select o.route_id, o.schedule_day, d.schedule_day, + o.commodity_id, o.vertex_id, d.vertex_id, + o.storage_max, o.storage_min, + o.cost_1, o.cost_2, 'storage', + o.source_facility_id + from vertices d, + (select v.vertex_id, v.schedule_day, + v.commodity_id, v.storage_vertex, v.source_facility_id, + t.* + from vertices v, + (select sr.route_name, o_name, d_name, cost_1,cost_2, travel_time, + storage_max, storage_min, rr.route_id, location_id, facility_id + from storage_routes sr, route_reference rr where sr.route_name = rr.route_name) t + where v.facility_id = t.facility_id + and v.storage_vertex = 1) o + where d.facility_id = o.facility_id + and d.schedule_day = o.schedule_day+o.travel_time + and d.commodity_id = o.commodity_id + and o.vertex_id != d.vertex_id + and d.storage_vertex = 1 + and d.source_facility_id = o.source_facility_id + ; + + insert or ignore into edges (route_id, start_day, end_day, + commodity_id, o_vertex_id, d_vertex_id, + edge_flow_cost, edge_type, + source_facility_id) + select s.route_id, s.schedule_day, p.schedule_day, + (case when s.commodity_id = {} then p.commodity_id else s.commodity_id end) commodity_id, + --inbound commodies start at storage and go into primary + --outbound starts at primary and goes into storage + --anything else is an error for a connector edge + (case when fc.io = 'i' then s.vertex_id + when fc.io = 'o' then p.vertex_id + else 0 end) as o_vertex, + (case when fc.io = 'i' then p.vertex_id + when fc.io = 'o' then s.vertex_id + else 0 end) as d_vertex, + 0, 'connector', + s.source_facility_id + from vertices p, facility_commodities fc, + --s for storage vertex info, p for primary vertex info + (select v.vertex_id, v.schedule_day, + v.commodity_id, v.storage_vertex, v.source_facility_id, + t.* + from vertices v, + (select sr.route_name, o_name, d_name, cost_1,cost_2, travel_time, + storage_max, storage_min, rr.route_id, location_id, facility_id + from storage_routes sr, route_reference rr where sr.route_name = rr.route_name) t --t is route data + where v.facility_id = t.facility_id + and v.storage_vertex = 1) s + --from storage into primary, same day = inbound connectors + where p.facility_id = s.facility_id + and p.schedule_day = s.schedule_day + and (p.commodity_id = s.commodity_id or p.commodity_id = {} ) + and p.facility_id = fc.facility_id + and fc.commodity_id = s.commodity_id + and p.storage_vertex = 0 + --either edge is inbound and aggregating, or kept separate by source, or primary vertex is not source tracked + and + (p.source_facility_id = 0 or p.source_facility_id = p.facility_id or p.source_facility_id = s.source_facility_id) + ;""".format(id_for_mult_commodities, id_for_mult_commodities)) + + for row_d in db_cur.execute("select count(distinct edge_id) from edges where edge_type = 'connector';"): + logger.info('{} connector edges created'.format(row_d[0])) + # clear any transport edges from table + db_cur.execute("delete from edges where edge_type = 'transport';") + + return + + +# =============================================================================== + + +def generate_first_edges_from_source_facilities(the_scenario, schedule_length, logger): + + logger.info("START: generate_first_edges_from_source_facilities") + # create edges table + # plan to generate start and end days based on nx edge time to traverse and schedule + # can still have route_id, but only for storage routes now; nullable + + # multi_commodity_name = "multicommodity" + transport_edges_created = 0 + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + db_cur = main_db_con.cursor() + edges_requiring_children = 0 + + counter = 0 + + # create transport edges, only between storage vertices and nodes, based on networkx graph + + commodity_mode_data = main_db_con.execute("select * from commodity_mode;") + commodity_mode_data = commodity_mode_data.fetchall() + commodity_mode_dict = {} + for row in commodity_mode_data: + mode = row[0] + commodity_id = int(row[1]) + commodity_phase = row[2] + allowed_yn = row[3] + commodity_mode_dict[mode, commodity_id] = allowed_yn + + + source_edge_data = main_db_con.execute("""select + ne.edge_id, + ifnull(CAST(fn.location_id as integer), 'NULL'), + ifnull(CAST(tn.location_id as integer), 'NULL'), + ne.mode_source, + ifnull(nec.phase_of_matter_id, 'NULL'), + nec.route_cost, + ne.from_node_id, + ne.to_node_id, + nec.dollar_cost, + ne.miles, + ne.capacity, + ne.artificial, + ne.mode_source_oid, + v.commodity_id, + v.schedule_day, + v.vertex_id, + v.source_facility_id, + tv.vertex_id, + ifnull(t.miles_travelled, 0), + ifnull(t.edge_count_from_source, 0), + t.mode + from networkx_edges ne, networkx_nodes fn, networkx_nodes tn, networkx_edge_costs nec, vertices v, + facility_type_id ft, facility_commodities fc + left outer join (--get facilities with incoming transport edges with tracked mileage + select vi.facility_id, min(ei.miles_travelled) miles_travelled, ei.source_facility_id, + ei.edge_count_from_source, ei.mode + from edges ei, vertices vi + where vi.vertex_id = ei.d_vertex_id + and edge_type = 'transport' + and ifnull(miles_travelled, 0) > 0 + group by vi.facility_id, ei.source_facility_id, ei.mode) t + on t.facility_id = v.facility_id and t.source_facility_id = v.source_facility_id and ne.mode_source = t.mode + left outer join vertices tv + on (CAST(tn.location_id as integer) = tv.location_id and v.source_facility_id = tv.source_facility_id) + where v.location_id = CAST(fn.location_id as integer) + and fc.facility_id = v.facility_id + and fc.commodity_id = v.commodity_id + and fc.io = 'o' + and ft.facility_type_id = v.facility_type_id + and (ft.facility_type = 'raw_material_producer' or t.facility_id = v.facility_id) + and ne.from_node_id = fn.node_id + and ne.to_node_id = tn.node_id + and ne.edge_id = nec.edge_id + and ifnull(ne.capacity, 1) > 0 + and v.storage_vertex = 1 + and v.source_facility_id != 0 --max commodity distance applies + ;""") + source_edge_data = source_edge_data.fetchall() + for row_a in source_edge_data: + + nx_edge_id = row_a[0] + from_location = row_a[1] + to_location = row_a[2] + mode = row_a[3] + phase_of_matter = row_a[4] + route_cost = row_a[5] + from_node = row_a[6] + to_node = row_a[7] + dollar_cost = row_a[8] + miles = row_a[9] + # max_daily_capacity = row_a[10] + # artificial = row_a[11] + mode_oid = row_a[12] + commodity_id = row_a[13] + origin_day = row_a[14] + vertex_id = row_a[15] + source_facility_id = row_a[16] + to_vertex = row_a[17] + previous_miles_travelled = row_a[18] + previous_edge_count = row_a[19] + previous_mode = row_a[20] + + simple_mode = row_a[3].partition('_')[0] + edge_count_from_source = 1 + previous_edge_count + total_route_cost = route_cost + miles_travelled = previous_miles_travelled + miles + + if counter % 10000 == 0: + for row_d in db_cur.execute("select count(distinct edge_id) from edges;"): + logger.info('{} edges created'.format(row_d[0])) + counter = counter + 1 + + tariff_id = 0 + if simple_mode == 'pipeline': + + # find tariff_ids + + sql = "select mapping_id from pipeline_mapping " \ + "where id = {} and id_field_name = 'source_OID' " \ + "and source = '{}' and mapping_id is not null;".format( + mode_oid, mode) + for tariff_row in db_cur.execute(sql): + tariff_id = tariff_row[0] + + + if mode in the_scenario.permittedModes and (mode, commodity_id) in commodity_mode_dict.keys() \ + and commodity_mode_dict[mode, commodity_id] == 'Y': + + # Edges are placeholders for flow variables + # 4-17: if both ends have no location, iterate through viable commodities and days, create edge + # for all days (restrict by link schedule if called for) + # for all allowed commodities, as currently defined by link phase of matter + + # days range from 1 to schedule_length + if origin_day in range(1, schedule_length+1): + if origin_day + fixed_route_duration <= schedule_length: + # if link is traversable in the timeframe + if simple_mode != 'pipeline' or tariff_id >= 0: + # for allowed commodities + # step 1 from source is from non-Null location to (probably) null location + + if from_location != 'NULL' and to_location == 'NULL': + # for each day and commodity, + # get the corresponding origin vertex id to include with the edge info + # origin vertex must not be "ultimate_destination + # transport link outgoing from facility - checking fc.io is more thorough + # than checking if facility type is 'ultimate destination' + # only connect to vertices with matching source_facility_id + # source_facility_id is zero for commodities without source tracking + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + o_vertex_id, + min_edge_capacity,edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, + tariff_id,phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, + {}, {}, {}, + '{}',{},'{}',{}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + vertex_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + miles_travelled, 'N', edge_count_from_source, total_route_cost)) + + + elif from_location != 'NULL' and to_location != 'NULL': + # for each day and commodity, get the corresponding origin and destination vertex ids + # to include with the edge info + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + o_vertex_id, d_vertex_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, {}, + {}, {}, {}, + '{}',{},'{}', {}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + vertex_id, to_vertex, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + miles_travelled, 'N', edge_count_from_source, total_route_cost)) + + for row in db_cur.execute("select count(distinct edge_id) from edges where edge_type = 'transport';"): + transport_edges_created = row[0] + logger.info('{} transport edges created'.format(transport_edges_created)) + for row in db_cur.execute("""select count(distinct edge_id), children_created from edges + where edge_type = 'transport' + group by children_created;"""): + if row[1] == 'N': + edges_requiring_children = row[0] + + elif row[1] == 'Y': + logger.info('{} transport edges that have already been checked for children'.format(row[0])) + edges_requiring_children = transport_edges_created - row[0] + # edges_requiring_children is updated under either condition here, since one may not be triggered + logger.info('{} transport edges that need children'.format(edges_requiring_children)) + + return + + +# =============================================================================== + + +def generate_all_edges_from_source_facilities(the_scenario, schedule_length, logger): + # method only runs for commodities with a max commodity constraint + + logger.info("START: generate_all_edges_from_source_facilities") + + multi_commodity_name = "multicommodity" + # initializations - all of these get updated if >0 edges exist + edges_requiring_children = 0 + endcap_edges = 0 + edges_resolved = 0 + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + db_cur = main_db_con.cursor() + transport_edges_created = 0 + nx_edge_count = 0 + source_based_edges_created = 0 + + for row_d in db_cur.execute("""select count(distinct e.edge_id), count(distinct e.nx_edge_id) + from edges e where e.edge_type = 'transport' + and children_created in ('N', 'Y', 'E');"""): + source_based_edges_created = row_d[0] + + for row_d in db_cur.execute("""select count(distinct e.edge_id), count(distinct e.nx_edge_id) + from edges e where e.edge_type = 'transport';"""): + transport_edges_created = row_d[0] + nx_edge_count = row_d[1] + + + commodity_mode_data = main_db_con.execute("select * from commodity_mode;") + commodity_mode_data = commodity_mode_data.fetchall() + commodity_mode_dict = {} + for row in commodity_mode_data: + mode = row[0] + commodity_id = int(row[1]) + commodity_phase = row[2] + allowed_yn = row[3] + commodity_mode_dict[mode, commodity_id] = allowed_yn + + current_edge_data = db_cur.execute("""select count(distinct edge_id), children_created + from edges + where edge_type = 'transport' + group by children_created + order by children_created asc;""") + current_edge_data = current_edge_data.fetchall() + for row in current_edge_data: + if row[1] == 'N': + edges_requiring_children = row[0] + elif row[1] == 'Y': + edges_resolved = row[0] + elif row[1] == 'E': + endcap_edges = row[0] + if source_based_edges_created == edges_resolved + endcap_edges: + edges_requiring_children = 0 + if source_based_edges_created == edges_requiring_children + endcap_edges: + edges_resolved = 0 + if source_based_edges_created == edges_requiring_children + edges_resolved: + endcap_edges = 0 + + logger.info( + '{} transport edges created; {} require children'.format(transport_edges_created, edges_requiring_children)) + + # set up a table to keep track of endcap nodes + sql = """ + drop table if exists endcap_nodes; + create table if not exists endcap_nodes( + node_id integer NOT NULL, + + location_id integer, + + --mode of edges that it's an endcap for + mode_source text NOT NULL, + + --facility it's an endcap for + source_facility_id integer NOT NULL, + + --commodities it's an endcap for + commodity_id integer NOT NULL, + + CONSTRAINT endcap_key PRIMARY KEY (node_id, mode_source, source_facility_id, commodity_id)) + --the combination of these four (excluding location_id) should be unique, + --and all fields except location_id should be filled + ;""" + db_cur.executescript(sql) + + # create transport edges, only between storage vertices and nodes, based on networkx graph + + edge_into_facility_counter = 0 + while_count = 0 + + while edges_requiring_children > 0: + while_count = while_count+1 + + # --get nx edges that align with the existing "in edges" - data from nx to create new edges --for each of + # those nx_edges, if they connect to more than one "in edge" in this batch, only consider connecting to + # the shortest -- if there is a valid nx_edge to build, the crossing node is not an endcap if total miles + # of the new route is over max transport distance, then endcap what if one child goes over max transport + # and another doesn't then the node will get flagged as an endcap, and another path may continue off it, + # allow both for now --check that day and commodity are permitted by nx + + potential_edge_data = main_db_con.execute(""" + select + ch.edge_id as ch_nx_edge_id, + ifnull(CAST(chfn.location_id as integer), 'NULL') fn_location_id, + ifnull(CAST(chtn.location_id as integer), 'NULL') tn_location_id, + ch.mode_source, + p.phase_of_matter, + nec.route_cost, + ch.from_node_id, + ch.to_node_id, + nec.dollar_cost, + ch.miles, + ch.capacity, + ch.artificial, + ch.mode_source_oid, + --parent edge into + p.commodity_id, + p.end_day, + --parent's dest. vertex if exists + ifnull(p.d_vertex_id,0) o_vertex, + p.source_facility_id, + p.leadin_miles_travelled, + + (p.edge_count_from_source +1) as new_edge_count, + (p.total_route_cost + nec.route_cost) new_total_route_cost, + p.edge_id leadin_edge, + p.nx_edge_id leadin_nx_edge, + + --new destination vertex if exists + ifnull(chtv.vertex_id,0) d_vertex, + + sc.max_transport_distance + + from + (select count(edge_id) parents, + min(miles_travelled) leadin_miles_travelled, + * + from edges + where children_created = 'N' + -----------------do not mess with this "group by" + group by to_node_id, source_facility_id, commodity_id, end_day + ------------------it affects which columns we're checking over for min miles travelled + --------------so that we only get the parent edges we want + order by parents desc + ) p, --parent edges to use in this batch + networkx_edges ch, + networkx_edge_costs nec, + source_commodity_ref sc, + networkx_nodes chfn, + networkx_nodes chtn + left outer join vertices chtv + on (CAST(chtn.location_id as integer) = chtv.location_id + and p.source_facility_id = chtv.source_facility_id + and chtv.commodity_id = p.commodity_id + and p.end_day = chtv.schedule_day + and chtv.iob = 'i') + + where p.to_node_id = ch.from_node_id + --and p.mode = ch.mode_source --build across modes, control at conservation of flow + and ch.to_node_id = chtn.node_id + and ch.from_node_id = chfn.node_id + and p.phase_of_matter = nec.phase_of_matter_id + and ch.edge_id = nec.edge_id + and ifnull(ch.capacity, 1) > 0 + and p.commodity_id = sc.commodity_id + ;""") + + # --should only get a single leadin edge per networkx/source/commodity/day combination + # leadin edge should be in route_data, current set of min. identifiers + # if we're trying to add an edge that has an entry in route_data, new miles travelled must be less + + potential_edge_data = potential_edge_data.fetchall() + + main_db_con.execute("update edges set children_created = 'Y' where children_created = 'N';") + + for row_a in potential_edge_data: + nx_edge_id = row_a[0] + from_location = row_a[1] + to_location = row_a[2] + mode = row_a[3] + phase_of_matter = row_a[4] + route_cost = row_a[5] + from_node = row_a[6] + to_node = row_a[7] + dollar_cost = row_a[8] + miles = row_a[9] + mode_oid = row_a[12] + commodity_id = row_a[13] + origin_day = row_a[14] + vertex_id = row_a[15] + source_facility_id = row_a[16] + leadin_edge_miles_travelled = row_a[17] + new_edge_count = row_a[18] + total_route_cost = row_a[19] + leadin_edge_id = row_a[20] + # leadin_nx_edge_id = row_a[21] + to_vertex = row_a[22] + max_commodity_travel_distance = row_a[23] + + # end_day = origin_day + fixed_route_duration + new_miles_travelled = miles + leadin_edge_miles_travelled + + if mode in the_scenario.permittedModes and (mode, commodity_id) in commodity_mode_dict.keys() \ + and commodity_mode_dict[mode, commodity_id] == 'Y': + + if new_miles_travelled > max_commodity_travel_distance: + # designate leadin edge as endcap + children_created = 'E' + # update the incoming edge to indicate it's an endcap + db_cur.execute( + "update edges set children_created = '{}' where edge_id = {}".format(children_created, + leadin_edge_id)) + if from_location != 'NULL': + db_cur.execute("""insert or ignore into endcap_nodes( + node_id, location_id, mode_source, source_facility_id, commodity_id) + VALUES ({}, {}, '{}', {}, {}); + """.format(from_node, from_location, mode, source_facility_id, commodity_id)) + else: + db_cur.execute("""insert or ignore into endcap_nodes( + node_id, mode_source, source_facility_id, commodity_id) + VALUES ({}, '{}', {}, {}); + """.format(from_node, mode, source_facility_id, commodity_id)) + + # create new edge + elif new_miles_travelled <= max_commodity_travel_distance: + + simple_mode = row_a[3].partition('_')[0] + tariff_id = 0 + if simple_mode == 'pipeline': + + # find tariff_ids + + sql = """select mapping_id + from pipeline_mapping + where id = {} + and id_field_name = 'source_OID' + and source = '{}' + and mapping_id is not null;""".format(mode_oid, mode) + for tariff_row in db_cur.execute(sql): + tariff_id = tariff_row[0] + + # if there are no edges yet for this day, nx, subc combination, + # AND this is the shortest existing leadin option for this day, nx, subc combination + # we'd be creating an edge for (otherwise wait for the shortest option) + # at this step, some leadin edge should always exist + + if origin_day in range(1, schedule_length+1): + if origin_day + fixed_route_duration <= schedule_length: + # if link is traversable in the timeframe + if simple_mode != 'pipeline' or tariff_id >= 0: + # for allowed commodities + + if from_location == 'NULL' and to_location == 'NULL': + # for each day and commodity, + # get the corresponding origin vertex id to include with the edge info + # origin vertex must not be "ultimate_destination + # transport link outgoing from facility - checking fc.io is more thorough than + # checking if facility type is 'ultimate destination' + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + min_edge_capacity,edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, + tariff_id,phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, {}, {}, + '{}',{},'{}',{}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + new_miles_travelled, 'N', new_edge_count, total_route_cost)) + # only create edge going into a location if an appropriate vertex exists + elif from_location == 'NULL' and to_location != 'NULL' and to_vertex > 0: + edge_into_facility_counter = edge_into_facility_counter + 1 + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + d_vertex_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, + {}, {}, {}, + '{}',{},'{}', {}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + to_vertex, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + new_miles_travelled, 'N', new_edge_count, total_route_cost)) + + # designate leadin edge as endcap + # this does, deliberately, allow endcap status to be + # overwritten if we've found a shorter path to a previous endcap + elif from_location != 'NULL' and to_location == 'NULL': + # for each day and commodity, get the corresponding origin vertex id + # to include with the edge info + # origin vertex must not be "ultimate_destination + # transport link outgoing from facility - checking fc.io is more thorough than + # checking if facility type is 'ultimate destination' + # new for bsc, only connect to vertices with matching source facility id + # (only limited for RMP vertices) + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + o_vertex_id, + min_edge_capacity,edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, + {}, {}, {}, + '{}',{},'{}',{}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + vertex_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + new_miles_travelled, 'N', new_edge_count, total_route_cost)) + + + elif from_location != 'NULL' and to_location != 'NULL': + # for each day and commodity, get the corresponding origin and + # destination vertex ids to include with the edge info + + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + o_vertex_id, d_vertex_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, + source_facility_id, + miles_travelled, children_created, edge_count_from_source, + total_route_cost) + VALUES ({}, {}, + {}, {}, {}, + {}, {}, + {}, {}, {}, + '{}',{},'{}', {}, + {},'{}',{},'{}', + {}, + {},'{}',{},{}); + """.format(from_node, to_node, + origin_day, origin_day + fixed_route_duration, commodity_id, + vertex_id, to_vertex, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, + miles, simple_mode, tariff_id, phase_of_matter, + source_facility_id, + new_miles_travelled, 'N', new_edge_count, total_route_cost)) + + for row_d in db_cur.execute("""select count(distinct e.edge_id), count(distinct e.nx_edge_id) + from edges e where e.edge_type = 'transport' + and children_created in ('N', 'Y', 'E');"""): + source_based_edges_created = row_d[0] + + for row_d in db_cur.execute("""select count(distinct e.edge_id), count(distinct e.nx_edge_id) + from edges e where e.edge_type = 'transport';"""): + transport_edges_created = row_d[0] + nx_edge_count = row_d[1] + + current_edge_data = db_cur.execute("""select count(distinct edge_id), children_created + from edges + where edge_type = 'transport' + group by children_created + order by children_created asc;""") + current_edge_data = current_edge_data.fetchall() + for row in current_edge_data: + if row[1] == 'N': + edges_requiring_children = row[0] + elif row[1] == 'Y': + edges_resolved = row[0] + elif row[1] == 'E': + endcap_edges = row[0] + logger.debug('{} endcap edges designated for candidate generation step'.format(endcap_edges)) + if source_based_edges_created == edges_resolved + endcap_edges: + edges_requiring_children = 0 + if source_based_edges_created == edges_requiring_children + endcap_edges: + edges_resolved = 0 + if source_based_edges_created == edges_requiring_children + edges_resolved: + endcap_edges = 0 + + if while_count % 1000 == 0 or edges_requiring_children == 0: + logger.info( + '{} transport edges on {} nx edges, created in {} loops, {} edges_requiring_children'.format( + transport_edges_created, nx_edge_count, while_count, edges_requiring_children)) + + # edges going in to the facility by re-running "generate first edges + # then re-run this method + + logger.info('{} transport edges on {} nx edges, created in {} loops, {} edges_requiring_children'.format( + transport_edges_created, nx_edge_count, while_count, edges_requiring_children)) + logger.info("all source-based transport edges created") + + logger.info("create an index for the edges table by nodes") + + sql = ("""CREATE INDEX IF NOT EXISTS edge_index ON edges ( + edge_id, route_id, from_node_id, to_node_id, commodity_id, + start_day, end_day, commodity_id, o_vertex_id, d_vertex_id, + max_edge_capacity, min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, source_facility_id);""") + db_cur.execute(sql) + + return + + +# =============================================================================== + + +def generate_all_edges_without_max_commodity_constraint(the_scenario, schedule_length, logger): + + global total_transport_routes + logger.info("START: generate_all_edges_without_max_commodity_constraint") + + multi_commodity_name = "multicommodity" + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + + db_cur = main_db_con.cursor() + + commodity_mode_data = main_db_con.execute("select * from commodity_mode;") + commodity_mode_data = commodity_mode_data.fetchall() + commodity_mode_dict = {} + for row in commodity_mode_data: + mode = row[0] + commodity_id = int(row[1]) + commodity_phase = row[2] + allowed_yn = row[3] + commodity_mode_dict[mode, commodity_id] = allowed_yn + + counter = 0 + # for row in db_cur.execute( + # "select commodity_id from commodities where commodity_name = '{}';""".format(multi_commodity_name)): + # id_for_mult_commodities = row[0] + logger.info("COUNTING TOTAL TRANSPORT ROUTES") + for row in db_cur.execute(""" + select count(*) from networkx_edges, shortest_edges + WHERE networkx_edges.from_node_id = shortest_edges.from_node_id + AND networkx_edges.to_node_id = shortest_edges.to_node_id + AND networkx_edges.edge_id = shortest_edges.edge_id + ; + """): + total_transport_routes = row[0] + + # for all commodities with no max transport distance + source_facility_id = 0 + + # create transport edges, only between storage vertices and nodes, based on networkx graph + # never touch primary vertices; either or both vertices can be null (or node-type) if it's a mid-route link + # iterate through nx edges: if neither node has a location, create 1 edge per viable commodity + # should also be per day, subject to nx edge schedule + # before creating an edge, check: commodity allowed by nx and max transport distance if not null + # will need nodes per day and commodity? or can I just check that with constraints? + # select data for transport edges + + sql = """select + ne.edge_id, + ifnull(fn.location_id, 'NULL'), + ifnull(tn.location_id, 'NULL'), + ne.mode_source, + ifnull(nec.phase_of_matter_id, 'NULL'), + nec.route_cost, + ne.from_node_id, + ne.to_node_id, + nec.dollar_cost, + ne.miles, + ne.capacity, + ne.artificial, + ne.mode_source_oid + from networkx_edges ne, networkx_nodes fn, networkx_nodes tn, networkx_edge_costs nec, shortest_edges se + + where ne.from_node_id = fn.node_id + and ne.to_node_id = tn.node_id + and ne.edge_id = nec.edge_id + and ne.from_node_id = se.from_node_id -- match to the shortest_edges table + and ne.to_node_id = se.to_node_id + and ifnull(ne.capacity, 1) > 0 + ;""" + nx_edge_data = main_db_con.execute(sql) + nx_edge_data = nx_edge_data.fetchall() + for row_a in nx_edge_data: + + nx_edge_id = row_a[0] + from_location = row_a[1] + to_location = row_a[2] + mode = row_a[3] + phase_of_matter = row_a[4] + route_cost = row_a[5] + from_node = row_a[6] + to_node = row_a[7] + dollar_cost = row_a[8] + miles = row_a[9] + # max_daily_capacity = row_a[10] + mode_oid = row_a[12] + simple_mode = row_a[3].partition('_')[0] + + counter = counter + 1 + + tariff_id = 0 + if simple_mode == 'pipeline': + + # find tariff_ids + + sql = "select mapping_id from pipeline_mapping " \ + "where id = {} and id_field_name = 'source_OID' and source = '{}' " \ + "and mapping_id is not null;".format( + mode_oid, mode) + for tariff_row in db_cur.execute(sql): + tariff_id = tariff_row[0] + + + if mode in the_scenario.permittedModes: + + # Edges are placeholders for flow variables + # for all days (restrict by link schedule if called for) + # for all allowed commodities, as currently defined by link phase of matter + + for day in range(1, schedule_length+1): + if day + fixed_route_duration <= schedule_length: + # if link is traversable in the timeframe + if simple_mode != 'pipeline' or tariff_id >= 0: + # for allowed commodities that can be output by some facility in the scenario + for row_c in db_cur.execute("""select commodity_id + from source_commodity_ref s + where phase_of_matter = '{}' + and max_transport_distance_flag = 'N' + and share_max_transport_distance = 'N' + group by commodity_id, source_facility_id""".format(phase_of_matter)): + db_cur4 = main_db_con.cursor() + commodity_id = row_c[0] + # source_facility_id = row_c[1] # fixed to 0 for all edges created by this method + if (mode, commodity_id) in commodity_mode_dict.keys() \ + and commodity_mode_dict[mode, commodity_id] == 'Y': + if from_location == 'NULL' and to_location == 'NULL': + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, source_facility_id) VALUES ({}, {}, + {}, {}, {}, + {}, {}, {}, + '{}',{},'{}',{},{},'{}',{},'{}',{}); + """.format(from_node, to_node, + day, day + fixed_route_duration, commodity_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, miles, simple_mode, + tariff_id, phase_of_matter, source_facility_id)) + + elif from_location != 'NULL' and to_location == 'NULL': + # for each day and commodity, get the corresponding origin vertex id + # to include with the edge info + # origin vertex must not be "ultimate_destination + # transport link outgoing from facility - checking fc.io is more thorough + # than checking if facility type is 'ultimate destination' + # new for bsc, only connect to vertices with matching source_facility_id + for row_d in db_cur4.execute("""select vertex_id + from vertices v, facility_commodities fc + where v.location_id = {} and v.schedule_day = {} + and v.commodity_id = {} and v.source_facility_id = {} + and v.storage_vertex = 1 + and v.facility_id = fc.facility_id + and v.commodity_id = fc.commodity_id + and fc.io = 'o'""".format(from_location, day, commodity_id, source_facility_id)): + from_vertex_id = row_d[0] + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + o_vertex_id, + min_edge_capacity,edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, source_facility_id) VALUES ({}, {}, + {}, {}, {}, + {}, + {}, {}, {}, + '{}',{},'{}',{},{},'{}',{},'{}',{}); + """.format(from_node, to_node, + day, day + fixed_route_duration, commodity_id, + from_vertex_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, miles, simple_mode, + tariff_id, phase_of_matter, source_facility_id)) + elif from_location == 'NULL' and to_location != 'NULL': + # for each day and commodity, get the corresponding destination vertex id + # to include with the edge info + for row_d in db_cur4.execute("""select vertex_id + from vertices v, facility_commodities fc + where v.location_id = {} and v.schedule_day = {} + and v.commodity_id = {} and v.source_facility_id = {} + and v.storage_vertex = 1 + and v.facility_id = fc.facility_id + and v.commodity_id = fc.commodity_id + and fc.io = 'i'""".format(to_location, day, commodity_id, source_facility_id)): + to_vertex_id = row_d[0] + main_db_con.execute("""insert or ignore into edges (from_node_id, to_node_id, + start_day, end_day, commodity_id, + d_vertex_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, source_facility_id) VALUES ({}, {}, + {}, {}, {}, + {}, + {}, {}, {}, + '{}',{},'{}',{},{},'{}',{},'{}',{}); + """.format(from_node, to_node, + day, day + fixed_route_duration, commodity_id, + to_vertex_id, + default_min_capacity, route_cost, dollar_cost, + 'transport', nx_edge_id, mode, mode_oid, miles, simple_mode, + tariff_id, phase_of_matter, source_facility_id)) + elif from_location != 'NULL' and to_location != 'NULL': + # for each day and commodity, get the corresponding origin and destination vertex + # ids to include with the edge info + for row_d in db_cur4.execute("""select vertex_id + from vertices v, facility_commodities fc + where v.location_id = {} and v.schedule_day = {} + and v.commodity_id = {} and v.source_facility_id = {} + and v.storage_vertex = 1 + and v.facility_id = fc.facility_id + and v.commodity_id = fc.commodity_id + and fc.io = 'o'""".format(from_location, day, commodity_id, source_facility_id)): + from_vertex_id = row_d[0] + db_cur5 = main_db_con.cursor() + for row_e in db_cur5.execute("""select vertex_id + from vertices v, facility_commodities fc + where v.location_id = {} and v.schedule_day = {} + and v.commodity_id = {} and v.source_facility_id = {} + and v.storage_vertex = 1 + and v.facility_id = fc.facility_id + and v.commodity_id = fc.commodity_id + and fc.io = 'i'""".format(to_location, day, commodity_id, source_facility_id)): + to_vertex_id = row_e[0] + main_db_con.execute("""insert or ignore into edges (from_node_id, + to_node_id, start_day, end_day, commodity_id, o_vertex_id, d_vertex_id, + min_edge_capacity, edge_flow_cost, edge_flow_cost2, edge_type, + nx_edge_id, mode, mode_oid, miles, simple_mode, tariff_id, + phase_of_matter, source_facility_id) VALUES ({}, {}, {}, {}, {}, {}, {}, + {}, {}, {}, '{}',{},'{}', {},{},'{}',{},'{}',{} + )""".format(from_node, + to_node, day, + day + fixed_route_duration, + commodity_id, + from_vertex_id, + to_vertex_id, + default_min_capacity, + route_cost, + dollar_cost, + 'transport', + nx_edge_id, mode, + mode_oid, miles, + simple_mode, + tariff_id, + phase_of_matter, + source_facility_id)) + + logger.debug("all transport edges created") + + logger.info("all edges created") + logger.info("create an index for the edges table by nodes") + index_start_time = datetime.datetime.now() + sql = ("""CREATE INDEX IF NOT EXISTS edge_index ON edges ( + edge_id, route_id, from_node_id, to_node_id, commodity_id, + start_day, end_day, commodity_id, o_vertex_id, d_vertex_id, + max_edge_capacity, min_edge_capacity, edge_flow_cost, edge_flow_cost2, + edge_type, nx_edge_id, mode, mode_oid, miles, source_facility_id);""") + db_cur.execute(sql) + logger.info("edge_index Total Runtime (HMS): \t{} \t ".format(get_total_runtime_string(index_start_time))) + return + + +# =============================================================================== + + +def set_edges_volume_capacity(the_scenario, logger): + logger.info("starting set_edges_volume_capacity") + with sqlite3.connect(the_scenario.main_db) as main_db_con: + logger.debug("starting to record volume and capacity for non-pipeline edges") + + main_db_con.execute( + "update edges set volume = (select ifnull(ne.volume,0) from networkx_edges ne " + "where ne.edge_id = edges.nx_edge_id ) where simple_mode in ('rail','road','water');") + main_db_con.execute( + "update edges set max_edge_capacity = (select ne.capacity from networkx_edges ne " + "where ne.edge_id = edges.nx_edge_id) where simple_mode in ('rail','road','water');") + logger.debug("volume and capacity recorded for non-pipeline edges") + + logger.debug("starting to record volume and capacity for pipeline edges") + ## + main_db_con.executescript("""update edges set volume = + (select l.background_flow + from pipeline_mapping pm, + (select id_field_name, cn.source_OID as link_id, min(cn.capacity) capac, + max(cn.volume) background_flow, source + from capacity_nodes cn + where cn.id_field_name = 'MASTER_OID' + and ifnull(cn.capacity,0)>0 + group by link_id) l + + where edges.tariff_id = pm.id + and pm.id_field_name = 'tariff_ID' + and pm.mapping_id_field_name = 'MASTER_OID' + and l.id_field_name = 'MASTER_OID' + and pm.mapping_id = l.link_id + and instr(edges.mode, l.source)>0) + where simple_mode = 'pipeline' + ; + + update edges set max_edge_capacity = + (select l.capac + from pipeline_mapping pm, + (select id_field_name, cn.source_OID as link_id, min(cn.capacity) capac, + max(cn.volume) background_flow, source + from capacity_nodes cn + where cn.id_field_name = 'MASTER_OID' + and ifnull(cn.capacity,0)>0 + group by link_id) l + + where edges.tariff_id = pm.id + and pm.id_field_name = 'tariff_ID' + and pm.mapping_id_field_name = 'MASTER_OID' + and l.id_field_name = 'MASTER_OID' + and pm.mapping_id = l.link_id + and instr(edges.mode, l.source)>0) + where simple_mode = 'pipeline' + ;""") + logger.debug("volume and capacity recorded for pipeline edges") + logger.debug("starting to record units and conversion multiplier") + main_db_con.execute("""update edges + set capacity_units = + (case when simple_mode = 'pipeline' then 'kbarrels' + when simple_mode = 'road' then 'truckload' + when simple_mode = 'rail' then 'railcar' + when simple_mode = 'water' then 'barge' + else 'unexpected mode' end) + ;""") + main_db_con.execute("""update edges + set units_conversion_multiplier = + (case when simple_mode = 'pipeline' and phase_of_matter = 'liquid' then {} + when simple_mode = 'road' and phase_of_matter = 'liquid' then {} + when simple_mode = 'road' and phase_of_matter = 'solid' then {} + when simple_mode = 'rail' and phase_of_matter = 'liquid' then {} + when simple_mode = 'rail' and phase_of_matter = 'solid' then {} + when simple_mode = 'water' and phase_of_matter = 'liquid' then {} + when simple_mode = 'water' and phase_of_matter = 'solid' then {} + else 1 end) + ;""".format(THOUSAND_GALLONS_PER_THOUSAND_BARRELS, + the_scenario.truck_load_liquid.magnitude, + the_scenario.truck_load_solid.magnitude, + the_scenario.railcar_load_liquid.magnitude, + the_scenario.railcar_load_solid.magnitude, + the_scenario.barge_load_liquid.magnitude, + the_scenario.barge_load_solid.magnitude, + )) + logger.debug("units and conversion multiplier recorded for all edges; starting capacity minus volume") + main_db_con.execute("""update edges + set capac_minus_volume_zero_floor = + 365*max((select (max_edge_capacity - ifnull(volume,0)) where max_edge_capacity is not null),0) + where max_edge_capacity is not null + ;""") + logger.debug("capacity minus volume (minimum set to zero) recorded for all edges") + return + + +# =============================================================================== + + +def pre_setup_pulp(logger, the_scenario): + logger.info("START: pre_setup_pulp") + + commodity_mode_setup(the_scenario, logger) + + # create table to track source facility of commodities with a max transport distance set + source_tracking_setup(the_scenario, logger) + + schedule_dict, schedule_length = generate_schedules(the_scenario, logger) + + generate_all_vertices(the_scenario, schedule_dict, schedule_length, logger) + + add_storage_routes(the_scenario, logger) + generate_connector_and_storage_edges(the_scenario, logger) + + # start edges for commodities that inherit max transport distance + generate_first_edges_from_source_facilities(the_scenario, schedule_length, logger) + + # replicate all_routes by commodity and time into all_edges dictionary + generate_all_edges_from_source_facilities(the_scenario, schedule_length, logger) + + # replicate all_routes by commodity and time into all_edges dictionary + generate_all_edges_without_max_commodity_constraint(the_scenario, schedule_length, logger) + logger.info("Edges generated for modes: {}".format(the_scenario.permittedModes)) + + set_edges_volume_capacity(the_scenario, logger) + + return + + +# =============================================================================== + + +def create_flow_vars(the_scenario, logger): + logger.info("START: create_flow_vars") + + # we have a table called edges. + # call helper method to get list of unique IDs from the Edges table. + # use the rowid as a simple unique integer index + edge_list = [] + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + edge_list_cur = db_cur.execute("""select edge_id--, commodity_id, start_day, source_facility_id + from edges;""") + edge_list_data = edge_list_cur.fetchall() + counter = 0 + for row in edge_list_data: + if counter % 500000 == 0: + logger.info( + "processed {:,.0f} records. size of edge_list {:,.0f}".format(counter, sys.getsizeof(edge_list))) + counter += 1 + # create an edge for each commodity allowed on this link - this construction may change + # as specific commodity restrictions are added + # TODO4-18 add days, but have no scheduel for links currently + # running just with nodes for now, will add proper facility info and storage back soon + edge_list.append((row[0])) + + + flow_var = LpVariable.dicts("Edge", edge_list, 0, None) + return flow_var + + +# =============================================================================== + + +def create_unmet_demand_vars(the_scenario, logger): + logger.info("START: create_unmet_demand_vars") + demand_var_list = [] + # may create vertices with zero demand, but only for commodities that the facility has demand for at some point + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + for row in db_cur.execute("""select v.facility_id, v.schedule_day, + ifnull(c.supertype, c.commodity_name) top_level_commodity_name, v.udp + from vertices v, commodities c, facility_type_id ft, facilities f + where v.commodity_id = c.commodity_id + and ft.facility_type = "ultimate_destination" + and v.storage_vertex = 0 + and v.facility_type_id = ft.facility_type_id + and v.facility_id = f.facility_id + and f.ignore_facility = 'false' + group by v.facility_id, v.schedule_day, ifnull(c.supertype, c.commodity_name) + ;""".format('')): + # facility_id, day, and simplified commodity name + demand_var_list.append((row[0], row[1], row[2], row[3])) + + unmet_demand_var = LpVariable.dicts("UnmetDemand", demand_var_list, None, None) + + return unmet_demand_var + + +# =============================================================================== + + +def create_candidate_processor_build_vars(the_scenario, logger): + logger.info("START: create_candidate_processor_build_vars") + processors_build_list = [] + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + for row in db_cur.execute( + """select f.facility_id from facilities f, facility_type_id ft + where f.facility_type_id = ft.facility_type_id and facility_type = 'processor' + and candidate = 1 and ignore_facility = 'false' group by facility_id;"""): + # grab all candidate processor facility IDs + processors_build_list.append(row[0]) + + processor_build_var = LpVariable.dicts("BuildProcessor", processors_build_list, 0, None, 'Binary') + + return processor_build_var + + +# =============================================================================== + + +def create_binary_processor_vertex_flow_vars(the_scenario, logger): + logger.info("START: create_binary_processor_vertex_flow_vars") + processors_flow_var_list = [] + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + for row in db_cur.execute("""select v.facility_id, v.schedule_day + from vertices v, facility_type_id ft + where v.facility_type_id = ft.facility_type_id + and facility_type = 'processor' + and storage_vertex = 0 + group by v.facility_id, v.schedule_day;"""): + # facility_id, day + processors_flow_var_list.append((row[0], row[1])) + + processor_flow_var = LpVariable.dicts("ProcessorDailyFlow", processors_flow_var_list, 0, None, 'Binary') + + return processor_flow_var + + +# =============================================================================== + + +def create_processor_excess_output_vars(the_scenario, logger): + logger.info("START: create_processor_excess_output_vars") + + excess_var_list = [] + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + xs_cur = db_cur.execute(""" + select vertex_id, commodity_id + from vertices v, facility_type_id ft + where v.facility_type_id = ft.facility_type_id + and facility_type = 'processor' + and storage_vertex = 1;""") + # facility_id, day, and simplified commodity name + xs_data = xs_cur.fetchall() + for row in xs_data: + excess_var_list.append(row[0]) + + excess_var = LpVariable.dicts("XS", excess_var_list, 0, None) + + return excess_var + + +# =============================================================================== + + +def create_opt_problem(logger, the_scenario, unmet_demand_vars, flow_vars, processor_build_vars): + + + logger.debug("START: create_opt_problem") + prob = LpProblem("Flow assignment", LpMinimize) + + unmet_demand_costs = [] + operation_costs = [] + flow_costs = {} + processor_build_costs = [] + for u in unmet_demand_vars: + # facility_id = u[0] + # schedule_day = u[1] + # demand_commodity_name = u[2] + udp = u[3] + unmet_demand_costs.append(udp * unmet_demand_vars[u]) + operation_costs.append(Operation_unitcost * (total_final_demand - unmet_demand_vars[u])) + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + # Flow cost memory improvements: only get needed data; dict instead of list; narrow in lpsum + flow_cost_var = db_cur.execute("select edge_id, edge_flow_cost from edges e group by edge_id;") + flow_cost_data = flow_cost_var.fetchall() + counter = 0 + for row in flow_cost_data: + edge_id = row[0] + edge_flow_cost = row[1] + counter += 1 + + # flow costs cover transportation and storage + flow_costs[edge_id] = edge_flow_cost + # flow_costs.append(edge_flow_cost * flow_vars[(edge_id)]) + + logger.info("check if candidate tables exist") + sql = "SELECT name FROM sqlite_master WHERE type='table' " \ + "AND name in ('candidate_processors', 'candidate_process_list');" + count = len(db_cur.execute(sql).fetchall()) + + if count == 2: + + processor_build_cost = db_cur.execute(""" + select f.facility_id, (p.cost_formula*c.quantity) build_cost + from facilities f, facility_type_id ft, candidate_processors c, candidate_process_list p + where f.facility_type_id = ft.facility_type_id + and facility_type = 'processor' + and candidate = 1 + and ignore_facility = 'false' + and f.facility_name = c.facility_name + and c.process_id = p.process_id + group by f.facility_id, build_cost;""") + processor_build_cost_data = processor_build_cost.fetchall() + for row in processor_build_cost_data: + candidate_proc_facility_id = row[0] + proc_facility_build_cost = row[1] + processor_build_costs.append( + proc_facility_build_cost * processor_build_vars[candidate_proc_facility_id]) + + prob += (lpSum(unmet_demand_costs) + lpSum(operation_costs) + lpSum(flow_costs[k] * flow_vars[k] for k in flow_costs) + lpSum( + processor_build_costs)), "Total Cost of Transport, storage, facility building, and penalties" + + logger.debug("FINISHED: create_opt_problem") + return prob + + +# =============================================================================== + + +def create_constraint_unmet_demand(logger, the_scenario, prob, flow_var, unmet_demand_var): + logger.debug("START: create_constraint_unmet_demand") + + # apply activity_level to get corresponding actual demand for var + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + # var has form(facility_name, day, simple_fuel) + # unmet demand commodity should be simple_fuel = supertype + + demand_met_dict = defaultdict(list) + actual_demand_dict = {} + + # demand_met = [] + # want to specify that all edges leading into this vertex + unmet demand = total demand + # demand primary (non-storage) vertices + + db_cur = main_db_con.cursor() + # each row_a is a primary vertex whose edges in contributes to the met demand of var + # will have one row for each fuel subtype in the scenario + unmet_data = db_cur.execute("""select v.vertex_id, v.commodity_id, + v.demand, ifnull(c.proportion_of_supertype, 1), ifnull(v.activity_level, 1), v.source_facility_id, + v.facility_id, v.schedule_day, ifnull(c.supertype, c.commodity_name), v.udp, e.edge_id + from vertices v, commodities c, facility_type_id ft, facilities f, edges e + where v.facility_id = f.facility_id + and ft.facility_type = 'ultimate_destination' + and f.facility_type_id = ft.facility_type_id + and f.ignore_facility = 'false' + and v.facility_type_id = ft.facility_type_id + and v.storage_vertex = 0 + and c.commodity_id = v.commodity_id + and e.d_vertex_id = v.vertex_id + group by v.vertex_id, v.commodity_id, + v.demand, ifnull(c.proportion_of_supertype, 1), ifnull(v.activity_level, 1), v.source_facility_id, + v.facility_id, v.schedule_day, ifnull(c.supertype, c.commodity_name), v.udp, e.edge_id + ;""") + + unmet_data = unmet_data.fetchall() + for row_a in unmet_data: + # primary_vertex_id = row_a[0] + # commodity_id = row_a[1] + var_full_demand = row_a[2] + proportion_of_supertype = row_a[3] + var_activity_level = row_a[4] + # source_facility_id = row_a[5] + facility_id = row_a[6] + day = row_a[7] + top_level_commodity = row_a[8] + udp = row_a[9] + edge_id = row_a[10] + var_actual_demand = var_full_demand * var_activity_level + + # next get inbound edges, apply appropriate modifier proportion to get how much of var's demand they satisfy + demand_met_dict[(facility_id, day, top_level_commodity, udp)].append( + flow_var[edge_id] * proportion_of_supertype) + actual_demand_dict[(facility_id, day, top_level_commodity, udp)] = var_actual_demand + + for key in unmet_demand_var: + if key in demand_met_dict: + # then there are some edges in + prob += lpSum(demand_met_dict[key]) == actual_demand_dict[key] - unmet_demand_var[ + key], "constraint set unmet demand variable for facility {}, day {}, commodity {}".format(key[0], + key[1], + key[2]) + else: + if key not in actual_demand_dict: + pdb.set_trace() + # no edges in, so unmet demand equals full demand + prob += actual_demand_dict[key] == unmet_demand_var[ + key], "constraint set unmet demand variable for facility {}, day {}, " \ + "commodity {} - no edges able to meet demand".format( + key[0], key[1], key[2]) + + logger.debug("FINISHED: create_constraint_unmet_demand and return the prob ") + return prob + + +# =============================================================================== + + +def create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, flow_var): + logger.debug("STARTING: create_constraint_max_flow_out_of_supply_vertex") + logger.debug("Length of flow_var: {}".format(len(list(flow_var.items())))) + + # create_constraint_max_flow_out_of_supply_vertex + # primary vertices only + # flow out of a vertex <= supply of the vertex, true for every day and commodity + + # for each primary (non-storage) supply vertex + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + for row_a in db_cur.execute("""select vertex_id, activity_level, supply + from vertices v, facility_type_id ft + where v.facility_type_id = ft.facility_type_id + and ft.facility_type = 'raw_material_producer' + and storage_vertex = 0;"""): + supply_vertex_id = row_a[0] + activity_level = row_a[1] + max_daily_supply = row_a[2] + actual_vertex_supply = activity_level * max_daily_supply + + flow_out = [] + db_cur2 = main_db_con.cursor() + # select all edges leaving that vertex and sum their flows + # should be a single connector edge + for row_b in db_cur2.execute("select edge_id from edges where o_vertex_id = {};".format(supply_vertex_id)): + edge_id = row_b[0] + flow_out.append(flow_var[edge_id]) + + prob += lpSum(flow_out) <= actual_vertex_supply, "constraint max flow of {} out of origin vertex {}".format( + actual_vertex_supply, supply_vertex_id) + # could easily add human-readable vertex info to this if desirable + + logger.debug("FINISHED: create_constraint_max_flow_out_of_supply_vertex") + return prob + + +# =============================================================================== + + +def create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_var, processor_build_vars, + processor_daily_flow_vars): + logger.debug("STARTING: create_constraint_daily_processor_capacity") + import pdb + # pdb.set_trace() + # primary vertices only + # flow into vertex is capped at facility max_capacity per day + # sum over all input commodities, grouped by day and facility + # conservation of flow and ratios are handled in other methods + + ### get primary processor vertex and its input quantityi + total_scenario_min_capacity = 0 + + i = np.load("scenario_num.npy") + t = np.load("time_horizon.npy") + logger.info("scenario {}".format(i)) + logger.info("time horizon {}".format(t)) + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + sql = """select f.facility_id, + ifnull(f.candidate, 0), ifnull(f.max_capacity, -1), v.schedule_day, v.activity_level + from facility_commodities fc, facility_type_id ft, facilities f, vertices v + where ft.facility_type = 'processor' + and ft.facility_type_id = f.facility_type_id + and f.facility_id = fc.facility_id + and fc.io = 'i' + and v.facility_id = f.facility_id + and v.storage_vertex = 0 + group by f.facility_id, ifnull(f.candidate, 0), f.max_capacity, v.schedule_day, v.activity_level + ; + """ + # iterate through processor facilities, one constraint per facility per day + # no handling of subcommodities + + processor_facilities = db_cur.execute(sql) + + processor_facilities = processor_facilities.fetchall() + + for row_a in processor_facilities: + + # input_commodity_id = row_a[0] + facility_id = row_a[0] + is_candidate = row_a[1] + max_capacity = facility_cap_noEarthquake[float(facility_id)-1][t+1][i]*row_a[2] + day = row_a[3] + daily_activity_level = row_a[4] + + if max_capacity >= 0: + daily_inflow_max_capacity = float(max_capacity) * float(daily_activity_level) + daily_inflow_min_capacity = daily_inflow_max_capacity / 2 + logger.debug( + "processor {}, day {}, input capacity min: {} max: {}".format(facility_id, day, daily_inflow_min_capacity, + daily_inflow_max_capacity)) + total_scenario_min_capacity = total_scenario_min_capacity + daily_inflow_min_capacity + flow_in = [] + + # all edges that end in that processor facility primary vertex, on that day + db_cur2 = main_db_con.cursor() + for row_b in db_cur2.execute("""select edge_id from edges e, vertices v + where e.start_day = {} + and e.d_vertex_id = v.vertex_id + and v.facility_id = {} + and v.storage_vertex = 0 + group by edge_id""".format(day, facility_id)): + input_edge_id = row_b[0] + flow_in.append(flow_var[input_edge_id]) + + logger.debug( + "flow in for capacity constraint on processor facility {} day {}: {}".format(facility_id, day, flow_in)) + prob += lpSum(flow_in) <= daily_inflow_max_capacity * processor_daily_flow_vars[(facility_id, day)], \ + "constraint max flow into processor facility {}, day {}, flow var {}".format( + facility_id, day, processor_daily_flow_vars[facility_id, day]) + + prob += lpSum(flow_in) >= daily_inflow_min_capacity * processor_daily_flow_vars[ + (facility_id, day)], "constraint min flow into processor {}, day {}".format(facility_id, day) + # else: + # pdb.set_trace() + + if is_candidate == 1: + # forces processor build var to be correct + # if there is flow through a candidate processor then it has to be built + prob += processor_build_vars[facility_id] >= processor_daily_flow_vars[ + (facility_id, day)], "constraint forces processor build var to be correct {}, {}".format( + facility_id, processor_build_vars[facility_id]) + + logger.debug("FINISHED: create_constraint_daily_processor_capacity") + return prob + + +# =============================================================================== + + +def create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow_var): + logger.debug("STARTING: create_primary_processor_vertex_constraints - conservation of flow") + # for all of these vertices, flow in always == flow out + # node_counter = 0 + # node_constraint_counter = 0 + i = np.load("scenario_num.npy") + t = np.load("time_horizon.npy") + logger.info("scenario {}".format(i)) + logger.info("time horizon {}".format(t)) + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + + # total flow in == total flow out, subject to conversion; + # dividing by "required quantity" functionally converts all commodities to the same "processor-specific units" + + # processor primary vertices with input commodity and quantity needed to produce specified output quantities + # 2 sets of constraints; one for the primary processor vertex to cover total flow in and out + # one for each input and output commodity (sum over sources) to ensure its ratio matches facility_commodities + + # the current construction of this method is dependent on having only one input commodity type per processor + # this limitation makes sharing max transport distance from the input to an output commodity feasible + + logger.debug("conservation of flow and commodity ratios, primary processor vertices:") + sql = """select v.vertex_id, + (case when e.o_vertex_id = v.vertex_id then 'out' + when e.d_vertex_id = v.vertex_id then 'in' else 'error' end) in_or_out_edge, + (case when e.o_vertex_id = v.vertex_id then start_day + when e.d_vertex_id = v.vertex_id then end_day else 0 end) constraint_day, + e.commodity_id, + e.mode, + e.edge_id, + nx_edge_id, fc.quantity, v.facility_id, c.commodity_name, + fc.io, + v.activity_level, + ifnull(f.candidate, 0) candidate_check, + e.source_facility_id, + v.source_facility_id, + v.commodity_id, + c.share_max_transport_distance + from vertices v, facility_commodities fc, facility_type_id ft, commodities c, facilities f + join edges e on (v.vertex_id = e.o_vertex_id or v.vertex_id = e.d_vertex_id) + where ft.facility_type = 'processor' + and v.facility_id = f.facility_id + and ft.facility_type_id = v.facility_type_id + and storage_vertex = 0 + and v.facility_id = fc.facility_id + and fc.commodity_id = c.commodity_id + and fc.commodity_id = e.commodity_id + group by v.vertex_id, + in_or_out_edge, + constraint_day, + e.commodity_id, + e.mode, + e.edge_id, + nx_edge_id, fc.quantity, v.facility_id, c.commodity_name, + fc.io, + v.activity_level, + candidate_check, + e.source_facility_id, + v.commodity_id, + v.source_facility_id, + ifnull(c.share_max_transport_distance, 'N') + order by v.facility_id, e.source_facility_id, v.vertex_id, fc.io, e.edge_id + ;""" + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + sql_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for :") + logger.info( + "execute for processor primary vertices, with their in and out edges - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + sql_data = sql_data.fetchall() + logger.info( + "fetchall processor primary vertices, with their in and out edges - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + # Nested dictionaries + # flow_in_lists[primary_processor_vertex_id] = dict of commodities handled by that processor vertex + + # flow_in_lists[primary_processor_vertex_id][commodity1] = + # list of edge ids that flow that commodity into that vertex + + # flow_in_lists[vertex_id].values() to get all flow_in edges for all commodities, a list of lists + # if edge out commodity inherits transport distance, then source_facility id must match. if not, aggregate + + flow_in_lists = {} + flow_out_lists = {} + inherit_max_transport = {} + # inherit_max_transport[commodity_id] = 'Y' or 'N' + + for row_a in sql_data: + + vertex_id = row_a[0] + in_or_out_edge = row_a[1] + # constraint_day = row_a[2] + commodity_id = row_a[3] + # mode = row_a[4] + edge_id = row_a[5] + # nx_edge_id = row_a[6] + facility_id = row_a[8] + quantity = facility_cap_noEarthquake[float(facility_id)-1][t+1][i]*float(row_a[7]) + # facility_id = row_a[8] + # commodity_name = row_a[9] + # fc_io_commodity = row_a[10] + # activity_level = row_a[11] + # is_candidate = row_a[12] + edge_source_facility_id = row_a[13] + vertex_source_facility_id = row_a[14] + # v_commodity_id = row_a[15] + inherit_max_transport_distance = row_a[16] + if commodity_id not in inherit_max_transport.keys(): + if inherit_max_transport_distance == 'Y': + inherit_max_transport[commodity_id] = 'Y' + else: + inherit_max_transport[commodity_id] = 'N' + + if in_or_out_edge == 'in': + # if the vertex isn't in the main dict yet, add it + # could have multiple source facilities + # could also have more than one input commodity now + flow_in_lists.setdefault(vertex_id, {}) + flow_in_lists[vertex_id].setdefault((commodity_id, quantity, edge_source_facility_id), []).append(flow_var[edge_id]) + # flow_in_lists[vertex_id] is itself a dict keyed on commodity, quantity (ratio) and edge_source_facility; + # value is a list of edge ids into that vertex of that commodity and edge source + + elif in_or_out_edge == 'out': + # for out-lists, could have multiple commodities as well as multiple sources + # some may have a max transport distance, inherited or independent, some may not + flow_out_lists.setdefault(vertex_id, {}) # if the vertex isn't in the main dict yet, add it + flow_out_lists[vertex_id].setdefault((commodity_id, quantity, edge_source_facility_id), []).append(flow_var[edge_id]) + + # Because we keyed on commodity, source facility tracking is merged as we pass through the processor vertex + + # 1) for each output commodity, check against an input to ensure correct ratio - only need one input + # 2) for each input commodity, check against an output to ensure correct ratio - only need one output; + # 2a) first sum sub-flows over input commodity + + # 1---------------------------------------------------------------------- + constrained_input_flow_vars = set([]) + # pdb.set_trace() + + for key, value in iteritems(flow_out_lists): + #value is a dictionary with commodity & source as keys + # set up a dictionary that will be filled with input lists to check ratio against + compare_input_dict = {} + compare_input_dict_commod = {} + vertex_id = key + zero_in = False + #value is a dictionary keyed on output commodity, quantity required, edge source + if vertex_id in flow_in_lists: + in_quantity = 0 + in_commodity_id = 0 + in_source_facility_id = -1 + for ikey, ivalue in iteritems(flow_in_lists[vertex_id]): + in_commodity_id = ikey[0] + in_quantity = ikey[1] + in_source = ikey[2] + # list of edges + compare_input_dict[in_source] = ivalue + # to accommodate and track multiple input commodities; does not keep sources separate + # aggregate lists over sources, by commodity + if in_commodity_id not in compare_input_dict_commod.keys(): + compare_input_dict_commod[in_commodity_id] = set([]) + for edge in ivalue: + compare_input_dict_commod[in_commodity_id].add(edge) + else: + zero_in = True + + + # value is a dict - we loop once here for each output commodity and source at the vertex + for key2, value2 in iteritems(value): + out_commodity_id = key2[0] + out_quantity = key2[1] + out_source = key2[2] + # edge_list = value2 + flow_var_list = value2 + # if we need to match source facility, there is only one set of input lists + # otherwise, use all input lists - this aggregates sources + # need to keep commodities separate, units may be different + # known issue - we could have double-counting problems if only some outputs have to inherit max + # transport distance through this facility + match_source = inherit_max_transport[out_commodity_id] + compare_input_list = [] + if match_source == 'Y': + if len(compare_input_dict_commod.keys()) >1: + error = "Multiple input commodities for processors and shared max transport distance are" \ + " not supported within the same scenario." + logger.error(error) + raise Exception(error) + + if out_source in compare_input_dict.keys(): + compare_input_list = compare_input_dict[out_source] + # if no valid input edges - none for vertex, or if output needs to match source and there are no + # matching source + if zero_in or (match_source == 'Y' and len(compare_input_list) == 0): + prob += lpSum( + flow_var_list) == 0, "processor flow, vertex {} has zero in so zero out of commodity {} " \ + "with source {} if applicable".format( + vertex_id, out_commodity_id, out_source) + else: + if match_source == 'Y': + # ratio constraint for this output commodity relative to total input of each commodity + required_flow_out = lpSum(flow_var_list) / out_quantity + # check against an input dict + prob += required_flow_out == lpSum( + compare_input_list) / in_quantity, "processor flow, vertex {}, source_facility {}," \ + " commodity {} output quantity" \ + " checked against single input commodity quantity".format( + vertex_id, out_source, out_commodity_id, in_commodity_id) + for flow_var in compare_input_list: + constrained_input_flow_vars.add(flow_var) + else: + for k, v in iteritems(compare_input_dict_commod): + # pdb.set_trace() + # as long as the input source doesn't match an output that needs to inherit + compare_input_list = list(v) + in_commodity_id = k + # ratio constraint for this output commodity relative to total input of each commodity + required_flow_out = lpSum(flow_var_list) / out_quantity + # check against an input dict + prob += required_flow_out == lpSum( + compare_input_list) / in_quantity, "processor flow, vertex {}, source_facility {}," \ + " commodity {} output quantity" \ + " checked against commodity {} input quantity".format( + vertex_id, out_source, out_commodity_id, in_commodity_id) + for flow_var in compare_input_list: + constrained_input_flow_vars.add(flow_var) + + for key, value in iteritems(flow_in_lists): + vertex_id = key + for key2, value2 in iteritems(value): + commodity_id = key2[0] + # out_quantity = key2[1] + source = key2[2] + # edge_list = value2 + flow_var_list = value2 + for flow_var in flow_var_list: + if flow_var not in constrained_input_flow_vars: + prob += flow_var == 0, "processor flow, vertex {} has no matching out edges so zero in of " \ + "commodity {} with source {}".format( + vertex_id, commodity_id, source) + + logger.debug("FINISHED: create_primary_processor_conservation_of_flow_constraints") + return prob + + +# =============================================================================== + + +def create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_var, processor_excess_vars): + logger.debug("STARTING: create_constraint_conservation_of_flow") + # node_counter = 0 + node_constraint_counter = 0 + storage_vertex_constraint_counter = 0 + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + + logger.info("conservation of flow, storage vertices:") + # storage vertices, any facility type + # these have at most one direction of transport edges, so no need to track mode + sql = """select v.vertex_id, + (case when e.o_vertex_id = v.vertex_id then 'out' + when e.d_vertex_id = v.vertex_id then 'in' else 'error' end) in_or_out_edge, + (case when e.o_vertex_id = v.vertex_id then start_day + when e.d_vertex_id = v.vertex_id then end_day else 0 end) constraint_day, + v.commodity_id, + e.edge_id, + nx_edge_id, v.facility_id, c.commodity_name, + v.activity_level, + ft.facility_type + + from vertices v, facility_type_id ft, commodities c, facilities f + join edges e on ((v.vertex_id = e.o_vertex_id or v.vertex_id = e.d_vertex_id) + and (e.o_vertex_id = v.vertex_id or e.d_vertex_id = v.vertex_id) and v.commodity_id = e.commodity_id) + + where v.facility_id = f.facility_id + and ft.facility_type_id = v.facility_type_id + and storage_vertex = 1 + and v.commodity_id = c.commodity_id + + group by v.vertex_id, + in_or_out_edge, + constraint_day, + v.commodity_id, + e.edge_id, + nx_edge_id,v.facility_id, c.commodity_name, + v.activity_level + + order by v.facility_id, v.vertex_id, e.edge_id + ;""" + + # get the data from sql and see how long it takes. + logger.info("Starting the long step:") + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + vertexid_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for :") + logger.info("execute for storage vertices, with their in and out edges - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + vertexid_data = vertexid_data.fetchall() + logger.info( + "fetchall nodes with no location id, with their in and out edges - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + flow_in_lists = {} + flow_out_lists = {} + for row_v in vertexid_data: + vertex_id = row_v[0] + in_or_out_edge = row_v[1] + constraint_day = row_v[2] + commodity_id = row_v[3] + edge_id = row_v[4] + # nx_edge_id = row_v[5] + # facility_id = row_v[6] + # commodity_name = row_v[7] + # activity_level = row_v[8] + facility_type = row_v[9] + + if in_or_out_edge == 'in': + flow_in_lists.setdefault((vertex_id, commodity_id, constraint_day, facility_type), []).append( + flow_var[edge_id]) + elif in_or_out_edge == 'out': + flow_out_lists.setdefault((vertex_id, commodity_id, constraint_day, facility_type), []).append( + flow_var[edge_id]) + + logger.info("adding processor excess variabless to conservation of flow") + for key, value in iteritems(flow_out_lists): + vertex_id = key[0] + # commodity_id = key[1] + # day = key[2] + facility_type = key[3] + if facility_type == 'processor': + flow_out_lists.setdefault(key, []).append(processor_excess_vars[vertex_id]) + + for key, value in iteritems(flow_out_lists): + + if key in flow_in_lists: + prob += lpSum(flow_out_lists[key]) == lpSum( + flow_in_lists[key]), "conservation of flow, vertex {}, commodity {}, day {}".format(key[0], key[1], + key[2]) + storage_vertex_constraint_counter = storage_vertex_constraint_counter + 1 + else: + prob += lpSum(flow_out_lists[key]) == lpSum( + 0), "conservation of flow (zero out), vertex {}, commodity {}, day {}".format(key[0], key[1], + key[2]) + storage_vertex_constraint_counter = storage_vertex_constraint_counter + 1 + + for key, value in iteritems(flow_in_lists): + + if key not in flow_out_lists: + prob += lpSum(flow_in_lists[key]) == lpSum( + 0), "conservation of flow (zero in), vertex {}, commodity {}, day {}".format(key[0], key[1], + key[2]) + storage_vertex_constraint_counter = storage_vertex_constraint_counter + 1 + + logger.info( + "total conservation of flow constraints created on nodes: {}".format(storage_vertex_constraint_counter)) + + logger.info("conservation of flow, nx_nodes:") + # for each day, get all edges in and out of the node. + # Sort edges by commodity and whether they're going in or out of the node + sql = """select nn.node_id, + (case when e.from_node_id = nn.node_id then 'out' + when e.to_node_id = nn.node_id then 'in' else 'error' end) in_or_out_edge, + (case when e.from_node_id = nn.node_id then start_day + when e.to_node_id = nn.node_id then end_day else 0 end) constraint_day, + e.commodity_id, + ifnull(mode, 'NULL'), + e.edge_id, nx_edge_id, + miles, + (case when ifnull(nn.source, 'N') == 'intermodal' then 'Y' else 'N' end) intermodal_flag, + e.source_facility_id, + e.commodity_id + from networkx_nodes nn + join edges e on (nn.node_id = e.from_node_id or nn.node_id = e.to_node_id) + where nn.location_id is null + order by nn.node_id, e.commodity_id, + (case when e.from_node_id = nn.node_id then start_day + when e.to_node_id = nn.node_id then end_day else 0 end), + in_or_out_edge, e.source_facility_id, e.commodity_id + ;""" + + logger.info("Starting the long step:") + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + nodeid_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for :") + logger.info( + "execute for nodes with no location id, with their in and out edges - Total Runtime (HMS): \t{} \t " + "".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + nodeid_data = nodeid_data.fetchall() + logger.info( + "fetchall nodes with no location id, with their in and out edges - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + flow_in_lists = {} + flow_out_lists = {} + + for row_a in nodeid_data: + node_id = row_a[0] + in_or_out_edge = row_a[1] + constraint_day = row_a[2] + # commodity_id = row_a[3] + mode = row_a[4] + edge_id = row_a[5] + # nx_edge_id = row_a[6] + # miles = row_a[7] + intermodal = row_a[8] + source_facility_id = row_a[9] + commodity_id = row_a[10] + + # node_counter = node_counter +1 + # if node is not intermodal, conservation of flow holds per mode; + # if intermodal, then across modes + if intermodal == 'N': + if in_or_out_edge == 'in': + flow_in_lists.setdefault( + (node_id, intermodal, source_facility_id, constraint_day, commodity_id, mode), []).append( + flow_var[edge_id]) + elif in_or_out_edge == 'out': + flow_out_lists.setdefault( + (node_id, intermodal, source_facility_id, constraint_day, commodity_id, mode), []).append( + flow_var[edge_id]) + else: + if in_or_out_edge == 'in': + flow_in_lists.setdefault((node_id, intermodal, source_facility_id, constraint_day, commodity_id), + []).append(flow_var[edge_id]) + elif in_or_out_edge == 'out': + flow_out_lists.setdefault((node_id, intermodal, source_facility_id, constraint_day, commodity_id), + []).append(flow_var[edge_id]) + + for key, value in iteritems(flow_out_lists): + node_id = key[0] + # intermodal_flag = key[1] + source_facility_id = key[2] + day = key[3] + commodity_id = key[4] + if len(key) == 6: + node_mode = key[5] + else: + node_mode = 'intermodal' + if key in flow_in_lists: + prob += lpSum(flow_out_lists[key]) == lpSum(flow_in_lists[ + key]), "conservation of flow, nx node {}, " \ + "source facility {}, commodity {}, " \ + "day {}, mode {}".format( + node_id, source_facility_id, commodity_id, day, node_mode) + node_constraint_counter = node_constraint_counter + 1 + else: + prob += lpSum(flow_out_lists[key]) == lpSum( + 0), "conservation of flow (zero out), nx node {}, source facility {}, commodity {}, day {}," \ + " mode {}".format(node_id, source_facility_id, commodity_id, day, node_mode) + node_constraint_counter = node_constraint_counter + 1 + + for key, value in iteritems(flow_in_lists): + node_id = key[0] + # intermodal_flag = key[1] + source_facility_id = key[2] + day = key[3] + commodity_id = key[4] + if len(key) == 6: + node_mode = key[5] + else: + node_mode = 'intermodal' + + if key not in flow_out_lists: + prob += lpSum(flow_in_lists[key]) == lpSum( + 0), "conservation of flow (zero in), nx node {}, source facility {}, commodity {}, day {}," \ + " mode {}".format(node_id, source_facility_id, commodity_id, day, node_mode) + node_constraint_counter = node_constraint_counter + 1 + + logger.info("total conservation of flow constraints created on nodes: {}".format(node_constraint_counter)) + + # Note: no consesrvation of flow for primary vertices for supply & demand - they have unique constraints + + logger.debug("FINISHED: create_constraint_conservation_of_flow") + + return prob + + +# =============================================================================== + + +def create_constraint_max_route_capacity(logger, the_scenario, prob, flow_var): + logger.info("STARTING: create_constraint_max_route_capacity") + logger.info("modes with background flow turned on: {}".format(the_scenario.backgroundFlowModes)) + # min_capacity_level must be a number from 0 to 1, inclusive + # min_capacity_level is only relevant when background flows are turned on + # it sets a floor to how much capacity can be reduced by volume. + # min_capacity_level = .25 means route capacity will never be less than 25% of full capacity, + # even if "volume" would otherwise restrict it further + # min_capacity_level = 0 allows a route to be made unavailable for FTOT flow if base volume is too high + # this currently applies to all modes + logger.info("minimum available capacity floor set at: {}".format(the_scenario.minCapacityLevel)) + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + # capacity for storage routes + sql = """select + rr.route_id, sr.storage_max, sr.route_name, e.edge_id, e.start_day + from route_reference rr + join storage_routes sr on sr.route_name = rr.route_name + join edges e on rr.route_id = e.route_id + ;""" + # get the data from sql and see how long it takes. + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + storage_edge_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for storage edges:") + logger.info("execute for edges for storage - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + storage_edge_data = storage_edge_data.fetchall() + logger.info("fetchall edges for storage - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + flow_lists = {} + + for row_a in storage_edge_data: + route_id = row_a[0] + aggregate_storage_capac = row_a[1] + storage_route_name = row_a[2] + edge_id = row_a[3] + start_day = row_a[4] + + flow_lists.setdefault((route_id, aggregate_storage_capac, storage_route_name, start_day), []).append( + flow_var[edge_id]) + + for key, flow in iteritems(flow_lists): + prob += lpSum(flow) <= key[1], "constraint max flow on storage route {} named {} for day {}".format(key[0], + key[2], + key[3]) + + logger.debug("route_capacity constraints created for all storage routes") + + # capacity for transport routes + # Assumption - all flowing material is in kgal, all flow is summed on a single non-pipeline nx edge + sql = """select e.edge_id, e.commodity_id, e.nx_edge_id, e.max_edge_capacity, e.start_day, e.simple_mode, e.phase_of_matter, + e.capac_minus_volume_zero_floor + from edges e + where e.max_edge_capacity is not null + and e.simple_mode != 'pipeline' + ;""" + # get the data from sql and see how long it takes. + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + route_capac_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for transport edges:") + logger.info("execute for non-pipeline edges for transport edge capacity - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + route_capac_data = route_capac_data.fetchall() + logger.info("fetchall non-pipeline edges for transport edge capacity - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + flow_lists = {} + + for row_a in route_capac_data: + edge_id = row_a[0] + commodity_id = row_a[1] + nx_edge_id = row_a[2] + nx_edge_capacity = row_a[3] + start_day = row_a[4] + simple_mode = row_a[5] + phase_of_matter = row_a[6] + capac_minus_background_flow = max(row_a[7], 0) + min_restricted_capacity = max(capac_minus_background_flow, nx_edge_capacity * the_scenario.minCapacityLevel) + + if simple_mode in the_scenario.backgroundFlowModes: + use_capacity = min_restricted_capacity + else: + use_capacity = nx_edge_capacity + + # flow is in thousand gallons (kgal), for liquid, or metric tons, for solid + # capacity is in truckload, rail car, barge, or pipeline movement per day + # if mode is road and phase is liquid, capacity is in truckloads per day, we want it in kgal + # ftot_supporting_gis tells us that there are 8 kgal per truckload, + # so capacity * 8 gives us correct units or kgal per day + # => use capacity * ftot_supporting_gis multiplier to get capacity in correct flow units + + multiplier = 1 # if units match, otherwise specified here + if simple_mode == 'road': + if phase_of_matter == 'liquid': + multiplier = the_scenario.truck_load_liquid.magnitude + elif phase_of_matter == 'solid': + multiplier = the_scenario.truck_load_solid.magnitude + elif simple_mode == 'water': + if phase_of_matter == 'liquid': + multiplier = the_scenario.barge_load_liquid.magnitude + elif phase_of_matter == 'solid': + multiplier = the_scenario.barge_load_solid.magnitude + elif simple_mode == 'rail': + if phase_of_matter == 'liquid': + multiplier = the_scenario.railcar_load_liquid.magnitude + elif phase_of_matter == 'solid': + multiplier = the_scenario.railcar_load_solid.magnitude + + converted_capacity = use_capacity * multiplier + + flow_lists.setdefault((nx_edge_id, converted_capacity, commodity_id, start_day), []).append(flow_var[edge_id]) + + for key, flow in iteritems(flow_lists): + prob += lpSum(flow) <= key[1], "constraint max flow on nx edge {} commodity {} for day {}".format(key[0], key[2], key[3]) + + logger.debug("route_capacity constraints created for all non-pipeline transport routes") + + logger.debug("FINISHED: create_constraint_max_route_capacity") + return prob + + +# =============================================================================== + + +def create_constraint_pipeline_capacity(logger, the_scenario, prob, flow_var): + logger.debug("STARTING: create_constraint_pipeline_capacity") + logger.debug("Length of flow_var: {}".format(len(list(flow_var.items())))) + logger.info("modes with background flow turned on: {}".format(the_scenario.backgroundFlowModes)) + logger.info("minimum available capacity floor set at: {}".format(the_scenario.minCapacityLevel)) + + with sqlite3.connect(the_scenario.main_db) as main_db_con: + db_cur = main_db_con.cursor() + + # capacity for pipeline tariff routes + # with sasc, may have multiple flows per segment, slightly diff commodities + sql = """select e.edge_id, e.tariff_id, l.link_id, l.capac, e.start_day, l.capac-l.background_flow allowed_flow, + l.source, e.mode, instr(e.mode, l.source) + from edges e, pipeline_mapping pm, + (select id_field_name, cn.source_OID as link_id, min(cn.capacity) capac, + max(cn.volume) background_flow, source + from capacity_nodes cn + where cn.id_field_name = 'MASTER_OID' + and ifnull(cn.capacity,0)>0 + group by link_id) l + + where e.tariff_id = pm.id + and pm.id_field_name = 'tariff_ID' + and pm.mapping_id_field_name = 'MASTER_OID' + and l.id_field_name = 'MASTER_OID' + and pm.mapping_id = l.link_id + and instr(e.mode, l.source)>0 + group by e.edge_id, e.tariff_id, l.link_id, l.capac, e.start_day, allowed_flow, l.source + ;""" + # capacity needs to be shared over link_id for any edge_id associated with that link + + logger.info("Starting the execute") + execute_start_time = datetime.datetime.now() + pipeline_capac_data = db_cur.execute(sql) + logger.info("Done with the execute fetch all for transport edges:") + logger.info("execute for edges for transport edge capacity - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(execute_start_time))) + + logger.info("Starting the fetchall") + fetchall_start_time = datetime.datetime.now() + pipeline_capac_data = pipeline_capac_data.fetchall() + logger.info("fetchall edges for transport edge capacity - Total Runtime (HMS): \t{} \t ".format( + get_total_runtime_string(fetchall_start_time))) + + flow_lists = {} + + for row_a in pipeline_capac_data: + edge_id = row_a[0] + # tariff_id = row_a[1] + link_id = row_a[2] + # Link capacity is recorded in "thousand barrels per day"; 1 barrel = 42 gall + # Link capacity * 42 is now in kgal per day, to match flow in kgal + link_capacity_kgal_per_day = THOUSAND_GALLONS_PER_THOUSAND_BARRELS * row_a[3] + start_day = row_a[4] + capac_minus_background_flow_kgal = max(THOUSAND_GALLONS_PER_THOUSAND_BARRELS * row_a[5], 0) + min_restricted_capacity = max(capac_minus_background_flow_kgal, + link_capacity_kgal_per_day * the_scenario.minCapacityLevel) + + # capacity_nodes_mode_source = row_a[6] + edge_mode = row_a[7] + # mode_match_check = row_a[8] + if 'pipeline' in the_scenario.backgroundFlowModes: + link_use_capacity = min_restricted_capacity + else: + link_use_capacity = link_capacity_kgal_per_day + + # add flow from all relevant edges, for one start; may be multiple tariffs + flow_lists.setdefault((link_id, link_use_capacity, start_day, edge_mode), []).append(flow_var[edge_id]) + + for key, flow in iteritems(flow_lists): + prob += lpSum(flow) <= key[1], "constraint max flow on pipeline link {} for mode {} for day {}".format( + key[0], key[3], key[2]) + + logger.debug("pipeline capacity constraints created for all transport routes") + + logger.debug("FINISHED: create_constraint_pipeline_capacity") + return prob + + +# =============================================================================== + + +def setup_pulp_problem(the_scenario, logger): + logger.info("START: setup PuLP problem") + + # flow_var is the flow on each edge by commodity and day. + # the optimal value of flow_var will be solved by PuLP + flow_vars = create_flow_vars(the_scenario, logger) + + # unmet_demand_var is the unmet demand at each destination, being determined + unmet_demand_vars = create_unmet_demand_vars(the_scenario, logger) + + # processor_build_vars is the binary variable indicating whether a candidate processor is used + # and thus whether its build cost is charged + processor_build_vars = create_candidate_processor_build_vars(the_scenario, logger) + + # binary tracker variables + processor_vertex_flow_vars = create_binary_processor_vertex_flow_vars(the_scenario, logger) + + # tracking unused production + processor_excess_vars = create_processor_excess_output_vars(the_scenario, logger) + + # THIS IS THE OBJECTIVE FUCTION FOR THE OPTIMIZATION + # ================================================== + + prob = create_opt_problem(logger, the_scenario, unmet_demand_vars, flow_vars, processor_build_vars) + + prob = create_constraint_unmet_demand(logger, the_scenario, prob, flow_vars, unmet_demand_vars) + + prob = create_constraint_max_flow_out_of_supply_vertex(logger, the_scenario, prob, flow_vars) + + # This constraint is being excluded because 1) it is not used in current scenarios and 2) it is not supported by + # this version - it conflicts with the change permitting multiple inputs + # adding back 12/2020 + prob = create_constraint_daily_processor_capacity(logger, the_scenario, prob, flow_vars, processor_build_vars, + processor_vertex_flow_vars) + + prob = create_primary_processor_vertex_constraints(logger, the_scenario, prob, flow_vars) + + prob = create_constraint_conservation_of_flow(logger, the_scenario, prob, flow_vars, processor_excess_vars) + + if the_scenario.capacityOn: + prob = create_constraint_max_route_capacity(logger, the_scenario, prob, flow_vars) + + prob = create_constraint_pipeline_capacity(logger, the_scenario, prob, flow_vars) + + del unmet_demand_vars + + del flow_vars + + # The problem data is written to an .lp file + prob.writeLP(os.path.join(the_scenario.scenario_run_directory, "debug", "LP_output_c2.lp")) + + logger.info("FINISHED: setup PuLP problem") + return prob + + +# =============================================================================== + + +def solve_pulp_problem(prob_final, the_scenario, logger): + import datetime + + logger.info("START: solve_pulp_problem") + start_time = datetime.datetime.now() + from os import dup, dup2, close + f = open(os.path.join(the_scenario.scenario_run_directory, "debug", 'probsolve_capture.txt'), 'w') + orig_std_out = dup(1) + dup2(f.fileno(), 1) + + # status = prob_final.solve (PULP_CBC_CMD(maxSeconds = i_max_sec, fracGap = d_opt_gap, msg=1)) + # CBC time limit and relative optimality gap tolerance + status = prob_final.solve(PULP_CBC_CMD(msg=1)) # CBC time limit and relative optimality gap tolerance + logger.info('Completion code: %d; Solution status: %s; Best obj value found: %s' % ( + status, LpStatus[prob_final.status], value(prob_final.objective))) + + dup2(orig_std_out, 1) + close(orig_std_out) + f.close() + # The problem is solved using PuLP's choice of Solver + + logger.info("completed calling prob.solve()") + logger.info( + "FINISH: prob.solve(): Runtime (HMS): \t{}".format(ftot_supporting.get_total_runtime_string(start_time))) + + # THIS IS THE SOLUTION + + # The status of the solution is printed to the screen + ##LpStatus key string value numerical value + ##LpStatusOptimal ?Optimal? 1 + ##LpStatusNotSolved ?Not Solved? 0 + ##LpStatusInfeasible ?Infeasible? -1 + ##LpStatusUnbounded ?Unbounded? -2 + ##LpStatusUndefined ?Undefined? -3 + logger.result("prob.Status: \t {}".format(LpStatus[prob_final.status])) + + logger.result( + "Total Scenario Cost = (transportation + unmet demand penalty + processor construction): \t ${0:,.0f}".format( + float(value(prob_final.objective)))) + + return prob_final + + +# =============================================================================== + +def save_pulp_solution(the_scenario, prob, logger, zero_threshold=0.00001): + i = np.load("scenario_num.npy") + t = np.load("time_horizon.npy") + logger.info("scenario {}".format(i)) + logger.info("time horizon {}".format(t)) + + import datetime + start_time = datetime.datetime.now() + logger.info("START: save_pulp_solution") + non_zero_variable_count = 0 + + with sqlite3.connect(the_scenario.main_db) as db_con: + + db_cur = db_con.cursor() + # drop the optimal_solution table + # ----------------------------- + db_cur.executescript("drop table if exists optimal_solution;") + + # create the optimal_solution table + # ----------------------------- + db_cur.executescript(""" + create table optimal_solution + ( + variable_name string, + variable_value real + ); + """) + + # insert the optimal data into the DB + # ------------------------------------- + for v in prob.variables(): + if v.varValue is None: + logger.debug("Variable value is none: " + str(v.name)) + else: + if v.varValue > zero_threshold: # eliminates values too close to zero + sql = """insert into optimal_solution (variable_name, variable_value) values ("{}", {});""".format( + v.name, float(v.varValue)) + db_con.execute(sql) + non_zero_variable_count = non_zero_variable_count + 1 + + # query the optimal_solution table in the DB for each variable we care about + # ---------------------------------------------------------------------------- + sql = "select count(variable_name) from optimal_solution where variable_name like 'BuildProcessor%';" + data = db_con.execute(sql) + optimal_processors_count = data.fetchone()[0] + logger.info("number of optimal_processors: {}".format(optimal_processors_count)) + + sql = "select count(variable_name) from optimal_solution where variable_name like 'UnmetDemand%';" + data = db_con.execute(sql) + optimal_unmet_demand_count = data.fetchone()[0] + logger.info("number facilities with optimal_unmet_demand : {}".format(optimal_unmet_demand_count)) + sql = "select ifnull(sum(variable_value),0) from optimal_solution where variable_name like 'UnmetDemand%';" + data = db_con.execute(sql) + optimal_unmet_demand_sum = data.fetchone()[0] + + unmet_demand_yearly = optimal_unmet_demand_sum + np.save("unmet_demand_yearly.npy", unmet_demand_yearly) + + + logger.info("Total Unmet Demand : {}".format(optimal_unmet_demand_sum)) + logger.info("Penalty per unit of Unmet Demand : ${0:,.0f}".format(the_scenario.unMetDemandPenalty)) + logger.info("Total Cost of Unmet Demand : \t ${0:,.0f}".format( + optimal_unmet_demand_sum * the_scenario.unMetDemandPenalty)) + #=================================non consider the UDP when UD < 0====================================== + if optimal_unmet_demand_sum < 0: + costs_yearly = (abs(float(value(prob.objective))- optimal_unmet_demand_sum * the_scenario.unMetDemandPenalty))*((1+Inflation_rate)**t) + sum(CatalystReplace_cost[:,t+1,i]) + + logger.result( + "Total Scenario Cost = (transportation + processor construction + operation + catalyst replacement): \t ${0:,.0f}" + "".format(costs_yearly)) + else: + costs_yearly = (abs(float(value(prob.objective))-optimal_unmet_demand_sum * the_scenario.unMetDemandPenalty))*((1+Inflation_rate)**t)+sum(CatalystReplace_cost[:,t+1,i])+(optimal_unmet_demand_sum*the_scenario.unMetDemandPenalty)*((1+Inflation_rate)**t) + logger.result( + "Total Scenario Cost = (transportation + unmet demand penalty + processor construction + operation+ catalyst replacement): \t ${0:,.0f}" + "".format(costs_yearly)) + + np.save("costs_yearly.npy", costs_yearly) + logger.info("costs_yearly: {}".format(costs_yearly)) + #=================================================================== + + sql = "select count(variable_name) from optimal_solution where variable_name like 'Edge%';" + data = db_con.execute(sql) + optimal_edges_count = data.fetchone()[0] + logger.info("number of optimal edges: {}".format(optimal_edges_count)) + + + logger.info( + "FINISH: save_pulp_solution: Runtime (HMS): \t{}".format(ftot_supporting.get_total_runtime_string(start_time))) + +# =============================================================================== + +def record_pulp_solution(the_scenario, logger): + logger.info("START: record_pulp_solution") + non_zero_variable_count = 0 + + with sqlite3.connect(the_scenario.main_db) as db_con: + + logger.info("number of solution variables greater than zero: {}".format(non_zero_variable_count)) + sql = """ + create table optimal_variables as + select + 'UnmetDemand' as variable_type, + cast(substr(variable_name, 13) as int) var_id, + variable_value, + null as converted_capacity, + null as converted_volume, + null as converted_capac_minus_volume, + null as edge_type, + null as commodity_name, + null as o_facility, + 'placeholder' as d_facility, + null as o_vertex_id, + null as d_vertex_id, + null as from_node_id, + null as to_node_id, + null as time_period, + null as commodity_id, + null as source_facility_id, + null as source_facility_name, + null as units, + variable_name, + null as nx_edge_id, + null as mode, + null as mode_oid, + null as miles, + null as original_facility, + null as final_facility, + null as prior_edge, + null as miles_travelled + from optimal_solution + where variable_name like 'UnmetDemand%' + union + select + 'Edge' as variable_type, + cast(substr(variable_name, 6) as int) var_id, + variable_value, + edges.max_edge_capacity*edges.units_conversion_multiplier as converted_capacity, + edges.volume*edges.units_conversion_multiplier as converted_volume, + edges.capac_minus_volume_zero_floor*edges.units_conversion_multiplier as converted_capac_minus_volume, + edges.edge_type, + commodities.commodity_name, + ov.facility_name as o_facility, + dv.facility_name as d_facility, + o_vertex_id, + d_vertex_id, + from_node_id, + to_node_id, + start_day time_period, + edges.commodity_id, + edges.source_facility_id, + s.source_facility_name, + commodities.units, + variable_name, + edges.nx_edge_id, + edges.mode, + edges.mode_oid, + edges.miles, + null as original_facility, + null as final_facility, + null as prior_edge, + edges.miles_travelled as miles_travelled + from optimal_solution + join edges on edges.edge_id = cast(substr(variable_name, 6) as int) + join commodities on edges.commodity_id = commodities.commodity_ID + left outer join vertices as ov on edges.o_vertex_id = ov.vertex_id + left outer join vertices as dv on edges.d_vertex_id = dv.vertex_id + left outer join source_commodity_ref as s on edges.source_facility_id = s.source_facility_id + where variable_name like 'Edge%' + union + select + 'BuildProcessor' as variable_type, + cast(substr(variable_name, 16) as int) var_id, + variable_value, + null as converted_capacity, + null as converted_volume, + null as converted_capac_minus_volume, + null as edge_type, + null as commodity_name, + 'placeholder' as o_facility, + 'placeholder' as d_facility, + null as o_vertex_id, + null as d_vertex_id, + null as from_node_id, + null as to_node_id, + null as time_period, + null as commodity_id, + null as source_facility_id, + null as source_facility_name, + null as units, + variable_name, + null as nx_edge_id, + null as mode, + null as mode_oid, + null as miles, + null as original_facility, + null as final_facility, + null as prior_edge, + null as miles_travelled + from optimal_solution + where variable_name like 'Build%'; + """ + db_con.execute("drop table if exists optimal_variables;") + db_con.execute(sql) + + logger.info("FINISH: record_pulp_solution") + +# =============================================================================== + + +def parse_optimal_solution_db(the_scenario, logger): + logger.info("starting parse_optimal_solution") + + optimal_processors = [] + optimal_processor_flows = [] + optimal_route_flows = {} + optimal_unmet_demand = {} + optimal_storage_flows = {} + optimal_excess_material = {} + + with sqlite3.connect(the_scenario.main_db) as db_con: + + # do the Storage Edges + sql = "select variable_name, variable_value from optimal_solution where variable_name like 'Edge%_storage';" + data = db_con.execute(sql) + optimal_storage_edges = data.fetchall() + for edge in optimal_storage_edges: + optimal_storage_flows[edge] = optimal_storage_edges[edge] + + # do the Route Edges + sql = """select + variable_name, variable_value, + cast(substr(variable_name, 6) as int) edge_id, + route_id, start_day time_period, edges.commodity_id, + o_vertex_id, d_vertex_id, + v1.facility_id o_facility_id, + v2.facility_id d_facility_id + from optimal_solution + join edges on edges.edge_id = cast(substr(variable_name, 6) as int) + join vertices v1 on edges.o_vertex_id = v1.vertex_id + join vertices v2 on edges.d_vertex_id = v2.vertex_id + where variable_name like 'Edge%_' and variable_name not like 'Edge%_storage'; + """ + data = db_con.execute(sql) + optimal_route_edges = data.fetchall() + for edge in optimal_route_edges: + + variable_name = edge[0] + + variable_value = edge[1] + + edge_id = edge[2] + + route_id = edge[3] + + time_period = edge[4] + + commodity_flowed = edge[5] + + od_pair_name = "{}, {}".format(edge[8], edge[9]) + + # first time route_id is used on a day or commodity + if route_id not in optimal_route_flows: + optimal_route_flows[route_id] = [[od_pair_name, time_period, commodity_flowed, variable_value]] + + else: # subsequent times route is used on different day or for other commodity + optimal_route_flows[route_id].append([od_pair_name, time_period, commodity_flowed, variable_value]) + + # do the processors + sql = "select variable_name, variable_value from optimal_solution where variable_name like 'BuildProcessor%';" + data = db_con.execute(sql) + optimal_candidates_processors = data.fetchall() + for proc in optimal_candidates_processors: + optimal_processors.append(proc) + + # do the processor vertex flows + sql = "select variable_name, variable_value from optimal_solution where variable_name like 'ProcessorVertexFlow%';" + data = db_con.execute(sql) + optimal_processor_flows_sql = data.fetchall() + for proc in optimal_processor_flows_sql: + optimal_processor_flows.append(proc) + + # do the UnmetDemand + sql = "select variable_name, variable_value from optimal_solution where variable_name like 'UnmetDemand%';" + data = db_con.execute(sql) + optimal_unmetdemand = data.fetchall() + for ultimate_destination in optimal_unmetdemand: + v_name = ultimate_destination[0] + v_value = ultimate_destination[1] + + search = re.search('\(.*\)', v_name.replace("'", "")) + + if search: + parts = search.group(0).replace("(", "").replace(")", "").split(",_") + + dest_name = parts[0] + commodity_flowed = parts[2] + if not dest_name in optimal_unmet_demand: + optimal_unmet_demand[dest_name] = {} + + if not commodity_flowed in optimal_unmet_demand[dest_name]: + optimal_unmet_demand[dest_name][commodity_flowed] = int(v_value) + else: + optimal_unmet_demand[dest_name][commodity_flowed] += int(v_value) + + + logger.info("length of optimal_processors list: {}".format(len(optimal_processors))) # a list of optimal processors + logger.info("length of optimal_processor_flows list: {}".format( + len(optimal_processor_flows))) # a list of optimal processor flows + logger.info("length of optimal_route_flows dict: {}".format( + len(optimal_route_flows))) # a dictionary of routes keys and commodity flow values + logger.info("length of optimal_unmet_demand dict: {}".format( + len(optimal_unmet_demand))) # a dictionary of route keys and unmet demand values + + return optimal_processors, optimal_route_flows, optimal_unmet_demand, optimal_storage_flows, optimal_excess_material + + diff --git a/program/ftot_scenario.py b/program/ftot_scenario.py index 49183f3..18ac6a9 100644 --- a/program/ftot_scenario.py +++ b/program/ftot_scenario.py @@ -89,6 +89,10 @@ def load_scenario_config_file(fullPathToXmlConfigFile, fullPathToXmlSchemaFile, raise Exception(error) scenario.scenario_name = xmlScenarioFile.getElementsByTagName('Scenario_Name')[0].firstChild.data + # Convert any commas in the scenario name to dashes + warning = "Replace any commas in the scenario name with dashes to accomodate csv files." + logger.debug(warning) + scenario.scenario_name = scenario.scenario_name.replace(",", "-") scenario.scenario_description = xmlScenarioFile.getElementsByTagName('Scenario_Description')[0].firstChild.data # SCENARIO INPUTS SECTION @@ -134,7 +138,7 @@ def load_scenario_config_file(fullPathToXmlConfigFile, fullPathToXmlSchemaFile, # solid and liquid vehicle loads try: - logger.debug("test: setting the vehicle loads for solid phase of matter pint") + logger.debug("test: setting the vehicle loads for solid phase of matter with pint") scenario.truck_load_solid = Q_(xmlScenarioFile.getElementsByTagName('Truck_Load_Solid')[0].firstChild.data).to(scenario.default_units_solid_phase) scenario.railcar_load_solid = Q_(xmlScenarioFile.getElementsByTagName('Railcar_Load_Solid')[0].firstChild.data).to(scenario.default_units_solid_phase) scenario.barge_load_solid = Q_(xmlScenarioFile.getElementsByTagName('Barge_Load_Solid')[0].firstChild.data).to(scenario.default_units_solid_phase) @@ -145,7 +149,7 @@ def load_scenario_config_file(fullPathToXmlConfigFile, fullPathToXmlSchemaFile, scenario.barge_load_liquid = Q_(xmlScenarioFile.getElementsByTagName('Barge_Load_Liquid')[0].firstChild.data).to(scenario.default_units_liquid_phase) scenario.pipeline_crude_load_liquid = Q_(xmlScenarioFile.getElementsByTagName('Pipeline_Crude_Load_Liquid')[0].firstChild.data).to(scenario.default_units_liquid_phase) scenario.pipeline_prod_load_liquid = Q_(xmlScenarioFile.getElementsByTagName('Pipeline_Prod_Load_Liquid')[0].firstChild.data).to(scenario.default_units_liquid_phase) - logger.debug("PASS: the setting the vehicle loads with pint passed") + logger.debug("PASS: setting the vehicle loads with pint passed") except Exception as e: logger.error("FAIL: {} ".format(e)) @@ -166,9 +170,18 @@ def load_scenario_config_file(fullPathToXmlConfigFile, fullPathToXmlSchemaFile, # SCRIPT PARAMETERS SECTION FOR NETWORK # ---------------------------------------------------------------------------------------- - #rail costs - scenario.solid_railroad_class_1_cost = format_number(xmlScenarioFile.getElementsByTagName('solid_Railroad_Class_I_Cost')[0].firstChild.data) - scenario.liquid_railroad_class_1_cost = format_number(xmlScenarioFile.getElementsByTagName('liquid_Railroad_Class_I_Cost')[0].firstChild.data) + # rail costs + try: + logger.debug("test: setting the base costs for rail with pint") + scenario.solid_railroad_class_1_cost = Q_(format_number(xmlScenarioFile.getElementsByTagName('solid_Railroad_Class_I_Cost')[0].firstChild.data), + "usd/tonne/mile").to("usd/{}/mile".format(scenario.default_units_solid_phase)) + scenario.liquid_railroad_class_1_cost = Q_(format_number(xmlScenarioFile.getElementsByTagName('liquid_Railroad_Class_I_Cost')[0].firstChild.data), + "usd/kgal/mile").to("usd/{}/mile".format(scenario.default_units_liquid_phase)) + logger.debug("PASS: setting the base costs for rail with pint passed") + + except Exception as e: + logger.error("FAIL: {} ".format(e)) + raise Exception("FAIL: {}".format(e)) # rail penalties scenario.rail_dc_7 = format_number(xmlScenarioFile.getElementsByTagName('Rail_Density_Code_7_Weight')[0].firstChild.data) @@ -180,9 +193,18 @@ def load_scenario_config_file(fullPathToXmlConfigFile, fullPathToXmlSchemaFile, scenario.rail_dc_1 = format_number(xmlScenarioFile.getElementsByTagName('Rail_Density_Code_1_Weight')[0].firstChild.data) scenario.rail_dc_0 = format_number(xmlScenarioFile.getElementsByTagName('Rail_Density_Code_0_Weight')[0].firstChild.data) - #truck costs - scenario.solid_truck_base_cost = format_number(xmlScenarioFile.getElementsByTagName('solid_Truck_Base_Cost')[0].firstChild.data) - scenario.liquid_truck_base_cost = format_number(xmlScenarioFile.getElementsByTagName('liquid_Truck_Base_Cost')[0].firstChild.data) + # truck costs + try: + logger.debug("test: setting the base costs for truck with pint") + scenario.solid_truck_base_cost = Q_(format_number(xmlScenarioFile.getElementsByTagName('solid_Truck_Base_Cost')[0].firstChild.data), + "usd/tonne/mile").to("usd/{}/mile".format(scenario.default_units_solid_phase)) + scenario.liquid_truck_base_cost = Q_(format_number(xmlScenarioFile.getElementsByTagName('liquid_Truck_Base_Cost')[0].firstChild.data), + "usd/kgal/mile").to("usd/{}/mile".format(scenario.default_units_liquid_phase)) + logger.debug("PASS: setting the base costs for truck with pint passed") + + except Exception as e: + logger.error("FAIL: {} ".format(e)) + raise Exception("FAIL: {}".format(e)) # road penalties scenario.truck_interstate = format_number(xmlScenarioFile.getElementsByTagName('Truck_Interstate_Weight')[0].firstChild.data) @@ -191,8 +213,17 @@ def load_scenario_config_file(fullPathToXmlConfigFile, fullPathToXmlSchemaFile, scenario.truck_local = format_number(xmlScenarioFile.getElementsByTagName('Truck_Local_Weight')[0].firstChild.data) # barge costs - scenario.solid_barge_cost = format_number(xmlScenarioFile.getElementsByTagName('solid_Barge_cost')[0].firstChild.data) - scenario.liquid_barge_cost = format_number(xmlScenarioFile.getElementsByTagName('liquid_Barge_cost')[0].firstChild.data) + try: + logger.debug("test: setting the base costs for barge with pint") + scenario.solid_barge_cost = Q_(format_number(xmlScenarioFile.getElementsByTagName('solid_Barge_cost')[0].firstChild.data), + "usd/tonne/mile").to("usd/{}/mile".format(scenario.default_units_solid_phase)) + scenario.liquid_barge_cost = Q_(format_number(xmlScenarioFile.getElementsByTagName('liquid_Barge_cost')[0].firstChild.data), + "usd/kgal/mile").to("usd/{}/mile".format(scenario.default_units_liquid_phase)) + logger.debug("PASS: setting the base costs for barge with pint passed") + + except Exception as e: + logger.error("FAIL: {} ".format(e)) + raise Exception("FAIL: {}".format(e)) # water penalties scenario.water_high_vol = format_number(xmlScenarioFile.getElementsByTagName('Water_High_Volume_Weight')[0].firstChild.data) @@ -201,8 +232,18 @@ def load_scenario_config_file(fullPathToXmlConfigFile, fullPathToXmlSchemaFile, scenario.water_no_vol = format_number(xmlScenarioFile.getElementsByTagName('Water_No_Volume_Weight')[0].firstChild.data) - scenario.transloading_dollars_per_ton = format_number(xmlScenarioFile.getElementsByTagName('transloading_dollars_per_ton')[0].firstChild.data) - scenario.transloading_dollars_per_thousand_gallons = format_number(xmlScenarioFile.getElementsByTagName('transloading_dollars_per_thousand_gallons')[0].firstChild.data) + # transloading costs + try: + logger.debug("test: setting the transloading costs with pint") + scenario.transloading_dollars_per_ton = Q_(format_number(xmlScenarioFile.getElementsByTagName('transloading_dollars_per_ton')[0].firstChild.data), + "usd/tonne/mile").to("usd/{}/mile".format(scenario.default_units_solid_phase)) + scenario.transloading_dollars_per_thousand_gallons = Q_(format_number(xmlScenarioFile.getElementsByTagName('transloading_dollars_per_thousand_gallons')[0].firstChild.data), + "usd/kgal/mile").to("usd/{}/mile".format(scenario.default_units_liquid_phase)) + logger.debug("PASS: setting the transloading costs with pint passed") + + except Exception as e: + logger.error("FAIL: {} ".format(e)) + raise Exception("FAIL: {}".format(e)) scenario.road_max_artificial_link_dist = xmlScenarioFile.getElementsByTagName('Road_Max_Artificial_Link_Distance_Miles')[0].firstChild.data scenario.rail_max_artificial_link_dist = xmlScenarioFile.getElementsByTagName('Rail_Max_Artificial_Link_Distance_Miles')[0].firstChild.data @@ -213,6 +254,16 @@ def load_scenario_config_file(fullPathToXmlConfigFile, fullPathToXmlSchemaFile, # RUN ROUTE OPTIMIZATION SCRIPT SECTION # ---------------------------------------------------------------------------------------- + # Setting flag for network density reduction based on 'NDR_On' field if it exists, or otherwise default to 'False' + if len(xmlScenarioFile.getElementsByTagName('NDR_On')): + if xmlScenarioFile.getElementsByTagName('NDR_On')[0].firstChild.data == "True": + scenario.ndrOn = True + else: + scenario.ndrOn = False + else: + logger.debug("NDR_On field not specified. Defaulting to False.") + scenario.ndrOn = False + scenario.permittedModes = [] if xmlScenarioFile.getElementsByTagName('Permitted_Modes')[0].getElementsByTagName('Road')[0].firstChild.data == "True": scenario.permittedModes.append("road") @@ -221,7 +272,7 @@ def load_scenario_config_file(fullPathToXmlConfigFile, fullPathToXmlSchemaFile, if xmlScenarioFile.getElementsByTagName('Permitted_Modes')[0].getElementsByTagName('Water')[0].firstChild.data == "True": scenario.permittedModes.append("water") - #TO DO ALO-- 10/17/2018-- make below compatible with distinct crude/product pipeline approach-- ftot_pulp.py changes will be needed. + # TODO ALO-- 10/17/2018-- make below compatible with distinct crude/product pipeline approach-- ftot_pulp.py changes will be needed. if xmlScenarioFile.getElementsByTagName('Permitted_Modes')[0].getElementsByTagName('Pipeline_Crude')[0].firstChild.data == "True": scenario.permittedModes.append("pipeline_crude_trf_rts") if xmlScenarioFile.getElementsByTagName('Permitted_Modes')[0].getElementsByTagName('Pipeline_Prod')[0].firstChild.data == "True": @@ -269,7 +320,7 @@ def load_scenario_config_file(fullPathToXmlConfigFile, fullPathToXmlSchemaFile, scenario.processor_candidates_commodity_data = os.path.join(scenario.scenario_run_directory, "debug", "ftot_generated_processor_candidates.csv") # this is the directory to store the shp files that a programtically generated for the networkx read_shp method - scenario.lyr_files_dir= os.path.join(scenario.scenario_run_directory, "temp_networkx_shp_files") + scenario.networkx_files_dir = os.path.join(scenario.scenario_run_directory, "temp_networkx_shp_files") return scenario @@ -355,6 +406,7 @@ def dump_scenario_info_to_report(the_scenario, logger): logger.config("xml_bargeCO2Emissions: \t{}".format(the_scenario.bargeCO2Emissions)) logger.config("xml_pipelineCO2Emissions: \t{}".format(the_scenario.pipelineCO2Emissions)) + logger.config("xml_ndrOn: \t{}".format(the_scenario.ndrOn)) logger.config("xml_permittedModes: \t{}".format(the_scenario.permittedModes)) logger.config("xml_capacityOn: \t{}".format(the_scenario.capacityOn)) logger.config("xml_backgroundFlowModes: \t{}".format(the_scenario.backgroundFlowModes)) @@ -459,8 +511,8 @@ def create_network_config_id_table(the_scenario, logger): the_scenario.water_max_artificial_link_dist, the_scenario.pipeline_crude_max_artificial_link_dist, the_scenario.pipeline_prod_max_artificial_link_dist, - the_scenario.liquid_railroad_class_1_cost, - the_scenario.solid_railroad_class_1_cost, + the_scenario.liquid_railroad_class_1_cost.magnitude, + the_scenario.solid_railroad_class_1_cost.magnitude, the_scenario.rail_dc_7, the_scenario.rail_dc_6, the_scenario.rail_dc_5, @@ -470,22 +522,22 @@ def create_network_config_id_table(the_scenario, logger): the_scenario.rail_dc_1, the_scenario.rail_dc_0, - the_scenario.liquid_truck_base_cost, - the_scenario.solid_truck_base_cost, + the_scenario.liquid_truck_base_cost.magnitude, + the_scenario.solid_truck_base_cost.magnitude, the_scenario.truck_interstate, the_scenario.truck_pr_art, the_scenario.truck_m_art, the_scenario.truck_local, - the_scenario.liquid_barge_cost, - the_scenario.solid_barge_cost, + the_scenario.liquid_barge_cost.magnitude, + the_scenario.solid_barge_cost.magnitude, the_scenario.water_high_vol, the_scenario.water_med_vol, the_scenario.water_low_vol, the_scenario.water_no_vol, - the_scenario.transloading_dollars_per_ton, - the_scenario.transloading_dollars_per_thousand_gallons) + the_scenario.transloading_dollars_per_ton.magnitude, + the_scenario.transloading_dollars_per_thousand_gallons.magnitude) db_con.execute(sql) @@ -546,8 +598,8 @@ def get_network_config_id(the_scenario, logger): the_scenario.pipeline_crude_max_artificial_link_dist, the_scenario.pipeline_prod_max_artificial_link_dist, - the_scenario.liquid_railroad_class_1_cost, - the_scenario.solid_railroad_class_1_cost, + the_scenario.liquid_railroad_class_1_cost.magnitude, + the_scenario.solid_railroad_class_1_cost.magnitude, the_scenario.rail_dc_7, the_scenario.rail_dc_6, the_scenario.rail_dc_5, @@ -557,22 +609,22 @@ def get_network_config_id(the_scenario, logger): the_scenario.rail_dc_1, the_scenario.rail_dc_0, - the_scenario.liquid_truck_base_cost, - the_scenario.solid_truck_base_cost, + the_scenario.liquid_truck_base_cost.magnitude, + the_scenario.solid_truck_base_cost.magnitude, the_scenario.truck_interstate, the_scenario.truck_pr_art, the_scenario.truck_m_art, the_scenario.truck_local, - the_scenario.liquid_barge_cost, - the_scenario.solid_barge_cost, + the_scenario.liquid_barge_cost.magnitude, + the_scenario.solid_barge_cost.magnitude, the_scenario.water_high_vol, the_scenario.water_med_vol, the_scenario.water_low_vol, the_scenario.water_no_vol, - the_scenario.transloading_dollars_per_ton, - the_scenario.transloading_dollars_per_thousand_gallons) + the_scenario.transloading_dollars_per_ton.magnitude, + the_scenario.transloading_dollars_per_thousand_gallons.magnitude) db_cur = db_con.execute(sql) network_config_id = db_cur.fetchone()[0] diff --git a/program/ftot_supporting.py b/program/ftot_supporting.py index 9892efb..4dfc476 100644 --- a/program/ftot_supporting.py +++ b/program/ftot_supporting.py @@ -4,7 +4,6 @@ # Purpose: # # --------------------------------------------------------------------------------------------------- -from __future__ import division # to make division work import os import math @@ -12,7 +11,7 @@ import datetime import sqlite3 from ftot import ureg, Q_ - +from six import iteritems # def create_loggers(dirLocation, task): @@ -241,13 +240,13 @@ def make_rmp_as_proc_slate(the_scenario, commodity_name, commodity_quantity_with io = row[5] if io == 'i': - if not facility_name in input_commodities.keys(): + if not facility_name in list(input_commodities.keys()): input_commodities[facility_name] = [] input_commodities[facility_name].append([a_commodity_name, quantity, units, phase_of_matter, io]) elif io == 'o': - if not facility_name in output_commodities.keys(): + if not facility_name in list(output_commodities.keys()): output_commodities[facility_name] = [] output_commodities[facility_name].append([a_commodity_name, quantity, units, phase_of_matter, io]) elif io == 'maxsize' or io == 'minsize': @@ -311,20 +310,20 @@ def get_max_fuel_conversion_process_for_commodity(commodity, the_scenario, logge ag_fuel_yield_dict, cropYield, bioWasteDict, fossilResources = load_afpat_tables(the_scenario, logger) - if ag_fuel_yield_dict.has_key(commodity): - processes_for_commodity.extend(ag_fuel_yield_dict[commodity].keys()) + if commodity in ag_fuel_yield_dict: + processes_for_commodity.extend(list(ag_fuel_yield_dict[commodity].keys())) # DO THE BIOWASTE RESOURCES - if bioWasteDict.has_key(commodity): - processes_for_commodity.extend(bioWasteDict[commodity].keys()) + if commodity in bioWasteDict: + processes_for_commodity.extend(list(bioWasteDict[commodity].keys())) # DO THE FOSSIL RESOURCES - fossil_keys = fossilResources.keys() + fossil_keys = list(fossilResources.keys()) for key in fossil_keys: if commodity.find(key) > -1: - processes_for_commodity.extend(fossilResources[key].keys()) + processes_for_commodity.extend(list(fossilResources[key].keys())) if processes_for_commodity == []: logger.warning("processes_for_commodity is empty: {}".format(processes_for_commodity)) @@ -345,7 +344,7 @@ def get_max_fuel_conversion_process_for_commodity(commodity, the_scenario, logge logger.warning("conversion_efficiency_dict is empty: {}".format(conversion_efficiency_dict)) else: - max_conversion_process = sorted(conversion_efficiency_dict.iteritems(), key=lambda (k, v): (v, k))[0] + max_conversion_process = sorted(iteritems(conversion_efficiency_dict), key=lambda k_v: (k_v[1], k_v[0]))[0] return max_conversion_process # just return the name of the max conversion process, not the conversion efficiency @@ -475,9 +474,9 @@ def get_input_and_output_commodity_quantities_from_afpat(commodity, process, the commodity = "hack-hack-hack" process = "hack-hack-hack" - elif ag_fuel_yield_dict.has_key(commodity): + elif commodity in ag_fuel_yield_dict: # print "in the wrong right place" - if ag_fuel_yield_dict[commodity].has_key(process): + if process in ag_fuel_yield_dict[commodity]: input_commodity_quantities = Q_(ag_fuel_yield_dict[commodity][process][8], "kg/day") @@ -496,9 +495,9 @@ def get_input_and_output_commodity_quantities_from_afpat(commodity, process, the "the commodity {} has no process {} in the AFPAT agricultural yield dictionary".format(commodity, process)) - elif bioWasteDict.has_key(commodity): + elif commodity in bioWasteDict: - if bioWasteDict[commodity].has_key(process): + if process in bioWasteDict[commodity]: input_commodity_quantities = Q_(bioWasteDict[commodity][process][1], "kg / year") @@ -521,13 +520,13 @@ def get_input_and_output_commodity_quantities_from_afpat(commodity, process, the process)) # DO THE FOSSIL RESOURCES - fossil_keys = fossilResources.keys() + fossil_keys = list(fossilResources.keys()) for key in fossil_keys: if commodity.find(key) > -1: - if fossilResources[key].has_key(process): + if process in fossilResources[key]: input_commodity_quantities = Q_(500e3, "oil_bbl / day") diff --git a/scenarios/ForestResiduals_SCR.zip b/scenarios/ForestResiduals_SCR.zip new file mode 100644 index 0000000..7ae8ef5 Binary files /dev/null and b/scenarios/ForestResiduals_SCR.zip differ diff --git a/scenarios/blank b/scenarios/blank new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/scenarios/blank @@ -0,0 +1 @@ +