Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Script to automate appending tasks in JSON #75

Open
niccolopaganini opened this issue Oct 5, 2023 · 15 comments
Open

Script to automate appending tasks in JSON #75

niccolopaganini opened this issue Oct 5, 2023 · 15 comments

Comments

@niccolopaganini
Copy link

Subset issue #70 created by @achasmita

Task that needs to be done from my end:
@niccolopaganini niccolopaganini moved this to Issues being worked on in OpenPATH Tasks Overview Oct 5, 2023
@niccolopaganini
Copy link
Author

Sampling with this to see how it is being appended

data_to_append = {
    "key1": "val1",
    "key2": "val2"
}

@niccolopaganini
Copy link
Author

update:
I checked ca-ebike.nrel-op.json, stage-study.nrel-op.json, and denver-casr.nrel-op.json randomly to see if the append process works and they seem to append successfully.

@niccolopaganini
Copy link
Author

niccolopaganini commented Oct 5, 2023 •
edited
Loading

let me try nested now:

"key_ob": {
        "key1": val1,
        "key2": val2
    }

@niccolopaganini
Copy link
Author

Sampling with this to see how it is being appended

data_to_append = {
    "key1": "val1",
    "key2": "val2"
}

This is how the output looks like:

{
    "version": 1,
    "ts": 1655143472,
    "server": {
        "connectUrl": "https://ca-ebike-openpath.nrel.gov/api/",
        "aggregate_call_auth": "user_only"
    },
    "opcode": {
        "autogen": false,
        "subgroups": [
            "test",
            "default"
        ]
    },
    "intro": {
        "program_or_study": "program",
        "start_month": "05",
        "start_year": "2023",
        "mode_studied": "e-bike",
        "program_admin_contact": "[email protected], 530-601-7624",
        "deployment_partner_name": "University of California Davis",
        "translated_text": {
            "en": {
                "deployment_partner_name": "University of California, Davis",
                "deployment_name": "California E-bike Incentive Program",
                "summary_line_1": "The California Air Resources Board (CARB) aims to incentivize people to adopt energy-efficient, non-polluting modes of transportation and maximize air quality benefits.",
                "summary_line_2": "In this program, it is providing e-bikes to low-income individuals in communities around the state.",
                "summary_line_3": "If you are a program Participant, this app can help you log your trips and compete against others to benefit our environment.",
                "short_textual_description": " As e-bike incentive programs are spreading, NREL would like to experiment with an automated trip tracking solution for understanding travel behavior. This program evaluation can allow NREL to understand variations in travel across centers, days of the week and months of the year without complex, time-consuming surveys.",
                "why_we_collect": "The data collected during the program will be used to understand how e-bikes are used including when and how often as well as distance travelled. This in turn will aid in the estimation of GHG reductio from such incentive programs and new technologies.",
                "research_questions": [
                    "Travel behavior modeling to support cost-benefit analyses of various potential incentive programs.",
                    "The impact of e-bikes on transportation energy efficiency and GHG emissions."
                ]
            },
            "es": {
                "deployment_partner_name": "Universidad de California Davis",
                "deployment_name": "Programa de Incentivos para Bicicletas El\u00e9ctricas de California",
                "summary_line_1": "El Consejo de Recursos del Aire de California (CARB) apunta a incentivar a personas adoptar modas de transportacion energ\u00eda eficientes y no contaminantes y a maximizar los beneficios del calidad del aire.",
                "summary_line_2": "En este programa, proporciona bicicletas el\u00e9ctricas a personas de bajos ingresos en comunidades de todo el estado.",
                "summary_line_3": "Si es un participante del programa, esta aplicaci\u00f3n puede ayudarlo a registrar sus viajes y competir contra otros para beneficiar nuestro medio ambiente.",
                "short_textual_description": "Como programas de incentivos para bicicletas el\u00e9ctiracas est\u00e1n extendiendo, NREL quiere experimentar con una soluci\u00f3n automatizada de seguimiento de viajes para entender comportamiento de viaje. El evaluaci\u00f3n del programa se puede permitir a NREL entender variaciones de viaje tras centros, d\u00edas del semana y meses del a\u00f1o sin encuestas complejos que cuestan mucho tiempo.",
                "why_we_collect": "Los datos recopilados durante el programa se utilizar\u00e1n para comprender c\u00f3mo se utilizan las bicicletas el\u00e9ctricas, incluido cu\u00e1ndo y con qu\u00e9 frecuencia, as\u00ed como la distancia recorrida. Esto, a su vez, ayudar\u00e1 en la estimaci\u00f3n de la reducci\u00f3n de GEI de dichos programas de incentivos y nuevas tecnolog\u00edas. ",
                "research_questions": [
                    "Modelado del comportamiento de viaje para apoyar el an\u00e1lisis de costo-beneficio de varios programas de incentivos potenciales.",
                    " El impacto de las bicicletas el\u00e9ctricas en la eficiencia energ\u00e9tica del transporte y las emisiones de GEI."
                ]
            }
        }
    },
    "survey_info": {
        "surveys": {
            "UserProfileSurvey": {
                "formPath": "json/demo-survey-v2.json",
                "version": 1,
                "compatibleWith": 1,
                "dataKey": "manual/demographic_survey",
                "labelTemplate": {
                    "en": "Answered",
                    "es": "Contestada"
                }
            }
        },
        "trip-labels": "MULTILABEL"
    },
    "display_config": {
        "use_imperial": true
    },
    "metrics": {
        "include_test_users": true
    },
    "profile_controls": {
        "support_upload": false,
        "trip_end_notification": false
    },
    "admin_dashboard": {
        "overview_users": true,
        "overview_active_users": true,
        "overview_trips": true,
        "overview_signup_trends": true,
        "overview_trips_trend": true,
        "data_uuids": true,
        "data_trips": true,
        "data_trips_columns_exclude": [
            "data.start_loc.coordinates",
            "data.end_loc.coordinates"
        ],
        "additional_trip_columns": [],
        "data_uuids_columns_exclude": [],
        "token_generate": false,
        "token_prefix": "nrelop",
        "map_heatmap": true,
        "map_bubble": true,
        "map_trip_lines": false,
        "push_send": false,
        "options_uuids": true,
        "options_emails": true
    },
    "key1": "val1",
    "key2": "val2"
}

@niccolopaganini
Copy link
Author

Seems to work fine. Output looks something like this (for ca-ebike.nrel-op.json):

{
    "version": 1,
    "ts": 1655143472,
    "server": {
        "connectUrl": "https://ca-ebike-openpath.nrel.gov/api/",
        "aggregate_call_auth": "user_only"
    },
    "opcode": {
        "autogen": false,
        "subgroups": [
            "test",
            "default"
        ]
    },
    "intro": {
        "program_or_study": "program",
        "start_month": "05",
        "start_year": "2023",
        "mode_studied": "e-bike",
        "program_admin_contact": "[email protected], 530-601-7624",
        "deployment_partner_name": "University of California Davis",
        "translated_text": {
            "en": {
                "deployment_partner_name": "University of California, Davis",
                "deployment_name": "California E-bike Incentive Program",
                "summary_line_1": "The California Air Resources Board (CARB) aims to incentivize people to adopt energy-efficient, non-polluting modes of transportation and maximize air quality benefits.",
                "summary_line_2": "In this program, it is providing e-bikes to low-income individuals in communities around the state.",
                "summary_line_3": "If you are a program Participant, this app can help you log your trips and compete against others to benefit our environment.",
                "short_textual_description": " As e-bike incentive programs are spreading, NREL would like to experiment with an automated trip tracking solution for understanding travel behavior. This program evaluation can allow NREL to understand variations in travel across centers, days of the week and months of the year without complex, time-consuming surveys.",
                "why_we_collect": "The data collected during the program will be used to understand how e-bikes are used including when and how often as well as distance travelled. This in turn will aid in the estimation of GHG reductio from such incentive programs and new technologies.",
                "research_questions": [
                    "Travel behavior modeling to support cost-benefit analyses of various potential incentive programs.",
                    "The impact of e-bikes on transportation energy efficiency and GHG emissions."
                ]
            },
            "es": {
                "deployment_partner_name": "Universidad de California Davis",
                "deployment_name": "Programa de Incentivos para Bicicletas El\u00e9ctricas de California",
                "summary_line_1": "El Consejo de Recursos del Aire de California (CARB) apunta a incentivar a personas adoptar modas de transportacion energ\u00eda eficientes y no contaminantes y a maximizar los beneficios del calidad del aire.",
                "summary_line_2": "En este programa, proporciona bicicletas el\u00e9ctricas a personas de bajos ingresos en comunidades de todo el estado.",
                "summary_line_3": "Si es un participante del programa, esta aplicaci\u00f3n puede ayudarlo a registrar sus viajes y competir contra otros para beneficiar nuestro medio ambiente.",
                "short_textual_description": "Como programas de incentivos para bicicletas el\u00e9ctiracas est\u00e1n extendiendo, NREL quiere experimentar con una soluci\u00f3n automatizada de seguimiento de viajes para entender comportamiento de viaje. El evaluaci\u00f3n del programa se puede permitir a NREL entender variaciones de viaje tras centros, d\u00edas del semana y meses del a\u00f1o sin encuestas complejos que cuestan mucho tiempo.",
                "why_we_collect": "Los datos recopilados durante el programa se utilizar\u00e1n para comprender c\u00f3mo se utilizan las bicicletas el\u00e9ctricas, incluido cu\u00e1ndo y con qu\u00e9 frecuencia, as\u00ed como la distancia recorrida. Esto, a su vez, ayudar\u00e1 en la estimaci\u00f3n de la reducci\u00f3n de GEI de dichos programas de incentivos y nuevas tecnolog\u00edas. ",
                "research_questions": [
                    "Modelado del comportamiento de viaje para apoyar el an\u00e1lisis de costo-beneficio de varios programas de incentivos potenciales.",
                    " El impacto de las bicicletas el\u00e9ctricas en la eficiencia energ\u00e9tica del transporte y las emisiones de GEI."
                ]
            }
        }
    },
    "survey_info": {
        "surveys": {
            "UserProfileSurvey": {
                "formPath": "json/demo-survey-v2.json",
                "version": 1,
                "compatibleWith": 1,
                "dataKey": "manual/demographic_survey",
                "labelTemplate": {
                    "en": "Answered",
                    "es": "Contestada"
                }
            }
        },
        "trip-labels": "MULTILABEL"
    },
    "display_config": {
        "use_imperial": true
    },
    "metrics": {
        "include_test_users": true
    },
    "profile_controls": {
        "support_upload": false,
        "trip_end_notification": false
    },
    "admin_dashboard": {
        "overview_users": true,
        "overview_active_users": true,
        "overview_trips": true,
        "overview_signup_trends": true,
        "overview_trips_trend": true,
        "data_uuids": true,
        "data_trips": true,
        "data_trips_columns_exclude": [
            "data.start_loc.coordinates",
            "data.end_loc.coordinates"
        ],
        "additional_trip_columns": [],
        "data_uuids_columns_exclude": [],
        "token_generate": false,
        "token_prefix": "nrelop",
        "map_heatmap": true,
        "map_bubble": true,
        "map_trip_lines": false,
        "push_send": false,
        "options_uuids": true,
        "options_emails": true
    },
    "key1": "val1",
    "key2": "val2",
    "key_ob": {
        "key1": "val1",
        "key2": "val2"
    }
}

@niccolopaganini
Copy link
Author

I will submit an initial PR and then modify the "hard-coded" part to substitute the appropriate value

@niccolopaganini niccolopaganini mentioned this issue Oct 5, 2023
Closed
@shankari
Copy link
Contributor

shankari commented Oct 5, 2023 •
edited
Loading

@niccolopaganini as I indicated in
#70 (comment)

the new key to add is data_trajectories_columns_exclude: ['user_id'], similar to the current data_trips_columns_exclude: key

Can you please update the testing done with it?

@niccolopaganini
Copy link
Author

On it

@niccolopaganini
Copy link
Author

Okay, my computer is lagging super bad and I am unable to work continuously. I already restarted it a couple of times but the issue is still persisting.

Working on the above issue

@niccolopaganini
Copy link
Author

Addressing the top-level comment on PR (e-mission/nrel-openpath-deploy-configs#40 (comment)) -> Now, they're under admin_dashboard for all the JSON files present inside config

@niccolopaganini
Copy link
Author

Pushing all the changes now...

@niccolopaganini
Copy link
Author

Listing program names for convenience:

  1. ca-ebike.nrel-op.json
  2. stage-program.nrel-op.json
  3. stage-study.nrel-op.json
  4. uprm-civic.nrel-op.json
  5. uprm-nicr.nrel-op.json
  6. usaid-laos-ev.nrel-op.json
  7. uue.nrel-op.json
  8. washingtoncommons.nrel-op.json

@niccolopaganini
Copy link
Author

Before I add what I tried, I also noted the weird indentation/ syntax issue occurring to the kv pair that we are trying to append.
With this information, I noticed that this issue was only occurring in keys that have any object array pair. With that information, here's what I did:

  1. Augment the parent program to just include this: separators=(',', ':') ❌
  2. I thought appending might be the issue here; so I copied the whole chunk (chunk below for reference) and tried to replace the existing chunk (idk for some reason/ 4:30 AM me thought this might work) ❌
"admin_dashboard": {
        "overview_users": true,
        "overview_active_users": true,
        "overview_trips": true,
        "overview_signup_trends": true,
        "overview_trips_trend": true,
        "data_uuids": true,
        "data_trips": true,
        "data_trips_columns_exclude": [
            "data.start_loc.coordinates",
            "data.end_loc.coordinates"
        ],
        "additional_trip_columns": [],
        "data_uuids_columns_exclude": [],
        "token_generate": false,
        "token_prefix": "nrelop",
        "map_heatmap": true,
        "map_bubble": true,
        "map_trip_lines": false,
        "push_send": false,
        "options_uuids": true,
        "options_emails": true,
        "data_trajectories_columns_exclude": ["user_id"]
  1. I thought of using the information I found above and incorporate it in a script addressing the weird spacing issue ❌
  2. With the latest push, I was only able to fix the last part because I was able to isolate it (because that's the part that is being appended to the list) 🤕

@niccolopaganini
Copy link
Author

I found this: microsoft/TypeScript#15034 (comment) @shankari do you think I might be facing a similar issue for square brackets?

@shankari
Copy link
Contributor

shankari commented Oct 6, 2023

@niccolopaganini

I found this: microsoft/TypeScript#15034 (comment) @shankari do you think I might be facing a similar issue for square brackets?

No, because that is typescript and this is JSON.

With this information, I noticed that this issue was only occurring in keys that have any object array pair. With that information, here's what I did:

That is a good point. As I said in the PR review:

If you can't figure out the JSON dump arguments to enable this, feel free to use jq to edit the files directly instead. If there is a reason why this doesn't work, please document it

So, I would like to see:

  • what you tried wrt JSON dump arguments to enable this, including a check of the JSON dump documentation
  • trying to use jq to edit the files (including the output)

If it is not possible to fix this, the original JSON format may not be compatible with JSON dump in the first place - much of that JSON was hand-created, so that would not be super suprising.

If you can make the argument for that, you can submit two PRs:

  • one to make all the configs compatible with JSON dump without changing anything in the config
  • a second one to add the changes

@shankari shankari moved this to Pending for Nitish in OpenPATH Tasks Overview Oct 21, 2023
@shankari shankari mentioned this issue Nov 12, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
OpenPATH Tasks Overview
Status: Pending for Nitish
Milestone
No milestone
Development

No branches or pull requests
2 participants
@shankari @niccolopaganini