-
Notifications
You must be signed in to change notification settings - Fork 4
Collectors
These are available at https://github.com/DECODEproject/bcnnow/tree/master/apps/backend/data
It is the system in charge of city equipment. This system fulfills two functions. One is to keep an up to date record of all city buildings and their usage, while the second is to keep a day to day agenda of all public activities held on public buildings. FILE. XML.
It is the Bicing stations (bike transport) system of the city of Barcelona. This data source includes elements relating to address (street and number), nearest stations, station status, number of parking spaces and also the number of bikes, mechanics and electrics, available. API. JSON.
Internal data lake of the city council, currently under final stages of development, scheduled to start working by January 2018. Currently, initial use cases are being considered for the platform, but eventually the data from ODI, Sentilo, IRIS, ASIA and many other city council sources will be connected to the infrastructure. ACCESS NOT AVAILABLE (via OAuth 2.0). Web Feature Service (WFS).
Proposals and meetings from PAM process at Decidim Barcelona through the GraphQL API. API. JSON.
This data source includes information on different typologies of equipments and related events, namely: sports, restaurants, health, cemeteries, comerce, culture, religion, associations, public administration, information centers, animals and plants, social services and communication. API. RDF.
It is an independent, non-commercial set of tools and data that allows users to explore how Airbnb is being used in cities. It is a mission-driven activist project which provides data and tools for an informed debate on the impact of Airbnb on residential communities, also for Barcelona. FILE. CSV.
It is the “citizen relation system” of the city. An infrastructure that handles all the citizen interactions with the city via phone, person, email or other online means. It stores citizens demands and requirements, which are classified according to a predefined ontology and sent to its allocated supervisors. The city council has then roughly 30 days to attend to those petitions, some of which are tagged with geolocated information. _FILE. JSON _
It is the Barcelona Housing Observatory (O-HB) (Observatori Metropolità de l'Habitatge de Barcelona (O-HB)). Provides several indicators grouped using the following themes: Housing stock; Construction and renovation; Socio-demographic and socio-economic factors; Housing market; Housing continuity access and maintenance issues; Housing policies; and Territory.
In BarcelonaNOW, 3 indicators are available and can be related to other external and public sources (e.g. Inside Airbnb).
- OHB1 - Housing for tourist use (apartments, housing used for tourism (HUTs) and rural tourism) for every 1,000 inhabitants.
- OHB2 - Average contract rental income per m2 (€ m2 / month).
- OHB3 - Offer of rooms in hotels and pensions for 1,000 inhabitants.
This source provides information related to the most singular tourist points of interest in the city of Barcelona. API. XML.
IoT sensor platform of the city. Currently a closed platform, accessible only to partners, which also has a public area. The platform allows to connect and consume sensors in a federated way. It contains mainly sensors from city council services and pilot projects that have been connected as proof. API. JSON.
It is a platform to generate participatory processes of people in the cities. Connecting data, people and knowledge, the objective of the platform is to build productive open indicators and distributed tools._ API. JSON. _
It is the “Tourist housing” of the city. This data source includes touristic historical resources ordered by year and term until the last week of the term are published._ API. CSV. _
This guide is based on the experience of DecidimCollector.
The collectors folder is organized in a way that the collectors which use pull technology and the collectors which use push technology are subdivided in two different folders.
Each collector has its folder with three files: a configuration file, a payload definition file, and an execution file. The first one includes a Python class that defines a JSON record with all the configuration parameters needed by the collector. The second one contains a Python class with all the record attributes which are specific of that kind of data source, getter and setter methods, and a method which returns a JSON record containing all the attributes. The third one contains a Python class with four methods: start() is the main method whose purpose is to loop for all the files to be accessed or all the API calls to be executed. For each file or API access, it calls the sendRequest() which takes a URL as parameter and returns the data retrieved from that URL. According to the access method, sendRequest() is able to get data with different formats (e.g. API, XML, CSV, JSON) by using the proper Python package. Once the data is retrieved, start() calls saveData(). This method has the purpose of accessing each record represented in the original format, calling buildRecord() to create an instance of the record coded in the internal unified format for that data source and using store() from StorageHelper to save the record into MongoDB. Sending records to Kafka is also possible just by updating the corresponding JSON attribute of the main configuration file of the project (i.e., bcnnow-master/config/Config.py).
The execution file of each collector should be inserted into the cron job whenever a collector requires to be periodically executed to get data from the original source (this is not the case for DecidimCollector).
In particular, the following files were added to the DecidimCollector folder:
- Config.py
- DecidimProposalCollector.py
- DecidimProposalPayload.py
- DecidimMeetingCollector.py
- DecidimMeetingPayload.py
Then, to allow citizens to explore the data, the dataset must be added to the list of datasets.
If it were of interest, the panel to be shown when a map marker is clicked can be customized at the getMakerPanel() method of the main JS.
To create a permanent dashboard in BarcelonaNow, the corresponding page should be added at the main HTML and the dashboard JS.
The code of the full process can be examined at this pull request