The Webservices project lets you take pretty much any structured data set and turn it into a search API without writing a line of code.
- understands CSV, TSV, JSON, XLS, or XML file formats
- smart guessing of data types and schemas based on heuristics
- versioned APIs by default
- ingest by URL
- easy polling/refresh of URL-based data sources
- simple YAML-based configuration for each data source
- customize the ETL process via built-in transformations
You'll need Ruby 2.3.
We use bundler to manage gems. You can install bundler and other required gems like this:
gem install bundler
bundle install
The charlock_holmes gem requires the UCI libraries to be installed. If you are using Homebrew, it's probably as simple as this:
brew install icu4c
More information about the gem can be found here
We're using ElasticSearch (== 6.6.0) for fulltext search. On a Mac, it's easy to install with Homebrew.
brew install elasticsearch
Otherwise, follow the instructions to download and run it.
You'll need to have redis installed on your machine. brew install redis, apt-get install redis-server, etc.
Create the indexes:
bundle exec rake db:create
Generate the default admin user with username [email protected] and password 1nitial_pwd:
bundle exec rake db:devseed
Fire up a server:
bundle exec rails s thin
Import some data:
bundle exec rake ita:import[ScreeningList]
Admin users can log in and monitor the progress of the Sidekiq import jobs via /sidekiq.
Since v2 of the API, an authentication token is required for every request. Pass it on the query string:
http://localhost:3000/consolidated_screening_list/search?api_key=devkey&size=5&offset=8
http://localhost:3000/consolidated_screening_list/search?api_key=devkey&q=john
http://localhost:3000/consolidated_screening_list/search?api_key=devkey&sources=SDN,EL
Or using http headers:
curl -H'Api-Key: devkey' 'http://localhost:3000/v2/consolidated_screening_list/search'
Admin users can create and administer search APIs from URLs. The file formats supported
include CSV, TSV, JSON, XLS, and XML. The initial admin user created with the db:devseed task has the admin flag
set to true already. To toggle an existing user, you can do this from the Rails console:
email = "[email protected]"
u = User.search(query: { constant_score: { filter: { term: { email: email } } } }).first
u.update_attribute(:admin, true)
To create an API, click the + next to the Dynamic APIs subnav heading.
To refresh a URL-based api, you can periodically call the rake task to check for updates. Pass the api field from the DataSource as a parameter:
bundle exec rake endpointme:import[business_service_providers]
bundle exec rspec
Elasticsearch must be running.
We track test coverage of the codebase over time to help identify areas where we could write better tests and to see when poorly tested code got introduced.
After running your tests, view the report by opening coverage/index.html.
Click around on the files that have less than 100% coverage to see what lines weren't exercised.
We use Mailcatcher to test emails sent from dev environments. Their advice is not to add the gem to your Gemfile, so in order to use it please do:
gem install mailcatcher
mailcatcher
If you use RVM, you should follow their specific instructions (search for "RVM") on how to install the gem.
If you want to run this app on AWS Opsworks, you may find this guide helpful.
