DashYUL

About

DashYUL is a data visualization package built for York University Libraries, combining a data dashboard and other interactive assessment and evaluation tools for library resources and services.

DashYUL is built and used by a devoted user of Emacs and Org. Everything will work even if you don’t use them, but it will all work better if you do.

Installation

Details to come. Work in progress.

Requirements

General

sudo apt install yaz

R

Some scripts are written in R (I am using 3.4.3 right now), and Shiny (built in R) is how the dashboard works.

Packages required.

• devtools
• flexdashboard
• knitr
• RCurl
• roxygen2
• shiny
• the entire tidyverse
• yulr, which offers small useful YUL-related functions in R

(Some of those aren’t actually needed; I will refine the list after testing.)

I use my Conforguration tool to get packages installed, but this will do it too:

sudo -i
PACKAGE_LIST="tidyverse devtools RCurl roxygen2 shiny flexdashboard knitr"
for PKG in $PACKAGE_LIST; do Rscript --vanilla -e "install.packages('$PKG', repos=c('https://cran.hafro.is/'))"; done
Rscript --vanilla -e "devtools::install_github('yorkulibraries/yulr')"

R and the packages need to be available system-wide to all users, including Shiny.

Ruby

A lot of text munging and data cleaning is done in Ruby. (I am using version 3.0 right now.)

Some gems are required:

• marc, for reading MARC records
• nokogiri, for reading XML and HTML
• ruby-oci8, for talking to Oracle (requires Oracle libraries, follow Oracle Instant Client instructions for how to get it working)
• rubyul is the Ruby parallel to yulr, and offers small useful YUL-related helper functions in Ruby

You can install them with Bundler, which installs gems listed in Gemfile:

sudo gem install bundler
bundle install

By default it will install ruby-oci8, but if you don’t have the Oracle libraries and don’t want to install all that, run:

bundle config set without 'oracle'

Ruby and the packages need to be available system-wide, not under RVM or rbenv.

Makefiles and shell scripts

There are a number of makefiles and bash scripts. If your system supports R and Ruby, it will have these.

Configuration

Environment variables are used to control where things are.

DASHYUL_HOME

where this source tree is, locally

DASHYUL_DATA

where the data is, locally

DASHYUL_LOGS

where logfiles are, locally

DASHYUL_SERVER_NAME

hostname of the production server

DASHYUL_SERVER_HOME

where the source tree is, in production

DASHYUL_SERVER_DATA

where the data is, in production

DASHYUL_SERVER_LOGS

where the logfiles are, in production

More environment variables are used for keys and cookies to get data:

ALMA_API_KEY

LIBSTATS_LOGIN_COOKIE

I have this in my .bashrc:

source ~/.bashrc.dashyul.rc

Then my local .bash.dashyul.rc, also in my home directory, is:

# LibStats: long-lived cookie so I can download data with curl
export LIBSTATS_LOGIN_COOKIE=12345678901234567890d6bff4ea7525

# Alma
export ALMA_API_KEY=abcdefghij12345678901234567890abcdefg

# DashYUL
export DASHYUL_HOME=~/src/dashyul
export DASHYUL_DATA=/usr/local/dashyul/data
export DASHYUL_LOGS=/usr/local/dashyul/logs
export DASHYUL_SHINY_DASHYUL=/usr/local/dashyul/shiny/dashyul
export DASHYUL_SHINY_DASHBOARD=/usr/local/dashyul/shiny/dashboard

export DASHYUL_SERVER_NAME=servername.library.yorku.ca
export DASHYUL_SERVER_HOME=/dashyul/src/dashyul
export DASHYUL_SERVER_DATA=/dashyul/data
export DASHYUL_SERVER_SHINY_DASHYUL=/dashyul/shiny/dashyul
export DASHYUL_SERVER_SHINY_DASHBOARD=/dashyul/shiny/dashboard

On the server’s .bash.dashyul.rc, what was SERVER becomes local:

# DashYUL
export DASHYUL_HOME=/dashyul/src/dashyul
export DASHYUL_DATA=/dashyul/data
export DASHYUL_LOGS=/dashyul/logs
export DASHYUL_SHINY_DASHYUL=/dashyul/shiny/dashyul
export DASHYUL_SHINY_DASHBOARD=/dashyul/shiny/dashboard

This source tree can be used locally or on the production server. Data files are kept separate.

To set things up on your local machine, get this source tree and create your local data directory:

cd ~/src
git pull https://github.com/yorkulibraries/dashyul.git
cd dashyul
make setup_directories

To set things up in production, do something like:

sudo mkdir -p /dashyul/src/
sudo chown -R ${USER}:${USER} /dashyul/
cd /dashyul/src
git clone https://github.com/yorkulibraries/dashyul.git
cd dashyul
make setup_directories

Crontabs

• Note PATHs and that /usr/local/bin needs to come first, since that’s where I installed Ruby and R (using system packages should just work)
• TODO: Update the cron job examples

Shiny

• Symphony scripts prepare-circ-metrics.R and prepare-item-circ-history.R are almost identical. Make it simpler, and document it, and then set up cron jobs.
• Use DASHYUL_SERVER_LOGS in makefiles when copying log files down from production.
• Set up the environment variables in the server, or, failing that, set them as variables in a script that every viz and script can load, so they are all defined in a single place, not hard-coded.
• Try shinyFeedback?
• Document what happens in September (almost everything) and October (rolling over Symphony logs). Ideally, automate it all.
• Change the links in the Gardener so they point to the relevant record in CircYUL, which has the circ history for that title, not the catalogue, which is much less informative.
• Use fs and purrr to read in a list of CSV files at once (as described here). Neater than my current method.

Guidelines

Notes to myself as I work on this.

• Makefiles
  • make should work anywhere and do whatever is necessary to prepare data for use
  • make push_app is for local use, and pushes a Shiny app up to production
• sources and viz here; data separate
• sources and viz both have scripts/
• Rely on DASHYUL_HOME and DASHYUL_DATA.
• Use _f and _d suffixes when storing file and directory names in variables.
• Have a README.org for every source and viz to explain what they are. Need to explain all the data somewhere, too.

See also:

• https://github.com/dmarx/make_for_datascience
• https://jdblischak.github.io/workflowr/articles/wflow-01-getting-started.html

License

GPL v3. See LICENSE.