This is small package aimed to simplify process of data downloading and postprocessing, without intermediate files storing. Additionally UrlDownload.jl provides progress bar for big files with long download time.
UrlDownload.jl has integrated support for the processing of various data formats, such as CSV, JSON, various image formats. Everything that is not supported by default, can be easily extended with custom parsers. Also, package automatically decompress and process various compressed data with the help of TranscodingStreams.jl. And it can process files located on http resources and local.
To install UrlDownload either do
using Pkg
Pkg.add("UrlDownload")or switch to Pkg mode with ] and issue
pkg> add UrlDownloadNote: this package uses many different packages for data processing, which should be installed separately. So, if you receive message like ERROR: ArgumentError: Package CSV not found in current path, install CSV.jl manually and error will go away. No additional work is needed, since UrlDownload.jl import necessary packages on it's own.
Usage is quite simple
using UrlDownload
using DataFrames
url = "https://raw.githubusercontent.com/Arkoniak/UrlDownload.jl/master/data/ext.csv"
df = urldownload(url) |> DataFrame
# 2×2 DataFrame
# │ Row │ x │ y │
# │ │ Int64 │ Int64 │
# ├─────┼───────┼───────┤
# │ 1 │ 1 │ 2 │
# │ 2 │ 3 │ 4 │More details and examples can be found in the documentation.