From 1bb54d1c3d46ca7aab8530be568b605fff022709 Mon Sep 17 00:00:00 2001
From: Michael Abbott <32575566+mcabbott@users.noreply.github.com>
Date: Thu, 24 Nov 2022 18:49:33 -0500
Subject: [PATCH 01/11] move 5 tutorials from fluxml.github.io
---
docs/make.jl | 7 +
.../2020-09-15-deep-learning-flux.md | 436 ++++++++++++++++++
docs/src/tutorials/2021-01-26-mlp.md | 169 +++++++
docs/src/tutorials/2021-02-07-convnet.md | 260 +++++++++++
docs/src/tutorials/2021-10-08-dcgan-mnist.md | 363 +++++++++++++++
docs/src/tutorials/2021-10-14-vanilla-gan.md | 280 +++++++++++
docs/src/tutorials/linear_regression.md | 6 +-
7 files changed, 1520 insertions(+), 1 deletion(-)
create mode 100755 docs/src/tutorials/2020-09-15-deep-learning-flux.md
create mode 100644 docs/src/tutorials/2021-01-26-mlp.md
create mode 100644 docs/src/tutorials/2021-02-07-convnet.md
create mode 100644 docs/src/tutorials/2021-10-08-dcgan-mnist.md
create mode 100644 docs/src/tutorials/2021-10-14-vanilla-gan.md
diff --git a/docs/make.jl b/docs/make.jl
index 5409a117c7..0847656b9d 100644
--- a/docs/make.jl
+++ b/docs/make.jl
@@ -42,7 +42,14 @@ makedocs(
"Functors.jl 📚 (`fmap`, ...)" => "models/functors.md",
],
"Tutorials" => [
+ # Roughly in order of increasing complexity? Not chronological.
"Linear Regression" => "tutorials/linear_regression.md",
+ "Julia & Flux: 60 Minute Blitz" => "tutorials/2020-09-15-deep-learning-flux.md",
+ "Multi-layer Perceptron" => "tutorials/2021-01-26-mlp.md",
+ "Simple ConvNet" => "tutorials/2021-02-07-convnet.md",
+ "Generative Adversarial Net" => "tutorials/2021-10-14-vanilla-gan.md",
+ "Deep Convolutional GAN" => "tutorials/2021-10-08-dcgan-mnist.md",
+ # Not really sure where this belongs... some in Fluxperimental, aim to delete?
"Custom Layers" => "models/advanced.md", # TODO move freezing to Training
],
"Performance Tips" => "performance.md",
diff --git a/docs/src/tutorials/2020-09-15-deep-learning-flux.md b/docs/src/tutorials/2020-09-15-deep-learning-flux.md
new file mode 100755
index 0000000000..efd89727c5
--- /dev/null
+++ b/docs/src/tutorials/2020-09-15-deep-learning-flux.md
@@ -0,0 +1,436 @@
+# Deep Learning with Julia & Flux: A 60 Minute Blitz
+
+This is a quick intro to [Flux](https://github.com/FluxML/Flux.jl) loosely based on [PyTorch's tutorial](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html). It introduces basic Julia programming, as well Zygote, a source-to-source automatic differentiation (AD) framework in Julia. We'll use these tools to build a very simple neural network.
+
+## Arrays
+
+The starting point for all of our models is the `Array` (sometimes referred to as a `Tensor` in other frameworks). This is really just a list of numbers, which might be arranged into a shape like a square. Let's write down an array with three elements.
+
+```julia
+x = [1, 2, 3]
+```
+
+Here's a matrix – a square array with four elements.
+
+```julia
+x = [1 2; 3 4]
+```
+
+We often work with arrays of thousands of elements, and don't usually write them down by hand. Here's how we can create an array of 5×3 = 15 elements, each a random number from zero to one.
+
+```julia
+x = rand(5, 3)
+```
+
+There's a few functions like this; try replacing `rand` with `ones`, `zeros`, or `randn` to see what they do.
+
+By default, Julia works stores numbers is a high-precision format called `Float64`. In ML we often don't need all those digits, and can ask Julia to work with `Float32` instead. We can even ask for more digits using `BigFloat`.
+
+```julia
+x = rand(BigFloat, 5, 3)
+
+x = rand(Float32, 5, 3)
+```
+
+We can ask the array how many elements it has.
+
+```julia
+length(x)
+```
+
+Or, more specifically, what size it has.
+
+```julia
+size(x)
+```
+
+We sometimes want to see some elements of the array on their own.
+
+```julia
+x
+
+x[2, 3]
+```
+
+This means get the second row and the third column. We can also get every row of the third column.
+
+```julia
+x[:, 3]
+```
+
+We can add arrays, and subtract them, which adds or subtracts each element of the array.
+
+```julia
+x + x
+
+x - x
+```
+
+Julia supports a feature called *broadcasting*, using the `.` syntax. This tiles small arrays (or single numbers) to fill bigger ones.
+
+```julia
+x .+ 1
+```
+
+We can see Julia tile the column vector `1:5` across all rows of the larger array.
+
+```julia
+zeros(5,5) .+ (1:5)
+```
+
+The x' syntax is used to transpose a column `1:5` into an equivalent row, and Julia will tile that across columns.
+
+```julia
+zeros(5,5) .+ (1:5)'
+```
+
+We can use this to make a times table.
+
+```julia
+(1:5) .* (1:5)'
+```
+
+Finally, and importantly for machine learning, we can conveniently do things like matrix multiply.
+
+```julia
+W = randn(5, 10)
+x = rand(10)
+W * x
+```
+
+Julia's arrays are very powerful, and you can learn more about what they can do [here](https://docs.julialang.org/en/v1/manual/arrays/).
+
+### CUDA Arrays
+
+CUDA functionality is provided separately by the [CUDA package](https://github.com/JuliaGPU/CUDA.jl). If you have a GPU and CUDA available, you can run `] add CUDA` in a REPL or IJulia to get it.
+
+Once CUDA is loaded you can move any array to the GPU with the `cu` function, and it supports all of the above operations with the same syntax.
+
+```julia
+using CUDA
+x = cu(rand(5, 3))
+```
+
+## Automatic Differentiation
+
+You probably learned to take derivatives in school. We start with a simple mathematical function like
+
+```julia
+f(x) = 3x^2 + 2x + 1
+
+f(5)
+```
+
+In simple cases it's pretty easy to work out the gradient by hand – here it's `6x+2`. But it's much easier to make Flux do the work for us!
+
+```julia
+using Flux: gradient
+
+df(x) = gradient(f, x)[1]
+
+df(5)
+```
+
+You can try this with a few different inputs to make sure it's really the same as `6x+2`. We can even do this multiple times (but the second derivative is a fairly boring `6`).
+
+```julia
+ddf(x) = gradient(df, x)[1]
+
+ddf(5)
+```
+
+Flux's AD can handle any Julia code you throw at it, including loops, recursion and custom layers, so long as the mathematical functions you call are differentiable. For example, we can differentiate a Taylor approximation to the `sin` function.
+
+```julia
+mysin(x) = sum((-1)^k*x^(1+2k)/factorial(1+2k) for k in 0:5)
+
+x = 0.5
+
+mysin(x), gradient(mysin, x)
+
+sin(x), cos(x)
+```
+
+You can see that the derivative we calculated is very close to `cos(x)`, as we expect.
+
+This gets more interesting when we consider functions that take *arrays* as inputs, rather than just a single number. For example, here's a function that takes a matrix and two vectors (the definition itself is arbitrary)
+
+```julia
+myloss(W, b, x) = sum(W * x .+ b)
+
+W = randn(3, 5)
+b = zeros(3)
+x = rand(5)
+
+gradient(myloss, W, b, x)
+```
+
+Now we get gradients for each of the inputs `W`, `b` and `x`, which will come in handy when we want to train models.
+
+Because ML models can contain hundreds of parameters, Flux provides a slightly different way of writing `gradient`. We instead mark arrays with `param` to indicate that we want their derivatives. `W` and `b` represent the weight and bias respectively.
+
+```julia
+using Flux: params
+
+W = randn(3, 5)
+b = zeros(3)
+x = rand(5)
+
+y(x) = sum(W * x .+ b)
+
+grads = gradient(()->y(x), params([W, b]))
+
+grads[W], grads[b]
+```
+
+
+We can now grab the gradients of `W` and `b` directly from those parameters.
+
+This comes in handy when working with *layers*. A layer is just a handy container for some parameters. For example, `Dense` does a linear transform for you.
+
+```julia
+using Flux
+
+m = Dense(10, 5)
+
+x = rand(Float32, 10)
+```
+
+We can easily get the parameters of any layer or model with params with `params`.
+
+```julia
+params(m)
+```
+
+This makes it very easy to calculate the gradient for all parameters in a network, even if it has many parameters.
+
+```julia
+x = rand(Float32, 10)
+m = Chain(Dense(10, 5, relu), Dense(5, 2), softmax)
+l(x) = sum(Flux.crossentropy(m(x), [0.5, 0.5]))
+grads = gradient(params(m)) do
+ l(x)
+end
+for p in params(m)
+ println(grads[p])
+end
+```
+
+You don't have to use layers, but they can be convient for many simple kinds of models and fast iteration.
+
+The next step is to update our weights and perform optimisation. As you might be familiar, *Gradient Descent* is a simple algorithm that takes the weights and steps using a learning rate and the gradients. `weights = weights - learning_rate * gradient`.
+
+```julia
+using Flux.Optimise: update!, Descent
+η = 0.1
+for p in params(m)
+ update!(p, -η * grads[p])
+end
+```
+
+While this is a valid way of updating our weights, it can get more complicated as the algorithms we use get more involved.
+
+Flux comes with a bunch of pre-defined optimisers and makes writing our own really simple. We just give it the learning rate η:
+
+```julia
+opt = Descent(0.01)
+```
+
+`Training` a network reduces down to iterating on a dataset mulitple times, performing these steps in order. Just for a quick implementation, let’s train a network that learns to predict `0.5` for every input of 10 floats. `Flux` defines the `train!` function to do it for us.
+
+```julia
+data, labels = rand(10, 100), fill(0.5, 2, 100)
+loss(x, y) = sum(Flux.crossentropy(m(x), y))
+Flux.train!(loss, params(m), [(data,labels)], opt)
+```
+
+You don't have to use `train!`. In cases where aribtrary logic might be better suited, you could open up this training loop like so:
+
+```julia
+ for d in training_set # assuming d looks like (data, labels)
+ # our super logic
+ gs = gradient(params(m)) do #m is our model
+ l = loss(d...)
+ end
+ update!(opt, params(m), gs)
+ end
+```
+
+## Training a Classifier
+
+Getting a real classifier to work might help cement the workflow a bit more. [CIFAR10](https://https://www.cs.toronto.edu/~kriz/cifar.html) is a dataset of 50k tiny training images split into 10 classes.
+
+We will do the following steps in order:
+
+* Load CIFAR10 training and test datasets
+* Define a Convolution Neural Network
+* Define a loss function
+* Train the network on the training data
+* Test the network on the test data
+
+### Loading the Dataset
+
+```julia
+using Statistics
+using Flux, Flux.Optimise
+using MLDatasets: CIFAR10
+using Images.ImageCore
+using Flux: onehotbatch, onecold
+using Base.Iterators: partition
+using CUDA
+```
+
+This image will give us an idea of what we are dealing with.
+
+
+
+```julia
+train_x, train_y = CIFAR10.traindata(Float32)
+labels = onehotbatch(train_y, 0:9)
+```
+
+The `train_x` contains 50000 images converted to 32 X 32 X 3 arrays with the third dimension being the 3 channels (R,G,B). Let's take a look at a random image from the train_x. For this, we need to permute the dimensions to 3 X 32 X 32 and use `colorview` to convert it back to an image.
+
+```julia
+using Plots
+image(x) = colorview(RGB, permutedims(x, (3, 2, 1)))
+plot(image(train_x[:,:,:,rand(1:end)]))
+```
+
+We can now arrange the training data in batches of say, 1000 and keep a validation set to track our progress. This process is called minibatch learning, which is a popular method of training large neural networks. Rather that sending the entire dataset at once, we break it down into smaller chunks (called minibatches) that are typically chosen at random, and train only on them. It is shown to help with escaping [saddle points](https://en.wikipedia.org/wiki/Saddle_point).
+
+The first 49k images (in batches of 1000) will be our training set, and the rest is for validation. `partition` handily breaks down the set we give it in consecutive parts (1000 in this case).
+
+```julia
+train = ([(train_x[:,:,:,i], labels[:,i]) for i in partition(1:49000, 1000)]) |> gpu
+valset = 49001:50000
+valX = train_x[:,:,:,valset] |> gpu
+valY = labels[:, valset] |> gpu
+```
+
+### Defining the Classifier
+
+Now we can define our Convolutional Neural Network (CNN).
+
+A convolutional neural network is one which defines a kernel and slides it across a matrix to create an intermediate representation to extract features from. It creates higher order features as it goes into deeper layers, making it suitable for images, where the strucure of the subject is what will help us determine which class it belongs to.
+
+```julia
+m = Chain(
+ Conv((5,5), 3=>16, relu),
+ MaxPool((2,2)),
+ Conv((5,5), 16=>8, relu),
+ MaxPool((2,2)),
+ x -> reshape(x, :, size(x, 4)),
+ Dense(200, 120),
+ Dense(120, 84),
+ Dense(84, 10),
+ softmax) |> gpu
+```
+
+We will use a crossentropy loss and an Momentum optimiser here. Crossentropy will be a good option when it comes to working with mulitple independent classes. Momentum gradually lowers the learning rate as we proceed with the training. It helps maintain a bit of adaptivity in our optimisation, preventing us from over shooting from our desired destination.
+
+```julia
+using Flux: crossentropy, Momentum
+
+loss(x, y) = sum(crossentropy(m(x), y))
+opt = Momentum(0.01)
+```
+
+We can start writing our train loop where we will keep track of some basic accuracy numbers about our model. We can define an `accuracy` function for it like so.
+
+```julia
+accuracy(x, y) = mean(onecold(m(x), 0:9) .== onecold(y, 0:9))
+```
+
+### Training the Classifier
+
+
+Training is where we do a bunch of the interesting operations we defined earlier, and see what our net is capable of. We will loop over the dataset 10 times and feed the inputs to the neural network and optimise.
+
+```julia
+epochs = 10
+
+for epoch = 1:epochs
+ for d in train
+ gs = gradient(params(m)) do
+ l = loss(d...)
+ end
+ update!(opt, params(m), gs)
+ end
+ @show accuracy(valX, valY)
+end
+```
+
+Seeing our training routine unfold gives us an idea of how the network learnt the function. This is not bad for a small hand-written network, trained for a limited time.
+
+### Training on a GPU
+
+The `gpu` functions you see sprinkled through this bit of the code tell Flux to move these entities to an available GPU, and subsequently train on it. No extra faffing about required! The same bit of code would work on any hardware with some small annotations like you saw here.
+
+### Testing the Network
+
+We have trained the network for 100 passes over the training dataset. But we need to check if the network has learnt anything at all.
+
+We will check this by predicting the class label that the neural network outputs, and checking it against the ground-truth. If the prediction is correct, we add the sample to the list of correct predictions. This will be done on a yet unseen section of data.
+
+Okay, first step. Let us perform the exact same preprocessing on this set, as we did on our training set.
+
+```julia
+test_x, test_y = CIFAR10.testdata(Float32)
+test_labels = onehotbatch(test_y, 0:9)
+
+test = gpu.([(test_x[:,:,:,i], test_labels[:,i]) for i in partition(1:10000, 1000)])
+```
+
+Next, display an image from the test set.
+
+```julia
+plot(image(test_x[:,:,:,rand(1:end)]))
+```
+
+The outputs are energies for the 10 classes. Higher the energy for a class, the more the network thinks that the image is of the particular class. Every column corresponds to the output of one image, with the 10 floats in the column being the energies.
+
+Let's see how the model fared.
+
+```julia
+ids = rand(1:10000, 5)
+rand_test = test_x[:,:,:,ids] |> gpu
+rand_truth = test_y[ids]
+m(rand_test)
+```
+
+This looks similar to how we would expect the results to be. At this point, it's a good idea to see how our net actually performs on new data, that we have prepared.
+
+```julia
+accuracy(test[1]...)
+```
+
+This is much better than random chance set at 10% (since we only have 10 classes), and not bad at all for a small hand written network like ours.
+
+Let's take a look at how the net performed on all the classes performed individually.
+
+```julia
+class_correct = zeros(10)
+class_total = zeros(10)
+for i in 1:10
+ preds = m(test[i][1])
+ lab = test[i][2]
+ for j = 1:1000
+ pred_class = findmax(preds[:, j])[2]
+ actual_class = findmax(lab[:, j])[2]
+ if pred_class == actual_class
+ class_correct[pred_class] += 1
+ end
+ class_total[actual_class] += 1
+ end
+end
+
+class_correct ./ class_total
+```
+
+The spread seems pretty good, with certain classes performing significantly better than the others. Why should that be?
+
+!!! info
+ Originally published at [fluxml.ai](https://fluxml.ai/tutorials/) on 15 November 2020.
+ Written by Saswat Das, Mike Innes, Andrew Dinhobl, Ygor Canalli, Sudhanshu Agrawal, João Felipe Santos.
+
diff --git a/docs/src/tutorials/2021-01-26-mlp.md b/docs/src/tutorials/2021-01-26-mlp.md
new file mode 100644
index 0000000000..54831834c8
--- /dev/null
+++ b/docs/src/tutorials/2021-01-26-mlp.md
@@ -0,0 +1,169 @@
+# Tutorial: Simple Multi-layer Perceptron
+
+In this example, we create a simple [multi-layer perceptron](https://en.wikipedia.org/wiki/Multilayer_perceptron#:~:text=A%20multilayer%20perceptron%20(MLP)%20is,artificial%20neural%20network%20(ANN).&text=An%20MLP%20consists%20of%20at,uses%20a%20nonlinear%20activation%20function.) (MLP) that classifies handwritten digits using the MNIST dataset. A MLP consists of at least *three layers* of stacked perceptrons: Input, hidden, and output. Each neuron of an MLP has parameters (weights and bias) and uses an [activation function](https://en.wikipedia.org/wiki/Activation_function) to compute its output.
+
+To run this example, we need the following packages:
+
+```julia
+using Flux, Statistics
+using Flux.Data: DataLoader
+using Flux: onehotbatch, onecold, logitcrossentropy, throttle, @epochs
+using Base.Iterators: repeated
+using Parameters: @with_kw
+using CUDA
+using MLDatasets
+if has_cuda() # Check if CUDA is available
+ @info "CUDA is on"
+ CUDA.allowscalar(false)
+end
+```
+
+We set default values for learning rate, batch size, epochs, and the usage of a GPU (if available) for our model:
+
+```julia
+@with_kw mutable struct Args
+ η::Float64 = 3e-4 # learning rate
+ batchsize::Int = 1024 # batch size
+ epochs::Int = 10 # number of epochs
+ device::Function = gpu # set as gpu, if gpu available
+end
+```
+
+
+If a GPU is available on our local system, then Flux uses it for computing the loss and updating the weights and biases when training our model.
+
+## Data
+
+We create the function `getdata` to load the MNIST train and test data sets from [MLDatasets](https://juliaml.github.io/MLDatasets.jl/latest/) and prepare them for the training process. In addition, we set mini-batches of the data sets by loading them onto a [DataLoader](https://fluxml.ai/Flux.jl/stable/data/dataloader/#Flux.Data.DataLoader) object.
+
+```julia
+function getdata(args)
+ ENV["DATADEPS_ALWAYS_ACCEPT"] = "true"
+
+ # Loading Dataset
+ xtrain, ytrain = MLDatasets.MNIST.traindata(Float32)
+ xtest, ytest = MLDatasets.MNIST.testdata(Float32)
+
+ # Reshape Data in order to flatten each image into a linear array
+ xtrain = Flux.flatten(xtrain)
+ xtest = Flux.flatten(xtest)
+
+ # One-hot-encode the labels
+ ytrain, ytest = onehotbatch(ytrain, 0:9), onehotbatch(ytest, 0:9)
+
+ # Batching
+ train_data = DataLoader((xtrain, ytrain), batchsize=args.batchsize, shuffle=true)
+ test_data = DataLoader((xtest, ytest), batchsize=args.batchsize)
+
+ return train_data, test_data
+end
+```
+
+
+`getdata` performs the following steps:
+
+* **Loads MNIST data set:** Loads the train and test set tensors. The shape of train data is `28x28x60000` and test data is `28X28X10000`.
+* **Reshapes the train and test data:** Uses the [flatten](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.flatten) function to reshape the train data set into a `784x60000` array and test data set into a `784x10000`. Notice that we reshape the data so that we can pass these as arguments for the input layer of our model (a simple MLP expects a vector as an input).
+* **One-hot encodes the train and test labels:** Creates a batch of one-hot vectors so we can pass the labels of the data as arguments for the loss function. For this example, we use the [logitcrossentropy](https://fluxml.ai/Flux.jl/stable/models/losses/#Flux.Losses.logitcrossentropy) function and it expects data to be one-hot encoded.
+* **Creates batches of data:** Creates two DataLoader objects (train and test) that handle data mini-batches of size `1024 ` (as defined above). We create these two objects so that we can pass the entire data set through the loss function at once when training our model. Also, it shuffles the data points during each iteration (`shuffle=true`).
+
+## Model
+
+As we mentioned above, a MLP consist of *three* layers that are fully connected. For this example, we define out model with the following layers and dimensions:
+
+* **Input:** It has `784` perceptrons (the MNIST image size is `28x28`). We flatten the train and test data so that we can pass them as arguments to this layer.
+* **Hidden:** It has `32` perceptrons that use the [relu](https://fluxml.ai/Flux.jl/stable/models/nnlib/#NNlib.relu) activation function.
+* **Output:** It has `10` perceptrons that output the model's prediction or probability that a digit is 0 to 9.
+
+
+We define our model with the `build_model` function:
+
+```julia
+function build_model(; imgsize=(28,28,1), nclasses=10)
+ return Chain(
+ Dense(prod(imgsize), 32, relu),
+ Dense(32, nclasses))
+end
+```
+
+
+Note that we use the functions [Dense](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.Dense) so that our model is *densely* (or fully) connected and [Chain](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.Chain) to chain the computation of the three layers.
+
+## Loss functions
+
+Now, we define the loss function `loss_all`. It expects a DataLoader object and the `model` function we defined aboved as arguments. Notice that this function iterates through the `dataloader` object in mini-batches and uses the function [logitcrossentropy](https://fluxml.ai/Flux.jl/stable/models/losses/#Flux.Losses.logitcrossentropy) to compute the difference between the predicted and actual values.
+
+```julia
+function loss_all(dataloader, model)
+ l = 0f0
+ for (x,y) in dataloader
+ l += logitcrossentropy(model(x), y)
+ end
+ l/length(dataloader)
+end
+```
+
+
+In addition, we define the function (`accuracy`) to report the accuracy of our model during the training process. To compute the accuray, we need to decode the output of our model using the [onecold](https://fluxml.ai/Flux.jl/stable/data/onehot/#Flux.onecold) function.
+
+```julia
+function accuracy(data_loader, model)
+ acc = 0
+ for (x,y) in data_loader
+ acc += sum(onecold(cpu(model(x))) .== onecold(cpu(y)))*1 / size(x,2)
+ end
+ acc/length(data_loader)
+end
+```
+
+
+## Train our model
+
+Finally, we create the `train` function that calls the functions we defined and trains the model.
+
+```julia
+function train(; kws...)
+ # Initializing Model parameters
+ args = Args(; kws...)
+
+ # Load Data
+ train_data,test_data = getdata(args)
+
+ # Construct model
+ m = build_model()
+ train_data = args.device.(train_data)
+ test_data = args.device.(test_data)
+ m = args.device(m)
+ loss(x,y) = logitcrossentropy(m(x), y)
+
+ ## Training
+ evalcb = () -> @show(loss_all(train_data, m))
+ opt = ADAM(args.η)
+
+ @epochs args.epochs Flux.train!(loss, params(m), train_data, opt, cb = evalcb)
+
+ @show accuracy(train_data, m)
+
+ @show accuracy(test_data, m)
+end
+```
+
+
+`train` performs the following steps:
+
+* **Initializes the model parameters:** Creates the `args` object that contains the defult values for training our model.
+* **Loads the train and test data:** Calls the function `getdata` we defined above.
+* **Constructs the model:** Builds the model and loads the train and test data sets, and our model onto the GPU (if available).
+* **Trains the model:** Defines the *callback* function `evalcb` to show the value of the `loss_all` function during the training process. Then, it sets [ADAM](https://fluxml.ai/Flux.jl/stable/training/optimisers/#Flux.Optimise.ADAM) as the optimiser for training out model. Finally, it runs the training process with the macro `@epochs` for `10` epochs (as defined in the `args` object) and shows the `accuracy` value for the train and test data.
+
+
+To see the full version of this example, see [Simple multi-layer perceptron - model-zoo](https://github.com/FluxML/model-zoo/blob/master/vision/mlp_mnist/mlp_mnist.jl).
+
+## Resources
+
+* [3Blue1Brown Neural networks videos](https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi).
+* [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/).
+
+!!! info
+ Originally published at [fluxml.ai](https://fluxml.ai/tutorials/) on 26 January 2021.
+ Written by Adarsh Kumar, Mike J Innes, Andrew Dinhobl, Jerry Ling, natema, Zhang Shitian, Liliana Badillo, Dhairya Gandhi
diff --git a/docs/src/tutorials/2021-02-07-convnet.md b/docs/src/tutorials/2021-02-07-convnet.md
new file mode 100644
index 0000000000..ed9996ccec
--- /dev/null
+++ b/docs/src/tutorials/2021-02-07-convnet.md
@@ -0,0 +1,260 @@
+# Tutorial: A Simple ConvNet
+
+In this tutorial, we build a simple Convolutional Neural Network (ConvNet) to classify the MNIST dataset. This model has a simple architecture with three feature detection layers (Conv -> ReLU -> MaxPool) followed by a final dense layer that classifies MNIST handwritten digits. Note that this model, while simple, should hit around 99% test accuracy after training for approximately 20 epochs.
+
+This example writes out the saved model to the file `mnist_conv.bson`. Also, it demonstrates basic model construction, training, saving, conditional early-exit, and learning rate scheduling.
+
+To run this example, we need the following packages:
+
+```julia
+using Flux, MLDatasets, Statistics
+using Flux: onehotbatch, onecold, logitcrossentropy
+using MLDatasets: MNIST
+using Base.Iterators: partition
+using Printf, BSON
+using Parameters: @with_kw
+using CUDA
+CUDA.allowscalar(false)
+```
+
+We set default values for learning rate, batch size, number of epochs, and path for saving the file `mnist_conv.bson`:
+
+```julia
+@with_kw mutable struct Args
+ lr::Float64 = 3e-3
+ epochs::Int = 20
+ batch_size = 128
+ savepath::String = "./"
+end
+```
+
+To train our model, we need to bundle images together with their labels and group them into mini-batches (makes the training process faster). We define the function `make_minibatch` that takes as inputs the images (`X`) and their labels (`Y`) as well as the indices for the mini-batches (`idx`):
+
+```julia
+function make_minibatch(X, Y, idxs)
+ X_batch = Array{Float32}(undef, size(X)[1:end-1]..., 1, length(idxs))
+ for i in 1:length(idxs)
+ X_batch[:, :, :, i] = Float32.(X[:,:,idxs[i]])
+ end
+ Y_batch = onehotbatch(Y[idxs], 0:9)
+ return (X_batch, Y_batch)
+end
+```
+
+`make_minibatch` takes the following steps:
+
+* Creates the `X_batch` array of size `28x28x1x128` to store the mini-batches.
+* Stores the mini-batches in `X_batch`.
+* One hot encodes the labels of the images.
+* Stores the labels in `Y_batch`.
+
+
+
+ `get_processed_data` loads the train and test data from `Flux.Data.MNIST`. First, it loads the images and labels of the train data set, and creates an array that contains the indices of the train images that correspond to each mini-batch (of size `args.batch_size`). Then, it calls the `make_minibatch` function to create all of the train mini-batches. Finally, it loads the test images and creates one mini-batch that contains them all.
+
+```julia
+function get_processed_data(args)
+ # Load labels and images
+ train_imgs, train_labels = MNIST.traindata()
+ mb_idxs = partition(1:length(train_labels), args.batch_size)
+ train_set = [make_minibatch(train_imgs, train_labels, i) for i in mb_idxs]
+
+ # Prepare test set as one giant minibatch:
+ test_imgs, test_labels = MNIST.testdata()
+ test_set = make_minibatch(test_imgs, test_labels, 1:length(test_labels))
+
+ return train_set, test_set
+
+end
+```
+
+
+Now, we define the `build_model` function that creates a ConvNet model which is composed of *three* convolution layers (feature detection) and *one* classification layer. The input layer size is `28x28`. The images are grayscale, which means there is only *one* channel (compared to 3 for RGB) in every data point. Combined together, the convolutional layer structure would look like `Conv(kernel, input_channels => output_channels, ...)`. Each convolution layer reduces the size of the image by applying the Rectified Linear unit (ReLU) and MaxPool operations.
+On the other hand, the classification layer outputs a vector of 10 dimensions (a dense layer), that is, the number of classes that the model will be able to predict.
+
+
+```julia
+function build_model(args; imgsize = (28,28,1), nclasses = 10)
+ cnn_output_size = Int.(floor.([imgsize[1]/8,imgsize[2]/8,32]))
+
+ return Chain(
+ # First convolution, operating upon a 28x28 image
+ Conv((3, 3), imgsize[3]=>16, pad=(1,1), relu),
+ MaxPool((2,2)),
+
+ # Second convolution, operating upon a 14x14 image
+ Conv((3, 3), 16=>32, pad=(1,1), relu),
+ MaxPool((2,2)),
+
+ # Third convolution, operating upon a 7x7 image
+ Conv((3, 3), 32=>32, pad=(1,1), relu),
+ MaxPool((2,2)),
+
+ # Reshape 3d array into a 2d one using `Flux.flatten`, at this point it should be (3, 3, 32, N)
+ flatten,
+ Dense(prod(cnn_output_size), 10))
+end
+```
+
+To chain the layers of a model we use the Flux function [Chain](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.Chain). It enables us to call the layers in sequence on a given input. Also, we use the function [flatten](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.flatten) to reshape the output image from the last convolution layer. Finally, we call the [Dense](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.Dense) function to create the classification layer.
+
+
+Before training our model, we need to define a few functions that will be helpful for the process:
+
+
+* `augment` augments the data by adding gaussian random noise to our image to make it more robust:
+
+ ```julia
+ augment(x) = x .+ gpu(0.1f0*randn(eltype(x), size(x)))
+ ```
+
+
+* `anynan` checks whether any element of the params is NaN or not:
+
+ ```julia
+ anynan(x) = any(y -> any(isnan, y), x)
+ ```
+
+* `accuracy` computes the accuracy of our ConvNet:
+
+ ```julia
+ accuracy(x, y, model) = mean(onecold(cpu(model(x))) .== onecold(cpu(y)))
+ ```
+
+Finally, we define the `train` function:
+
+```julia
+function train(; kws...)
+ args = Args(; kws...)
+
+ @info("Loading data set")
+ train_set, test_set = get_processed_data(args)
+
+ # Define our model. We will use a simple convolutional architecture with
+ # three iterations of Conv -> ReLU -> MaxPool, followed by a final Dense layer.
+ @info("Building model...")
+ model = build_model(args)
+
+ # Load model and datasets onto GPU, if enabled
+ train_set = gpu.(train_set)
+ test_set = gpu.(test_set)
+ model = gpu(model)
+
+ # Make sure our model is nicely precompiled before starting our training loop
+ model(train_set[1][1])
+
+ # `loss()` calculates the crossentropy loss between our prediction `y_hat`
+ # (calculated from `model(x)`) and the ground truth `y`. We augment the data
+ # a bit, adding gaussian random noise to our image to make it more robust.
+ function loss(x, y)
+ x̂ = augment(x)
+ ŷ = model(x̂)
+ return logitcrossentropy(Å·, y)
+ end
+
+ # Train our model with the given training set using the ADAM optimizer and
+ # printing out performance against the test set as we go.
+ opt = ADAM(args.lr)
+
+ @info("Beginning training loop...")
+ best_acc = 0.0
+ last_improvement = 0
+ for epoch_idx in 1:args.epochs
+ # Train for a single epoch
+ Flux.train!(loss, params(model), train_set, opt)
+
+ # Terminate on NaN
+ if anynan(Flux.params(model))
+ @error "NaN params"
+ break
+ end
+
+ # Calculate accuracy:
+ acc = accuracy(test_set..., model)
+
+ @info(@sprintf("[%d]: Test accuracy: %.4f", epoch_idx, acc))
+ # If our accuracy is good enough, quit out.
+ if acc >= 0.999
+ @info(" -> Early-exiting: We reached our target accuracy of 99.9%")
+ break
+ end
+
+ # If this is the best accuracy we've seen so far, save the model out
+ if acc >= best_acc
+ @info(" -> New best accuracy! Saving model out to mnist_conv.bson")
+ BSON.@save joinpath(args.savepath, "mnist_conv.bson") params=cpu.(params(model)) epoch_idx acc
+ best_acc = acc
+ last_improvement = epoch_idx
+ end
+
+ # If we haven't seen improvement in 5 epochs, drop our learning rate:
+ if epoch_idx - last_improvement >= 5 && opt.eta > 1e-6
+ opt.eta /= 10.0
+ @warn(" -> Haven't improved in a while, dropping learning rate to $(opt.eta)!")
+
+ # After dropping learning rate, give it a few epochs to improve
+ last_improvement = epoch_idx
+ end
+
+ if epoch_idx - last_improvement >= 10
+ @warn(" -> We're calling this converged.")
+ break
+ end
+ end
+end
+```
+
+`train` calls the functions we defined above and trains our model. It stops when the model achieves 99% accuracy (early-exiting) or after performing 20 steps. More specifically, it performs the following steps:
+
+ * Loads the MNIST dataset.
+ * Builds our ConvNet model (as described above).
+ * Loads the train and test data sets as well as our model onto a GPU (if available).
+ * Defines a `loss` function that calculates the crossentropy between our prediction and the ground truth.
+ * Sets the [ADAM optimiser](https://fluxml.ai/Flux.jl/stable/training/optimisers/#Flux.Optimise.ADAM) to train the model with learning rate `args.lr`.
+ * Runs the training loop. For each step (or epoch), it executes the following:
+ * Calls `Flux.train!` function to execute one training step.
+ * If any of the parameters of our model is `NaN`, then the training process is terminated.
+ * Calculates the model accuracy.
+ * If the model accuracy is >= 0.999, then early-exiting is executed.
+ * If the actual accuracy is the best so far, then the model is saved to `mnist_conv.bson`. Also, the new best accuracy and the current epoch is saved.
+ * If there has not been any improvement for the last 5 epochs, then the learning rate is dropped and the process waits a little longer for the accuracy to improve.
+ * If the last improvement was more than 10 epochs ago, then the process is terminated.
+
+
+Finally, to test our model we define the `test` function:
+
+```julia
+function test(; kws...)
+ args = Args(; kws...)
+
+ # Loading the test data
+ _,test_set = get_processed_data(args)
+
+ # Re-constructing the model with random initial weights
+ model = build_model(args)
+
+ # Loading the saved parameters
+ BSON.@load joinpath(args.savepath, "mnist_conv.bson") params
+
+ # Loading parameters onto the model
+ Flux.loadparams!(model, params)
+
+ test_set = gpu.(test_set)
+ model = gpu(model)
+ @show accuracy(test_set...,model)
+end
+```
+
+`test` loads the MNIST test data set, reconstructs the model, and loads the saved parameters (in `mnist_conv.bson`) onto it. Finally, it computes our model's predictions for the test set and shows the test accuracy (around 99%).
+
+To see the full version of this example, see [Simple ConvNets - model-zoo](https://github.com/FluxML/model-zoo/blob/master/vision/conv_mnist/conv_mnist.jl).
+
+## Resources
+
+* [Neural Networks in Flux.jl with Huda Nassar (working with the MNIST dataset)](https://youtu.be/Oxi0Pfmskus)
+* [Convolutional Neural Networks (CNNs / ConvNets)](https://cs231n.github.io/convolutional-networks/).
+* [Convolutional Neural Networks Tutorial in PyTorch](https://adventuresinmachinelearning.com/convolutional-neural-networks-tutorial-in-pytorch/).
+
+!!! info
+ Originally published at [fluxml.ai](https://fluxml.ai/tutorials/) on 7 February 2021.
+ Written by Elliot Saba, Adarsh Kumar, Mike J Innes, Dhairya Gandhi, Sudhanshu Agrawal, Sambit Kumar Dash, fps.io, Carlo Lucibello, Andrew Dinhobl, Liliana Badillo
diff --git a/docs/src/tutorials/2021-10-08-dcgan-mnist.md b/docs/src/tutorials/2021-10-08-dcgan-mnist.md
new file mode 100644
index 0000000000..31b1ebb914
--- /dev/null
+++ b/docs/src/tutorials/2021-10-08-dcgan-mnist.md
@@ -0,0 +1,363 @@
+# Tutorial: Deep Convolutional Generative Adversarial Network (DCGAN)
+
+This is a beginner level tutorial for generating images of handwritten digits using a [Deep Convolutional Generative Adversarial Network](https://arxiv.org/pdf/1511.06434.pdf) inspired by the [TensorFlow tutorial on DCGAN](https://www.tensorflow.org/tutorials/generative/dcgan).
+
+## What are GANs?
+[Generative Adversarial Neural Networks or simply GANs](https://arxiv.org/abs/1406.2661) introduced by Goodfellow et al. is one of the most innovative ideas in modern-day machine learning. GANs are used extensively in the field of image and audio processing to generate high-quality synthetic data that can easily be passed off as real data.
+
+A GAN is composed of two sub-models - the **generator** and the **discriminator** acting against one another. The generator can be considered as an artist who draws (generates) new images that look real, whereas the discriminator is a critic who learns to tell real images apart from fakes.
+
+
+
+The GAN starts with a generator and discriminator which have very little or no idea about the underlying data. During training, the generator progressively becomes better at creating images that look real, while the discriminator becomes better at telling them apart. The process reaches equilibrium when the discriminator can no longer distinguish real images from fakes.
+
+
+
+
+[[source]](https://www.tensorflow.org/tutorials/generative/dcgan)
+
+This tutorial demonstrates the process of training a DC-GAN on the [MNIST dataset for handwritten digits](http://yann.lecun.com/exdb/mnist/). The following animation shows a series of images produced by the generator as it was trained for 25 epochs. The images begin as random noise, but over time, the images become increasingly similar to handwritten numbers.
+
+~~~
+
+
+
+
+~~~
+
+## Setup
+
+We need to install some Julia packages before we start with our implementation of DCGAN.
+
+```julia
+using Pkg
+
+# Activate a new project environment in the current directory
+Pkg.activate(".")
+# Add the required packages to the environment
+Pkg.add(["Images", "Flux", "MLDatasets", "CUDA", "Parameters"])
+```
+*Note: Depending on your internet speed, it may take a few minutes for the packages install.*
+
+
+After installing the libraries, load the required packages and functions:
+```julia
+using Base.Iterators: partition
+using Printf
+using Statistics
+using Random
+using Images
+using Parameters: @with_kw
+using Flux
+using Flux.Data: DataLoader
+using Flux.Optimise: update!
+using Flux.Losses: logitbinarycrossentropy
+using MLDatasets: MNIST
+using CUDA
+```
+
+Now we set default values for the learning rates, batch size, epochs, the usage of a GPU (if available) and other hyperparameters for our model.
+
+```julia
+@with_kw struct HyperParams
+ batch_size::Int = 128
+ latent_dim::Int = 100
+ epochs::Int = 25
+ verbose_freq::Int = 1000
+ output_dim::Int = 5
+ disc_lr::Float64 = 0.0002
+ gen_lr::Float64 = 0.0002
+ device::Function = gpu
+end
+```
+
+## Loading the data
+As mentioned before, we will be using the MNIST dataset for handwritten digits. So we begin with a simple function for loading and pre-processing the MNIST images:
+```julia
+function load_MNIST_images(hparams)
+ images = MNIST.traintensor(Float32)
+
+ # Normalize the images to (-1, 1)
+ normalized_images = @. 2f0 * images - 1f0
+ image_tensor = reshape(normalized_images, 28, 28, 1, :)
+
+ # Create a dataloader that iterates over mini-batches of the image tensor
+ dataloader = DataLoader(image_tensor, batchsize=hparams.batch_size, shuffle=true)
+
+ return dataloader
+end
+```
+To learn more about loading images in Flux, you can check out [this tutorial](https://fluxml.ai/tutorials/2021/01/21/data-loader.html).
+
+*Note: The data returned from the dataloader is loaded is on the CPU. To train on the GPU, we need to transfer the data to the GPU beforehand.*
+
+## Create the models
+
+
+### The generator
+
+Our generator, a.k.a. the artist, is a neural network that maps low dimensional data to a high dimensional form.
+
+- This low dimensional data (seed) is generally a vector of random values sampled from a normal distribution.
+- The high dimensional data is the generated image.
+
+The `Dense` layer is used for taking the seed as an input which is upsampled several times using the [ConvTranspose](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.ConvTranspose) layer until we reach the desired output size (in our case, 28x28x1). Furthermore, after each `ConvTranspose` layer, we apply the Batch Normalization to stabilize the learning process.
+
+We will be using the [relu](https://fluxml.ai/Flux.jl/stable/models/nnlib/#NNlib.relu) activation function for each layer except the output layer, where we use `tanh` activation.
+
+We will also apply the weight initialization method mentioned in the original DCGAN paper.
+
+```julia
+# Function for intializing the model weights with values
+# sampled from a Gaussian distribution with μ=0 and σ=0.02
+dcgan_init(shape...) = randn(Float32, shape) * 0.02f0
+```
+
+```julia
+function Generator(latent_dim)
+ Chain(
+ Dense(latent_dim, 7*7*256, bias=false),
+ BatchNorm(7*7*256, relu),
+
+ x -> reshape(x, 7, 7, 256, :),
+
+ ConvTranspose((5, 5), 256 => 128; stride = 1, pad = 2, init = dcgan_init, bias=false),
+ BatchNorm(128, relu),
+
+ ConvTranspose((4, 4), 128 => 64; stride = 2, pad = 1, init = dcgan_init, bias=false),
+ BatchNorm(64, relu),
+
+ # The tanh activation ensures that output is in range of (-1, 1)
+ ConvTranspose((4, 4), 64 => 1, tanh; stride = 2, pad = 1, init = dcgan_init, bias=false),
+ )
+end
+```
+
+Time for a small test!! We create a dummy generator and feed a random vector as a seed to the generator. If our generator is initialized correctly it will return an array of size (28, 28, 1, `batch_size`). The `@assert` macro in Julia will raise an exception for the wrong output size.
+
+```julia
+# Create a dummy generator of latent dim 100
+generator = Generator(100)
+noise = randn(Float32, 100, 3) # The last axis is the batch size
+
+# Feed the random noise to the generator
+gen_image = generator(noise)
+@assert size(gen_image) == (28, 28, 1, 3)
+```
+
+Our generator model is yet to learn the correct weights, so it does not produce a recognizable image for now. To train our poor generator we need its equal rival, the *discriminator*.
+
+### Discriminator
+
+The Discriminator is a simple CNN based image classifier. The `Conv` layer a is used with a [leakyrelu](https://fluxml.ai/Flux.jl/stable/models/nnlib/#NNlib.leakyrelu) activation function.
+
+```julia
+function Discriminator()
+ Chain(
+ Conv((4, 4), 1 => 64; stride = 2, pad = 1, init = dcgan_init),
+ x->leakyrelu.(x, 0.2f0),
+ Dropout(0.3),
+
+ Conv((4, 4), 64 => 128; stride = 2, pad = 1, init = dcgan_init),
+ x->leakyrelu.(x, 0.2f0),
+ Dropout(0.3),
+
+ # The output is now of the shape (7, 7, 128, batch_size)
+ flatten,
+ Dense(7 * 7 * 128, 1)
+ )
+end
+```
+For a more detailed implementation of a CNN-based image classifier, you can refer to [this tutorial](https://fluxml.ai/tutorials/2021/02/07/convnet.html).
+
+Now let us check if our discriminator is working:
+
+```julia
+# Dummy Discriminator
+discriminator = Discriminator()
+# We pass the generated image to the discriminator
+logits = discriminator(gen_image)
+@assert size(logits) == (1, 3)
+```
+
+Just like our dummy generator, the untrained discriminator has no idea about what is a real or fake image. It needs to be trained alongside the generator to output positive values for real images, and negative values for fake images.
+
+## Loss functions for GAN
+
+In a GAN problem, there are only two labels involved: fake and real. So Binary CrossEntropy is an easy choice for a preliminary loss function.
+
+But even if Flux's `binarycrossentropy` does the job for us, due to numerical stability it is always preferred to compute cross-entropy using logits. Flux provides [logitbinarycrossentropy](https://fluxml.ai/Flux.jl/stable/models/losses/#Flux.Losses.logitbinarycrossentropy) specifically for this purpose. Mathematically it is equivalent to `binarycrossentropy(σ(ŷ), y, kwargs...).`
+
+### Discriminator Loss
+
+The discriminator loss quantifies how well the discriminator can distinguish real images from fakes. It compares
+
+- discriminator's predictions on real images to an array of 1s, and
+- discriminator's predictions on fake (generated) images to an array of 0s.
+
+These two losses are summed together to give a scalar loss. So we can write the loss function of the discriminator as:
+
+```julia
+function discriminator_loss(real_output, fake_output)
+ real_loss = logitbinarycrossentropy(real_output, 1)
+ fake_loss = logitbinarycrossentropy(fake_output, 0)
+ return real_loss + fake_loss
+end
+```
+
+### Generator Loss
+
+The generator's loss quantifies how well it was able to trick the discriminator. Intuitively, if the generator is performing well, the discriminator will classify the fake images as real (or 1).
+
+```julia
+generator_loss(fake_output) = logitbinarycrossentropy(fake_output, 1)
+```
+
+We also need optimizers for our network. Why you may ask? Read more [here](https://towardsdatascience.com/overview-of-various-optimizers-in-neural-networks-17c1be2df6d5). For both the generator and discriminator, we will use the [ADAM optimizer](https://fluxml.ai/Flux.jl/stable/training/optimisers/#Flux.Optimise.ADAM).
+
+## Utility functions
+
+The output of the generator ranges from (-1, 1), so it needs to be de-normalized before we can display it as an image. To make things a bit easier, we define a function to visualize the output of the generator as a grid of images.
+
+```julia
+function create_output_image(gen, fixed_noise, hparams)
+ fake_images = cpu(gen.(fixed_noise))
+ image_array = reduce(vcat, reduce.(hcat, partition(fake_images, hparams.output_dim)))
+ image_array = permutedims(dropdims(image_array; dims=(3, 4)), (2, 1))
+ image_array = @. Gray(image_array + 1f0) / 2f0
+ return image_array
+end
+```
+
+## Training
+
+For the sake of simplifying our training problem, we will divide the generator and discriminator training into two separate functions.
+
+```julia
+function train_discriminator!(gen, disc, real_img, fake_img, opt, ps, hparams)
+
+ disc_loss, grads = Flux.withgradient(ps) do
+ discriminator_loss(disc(real_img), disc(fake_img))
+ end
+
+ # Update the discriminator parameters
+ update!(opt, ps, grads)
+ return disc_loss
+end
+```
+
+We define a similar function for the generator.
+
+```julia
+function train_generator!(gen, disc, fake_img, opt, ps, hparams)
+
+ gen_loss, grads = Flux.withgradient(ps) do
+ generator_loss(disc(fake_img))
+ end
+
+ update!(opt, ps, grads)
+ return gen_loss
+end
+```
+
+Now that we have defined every function we need, we integrate everything into a single `train` function where we first set up all the models and optimizers and then train the GAN for a specified number of epochs.
+
+```julia
+function train(hparams)
+
+ dev = hparams.device
+ # Check if CUDA is actually present
+ if hparams.device == gpu
+ if !CUDA.has_cuda()
+ dev = cpu
+ @warn "No gpu found, falling back to CPU"
+ end
+ end
+
+ # Load the normalized MNIST images
+ dataloader = load_MNIST_images(hparams)
+
+ # Initialize the models and pass them to correct device
+ disc = Discriminator() |> dev
+ gen = Generator(hparams.latent_dim) |> dev
+
+ # Collect the generator and discriminator parameters
+ disc_ps = params(disc)
+ gen_ps = params(gen)
+
+ # Initialize the ADAM optimizers for both the sub-models
+ # with respective learning rates
+ disc_opt = ADAM(hparams.disc_lr)
+ gen_opt = ADAM(hparams.gen_lr)
+
+ # Create a batch of fixed noise for visualizing the training of generator over time
+ fixed_noise = [randn(Float32, hparams.latent_dim, 1) |> dev for _=1:hparams.output_dim^2]
+
+ # Training loop
+ train_steps = 0
+ for ep in 1:hparams.epochs
+ @info "Epoch $ep"
+ for real_img in dataloader
+
+ # Transfer the data to the GPU
+ real_img = real_img |> dev
+
+ # Create a random noise
+ noise = randn!(similar(real_img, (hparams.latent_dim, hparams.batch_size)))
+ # Pass the noise to the generator to create a fake imagae
+ fake_img = gen(noise)
+
+ # Update discriminator and generator
+ loss_disc = train_discriminator!(gen, disc, real_img, fake_img, disc_opt, disc_ps, hparams)
+ loss_gen = train_generator!(gen, disc, fake_img, gen_opt, gen_ps, hparams)
+
+ if train_steps % hparams.verbose_freq == 0
+ @info("Train step $(train_steps), Discriminator loss = $(loss_disc), Generator loss = $(loss_gen)")
+ # Save generated fake image
+ output_image = create_output_image(gen, fixed_noise, hparams)
+ save(@sprintf("output/dcgan_steps_%06d.png", train_steps), output_image)
+ end
+ train_steps += 1
+ end
+ end
+
+ output_image = create_output_image(gen, fixed_noise, hparams)
+ save(@sprintf("output/dcgan_steps_%06d.png", train_steps), output_image)
+
+ return nothing
+end
+```
+
+Now we finally get to train the GAN:
+
+```julia
+# Define the hyper-parameters (here, we go with the default ones)
+hparams = HyperParams()
+train(hparams)
+```
+
+## Output
+The generated images are stored inside the `output` folder. To visualize the output of the generator over time, we create a gif of the generated images.
+
+```julia
+folder = "output"
+# Get the image filenames from the folder
+img_paths = readdir(folder, join=true)
+# Load all the images as an array
+images = load.(img_paths)
+# Join all the images in the array to create a matrix of images
+gif_mat = cat(images..., dims=3)
+save("./output.gif", gif_mat)
+```
+
+
+
+
+
+## Resources & References
+- [The DCGAN implementation in Model Zoo.](http=s://github.com/FluxML/model-zoo/blob/master/vision/dcgan_mnist/dcgan_mnist.jl)
+
+!!! info
+ Originally published at [fluxml.ai](https://fluxml.ai/tutorials/) on 8 October 2021,
+ by Deeptendu Santra
+
diff --git a/docs/src/tutorials/2021-10-14-vanilla-gan.md b/docs/src/tutorials/2021-10-14-vanilla-gan.md
new file mode 100644
index 0000000000..2647504a84
--- /dev/null
+++ b/docs/src/tutorials/2021-10-14-vanilla-gan.md
@@ -0,0 +1,280 @@
+# Tutorial: Generative Adversarial Networks
+
+This tutorial describes how to implement a vanilla Generative Adversarial
+Network using Flux and how train it on the MNIST dataset. It is based on this
+[Pytorch tutorial](https://medium.com/ai-society/gans-from-scratch-1-a-deep-introduction-with-code-in-pytorch-and-tensorflow-cb03cdcdba0f). The original GAN [paper](https://arxiv.org/abs/1406.2661) by Goodfellow et al. is a great resource that describes the motivation and theory behind GANs:
+
+> In the proposed adversarial nets framework, the generative model is pitted against an adversary: a
+> discriminative model that learns to determine whether a sample is from the model distribution or the
+> data distribution. The generative model can be thought of as analogous to a team of counterfeiters,
+> trying to produce fake currency and use it without detection, while the discriminative model is
+> analogous to the police, trying to detect the counterfeit currency. Competition in this game drives
+> both teams to improve their methods until the counterfeits are indistinguishable from the genuine
+> articles.
+
+Let's implement a GAN in Flux. To get started we first import a few useful packages:
+
+
+```julia
+using MLDatasets: MNIST
+using Flux.Data: DataLoader
+using Flux
+using CUDA
+using Zygote
+using UnicodePlots
+```
+
+To download a package in the Julia REPL, type `]` to enter package mode and then
+type `add MLDatasets` or perform this operation with the Pkg module like this
+
+```julia
+> import Pkg
+> Pkg.add(MLDatasets)
+```
+
+While [UnicodePlots]() is not necessary, it can be used to plot generated samples
+into the terminal during training. Having direct feedback, instead of looking
+at plots in a separate window, use fantastic for debugging.
+
+
+Next, let us define values for learning rate, batch size, epochs, and other
+hyper-parameters. While we are at it, we also define optimizers for the generator
+and discriminator network. More on what these are later.
+
+```julia
+ lr_g = 2e-4 # Learning rate of the generator network
+ lr_d = 2e-4 # Learning rate of the discriminator network
+ batch_size = 128 # batch size
+ num_epochs = 1000 # Number of epochs to train for
+ output_period = 100 # Period length for plots of generator samples
+ n_features = 28 * 28# Number of pixels in each sample of the MNIST dataset
+ latent_dim = 100 # Dimension of latent space
+ opt_dscr = ADAM(lr_d)# Optimizer for the discriminator
+ opt_gen = ADAM(lr_g) # Optimizer for the generator
+```
+
+
+In this tutorial I'm assuming that a CUDA-enabled GPU is available on the
+system where the script is running. If this is not the case, simply remove
+the `|>gpu` decorators: [piping](https://docs.julialang.org/en/v1/manual/functions/#Function-composition-and-piping).
+
+## Data loading
+The MNIST data set is available from [MLDatasets](https://juliaml.github.io/MLDatasets.jl/latest/). The first time you instantiate it you will be prompted
+if you want to download it. You should agree to this.
+
+GANs can be trained unsupervised. Therefore only keep the images from the training
+set and discard the labels.
+
+After we load the training data we re-scale the data from values in [0:1]
+to values in [-1:1]. GANs are notoriously tricky to train and this re-scaling
+is a recommended [GAN hack](https://github.com/soumith/ganhacks). The
+re-scaled data is used to define a data loader which handles batching
+and shuffling the data.
+
+```julia
+ # Load the dataset
+ train_x, _ = MNIST.traindata(Float32);
+ # This dataset has pixel values ∈ [0:1]. Map these to [-1:1]
+ train_x = 2f0 * reshape(train_x, 28, 28, 1, :) .- 1f0 |>gpu;
+ # DataLoader allows to access data batch-wise and handles shuffling.
+ train_loader = DataLoader(train_x, batchsize=batch_size, shuffle=true);
+```
+
+
+## Defining the Networks
+
+
+A vanilla GAN, the discriminator and the generator are both plain, [feed-forward
+multilayer perceptrons](https://boostedml.com/2020/04/feedforward-neural-networks-and-multilayer-perceptrons.html). We use leaky rectified linear units [leakyrelu](https://fluxml.ai/Flux.jl/stable/models/nnlib/#NNlib.leakyrelu) to ensure out model is non-linear.
+
+Here, the coefficient `α` (in the `leakyrelu` below), is set to 0.2. Empirically,
+this value allows for good training of the network (based on prior experiments).
+It has also been found that Dropout ensures a good generalization of the learned
+network, so we will use that below. Dropout is usually active when training a
+model and inactive in inference. Flux automatically sets the training mode when
+calling the model in a gradient context. As a final non-linearity, we use the
+`sigmoid` activation function.
+
+```julia
+discriminator = Chain(Dense(n_features, 1024, x -> leakyrelu(x, 0.2f0)),
+ Dropout(0.3),
+ Dense(1024, 512, x -> leakyrelu(x, 0.2f0)),
+ Dropout(0.3),
+ Dense(512, 256, x -> leakyrelu(x, 0.2f0)),
+ Dropout(0.3),
+ Dense(256, 1, sigmoid)) |> gpu
+```
+
+Let's define the generator in a similar fashion. This network maps a latent
+variable (a variable that is not directly observed but instead inferred) to the
+image space and we set the input and output dimension accordingly. A `tanh` squashes
+the output of the final layer to values in [-1:1], the same range that we squashed
+the training data onto.
+
+```julia
+generator = Chain(Dense(latent_dim, 256, x -> leakyrelu(x, 0.2f0)),
+ Dense(256, 512, x -> leakyrelu(x, 0.2f0)),
+ Dense(512, 1024, x -> leakyrelu(x, 0.2f0)),
+ Dense(1024, n_features, tanh)) |> gpu
+```
+
+
+
+## Training functions for the networks
+
+To train the discriminator, we present it with real data from the MNIST
+data set and with fake data and reward it by predicting the correct labels for
+each sample. The correct labels are of course 1 for in-distribution data
+and 0 for out-of-distribution data coming from the generator.
+[Binary cross entropy](https://fluxml.ai/Flux.jl/stable/models/losses/#Flux.Losses.binarycrossentropy)
+is the loss function of choice. While the Flux documentation suggests to use
+[Logit binary cross entropy](https://fluxml.ai/Flux.jl/stable/models/losses/#Flux.Losses.logitcrossentropy),
+the GAN seems to be difficult to train with this loss function.
+This function returns the discriminator loss for logging purposes. We can
+calculate the loss in the same call as evaluating the pullback and resort
+to getting the pullback directly from Zygote instead of calling
+`Flux.train!` on the model. To calculate the gradients of the loss
+function with respect to the parameters of the discriminator we then only have to
+evaluate the pullback with a seed gradient of 1.0. These gradients are used
+to update the model parameters
+
+
+```julia
+function train_dscr!(discriminator, real_data, fake_data)
+ this_batch = size(real_data)[end] # Number of samples in the batch
+ # Concatenate real and fake data into one big vector
+ all_data = hcat(real_data, fake_data)
+
+ # Target vector for predictions: 1 for real data, 0 for fake data.
+ all_target = [ones(eltype(real_data), 1, this_batch) zeros(eltype(fake_data), 1, this_batch)] |> gpu;
+
+ ps = Flux.params(discriminator)
+ loss, pullback = Zygote.pullback(ps) do
+ preds = discriminator(all_data)
+ loss = Flux.Losses.binarycrossentropy(preds, all_target)
+ end
+ # To get the gradients we evaluate the pullback with 1.0 as a seed gradient.
+ grads = pullback(1f0)
+
+ # Update the parameters of the discriminator with the gradients we calculated above
+ Flux.update!(opt_dscr, Flux.params(discriminator), grads)
+
+ return loss
+end
+```
+
+
+Now we need to define a function to train the generator network. The job of the
+generator is to fool the discriminator so we reward the generator when the discriminator
+predicts a high probability for its samples to be real data. In the training function
+we first need to sample some noise, i.e. normally distributed data. This has
+to be done outside the pullback since we don't want to get the gradients with
+respect to the noise, but to the generator parameters. Inside the pullback we need
+to first apply the generator to the noise since we will take the gradient with respect
+to the parameters of the generator. We also need to call the discriminator in order
+to evaluate the loss function inside the pullback. Here we need to remember to deactivate
+the dropout layers of the discriminator. We do this by setting the discriminator into
+test mode before the pullback. Immediately after the pullback we set it back into training
+mode. Then we evaluate the pullback, call it with a seed gradient of 1.0 as above, update the
+parameters of the generator network and return the loss.
+
+
+```julia
+function train_gen!(discriminator, generator)
+ # Sample noise
+ noise = randn(latent_dim, batch_size) |> gpu;
+
+ # Define parameters and get the pullback
+ ps = Flux.params(generator)
+ # Set discriminator into test mode to disable dropout layers
+ testmode!(discriminator)
+ # Evaluate the loss function while calculating the pullback. We get the loss for free
+ loss, back = Zygote.pullback(ps) do
+ preds = discriminator(generator(noise));
+ loss = Flux.Losses.binarycrossentropy(preds, 1.)
+ end
+ # Evaluate the pullback with a seed-gradient of 1.0 to get the gradients for
+ # the parameters of the generator
+ grads = back(1.0f0)
+ Flux.update!(opt_gen, Flux.params(generator), grads)
+ # Set discriminator back into automatic mode
+ trainmode!(discriminator, mode=:auto)
+ return loss
+end
+```
+
+## Training
+Now we are ready to train the GAN. In the training loop we keep track
+of the per-sample loss of the generator and the discriminator, where
+we use the batch loss returned by the two training functions defined above.
+In each epoch we iterate over the mini-batches given by the data loader.
+Only minimal data processing needs to be done before the training functions
+can be called.
+
+```julia
+lossvec_gen = zeros(num_epochs)
+lossvec_dscr = zeros(num_epochs)
+
+for n in 1:num_epochs
+ loss_sum_gen = 0.0f0
+ loss_sum_dscr = 0.0f0
+
+ for x in train_loader
+ # - Flatten the images from 28x28xbatchsize to 784xbatchsize
+ real_data = flatten(x);
+
+ # Train the discriminator
+ noise = randn(latent_dim, size(x)[end]) |> gpu
+ fake_data = generator(noise)
+ loss_dscr = train_dscr!(discriminator, real_data, fake_data)
+ loss_sum_dscr += loss_dscr
+
+ # Train the generator
+ loss_gen = train_gen!(discriminator, generator)
+ loss_sum_gen += loss_gen
+ end
+
+ # Add the per-sample loss of the generator and discriminator
+ lossvec_gen[n] = loss_sum_gen / size(train_x)[end]
+ lossvec_dscr[n] = loss_sum_dscr / size(train_x)[end]
+
+ if n % output_period == 0
+ @show n
+ noise = randn(latent_dim, 4) |> gpu;
+ fake_data = reshape(generator(noise), 28, 4*28);
+ p = heatmap(fake_data, colormap=:inferno)
+ print(p)
+ end
+end
+```
+
+For the hyper-parameters shown in this example, the generator produces useful
+images after about 1000 epochs. And after about 5000 epochs the result look
+indistinguishable from real MNIST data. Using a Nvidia V100 GPU on a 2.7
+GHz Power9 CPU with 32 hardware threads, training 100 epochs takes about 80
+seconds when using the GPU. The GPU utilization is between 30 and 40%.
+To observe the network more frequently during training you can for example set
+`output_period=20`. Training the GAN using the CPU takes about 10 minutes
+per epoch and is not recommended.
+
+## Results
+
+Below you can see what some of the images output may look like after different numbers of epochs.
+
+
+
+
+
+
+
+
+
+## Resources
+
+* [A collection of GANs in Flux](https://github.com/AdarshKumar712/FluxGAN)
+* [Wikipedia](https://en.wikipedia.org/wiki/Generative_adversarial_network)
+* [GAN hacks](https://github.com/soumith/ganhacks)
+
+!!! info
+ Originally published at [fluxml.ai](https://fluxml.ai/tutorials/) on 14 October 2021,
+ by Ralph Kube.
diff --git a/docs/src/tutorials/linear_regression.md b/docs/src/tutorials/linear_regression.md
index d73c20cc38..993c66e494 100644
--- a/docs/src/tutorials/linear_regression.md
+++ b/docs/src/tutorials/linear_regression.md
@@ -1,4 +1,4 @@
-# [Linear Regression](@id man-linear-regression)
+# [Tutorial: Linear Regression](@id man-linear-regression)
Flux is a pure Julia ML stack that allows you to build predictive models. Here are the steps for a typical Flux program:
@@ -388,3 +388,7 @@ Summarising this tutorial, we started by generating a random yet correlated data
Next, we trained the model by manually writing down the Gradient Descent algorithm and optimising the loss. We also saw how `Flux` provides various wrapper functionalities and keeps the API extremely intuitive and simple for the users.
After getting familiar with the basics of `Flux` and `Julia`, we moved ahead to build a machine learning model for a real dataset. We repeated the exact same steps, but this time with a lot more features and data points, and by harnessing `Flux`'s full capabilities. In the end, we developed a training loop that was smarter than the hardcoded one and ran the model on our normalised dataset to conclude the tutorial.
+
+!!! info
+ Originally published on 21 November 2022,
+ by Saransh Chopra.
From 3601ac2554dccd4e210f38316e01b194a4328478 Mon Sep 17 00:00:00 2001
From: Michael Abbott <32575566+mcabbott@users.noreply.github.com>
Date: Thu, 24 Nov 2022 19:52:37 -0500
Subject: [PATCH 02/11] more heading links
---
docs/src/tutorials/2020-09-15-deep-learning-flux.md | 2 +-
docs/src/tutorials/2021-01-26-mlp.md | 2 +-
docs/src/tutorials/2021-02-07-convnet.md | 2 +-
docs/src/tutorials/2021-10-08-dcgan-mnist.md | 2 +-
docs/src/tutorials/2021-10-14-vanilla-gan.md | 2 +-
5 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/docs/src/tutorials/2020-09-15-deep-learning-flux.md b/docs/src/tutorials/2020-09-15-deep-learning-flux.md
index efd89727c5..fc5d3da357 100755
--- a/docs/src/tutorials/2020-09-15-deep-learning-flux.md
+++ b/docs/src/tutorials/2020-09-15-deep-learning-flux.md
@@ -1,4 +1,4 @@
-# Deep Learning with Julia & Flux: A 60 Minute Blitz
+# [Deep Learning with Julia & Flux: A 60 Minute Blitz](@id man-blitz)
This is a quick intro to [Flux](https://github.com/FluxML/Flux.jl) loosely based on [PyTorch's tutorial](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html). It introduces basic Julia programming, as well Zygote, a source-to-source automatic differentiation (AD) framework in Julia. We'll use these tools to build a very simple neural network.
diff --git a/docs/src/tutorials/2021-01-26-mlp.md b/docs/src/tutorials/2021-01-26-mlp.md
index 54831834c8..9601429c48 100644
--- a/docs/src/tutorials/2021-01-26-mlp.md
+++ b/docs/src/tutorials/2021-01-26-mlp.md
@@ -1,4 +1,4 @@
-# Tutorial: Simple Multi-layer Perceptron
+# [Tutorial: Simple Multi-layer Perceptron](@id man-mlp-tutorial)
In this example, we create a simple [multi-layer perceptron](https://en.wikipedia.org/wiki/Multilayer_perceptron#:~:text=A%20multilayer%20perceptron%20(MLP)%20is,artificial%20neural%20network%20(ANN).&text=An%20MLP%20consists%20of%20at,uses%20a%20nonlinear%20activation%20function.) (MLP) that classifies handwritten digits using the MNIST dataset. A MLP consists of at least *three layers* of stacked perceptrons: Input, hidden, and output. Each neuron of an MLP has parameters (weights and bias) and uses an [activation function](https://en.wikipedia.org/wiki/Activation_function) to compute its output.
diff --git a/docs/src/tutorials/2021-02-07-convnet.md b/docs/src/tutorials/2021-02-07-convnet.md
index ed9996ccec..99a64ed2d0 100644
--- a/docs/src/tutorials/2021-02-07-convnet.md
+++ b/docs/src/tutorials/2021-02-07-convnet.md
@@ -1,4 +1,4 @@
-# Tutorial: A Simple ConvNet
+# [Tutorial: A Simple ConvNet](@id man-convnet-tutorial)
In this tutorial, we build a simple Convolutional Neural Network (ConvNet) to classify the MNIST dataset. This model has a simple architecture with three feature detection layers (Conv -> ReLU -> MaxPool) followed by a final dense layer that classifies MNIST handwritten digits. Note that this model, while simple, should hit around 99% test accuracy after training for approximately 20 epochs.
diff --git a/docs/src/tutorials/2021-10-08-dcgan-mnist.md b/docs/src/tutorials/2021-10-08-dcgan-mnist.md
index 31b1ebb914..c0960bf636 100644
--- a/docs/src/tutorials/2021-10-08-dcgan-mnist.md
+++ b/docs/src/tutorials/2021-10-08-dcgan-mnist.md
@@ -1,4 +1,4 @@
-# Tutorial: Deep Convolutional Generative Adversarial Network (DCGAN)
+# Deep Convolutional Generative Adversarial Network (DCGAN)
This is a beginner level tutorial for generating images of handwritten digits using a [Deep Convolutional Generative Adversarial Network](https://arxiv.org/pdf/1511.06434.pdf) inspired by the [TensorFlow tutorial on DCGAN](https://www.tensorflow.org/tutorials/generative/dcgan).
diff --git a/docs/src/tutorials/2021-10-14-vanilla-gan.md b/docs/src/tutorials/2021-10-14-vanilla-gan.md
index 2647504a84..b2c7bb6f4c 100644
--- a/docs/src/tutorials/2021-10-14-vanilla-gan.md
+++ b/docs/src/tutorials/2021-10-14-vanilla-gan.md
@@ -1,4 +1,4 @@
-# Tutorial: Generative Adversarial Networks
+# [Tutorial: Generative Adversarial Networks](](@id man-gan-tutorial))
This tutorial describes how to implement a vanilla Generative Adversarial
Network using Flux and how train it on the MNIST dataset. It is based on this
From 55836fff986881b947bf6ad06d155fb6a045d1b9 Mon Sep 17 00:00:00 2001
From: Michael Abbott <32575566+mcabbott@users.noreply.github.com>
Date: Thu, 24 Nov 2022 20:02:00 -0500
Subject: [PATCH 03/11] somehow fix images + links for DCGAN
---
docs/src/tutorials/2021-10-08-dcgan-mnist.md | 17 +++++------------
1 file changed, 5 insertions(+), 12 deletions(-)
diff --git a/docs/src/tutorials/2021-10-08-dcgan-mnist.md b/docs/src/tutorials/2021-10-08-dcgan-mnist.md
index c0960bf636..be966bebe1 100644
--- a/docs/src/tutorials/2021-10-08-dcgan-mnist.md
+++ b/docs/src/tutorials/2021-10-08-dcgan-mnist.md
@@ -7,7 +7,7 @@ This is a beginner level tutorial for generating images of handwritten digits us
A GAN is composed of two sub-models - the **generator** and the **discriminator** acting against one another. The generator can be considered as an artist who draws (generates) new images that look real, whereas the discriminator is a critic who learns to tell real images apart from fakes.
-
+
The GAN starts with a generator and discriminator which have very little or no idea about the underlying data. During training, the generator progressively becomes better at creating images that look real, while the discriminator becomes better at telling them apart. The process reaches equilibrium when the discriminator can no longer distinguish real images from fakes.
@@ -18,12 +18,7 @@ The GAN starts with a generator and discriminator which have very little or no i
This tutorial demonstrates the process of training a DC-GAN on the [MNIST dataset for handwritten digits](http://yann.lecun.com/exdb/mnist/). The following animation shows a series of images produced by the generator as it was trained for 25 epochs. The images begin as random noise, but over time, the images become increasingly similar to handwritten numbers.
-~~~
-
-
-
-
-~~~
+
## Setup
@@ -94,7 +89,7 @@ To learn more about loading images in Flux, you can check out [this tutorial](ht
## Create the models
-### The generator
+### Generator
Our generator, a.k.a. the artist, is a neural network that maps low dimensional data to a high dimensional form.
@@ -350,12 +345,10 @@ gif_mat = cat(images..., dims=3)
save("./output.gif", gif_mat)
```
-
-
-
+
## Resources & References
-- [The DCGAN implementation in Model Zoo.](http=s://github.com/FluxML/model-zoo/blob/master/vision/dcgan_mnist/dcgan_mnist.jl)
+- [The DCGAN implementation in the Model Zoo.](https://github.com/FluxML/model-zoo/blob/master/vision/dcgan_mnist/dcgan_mnist.jl)
!!! info
Originally published at [fluxml.ai](https://fluxml.ai/tutorials/) on 8 October 2021,
From 009192b26278d01bb5e654b6d86e2fe0790c700a Mon Sep 17 00:00:00 2001
From: Michael Abbott <32575566+mcabbott@users.noreply.github.com>
Date: Thu, 24 Nov 2022 21:52:23 -0500
Subject: [PATCH 04/11] using Flux: params
---
docs/src/tutorials/2021-01-26-mlp.md | 2 +-
docs/src/tutorials/2021-02-07-convnet.md | 2 +-
docs/src/tutorials/2021-10-08-dcgan-mnist.md | 3 +--
3 files changed, 3 insertions(+), 4 deletions(-)
diff --git a/docs/src/tutorials/2021-01-26-mlp.md b/docs/src/tutorials/2021-01-26-mlp.md
index 9601429c48..d2df9ddf84 100644
--- a/docs/src/tutorials/2021-01-26-mlp.md
+++ b/docs/src/tutorials/2021-01-26-mlp.md
@@ -7,7 +7,7 @@ To run this example, we need the following packages:
```julia
using Flux, Statistics
using Flux.Data: DataLoader
-using Flux: onehotbatch, onecold, logitcrossentropy, throttle, @epochs
+using Flux: onehotbatch, onecold, logitcrossentropy, throttle, @epochs, params
using Base.Iterators: repeated
using Parameters: @with_kw
using CUDA
diff --git a/docs/src/tutorials/2021-02-07-convnet.md b/docs/src/tutorials/2021-02-07-convnet.md
index 99a64ed2d0..4824365a3d 100644
--- a/docs/src/tutorials/2021-02-07-convnet.md
+++ b/docs/src/tutorials/2021-02-07-convnet.md
@@ -8,7 +8,7 @@ To run this example, we need the following packages:
```julia
using Flux, MLDatasets, Statistics
-using Flux: onehotbatch, onecold, logitcrossentropy
+using Flux: onehotbatch, onecold, logitcrossentropy, params
using MLDatasets: MNIST
using Base.Iterators: partition
using Printf, BSON
diff --git a/docs/src/tutorials/2021-10-08-dcgan-mnist.md b/docs/src/tutorials/2021-10-08-dcgan-mnist.md
index be966bebe1..250ed22c3e 100644
--- a/docs/src/tutorials/2021-10-08-dcgan-mnist.md
+++ b/docs/src/tutorials/2021-10-08-dcgan-mnist.md
@@ -43,8 +43,7 @@ using Statistics
using Random
using Images
using Parameters: @with_kw
-using Flux
-using Flux.Data: DataLoader
+using Flux: params, DataLoader
using Flux.Optimise: update!
using Flux.Losses: logitbinarycrossentropy
using MLDatasets: MNIST
From 2aa2a7a7561de913a6fb2e131dd87aa905c1d97c Mon Sep 17 00:00:00 2001
From: Michael Abbott <32575566+mcabbott@users.noreply.github.com>
Date: Fri, 25 Nov 2022 08:27:02 -0500
Subject: [PATCH 05/11] rm indents
---
docs/src/tutorials/2021-02-07-convnet.md | 23 +++++++++++------------
1 file changed, 11 insertions(+), 12 deletions(-)
diff --git a/docs/src/tutorials/2021-02-07-convnet.md b/docs/src/tutorials/2021-02-07-convnet.md
index 4824365a3d..1462e1348a 100644
--- a/docs/src/tutorials/2021-02-07-convnet.md
+++ b/docs/src/tutorials/2021-02-07-convnet.md
@@ -104,22 +104,21 @@ Before training our model, we need to define a few functions that will be helpfu
* `augment` augments the data by adding gaussian random noise to our image to make it more robust:
- ```julia
- augment(x) = x .+ gpu(0.1f0*randn(eltype(x), size(x)))
- ```
-
-
+```julia
+augment(x) = x .+ gpu(0.1f0*randn(eltype(x), size(x)))
+```
+
* `anynan` checks whether any element of the params is NaN or not:
- ```julia
- anynan(x) = any(y -> any(isnan, y), x)
- ```
+```julia
+ anynan(x) = any(y -> any(isnan, y), x)
+```
* `accuracy` computes the accuracy of our ConvNet:
- ```julia
- accuracy(x, y, model) = mean(onecold(cpu(model(x))) .== onecold(cpu(y)))
- ```
+```julia
+ accuracy(x, y, model) = mean(onecold(cpu(model(x))) .== onecold(cpu(y)))
+```
Finally, we define the `train` function:
@@ -222,7 +221,7 @@ end
Finally, to test our model we define the `test` function:
-
+
```julia
function test(; kws...)
args = Args(; kws...)
From 02e9d2066528ea8aa734ab2d00aeb7c44f114593 Mon Sep 17 00:00:00 2001
From: Michael Abbott <32575566+mcabbott@users.noreply.github.com>
Date: Fri, 25 Nov 2022 08:49:52 -0500
Subject: [PATCH 06/11] ADAM
---
docs/src/tutorials/2021-01-26-mlp.md | 4 ++--
docs/src/tutorials/2021-02-07-convnet.md | 6 +++---
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/docs/src/tutorials/2021-01-26-mlp.md b/docs/src/tutorials/2021-01-26-mlp.md
index d2df9ddf84..c9ee9897a0 100644
--- a/docs/src/tutorials/2021-01-26-mlp.md
+++ b/docs/src/tutorials/2021-01-26-mlp.md
@@ -138,7 +138,7 @@ function train(; kws...)
## Training
evalcb = () -> @show(loss_all(train_data, m))
- opt = ADAM(args.η)
+ opt = Adam(args.η)
@epochs args.epochs Flux.train!(loss, params(m), train_data, opt, cb = evalcb)
@@ -154,7 +154,7 @@ end
* **Initializes the model parameters:** Creates the `args` object that contains the defult values for training our model.
* **Loads the train and test data:** Calls the function `getdata` we defined above.
* **Constructs the model:** Builds the model and loads the train and test data sets, and our model onto the GPU (if available).
-* **Trains the model:** Defines the *callback* function `evalcb` to show the value of the `loss_all` function during the training process. Then, it sets [ADAM](https://fluxml.ai/Flux.jl/stable/training/optimisers/#Flux.Optimise.ADAM) as the optimiser for training out model. Finally, it runs the training process with the macro `@epochs` for `10` epochs (as defined in the `args` object) and shows the `accuracy` value for the train and test data.
+* **Trains the model:** Defines the *callback* function `evalcb` to show the value of the `loss_all` function during the training process. Then, it sets [Adam](@ref Flux.Optimise.Adam) as the optimiser for training out model. Finally, it runs the training process with the macro `@epochs` for `10` epochs (as defined in the `args` object) and shows the `accuracy` value for the train and test data.
To see the full version of this example, see [Simple multi-layer perceptron - model-zoo](https://github.com/FluxML/model-zoo/blob/master/vision/mlp_mnist/mlp_mnist.jl).
diff --git a/docs/src/tutorials/2021-02-07-convnet.md b/docs/src/tutorials/2021-02-07-convnet.md
index 1462e1348a..7fa32e58bb 100644
--- a/docs/src/tutorials/2021-02-07-convnet.md
+++ b/docs/src/tutorials/2021-02-07-convnet.md
@@ -151,9 +151,9 @@ function train(; kws...)
return logitcrossentropy(Å·, y)
end
- # Train our model with the given training set using the ADAM optimizer and
+ # Train our model with the given training set using the Adam optimizer and
# printing out performance against the test set as we go.
- opt = ADAM(args.lr)
+ opt = Adam(args.lr)
@info("Beginning training loop...")
best_acc = 0.0
@@ -209,7 +209,7 @@ end
* Builds our ConvNet model (as described above).
* Loads the train and test data sets as well as our model onto a GPU (if available).
* Defines a `loss` function that calculates the crossentropy between our prediction and the ground truth.
- * Sets the [ADAM optimiser](https://fluxml.ai/Flux.jl/stable/training/optimisers/#Flux.Optimise.ADAM) to train the model with learning rate `args.lr`.
+ * Sets the [Adam optimiser](https://fluxml.ai/Flux.jl/stable/training/optimisers/#Flux.Optimise.Adam) to train the model with learning rate `args.lr`.
* Runs the training loop. For each step (or epoch), it executes the following:
* Calls `Flux.train!` function to execute one training step.
* If any of the parameters of our model is `NaN`, then the training process is terminated.
From a1ab2d0a551633b4751c6d0ceaf4884dedf013fe Mon Sep 17 00:00:00 2001
From: Michael Abbott <32575566+mcabbott@users.noreply.github.com>
Date: Fri, 25 Nov 2022 08:58:21 -0500
Subject: [PATCH 07/11] rm using Parameters, and use distinct names
---
docs/src/tutorials/2021-01-26-mlp.md | 7 +++----
docs/src/tutorials/2021-02-07-convnet.md | 7 +++----
docs/src/tutorials/2021-10-08-dcgan-mnist.md | 3 +--
3 files changed, 7 insertions(+), 10 deletions(-)
diff --git a/docs/src/tutorials/2021-01-26-mlp.md b/docs/src/tutorials/2021-01-26-mlp.md
index c9ee9897a0..7b2b530518 100644
--- a/docs/src/tutorials/2021-01-26-mlp.md
+++ b/docs/src/tutorials/2021-01-26-mlp.md
@@ -9,7 +9,6 @@ using Flux, Statistics
using Flux.Data: DataLoader
using Flux: onehotbatch, onecold, logitcrossentropy, throttle, @epochs, params
using Base.Iterators: repeated
-using Parameters: @with_kw
using CUDA
using MLDatasets
if has_cuda() # Check if CUDA is available
@@ -21,8 +20,8 @@ end
We set default values for learning rate, batch size, epochs, and the usage of a GPU (if available) for our model:
```julia
-@with_kw mutable struct Args
- η::Float64 = 3e-4 # learning rate
+Base.@kwdef mutable struct Args
+ rate::Float64 = 3e-4 # learning rate
batchsize::Int = 1024 # batch size
epochs::Int = 10 # number of epochs
device::Function = gpu # set as gpu, if gpu available
@@ -138,7 +137,7 @@ function train(; kws...)
## Training
evalcb = () -> @show(loss_all(train_data, m))
- opt = Adam(args.η)
+ opt = Adam(args.rate)
@epochs args.epochs Flux.train!(loss, params(m), train_data, opt, cb = evalcb)
diff --git a/docs/src/tutorials/2021-02-07-convnet.md b/docs/src/tutorials/2021-02-07-convnet.md
index 7fa32e58bb..3620d27205 100644
--- a/docs/src/tutorials/2021-02-07-convnet.md
+++ b/docs/src/tutorials/2021-02-07-convnet.md
@@ -12,7 +12,6 @@ using Flux: onehotbatch, onecold, logitcrossentropy, params
using MLDatasets: MNIST
using Base.Iterators: partition
using Printf, BSON
-using Parameters: @with_kw
using CUDA
CUDA.allowscalar(false)
```
@@ -20,7 +19,7 @@ CUDA.allowscalar(false)
We set default values for learning rate, batch size, number of epochs, and path for saving the file `mnist_conv.bson`:
```julia
-@with_kw mutable struct Args
+Base.@kwdef mutable struct TrainArgs
lr::Float64 = 3e-3
epochs::Int = 20
batch_size = 128
@@ -124,7 +123,7 @@ Finally, we define the `train` function:
```julia
function train(; kws...)
- args = Args(; kws...)
+ args = TrainArgs(; kws...)
@info("Loading data set")
train_set, test_set = get_processed_data(args)
@@ -224,7 +223,7 @@ Finally, to test our model we define the `test` function:
```julia
function test(; kws...)
- args = Args(; kws...)
+ args = TrainArgs(; kws...)
# Loading the test data
_,test_set = get_processed_data(args)
diff --git a/docs/src/tutorials/2021-10-08-dcgan-mnist.md b/docs/src/tutorials/2021-10-08-dcgan-mnist.md
index 250ed22c3e..9a71a8ade8 100644
--- a/docs/src/tutorials/2021-10-08-dcgan-mnist.md
+++ b/docs/src/tutorials/2021-10-08-dcgan-mnist.md
@@ -42,7 +42,6 @@ using Printf
using Statistics
using Random
using Images
-using Parameters: @with_kw
using Flux: params, DataLoader
using Flux.Optimise: update!
using Flux.Losses: logitbinarycrossentropy
@@ -53,7 +52,7 @@ using CUDA
Now we set default values for the learning rates, batch size, epochs, the usage of a GPU (if available) and other hyperparameters for our model.
```julia
-@with_kw struct HyperParams
+Base.@kwdef struct HyperParams
batch_size::Int = 128
latent_dim::Int = 100
epochs::Int = 25
From 52b96535d9d311f927ac92699e237ce2cd525e72 Mon Sep 17 00:00:00 2001
From: Michael Abbott <32575566+mcabbott@users.noreply.github.com>
Date: Fri, 25 Nov 2022 09:06:59 -0500
Subject: [PATCH 08/11] local image files
---
.../assets/2021-10-08-dcgan-mnist/cat_gan.png | Bin 0 -> 1173514 bytes
.../assets/2021-10-08-dcgan-mnist/output.gif | Bin 0 -> 182397 bytes
2 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 docs/src/assets/2021-10-08-dcgan-mnist/cat_gan.png
create mode 100644 docs/src/assets/2021-10-08-dcgan-mnist/output.gif
diff --git a/docs/src/assets/2021-10-08-dcgan-mnist/cat_gan.png b/docs/src/assets/2021-10-08-dcgan-mnist/cat_gan.png
new file mode 100644
index 0000000000000000000000000000000000000000..a3b59c5adb5eb0734eb90d4d7bd4051b68542dc5
GIT binary patch
literal 1173514
zcmeEP2VjoJ8^7!!_8uWNtq`%Ju{W`6mWm{Z+G6)#dsk7EDn(J$EJ~}bJ!{t9d+(i{
z{GZ=*@B1Z2l@LvMPTudm_wKppp8MS=_ses4?~SZmyJ}MBOwLNFq}8ics;`vwIHjz9
zceJOJpyG2&DwW!)Z^I@*^?iG}b?Mf*O<;!rx1c^<0^9<62ewhF_wl(?8~4ohTguAO
zKRdfyl^Jkpz3raAldb&xRJnA=f{M&;G@)X;-#hoJZWjgs@
ze-O4N+R3Kuo-Zyv?sM!+lzXMl=`U9=+}?S|gVH}fF5q^oPM5%=4#B<7Z5)@W|F(mn
zw;KE}X_azcj5-jWK4X_{-%RcQ$JGXnPDh7*z4)H3hNb#nc&WCly9e(oSY^qOHEWAMTD2uS{SluZmo50t
zFYr*3vO}|9{G!*|%pd=mZ|JSbGd^FsuVStb^IlxG$uaNM|F!V_d2^OSmCudMkUrze
z63xFpdZETQ>pGk{S1J4O?>cNx0Adt}EPZS<_(wd%ozNF{9I!vQo=XR=6-hu`GDC3>Mf1SNsrCKeT)T-t`VA0>1XW17jT*J|J`S_U|_WS+S_3MFk
zYn&|Ld3oXozXksN;X>suy$mZPw0ejrI%;8-H^Cta|l#%$?Y5
z&dU8e=P$V2to`EpNz)9^wWIc^hA9L02dq5U?Zd^z0~bFk-uI@{z5;*tytX*FjD7cH
zRkEJ+{soD%w7shYe`!O^y#
zCtYe^r)}f=X={F#JI|?+LmFT6z1(0!nPcH!dh{*SV_VVESDS^UxjR1~(}PNnrVg_`
zRI+w(z~bW%J?c+g{G`a!YI&T|`NnsBs%zfaSJqt|xIJUf4}ncfpGw;eq&
zcb03eUtP@c=-`mmfqSf12K+dC@Rr41O>4JUv#?k4(>v7%?E^Me964>uoXORkE}ahg
z^5nrbHE!*y*YzuxtyxG*qX@1VFi(YJ!?ck!K
zms|OJe$ze4twomtc7L(vvkKEE{gH2Pn!g7X3Ewxechg+6Z+z6Ee*WTaKF+ODx6M)Z
zr}diy?v)NG^KA?7
zO=?jyfOdS$;iRSC!}7GedCe7SHHX1ysUq-
zfs6V%{cq$R&uaa~t=dt|K7FkP-?t7ewasyP-nrYpN_pt`HT$D0Po=)vr(M5uZNm<4
zTT-LmXSFilZL_K3fZ}esYp*HmGIQzL#)pUZ&G&`R{;Cdj=8PG;BF{J5hxf7R-tEVm
zI~p}j(=^G>q~s%(#*D=MTI+o7*=menE|Gn{wR3+}~YG-ejSFl3OPy
zwJwym_7b)CTj!J;{>nak+LfE5U2AL)ueE+p@!eGxSFj77m^Wgx|Bt8q+z+-LR{4JM
z5ck}_CjH@q3qPtzr&vCb65KO(xmE$D*dLryEJ3q;)
z)|{J8MavCy^7JUX-eKTh^G0>(*8AtihZ+`3Iq8bK7L2PPTe$X$GHtM9x2$Q
zUYdo~UF+4n_HcUJUyJ6yc_OIS(Y&Kw8t2SjA!vqIwM-ja*ZgvKM!l2ctwPuL{CMG+
zV?Q47>esGc&UU^Z&Oh9GK!>0tAz4(QYC10b?&-kaSN-p7anHJ$*V|vuc_n%JApbos
zbyBXV>Q*jeyKlT&%ve!y=lqculGRZsKKZ-noFmuswWx6|$CUs^BNv-JIHI9RB6vG6e=V{AlKqMe~0M8`Nz4
z`kj~Zjmi9=x6S-U(W7=O`b14>opDmJPQlxT9M0spw9?dc1H987t+3bULYlwo@9Xv9
zSM@fuNR#Fl*8$n~Wy!K*b_;!feY9`S$f2dW4Yoi2+1E!}TlYAZwqg3AHNPF`oTr=T374AnlBK-%*I&iE
zI(uiE^X<=fwwB+&wd@ylJ;&$EeDH3&jh=bu)=AZRQ;8pcbsn=Pd6P}c$9-|^XP5REeiP6n&8HjB6-iewNtLg^>w2zD
zrg33gH|O8Euwn63g>rr}X4K(ncFPy1@8@;9+=9C4R+ebtI44c
znkIL(^2&RpQSqrOXD?k-(6OaYCw0c|!I+*^l5L1eJ0@A1T-lqQs?#h(VfzyMZVXyb
zKFDoLlEI!+*G;NbaZu&yc7K1Gt6Z<9qlP**J-7F(*3$#;{o*suuW-8aTQ)2i(BwPk
z+?#G$f9P_u+IgRuf0QeDD`s^HVtjetY^Go}W@)p#^kmpb~TTjZsP&DADc-M?s9@Sz`9
zY+aunGk1G9F!-}^%l<4*A?7GL(4*g@tpZS02WIZg`wY04rymqUV
zy(bJ&<*KJFRB~VYPM)(edEA-5^I_KC+xK){?!7OA^S*jJ3mvrSlezGmUMt4<(JYaRkYcn2=sF*TG;jW`b1g$SrH+QB2
zD~ErTGtaUz+kZ*ktx%ZH829P}Kd97e<2pOHFSm?4*y5pgd!NzXD&Ls_7ng@QsxFw|i`?bSq<{
z>5c{GoM^FUPn(@{rzBsRcKUC#4rQvdEMVWKHu)PYXjQ9y-qGzNmN#teacSeA>F2N8
zl^n8uR=GtT%kSGiphA*Vm48`bJ!H|;rEU&GH=iHbwVqwp5!Hr#-{kWVdoE_bHK}@s
z)|s}aNtf#K#{7G4-O6XRHOcl4HCAsuoz8C6uVoshX--;pIT;@Yw>2*
zBP)%$P^R>4*FMSHtRJE5tgQOGUbjD2sdk^aU+oKWw>oyU%QvZ8d2Uvxzt}p|cFXt6
zJ2sn=aaoE2HELZ8@z~O0{3PH0pBDe0{hpbb8myX-fB%@0gF3dV@kq7%WmBG9xkhcB
z@UWQMD7&qfrew;LD)|PNu$6q!Q;tzZ(#J*UNfxj^+;+%WOMj3|F@l+HXNAT`@D_Ek2ObSt}w8ie+obMYF&;eoBXBA4F}tD^I8QiSdeVR
z=C4l8+*@mHlfJ*@9dq*X#s!&!SN~aMRp|}w4n)=by>R3C-WTl8^(;3p}={Fq}?n%s8%ca0jg6HVT?mzciNXnBv-HzX>JND|T;oWvy{}#0;
z;`g!hREzMfRabmIq;#G;&JIOB==yoc_B_94E0ZnKBgN>w!TG0;Pv7t73oSY)`K0mC
zG48HEZ93`{;azx=hwbsBwU>1ZIyyE@kukf@7FoY2r2O(P4qv{|)H~g>S&QoK@*DQe
z{EMB=kL$nTWB)k?PJS?G^Q`Z!n&j$Pb>XHGE5|J;k^6e?fvr+FXL9Oub^M`0eJceO
zJKQ#6r0X9iN^H8b@4%G2{aRZ8l(*LS=~dE97`|p;__vK)+Wb{%bjR=Nm+Llh&TnC^
zlaDX>t4fFT4hyqCn7Vq?=D-cPO8z?3|K^A?13HYjH|d9YLz~Y((SGv%;SXlbwEyTr
z_?i(<+n$7k?ub#y?PF7Cza-TO3cwaTHs
zo+&CNDO`5efx&wZ54qj5cY|^p_B_7nRCRUg;f<^c@2~UKwRz=+H=dJu$X6rs&uTq%
zy?3|iO-uc38+Lv~?M3C&9kt$RmDzDer)@QRRBM}aY>HFI_s;g*)~ELJ?H4`Pe3E8s
z(Lay7?pU*7#qe5nj@2vGzW6!cQbjs!w@uaOm+{_fezu>tIrPCG7pIva-yiVvdb~EE
z@BOCRKg(Er*}D5TB9^(>touD#gATp2ue$kZl0NR6J}9xLsz?9k+gE&9dP9q$Ug-vo
zI-Yjiu+vM^+jn7;QsrFx
zjHv5(U_`}@pR2VeR*k+ryv*fHQC|+oJ9Nj*`JZP!v2WpOtKsupc4fNWCymp_D3x>E
zgX$`M(~9n8^BxTtvikQK{mPyWJy+(#=qcG&T{wJX%@VT~Fy&;2$lRGoKAeJ48e;C&Tb+;*hhU*7ji
zySY;*R7vSG_JcYLr+QiE&$N8gktI8`mQ8vz@WX-4MizHi6;$eK=R<{Wj(3Zg+-#K
zH*?GP)ob;eS@}WI=6gG2I525qr6hgrlU1E!Q`^1Kk)_FO^ORpUG{c(IR(tZ?Tkblt
zaPxVePpDGswoof)m?SI^RVgl3ode>TRz=b*A3nCcHJ?h+sL#v+oasscU+c;4jwy3PP%b)|KDAG
zQ*UWg$0JFyl$pnVv9Z|k38U@~=)XAls*nZ9sps`iGkfKs+qc)9DC_m@`Rp#4$U`(r~ce_s`UCZi#!kA%xKeax2NZV
zIpcONb)MCDNXp8#MI3AFI(Vg0;iP--^+=xW(>tj?+8KUw#_T%zr?}j8+PtDxi2~=7
zo?qZo^U5D}viJp5&t4{P)?!x&Z65o{y+!=06faUIOU`sPS`_G2_`}Q}
z+{@MVlX}Je3cm75qmrGwHd@%@@Y=lnedo+vT%h~(%PHzN*_^79>|eIc_HI8E
zTTy%uIWIE>l9RHDPc1LM&K(Q-`gd*}P_TE$F66{i>Z7u~yZHKj77*mtI-qS}C+{40
zeqWcvEzsXPN8@5X9zIg`1eItB#!y7lhZ
zp;LFS-rhM_uNTFUTquVdiv)eXbSDY-?tMB1`Sy12
z)IBFuXdRUTy8Cqt>=G2%xsw}o`nK-eBgi{P4$5~k?d#aZ$H$_+Q+ERe!lO`c-!6p;
z7xXC9v16gwCAtSy?nz1vgPvQWdqeW|7OEf6y>pLlegTzx26PI_8Cylw>OOTX8qlP+
zfgQVWqX@hwD)aZV$aU$_tpm&W`xOf45YUm7b|?D^KOr9!*d_+7Cu~CwHz&3sRGmft
z6VhheS(v1Je7q`k_UnO4t5@>Q0eijtJNpItdvUZxk)i>f{(i;WOBM<6bT8`Lx~RLa
zho`T55&xndtv!piE)vkDR4i51J9Q88?L_~@NEInB7)W~Bl=3f8vUSl??#2B}2DlgX
z_but}TdJh5yGN-KrAoAEQ@EIakz%n_)ae#T-%8&Ov8{qCe^OPXcnMEW&o+hK{fm$d
zMSY9-xtA>LS=8Ob)5EV#@sh<#mhf%Es?ZN!C~2spj*7CF$AnHzNe&_)2S_96}tua26gV%uyf}Q-Z@~a8&vYw
z+wCLWDb)hIlSChUZv3_N^7Ye|bocEU;LjCNIi2O#&bL$B
z0Q&!Je4)(rz~^*UQBMzl-!{I5-2+EFh;q@RDWHbtM*
z-o101pkBV+0?N0gmQriz3&gE)%f}e!e^Ri)Ms8MmaP44N^$B^t!a0qlBC9ErKV2HK#2}P)qN^ETyjqCQrdBL2nd0UcZlEh3gnegZ?T8xwsE0j1N0dIZ?Rgb
zN)QkP1OY)n5cp>VgiQW3Hj?vF2*~T90FzMam-@1ZkRTuk2m*q@JA!}^$akb8r5XP%
z0z#wzTl}SVK|l}?1O$P%i+~Www~M#bCI|=u{}BQ~4!M0~Ex>;SzEmp+2mOn@2)tPYWWN5*;wjn$0YN|z
zcoz^50{JfVqO?H}5CjB)ctT)iw&NRQEkHbhD{2LScLV_;lkZ4JN;3ojK|l}?1Ox#=
zAU+Y0#XYgn6#^L>4T%c^f`A|(2nYg#fFST(1Y~hfY*>Uq#zsTpfX}2=x5Ub)u{VNT|IdcD+FagiO9cFh$k7g}|Kk
zx0$d_{q5!x4V$#;hUz)%;uC2J?cvb*V75JQ^7$uB+j{h&s@(s?6KOAc@Sk{2t=@g_
zMIw?X2nYg#fFST5A|M3vJ?wO8^2;NT+}TF`wB@$Cb~pUx3%#afMC6-Wb93Qlm|Az>
z!E0Ldc8W(uDHZkRn*Mf-mRbY>K|m0APZ6+vPunfc76e`jfvjomRodjX3SP+nHO~0l
zGQnX{oWJ?-L-m;AA7*t_-Rh)Q8B*A>eEs34RNs#?s-n4^`T54Z2vvW?DYd$PZsp==
ztuTFyG#mPJ?##6?mg`t6of@|2s@iq(k$M;$#Ws9WE|se8nf$pn{8OTD#nfubuQ$|3
zAGjzxn`hVZyvTOgb^2PUnoQ+qt-GaiXLM9SbqHG$!yi`Q;$`bX%|$V
zh8a|iBFR-GF~AywPO5SKt}1^PM|J#ih?=nShB|pAL?t6KUbAR&^-0;(%9`5xFeFM<
zA9zA_teH;zw)eie|2R^8+dk_vO_{a!mim?QM}L}GWl3ZIOui%t0)oK1h(NV^+3Ltz
zfW*?XLM9VSPl=uX83OI9rB$DNbw)KTostRC(}oQB>52*`58}6&%tGkw~3`Mdsfk
zwTq`vOLyE=i?`lUO}tY*BmN@WI}xGXe>PY(D3wB0C!%PEoEaR5o{{)GqO=74ZAP4{fna-B)fgtcN2xLrY
zM14SMD02K_?5O-Bo=(A|2SOJ+#MSnqxsot(gf0+C|H^5|EBQ!86IZ~q
zXAWn!8QxiVTj6~KIn18kL4i1~*!kr8%I0NLJwqPhsSKKNQ9TNcV$Y_K$JjlWnBqbn
z6H{-At%AU-ARr#dS3yf;-&+J)R!FUyj6Tg|F*XR7?}RHW8Y8<-zYsfFIoeyPTlXUr
zymc=V($duM(_z8pJ2cIEPu+bKscfi{yF>up(!ct2ZmE1p*dsXe&s(ZkZat-ry5ed(
zrsapz?&$}?k?d70SHOk6t|zaCGO?^&C>b~9I+5#4sq7dZvu&AE+o|jKo?Kg+Ho2bm
zH)U{#=ksAmr24+2n{u!d5)vC5i3
z>kP3@uq1f~&5$95{x3ACZB~`yAEsNK9IRDX1nnF#0!ZQ0;c2>r>FxQOZnGyY8?Dbo
z=tEQF@c6|gbfEQzZN53D;8``p%h>+B>G(p~o!PUwc-vitIHs?Qk%w~Qp$7`n*?k&h
z;40G5n$oL8wqu9Ox5HJsr^w^eZHK%U5#U@jZQtp;3u;X3EF6!mN8*BjARq_`0{<}r
zK0}xHl{xMIxD%xM#DhSc5-E6j*YAJdw-}wCcF}y4--Tar2p1dzTy<;rKj7Uac+N))
z1h*V{gea%Y-*}tjFZzHEEU?T%EmPU0UcQ4*qBtZ}m1Ox#=AmJb&Zpehw7h*{QN5GaA_O+*7E^r6N4xcy5s4!pL
zaq4+>=V7G6dx1Q1IH__4lCoHvs%g~Fg;&+5zpL`mmg}*8u53avB8+=!fhHD6A{5T)
ztU|)0RXO*VIdkCf6}b>6(qd99$^;>!gg_>aeiADM0YN|z5CjAPK|l~lFbD{NOfX%MAPl*`
zZg5Un3y>hXLW~du1OY)n5D)|s2LeJS6GuOZm4bjEAP5Kof`A|(2qYK;gg_>kt`I{K
zD*|ol<%=tyI)){&wngj~1Ox#=KoAfFUIqankS~Lh$h>n11cyZ_?>@(#c?))28j5rL
z3eaKK|7&^iV0^Cyn6&DK8b+@QdLP64-1hgezopHBfFO`~5Ri98CZ4_$OaB7|BIqUA
zct_-mg85F!7s*K8lL$oN1RsL+c}LV;fb;*3
z(D{5B=(HId8>Mg(O;hhw%J=KDs(9`se0JZRhY_lF@f3W{;ga4tRZ15Uq=2`pHX3zW
z`P0k0s}@eCFrDpFECrv!g0l)It-7wN6;7@;A9=_p*Lc#4xP$7Z=d%G({=m6lHTLIg
z3g`djpaZt?67fIjbekW#xFP$A@CTiYGiKQ}6-9NL_0R5_9o&mw;Qi~~`|43hl**Hd
z&Ij~Kr*fo!{(;@$5z%TQ*@b%Sh!EE=l|ucx=f1)V$V~#ghh82&^|zaJs!S-q6ur?$
zsZ_m^DcGKuX&2fvfnF?*lVxxYV%ZOqs&>`WC`UVMIt)8XRUddlb)>Twe(mm+SknK8zZ2)Bm`s{va}l(_lb?
zXVPgfU8i4EKX%WqlF%XDPlp+Fn$E#latY|XIm%wsi$
zh$BcW!iAgRl%HPZnb)oHAagAMk>SUoQEK?2t4yAUwaBdg35P{n?We#zPNj+FvQZ%4rxE9D5j0CG8XCp{F?=rDzgKJQvFDq&{$U=vJ|i6Dv312X3g>}iIvh4*Z7N8=1VKO$5CjB)w}gPt(B*yQZ}ztYwp5dN
z5J>x!E32G)QZ;_%b+!L&u*#Oso|)ixpJn^U2I0W#*a2)4bGQgM?nkf-F?N226tDK*
z5OeA`Nl)g~b`~{Ye$aEB=6IhDkiYfd$$IUHd8TrSsrT}oaCQ1>sM`P48B6gnng`F6
zDyEJXwF~8MJ&07zW0l7|JXjufGnpZ6@)&(cJ4bWc-~E15oxBpFP@ke7xFazK4|e=?
zKwGX*UEi&_sb;OaWyubvGWqF%ZwZ2cARq_`0&fR_m=Wad)F-tF0to^Et7o~1#jdBb
zK(?yUTrFIYaDCx|`!FscEx7^L-;J3|hK{RupCx^aH$^C`^vIafR
zo5zJb=I5_cy?HZhk_EDMn-+bV0rTQ+M2@@GPH!f?(jDb9l6&=e3p=3*iy{%8#&*E`
z%RGnvzeX9^eTljC*0eB{n#19FZBI`mw0(`V|G1{WDEfZfZoAw(Y_0`(KUyrU6a)l;
z1c*SaS*-+UotPvD#1jITYsNyt^R%<$Wdi24*ByAEmTkYQYSV6-xCU>Uf4xfHG`wIE
z1n?Aj09X8V_l4@x5D~n1D@^gWW4W!AmJY5D)|e
z0YTvHAdo=3k?=h{?WcG<07xzGF9PkVr&YrjUsc#0g6+t@6;uBs7}?U=lPAwX9jDzc
z6>0HcTmzZawnoA5~VNZQvUZKTx%XoFvj2tyc8QMGG8L
zE8=;o%d`u;+b2&ZNA;ComgnxSfnr};MEEe|x@tw9K5TJr;FUt9r|EBVfnq+I3Leoi
zGw<9Px#TJnGqf4?yi5QY|2n{xcGTQZ&BmNzOtCBgMAs|dKWpewFTI*b+qxHRy~~R#
zD-}$}yOGQi&uja~UMZCWd5DM7i!-kgX>_1J2J&X1E$X#mcJi3zU@-1D?JXs)l}&`DKf|wj8{6
z)d!tWSl1cXsrlD7LKM7{2qd20Gwr^awo29lypvx-Y1CUs;6Iqo7O&%5?`5MxK|b!t
zU$_~@)7h9xeckZaiAU<5vDpS&x0lk(5Ai;m_zYMFi78)9q2nDy@Bm_aH@0F+5Cq;N
z0`hz2O`<8f1c8K$KmwW0PPhPxWrD!xb1$pEjz3bLluPxGZ}ff@lnxVV9P-l@-i?eM
z(k_W!czlKk+8jQ9#dThzh&OwGK-1U5$oq-i(-IOJ0*S}pm)MXMdj$bO;N=nc
z52mwU-Z+sI1Ox#=KoEEZ1cY0@0$3vY8VEG(-IX>1x~b}w$~SxsHB$6lLEt}_&VE;*
zlXeIKf`A~9zz`7casulRF-H)1cM%W*`R?|uv@D(x!21@x`yAs}?>w)n9W(`AfA}d&
z77p6RYkd%KbWM{lPh0X5dIY^3o45O`M+_+-itli#(q0J6E}
zUH!I7yIzO@UZ`zLn@rwjC|4kUH_*gqTMA@NcqBMJ+bg;S0YN|z5CmR~fDp(R<0HB6
z2m+Ws#w%^`8XcUV;%INJCb!F~F4OUW6Da)>)-tL@@st?)wcHdWIwf&QA#1jl>o!%=
zsG3D}TTvD-F+X}KL`Bdtfb8^=^^P^usSjLZ&b!7*Kyx?VCafbF1Dpr$OfNAvk7Ilh
zCOCm--16(ROyB|UdQa+P!}VA8NTeRrXL>*q1OY)n5J+GM_zYd%S6E~M>l863{ty5Q
zN#ba&W^{7nN^n$7r|&KhD@>*aG|9v-PaH7sGBKu`s*8_4{-i)wNA(#UZIjg5M(w1R
zJNBJ@N#)AmsB&j|b1zr?nO+hJ1~=-{%zVK0%h(kgC7dzX@w@ZPF*%u1*{SpNvdiAHE;1wjm|l2^V_x>34OR_GrBK!Bc$rrTYxh6kV`^~7
zb=PSZRluZksxj$Y+9#*Ni!l2WmR@uQV@ekrK5C~z49(qeo3UKbBL|;%
z2>B_$-B4e(c2zh&XFZ)SIhfvqRJ*vg>EAHHocxgs!Rp8EIg|_a@$Cl@3g;|Xc?l%!|$n@mZH|XrYJL(&9kLJy+-E%KvSM2&B
zZ$pgkvl?mn{6n}_-Dx4F=~k^!FsaI%+D?J=;Y`DNB~z;1r(VAet?5j~)O5Hv$SMxd
zb|Mm6w`2;&5(kCDEeLlfLi!YT3TfyAoVnPF&Q%0KJa(i_2XQ5CjAPLEsG|Al}F~45+9}xCo>ouNdY@
z&2X6xU%vGqQZ@hTjG4qox`4n06S;9Bva?|G9XinYp1S)el1bWKI%>u({hJ{scOFKn
zCEM<*u65F>3`DTZ@G^FpvbbNRc4Q)gXbpz=oUrZaBQrwFzmNFmna+AqWTpf`B0KMiF@OH~t${t(3g{_6-9n>RtW40F&(({Cl=F#rjq`W45RpDKoej_7n+U0r+NgdXXH5!6
zbWtA`R+>TzNreNVE`~HucdN;yJMEo8cBqIv&$%Ixj{aXPR5=>
zQ{!k4;s?(K^IA`MRimk4xo9mXNFxsJHeXt4D!`O9)=b7{0G1%VSQ>zU^`Mx3HbYe+
z(!cGw$7?!q$oD#08+m}*V1|Fg1c!s4yb`LSNF{dVU=6IfHq_K1cR}EtML-7LceZO^
zzhUN|?60rsX$|q}jeNQ?c@_i`5&}txXnY;u%G-~djXA^9yx4|agEpCX%(JF=>7(=VuCB7pFyy~wVo>qk4rQ#;^+Yc%?_ilp^`
zIWs!YW}PhTZEaUAEpH-9LF6qdZPCWdA9vHXb2Bt6n@Wvbe3iFxV`Ix$KUXvBjop1|
z?c~sfS5)f>XL*?bHr-&2q#246u2_rO|HsQjEcF)cvISgt?IdLWt*>s#EgJ=TVJF1;_t5uIY9e0xmm?;le({KS>i1(-pIt;cVf99@J=Dnlip!@@LUMD39lo<
z8_V$~a|wcgARq_`0tpWRA&?2LPsAcYKoAfF1OY)n5D)|sGy*c6ouIo%j1vSB0s_UF
zOwJ=~0TMz_hz){(ARq`NL|`2{|Nyh
zlmCQ_q{kluGQPwgwxU@ONKgn^B`9me7(qY~cykB{fqZjVil&!EK*p4pMM~rZ0YN|z
z5CjB)1c`tU$OPFfVw4~t2nYg#fFK|UymbT~x1KUc)&jgT>_R4A88Rs#2nYg#fFK|U
z2m*q@`;CAQ$oISdrS*bl{H-^T7(m@KyiH1PZZkE?bJGzm{$~lR%a&n@}(MdTvI4B46PWiTWcFNw)LD|^aav5tI
zYmQr6lihUTyJ1BN>1GLtQuP2t+saOw~4n}{$KM*R>$
zeT2S=h>9dR(qZ&3)Z^k$9pUzJ5>nX!j_`Mfg@GL<0V8~e&}&xQcX(E+SEn9k@i^U|
zAd_*@AcX`0K|l}?1O$OZf`B>Df?()CcMQJfGv6^n8~Hpi>v&9=XB(@jim4K03Dkv#
zZ;M`I)Q1S`)UqHu7|eOV=OkSgc5<>ItDlJi#3qj*19&ExC?*`5pn&LsNJfH)FhSI$
zk;!AYK`IS`5=nQYM;RB>{N~p)}-!@G$W)$OniJlM!wbVn$o&W(8ZwCMzP$xX3U{
zjaCs5A>tKn4QaFyL<}+LqJVbN4?i*S($7fewxbxALIl`EE2si3(uk2R(nr(|N;!b2
zLK^+1i7N=Kj%%_@&mg))46UqeSb<3nxvxnUcQejP>Dkn;Ye8ky8hzs=Z6QUCG~94;
zKU*ZI6cQ2m4+|hdB`J;n*3dbI4o$P8Ne(cfE+&SoRSS}p6rp^&0e6iHl>>grLOUp8
zWkdBK0g7HT2!VV}0Ht`sN8p!Veo+@LT(DdhFJ4sl@83`OhKq%9Mj%nIst6
zC~j~D&Wfo6w1|ZD4-+TkX#zz=jYy#;i*$!)iijGB9r8`W7{vq-1k&^>f;^IE5#BA_
zK`N;%u1KVDJe)m|@HWCL1~SPW%1C+!v7|bn59CgJ{HQ*vn=0le!o#OqO+<`5ZdTFS
z!x|k0!iX{?%DA8iBncBZXVDkbR!sn@^^pjN<$pD~=TXNEM?+Y>7eOxCq;%
z%NpfygHf@{@*~QU0j4_rRiP(}Sd&rkWa@x6AO`Y`e51{rhE&o)0HG}!;7FDSRPgh1
z9Vkb^D8li$eh7h#D{@bl`@PC60@F=ZuT;L_(|PjzpChno(@V`0@YT7BS%M
zB7i>@4j(?ums#`NxpP%uV4(8$_BNB`{$3z}fmo9T5Sb?ekJ>SVxX$At*@#kEJu=hn
zsj?)Ylh}u3QTwcvuHPbHM3^8!n6jBPAswQ0MIqgAQJO@O$s?t~(+6V5UPvN>Adhr2
z2%nxF1py3)*AY{sG%d~3r9=!(&m@Q7ckMp4yvhNhzs
zMyIin+$Pdu3HbWpD}}ZqjzAjd*?6F5eXyr
zo;-O}nKET8UMYB~Ql(0j04*1j-V_42Z{Jq%I-6eTjT<-e9}-==c2(7@S64lH^ic3v
zN_ZC#;8F7h4Z%Q3!!!trWzc+b@HEHWSmWyx`eMpGk!(De!!)V^9nciw8A%NMCd)A6
zR8-fk2?EM6KOkF?4lk5Ro?_P0!6ONeAv}psuCc>9NG6XUencQa9>YK)!$2fyonsg-
z5K77?@~Ee)m1U}$i7E6_8OVTeMl$gXCtJcbeufv%nx=nki1D$#uoh$v-afk9k_VC&
z5mFj?{-_PSmXL74#Ix=Kv`_Z|x8DGZBq`s>qdTTz5x5SThP9z*o}L9!q_#3X6lX6b
z)(!$UM4*Ou6Hkc};2aAnJhAn73E^_Hw{s4l3Xr~ci4eKGyK*R|}DnfDe4?QD|?It})
ze^WsSAwUf1Q=8?nDT=;h{h_5B#hJeh#hFa)mgFAbqafjC~6^?G^q0w
zDVHaCi#9Ylzxjp6y@I}s&JEz^`Ph#oFXQL_h<$P?sv
zZ6lGsFpx(kksy*`Dujq*XlMwLM_fb>$-{`v8z7L_pu!t#s62>cwBfl7C*lYa%IhOZ
z7rct_5L(;X@kSXN8$0rt*&5z0cpO3QsJ#*7IkdraF*c;wDO+1K&Q3)V?l#6WcN8;1
z>I)))KvnZ227#u@ff)jxbqroYV2!}jxzs0^Vh2w}x-DI?F@q$CCi3991@p7Bw^I&w
z_Pk-n))wASv#0T$(3R@i^e0!w^_$NiUuGFthji1%m?zzjy0q?CEymVcJ`kac$s8rNqUJHOJY>)*I$)!t|s!W+OykvuI?GGy?N~jCCcq;~`uL6n
zec{I5NKT*!3fvU*7z*N<5;#uJoCG~)oHCHl8AclVR`)lODG?ZQ9(^#u#CM8vbVV^W
z8+t7D>e8k`u!LoGG3bh|0By6RP+r`Fkjc1XCnf(0fmq+ye-eqWbgX|wfEQB8V|<~U
zc=h5tf$zq|i4)l~DFia{_T|f44)THl)VvTi(l033V{%DFmxbpj_^-LDOt-fX)wyfkO`^*#Dd8qkuSRQx<4Y1@T|cD$(~3MK1?^$
z1p*1eh|M!t=g0&SYZ@_)O;>POFq250&IU1r$C1hd2DAeDq1&9o29eYoa%dWxJd_de
zvf>|v71m;6!w2?T(-b$OgO&g5Drfmd~H6oU_MBXsf
zO~CPH7u>0Aq(R(h0-W`**NkG=45PWY){jpnc#zT$+<)*c0#?n6vatQF)!7GTZ
z9M;rGqNJ1tFDHROB-om+NXn0-y2C*VvB8G=1{W};B&~zurU#JMQG%F51_YBT)%|JQ
zxh!Q<{R9)=*+Png&KlRC<1RJJXJ&)V~cHyPeI(@?^G|zaPtkDkbq(c{H{pK#CJ?5AdxN;
z5K9@I9oJ7GlW|2($|X1iOfrD?14w%fM?qV_<~`
z%y`r{J@8rvWEzNhm#js`kfvK~86Ks<2z13p9taPSQ1DPAfJoudaO5Fo{D4pxamwUZ
z>d>7}YoT=ZMq(P9Jc+#Ck-Uo_hoLlejLkD4wC*uD7n<%i2p0+99h5YC
zO&&;U6HRX;fOth)({#8&0zp1Gou08y5>1B?zm^AilweL9!)6krBy8jX?bJpv)chuM
z)?^`7@OGM=YG;cb;~=hfbfF!D0op-2up9uGI67J}X31QVu@~5uZPNxaanw;%V$o|7
zM<$e9nL*;Xp=MdoSJ-W|1sLET+km_F1k;WBnc_sO8GnjXg!3(bgpek9ZdoBWoHc8M
zD2i$|u{P=>8A>t&1n@&?#@)<`>s<(BT#=J<2@V0144C8~!5Jt9B^U(E?}+)GNic?q
zA#W4`49W(VUPMf56Dh>2d*E>l4hc5M<0JL(
z(F3|3st}OL$M-4y5pSxAr0vg4cnp%mb%0bF_E^)}L9AOud-Qrmn53s!vHsG0PbEBe
zcC^dcmUcZ8FVle)8;t;gvji|a8-f;ui1g82Z-}7>0uugRdM!5nV|qj~Hh_V$bP5O$
zY(X#als4Tta3>J1=nD`slmhWWI(ROVG@ws}Hrz@@kgaw^xUJ|8JYo3#`}ddxKO(GB
zr%ug;97iE?kCGa*U(ILM&DB$i2^wQb>excm3junfJBuJn|JXuGFzwR)8$%d%8}u{2
z7kK6X#S}D=)N509)#xLwo9$$~w5&-a4MZ*NU=~ic0NK|K364OT9q_shlk)E$0(c)F
z0^a{9K@fNm0_JxlsAbYoO6j!#=J)4CihA^=Wd3EBlikI#-c7BM4YPG6#|>CZ6-
zYbGofE$|E>#~AccfCp(!*m&@VPF&zt=G&y1Txd(!9A*ZZ6hJ>LjASyQy;FKq20VnC
zRD#suRXmpUkmQLBWikmb}gtWNxPpTB51=3y-vs8hR&d-DPa)P
zD4OPuAdEmBZR!1zR+NT22qTkhVnd)u*8H9zqB!g|ggsOPZO-9tgFIEtQ`%GtG5}TE
zM`iFx76C@|7p#Ox6rG%dbiJvD)?$J}kqnbG(*NKtd2GWT(fUuNT#~t{2M-^plqplQ
zUb7BpCUr(IB4|7X^gQ=58wtgfZ}g=$N(*X>Kr9i%KMF?wnxv86cL{_k9o|yj@I-?A
z$|C-bp$zGVcbC%9b}q|}r*wnR8epIXwoltmxO1(rA67#i@z)GKLznkWaE$T|wCgot
zk>ZIJ0rN~ld7F4*?ZLNaxA`40zcX*IW~u3IBA`cGjM*4ja0kCI##75EYUXPZ%^r}3
zKzEGXbmy_3qI!%n^zk!tOs!mqKbTk`#s!R#ACE9fYonX42RE*PBIbNE^5De6o##2h
zAo(3Xv=askB(cDeF1RB@u??H%v%w@oLNtpEdHjeOWC*cH?5x37X)s7;XRw&-B;&x=
z=*PlJOc}<9GGHxk^Z^+#N&Fz-<&IIIv~8RgW7^t=(K~bCqO@UGCElN-w?)InNpWkv
z9omkzXh(6}hPP#7K_%W^ZWT?75{dM&YJ(;610|rxP&aI5dPp}sBOkkFbfA1h=|U3c
zQvoc1WLt>Ug4x;<+ocqB|K5Fd?fO-E_m4fX?qHSlQBqZ;NHMZi&xNxpLmzqo?yqQiGbMW|nGj(<7VbrA3)u~_
z$Xt!?z}f6`i0WNBnoxphn!9vD9*3wCCr?t{HtNF73)C-GDpTf6%GoIiw+EVZAF;+5
zUdC2&%_M?`!%EnJe${ae2vW5M#B_iR$`}_VTArx^+&}0Kqi;2VB(}(Pl09T2+KUCH
zz}uR4E5S~%ZqjM&EJ9tTC49Ghz#e+Qk0lM#aSuWu#k
z?*(8Y#|!IdAmstnax?lcb|MB_kQ8IkWeFqSEQ~bPhX{xm<(ncXMi0z)Vx?K*$QF4zGEDq${FPjUT#g_V4v1nIwf=dL<^`m8#0>J-nd-+ORRWy$JFbK1$cEbkzK
zO4EK)gJ9e&akKBazd02jz{&s`hQMOF@gD}rHk5D(Q8^C4vNb!IG+Lr99SyOeO_LHc1#{jsI7a4^o4F
zvFqgln^=Y^sg=fjnku11mj=f|44)J?NFVmY5tJM_TNF{Ma(|
zXaI^-^e}&N8tg>pAXUeibP`VZ=CC~EfHU1$D?o=l-Jyw|v@CtsAPPJ=1F5<~{zC>)
z^P=c$1tE~H1>Y+c=1zUZG9oSryfFmuYePcfL_mIdyjH&v%dij)jIQi9!jOvqY<1u)
z`fq|k+@uanN+4s3kr2}GsQ*Vn8uA#>L1HL_WhsJsv=9&8IvsV9JY}IA2jix5P)z3&
z%hGc6Y!V8~nmHmD@KS+1^1?*A@xjz|XJUqYc>PQdB&TaaYE~Nw6JeieBgr67dpl55vwM#^
zd*-~ldG`UWBaKjYBu0E%UBCZO{l5NBm5)xVfOi*oXfg_|s1cVAj2As41=XTvv>NGb
z29`IlR#>C$f&|O!!uq>tej%Y2Vr(}QQYpSP!R9y>h6bX}Y!?m=N1-Uf1@h1hstJ6l
z+jXBB5z2&etDS6MjES8ueNLstz?i
zTdR@Cqd&~Wz^}LKA|68`asD8l$x7lq@ao}!Yks*TlS?gYZq9(Eu$9Chtf!LAIw%R$
zhuH_44k8m{8$*V|rU+z=XYd%rP0|%ZC?QQsSP-owAyNb@G=D-Nove?Kp&WxO^7=j^
zL`X+@lVoCBHxo;e<28)S9!9{R%%0tQ)v;42)JJJbY2)bgiC~xwSuTJ!P(-84RU?5?Sb}lX2_(a2k|bww
zL5+bHHDu`lfSWg?jXJ>~i@dEJH9&}nCG{!4T$3x}N(eK;0(vZg;=mJWz|_dn-*Jmj
zoFYOd;{=-&5(Hj^0RD#>za(BHCV7H@AQ1lu7=tNqfH3}FVnBZ;Skg2lFlCJKmIq&E
zUl@!Hf*GU3{67zthI~?H(Wj-f_mSeJs70|BXO$EM8G;rl&>}R{n}mQ!1L%$>lki01
z1&-uuBa(zP+?jadPGk~m1kEXBUdzaONvr;06G7_qj&al%Lq1R?S^`;+5^XpupbWwY
z<;**X37+{*5tK1RL1sYata%>Y*!fI^5c2#oP7p?_iK&>O14XqikV)vKJIEtElz0Y7
zf{b~AfZ_r5BA=qzT!ggC7cQ$!8@H;<*KQKwic-#0)|UHFsZfwG`UgUG|L!Ao;`DLl
zU8=NJ#9GV>=x#o99R?ay0otc)ghh;?$7TX@*FpjbJLEM?}FJIGA@z)kKW$@FJ1`d)SC1
z!4s)pD2X_TBg#Vt=4#tPBJqAo<}xrH7v)2}5hWWfmFWx^fJ`IAkVQ%+9t0?m4c!PP
zapQ{VnbmQnh-=adFC|$q5FFx!MlY
zXGOSNyLwIi`R4|8{n{<%OhujSh=7vqFq!rqDLZHph>u3Jl5OkXC#t6xIr)HLm?tsf#xn0#}D6-j>~)|FClqQBO__eqcJs&^?iI?
z4Tvz4BI*lji%GJgq6YX!O;Ad&1t6~`IW(D|kxor6Sv{Gd4bsFdO5M48Pp$oPqq=zc
zhRWzbw1=|UlSy*WO}vs`Xl|wKQGZy3dPHeguifAS$DxCfXPs0KKUzJbs^nSgjgd}x$}%nVil0IzUlA}77X$BvJal`t6uMtYN3>TLbVs$&&0^C*e4P>j+v>BDf22jfBp7y}C!cjJZ$P@O@Uuz`NG
z_l?#F^2QYH8sv3?T%PD8$RybbIS@b`SQ5!Lvoedum?s^BIc76WA&?H@Seks$o#Qc?
z)C4L;Kd=+fz&@D4#0GMB=8Fgpy#okS(pcw+IBdi@gs6l6ff&KNYG-&Ti9p%XtAC>C
zqIUq%6fSMNiNvNGIxUgPV51In^QIC4&|F7(lZfE*rN5}_H}9wnj<(9d7S_Q`BG-6-
zB^d{T6%tCiDb<-=lpzt(^m29_{S(R!(R7T9p&*em*sq_BW*V91jWS6m$|3)Wz`z(O
z6i40YBi2i8v@AuSw(~kykZFoDnM7IO$N#7ZrQ=mfMl#EDpFkd%82!wDlyFfPjR@MT
zG16t9vGNI%U*FXz(I7IaYr$(
zL+6>YNM=FUfdWQg8(=4qUTZq7#+vT10z^_-(MdJbF8EL*0Wj@!p%`TwWDNEaL9(Hp
z<~DTBVk8du1|Fo3sUZ(*KezgPPQtgv%LxjNh6&On+30FcB|J1C
zBnc!y68%K&LV|@bf(YU=(xBEt3aPPJ(+P_)EsrA|ZKxioz#V0Pp+=oPIRY#xf=c;D
zcLdJU(YSsJfs8A1Qce(f4-vqxo;UD&=RHJDn*5(3V0>Xs;s-mB#KS7*=>b)X#eBng
z&^7ZZ89A6L)^gYY0u4q$#3*WdFEPv;93rL+ZJ|Y}*yN5inC0n#IVQNRi)*$
zuvrMX9JozvtA1wk$dwrdP#B8KU?3JaJj$2t;@(vGE3ZqzCOrr65bx
zG)f}MNoy%-FFNPbzH}mwq3X}y*Q=9fE~!-Hb#x)Jh$@`z=@mir>}c;mg+b8Bv#G+B
zGkGUfC=LlHFC?`Vgi1H(e6uhJdES;W??NKrXKC+eTKwx=$X!orn8tabi-qW5*V7zO%~=rZmAmi#4^|d
z5;ki>?TGU;eLOL|vr?^D6Ew;vgG_H6rxC$mnr0GB)B~kpU3_WVK&;?>dm^ws&`GJd
zsD8GC;%1U5&oU501_j7z@(f|Jni7~Fa;~8Oa+Ho&=~xpGun4pe7|?t)ekcZ6kY(c{
zA{j*_)6R;HLPXo>8Tla3n9JoG)yHi|Y3QeSKknYOPaQjRQ8`kc6MJgWdiXiD4XD$>
z&VlWZ0D)6s%07~8pgs))4y5DO?He$NEySnsprcQHqFwqA11`Y1oW=^EpZ;kfz9Sy{
znMov*MzRul6z4UNNF)37rX$k?`s0AWwFbka_Tf|_D|+1z6G?hzRBIV;k!4Fi@BuQxzU5Ij^V@^=Cu1-Eg@vgQO2Uqs<8DlYRUCRy+Sb%&aQ8owZ7lgxrv{01|Qzb~Y-BvlI1~i*j&E
z%5&Tfjt|wN2lwd+pGS1~IQ?Zna#$RR`EA`AQcO0-{tLj8K-Dr4R)nMILaKxgIxn6oEyQ7(=gK-ls
z@M}|h7%yOi3Q3+m`J$9J5`;j$5g0{@ARq_`0)jw7ML-Xjv4>3zr3gk223a0jjTA!^
zQXJ)MkVtZkMbjd?@No6`;Uju?%oUY7bs8R^9UO>6!d->{5z!{W%-z7WPt>6;5;f%Btp++>v>_229OAx^M?hJOdgO%g-}Nn*#q~Vu9P+L
z10>i5hz$DSTIjDp{O~9%9gg%V1y(3B@CcB!NgqThY!K{s*OG
z1s#?RiUa-7p_>d+MQOxTki&IbJ`GICsLK~GtKEAKDJyc#rK5T3qz=?iG#_qHd#~*s
z>}e+t#T}iMWAZdAS<)12TNv%0adf^#`Y)5l$0|A`oE90%Z>6v%mQ19vX!;Oo&Acy{-Cv-QJ
zv8MhfVIt7ip^-6ACwcs;na850JXIFiYLcW*GTS=j}rkQwNdL-ecJLl
z3g~93ykz&@caPn0=w@qfYq4x57ecGBnLCIGy|8(+`D;vp1MgfN3LbbjUMqY=04bh;
z3InPN2jvA40gW+7GAK!g^B7o7kmxmFxTwk2Zl=BZAHRsTWCK|6t
z&IpBDlvI@qPPGU|d#Qj9*abin#ue8v21%m9Di`H}T=5$Xl>mX96dbhmb!6F?Y`dZ#
zZC(DJ5&#wDH7Fx`UsyVE#E(aEK)~eX6DVn!P#~c|LV<(=&ut0_s)emT?BB8J
zwXe!I<*>QxGu~Z=71U~bOF;o`!V+!QAu?Smm)XE;?1O*x0lV||JFK<6+nO31t+S&O
zpjpobFRme@a17%M2C0`Y5C7b@nTIQs`kd4vAhj?>WkP3mITqJg9ZS{rsakP_f;?&6LgWYiZof0bFwri4XUE(AT%dHr}^vXoGp$qZH0kU
z=0UvUHoj_~fg|NLEQ|z91(I~h{rqSxyw#7AII9{~{Rv1(`EL!dS>rKU0zEX_efI6Ie2sN=by*fMssKP8T(LNIq59qI
zS^WuLc##J7$;6kvrcT7o)1|;=6c~`77lFoF^!`F#fu(~UAwWZ5k~%(j<_!)BZw3O#
z0A51Zv0kQ@rX;{4@Ca^lup*G68L7JFshJ9N*7|(7!k?O|jxr8($O^#75=nqgSJkh;
zP%ZBoz>z?na{*KJ9Me`0;I6~sr5+Bi)CS^p2?%Qy64&||tl2>r;d;)TIb-)fd<_3Q
zwN_;9nKafk*)*WB9v#ROz_Af-an>3TVQ0~UY{=*Ft6e3(wBRZLTAI`W
zEoi@NA}RtV+5Ta(>FW
zI5{B?Uh?bysr{EESmmIKbAdW-+Qr6Z`^x=?Z3-{f;!f7#QTx=16ZX*IBX;ez*V+v?
z9kLhQc{lhWa1O;+3Q0%aI;ek+J}n(p6deW1lr%T5rgMNo0F4qY2LS0nL14iFCR{n^
zOad4LEa+=ZK!wRWmWk$|yhBBf{0ZYw-~T9&Xv1(S_C$N}dCLBVamFlRHrp8;D
z0Gv582qRNY9-vf%A&diE08YIEV#@=$9|4m9OL-(0CszPd;8^dJXw8~atWUF9MK^V5
zaKt|SnR{#S#w?1R#Romqe~f9;LFgGLW+st1@Vb4Z9PLymOxe&
zKj*$dFO9p?lz_>dQZQ+sP#~c|LV@Qy1+??|#`)Y^C|l@7Q)e-cxgKrp+Hj+dmsWnR
z)@;_&Ba>LrlfV-jvbcb=GxPTF(WAC7x9Hsp;wT+IHDF);%Ga#rZ=3C#U-?yg^{c?3-HNLo#!WK$euMpyfIEVp3t5AO-sh$HuXhT0R!vJ6$89N##RCYRrI*y?@
zSCL?jgCBq-=Fr}!17ocVkmxUtwOG@{KvKpoDb^D>t^%g(RxVZSt6x2Ai{*+na;*XD
z>lAZXAu9Iz#;Pr^FIr1;Gk_9~VntD;Pt7QJ_b_LuG0!@DX58n1aBK_o=lq?q*UO&K(5Ok3`+-ua<#qew=Lm*9G=kS~
z3}VnrE#xH1#$YBbE@sLKfWRE0>ns-Y4FJe|zQ6!$v^nS8%-)_pYeH;ZSz5N5-UiDS
zW^8rvjLmYtoU`muOLN~!TV&9pLB{r7*Z}c&AGWXEf4{x)
z4R5fQz3k<`0W&nq)B@a~{>E9!{gyDj>fOr|um3kPkj`3Xa@{wGoyk
zPfSDUKc)eW&CN|%TQ^xMokI_@U{h0LHa0n8>!TQ;EW^3P&_>+8^MwZdM&_(gDA>}%
zB4KM5g4I0$w9wGR8UiQ_&x=!o6+{FGOAEx%;+!7QsS_wF3@rixX0EZ=dP5>>0ZHdL
zN+(i!keCVyL@uv-48|2et-xduIjd9VpaXQEHW@vuDu8aTuqyN|y;D8K+;4#825};#
zLLm;O&yG1jU9G}fDiY34QvxP;O2MRkLV<(=2?Y`gJnJbCK##+K&AB=NiolRyf_7G&
z#!ylf!-mVY%r)tqg^inynU^?4HNoP<>j_bTA2GIJs}}fS!*4-9Z-utAOI1Sct=q%*
zAGM!5`c8Y}>yOx*-uxCgXxYtC)JY6iwC!s$K5qbx0OFd&)c9DGxLUe?fg>r=HGq*a
z=@rVrN1RCa9Z8uudIDw=7mE*tSqn@m?_DE=F?0+Aa#Tj^4D<-x#ZWBTI6VYLl%bx}
z3WO-}s*2)r4dHdYj2!gPeCq*<3M4Kl1WIB}07Jm;#s=YOC=2xDT0~p7qF%#~Bfx8X
zbjm*c@lO-0FmPrYvh~)|n8n$9&UWqEXT|1L04M#>pYBkhVXseI;B#?)!bRBYtMyha
zHhD}$H}Zi?DZi2e^L43ZYoIT2TQ`&PwD%(JwWt`@t+hf+ab&mG*HxNaz>TCp=MEeN
z9uc`aa1;j;{m@m!?K1kQE>}%HELS4%=v-pj#RgLn!#hCK!g@oiL1DF(YZKw
zGEPyM^aAS$7F|!dfToawz${$|2Vp-V5UF*EBAjciXjx7pA6peP4@!g}Re^KW->OTF
zJ{-AlMHbtt>SgqCLE=
z;~Olzm&#U|n*_M6S_vPMT;KP3S2@@bc}Oem00;Hjj*+iGrJF-vp-o=O_OK>K!}
zqMg@Me2mY+c`O->DutSW9r1D;Xhj!PTfj+xCYKjCkxiXcWutel92@vDpfLw$5LG=|
zQ$q|k^+b6yU~d7=R2K{@kx6ZE9;)0U1mJP%3Vlpc{j1Qp7WkP2w`G(ralt
z4n(y|twVL4W5+th2@`hFd_9F`k$7AcL$R*r0p8LJl>bHnUCk|nK4V(0d30pHY06}U
zEwQ%T)U2*8+dM$>-uv#iGpEmEMXj}jPL~13nXIDH#yyT}W|P*Wfy?;M{zr8n}@Hl`A+yUk6O9
zS$O8OR$!oh42afx={_=k1$DXvd%R}Wpm5u~3S
zeaZ;LfgB)F#vVC|vjL6jI+a>+qXtD8N#mBLzPe(Q<8$^iKl7jLJHGQ(_NE_r3!rLC
z&KSc0aqpzdR*Ned=S-qwxvBpcbMqM2bs+^fd8^Hq6LJAdm1!OUI0*RZr8R1qD%Bi)
z$;y!8GS7!*T-U{LvI!nVPbDsakEkP18F9xrDi8i~PFn+8zqlWL)Q4?;dctZN8ZFn;i4zDHu%)*L
zw?7R|gfL?kJWV$kT5W34)O@WzL@kujK*K4fs?BVUjLkm5$d
zf#falC{QVmp?pa9iNfGlYqFl;#YFr>uOq|1+gOxL6MhFzocbCfrJ7H1riE8iz(m$z<2*>SB+RVE)evcmdS$ze6OWmVDbnW
z#Hc}+;$7FFXV{gw9_rQ*o~GUM4QX31RV@vY%YbskGHO7MDnz-;m?403&Iqb!wJ7ba
z!h%eL9*P8e&Mhw6%v{O-=O6yL9ml!kPyC0Uw9d{hAIFh|&>yIxRt_172`EqBwOCNp
zfF#{74%dYcg&ZXx7IF-%%HV*pwiabS*p;qW#~QU7Uum3YsQ^kX)02>ov-zU)u;A#L
zDseqjR9{hB6;Q6WdaIfvN$4PDQZaHx3(dD4OPJoCZkwH(a}VITY%rr18Wf6@Q%Ipq
zu_4EX$L-9(89R3DxK+zl+t*U`zC`_wmL^AUawsVuL+khj+PCit&(~3Qc4)xt3^h?Y
z%t7b=eTS^8yUP}5N31+GOUUeXoO&+WG75=I(XDa{%`J{%wKA1$t$l0VfquG34KEg&
z6YGz$^wt%QqK)pG@~SIjC<~@Fh%HBu?!O1b82tGNJ;HvJr41-kb(QqQsUMkT>rABz
zSF^E!@+lqNLm!*w9g)+sG+`O@#W%jx`VRjsMmJ$HDGx{a16AC00NGR~^
zqCoN2K7Yt8(FM5Zu4q#gE>zMMx|zI8yQ}`hETbTj1G-Y6*A5%6;DNW*Cah}I4p@^b
zG?in*1J@j|Q}-P~CaQRJDA@sw(g2#KmKJO8>a}^C63TRVc6!E2OY^q4Sf-yc!JjME
znity=$CD^ce*QD}+UU?v+mHU(57|vO-^pM;Q3n750eTBChPZ)ZPfEV4`OCCffKtF!
z$4FsPa&6`#>l+80-X4e&x{cPuIz)$fZ2~2{0_afBA%%783Q2W66ONu&#pta|l6^BB
z`T^d^;z&xH%uOJ1Z{Hp}Q9bJ3y|d6M3&2#^ntA}?^3sYeR2Gr*uG+Cjj@s#gvsNmV
zVKJs`9RO61lu7vBt5gaXVyY54niSDz&rRD%b(x)OIDUQ0}
zI^Lxmtb1%v9!!<^ebMDOZ59X;>$%xkO2FjK=$EukC~(OXm|I-4Klu2hec_Q38sTf!
z-k7qR_vh{PcemQ^ju0;4k~@8h8hrRGbN1OI3-(`sVAoSLLegzQfv1iF+I$^p_;#z4
z3zN3dkOa<@i%^jl#*I}*KE4zea}>yGXeii=U-S|?_0>mIU|A{l0o;1J`>m_L&sHiG
zYwGF*?PRS&Y{YcF1*ekrRz==j0yvhID(Db~^Ya1yC~l#jQXcXEMF)^@t#nQE=a?42B2-Hi0O|Gpf?vXVMaV;adS!h?$>C!F
z$+{LTKhEl0eW}>-h%V?I9aJVvtLqLR$qU%l3qV4f9A?#HBZGiOaU8`v1L(k=TLD0?
zBeS0&820S!tj$bK;}dDn!FF$R5&t}Z?AelA%%szGwumBOZ)b};gKR1g-w^*ijg8{;
zAs^A0>e2JaZLjTgOr-&)S#|uV_iW
zE-g@N_T-Bf7>o;)*J|UW^T0_|KYopK9aYOYhRX96P}H~qP>NGYUffE`Vr9#CT(uVD
zQy+DhV2kkxxjb!vBG=(q%5kSm+I<|(eN3HD;w$3D=u|Tg5LDMLKP3S2@@bc}Oek>4
z6j-dT+mF8cob|NS+q>S-XL|sQ3*|MyugrG0(Sx
zK;L}BE%s+9_{r)){$eE7&JoO6&;Hf7-irC{nhl*eYRzrUI3=vuB9;`hm=v#}1gDL?
zjPjq%xSIi2`gfd}oU?blUB
z`c~|~EbYt8%PXl#w6Fjx;8i&qiF(sgsi3wRN^h?owb3L(&-x4?QAeLi)OQevOYB|)
z;DZ|qaLi}$?48RwedLKrU9h-FZA
z%mF3`PJPYJo;;32#&t}&bIuu_0kFzMx>#&PoUV0?e;2JOtSk^g#PAd?)URzLjIM1v
zb?~SnebI8k6;a9Sr7zzAAR`dTJg;tdYxVh@r~KU?nXt;rx_$Sr
zp0;AX*53EFy|x4h*4(~$w8WQwjeYAaO%8koME~ZVS?8=aGAGUByI
z0JJu&ud@ekSk6}F7669J){YKev9ZPa`uc3Sg4r!Pc>M>jCRXC44Gf&NI)JBu&thrb
z(3rFa#I#H3{*{+ktN~7&JeAK=_LqP8H*n)B_R4SjHu~BHzyK+*rT`Ym`W)d%@_*>y
zPwQh+VQzRNhvT@6ZlHjO2dnlUc477cOmR|wn@MEhR9Ah~-WlCeAdhdzfSo*s7qgAlQS5OimK%5&m+oQ}AX!xfGzE?r|2fDiz<&SVOx2jnryZs6G7*6J2J^*PG1wy6^1aCvgcG7{tC
zc{~rNa*g{k3+J)5tKTwkgcl~qG2b3VrJxLr7jWjeV$E)zt%U?Y886D07Ol@$N#1G3
zy$C!)L0YGlUyeV*^AVkis49s1i2-HIG4!MNK?MMyl7mpyr)vRCcb+NwBggh2pPFQ0bLcuPI5(X;rB&n20_lcT`BCl~Ey
z*A_N`LDcQS!bhnGq`7W_Z0O>EhzSnlQryT(P#2=owL%XwfRbSSA
zhxe;BvP`BnSC+xI)LeEa+Gy8#IO?w7Rqa=v_g(SGZP`t9exr^om2$68}j
zmQdj7r+_wQ-u|U2xWy^2aH?{Fo=q4=xnH4H*vx$+rpPybU5e0w_*R`AowjFBznwjN
z1QXx9^`P%pXaZ2;J-UJLDA)xf#?D)L9QBp
zNxV>Oq5nGT>g%f4;)RI*9chF
ziSzDBeTmAE4=L|hJ?|_f0CH#adq!J_NAek+LNfRzQebXr&DtWap_<-%a>;)2gCpL{
zzOJd)H$aedLtZ`zkPz3(};Xy1Bklb8M5o0}Xw>g2Ag8tgqEo%Hf2
z)ApNhYIN|Z2KxTG;|eyoC|$$2-yqO>rtAPRF5iCq=LZ(-cYnCw3R!;b^Zx#qx7q*t
z!ptUsytY5*^HrPZdvkm9B%e^=X`_H|zWxyUt{ZmaX!HxVPi@+&;v2e7^tS~j)kK@V
zK%`$(z0D3b+?UVQ*Bt^tKH{7EG8`}<7d~n#HiuYty)J_eAX0>u4yz#k)>dARCGu(&
zz%o>C!=ppa@~Z;28sVU&;M!FXaqA~p1NjH;d&u7TOYgKdz3ENXzk9c3af&EQ-dH^9
zF$Yrrpkuow2rt7A%b6E}9%ZHqfpWKRgAzXv1ZMcqPD|fZ!_g
zsl#HtqqD>2@%RnQfffKp!09wTcNPJbON&+OY${q;Yl~F@hyu;B+Li%I13r32N5&Xy
z1&(B^JC>Zr7$#4k>N%cAh(JOMup9{QEQ!cv}yGKL$>em@pA>Hp}9S>H_41>Y@~Ks;{IYAYBIlw(4$w
z`)%Tyo-HNCCv=>rbDu{9@3|N9Chlqi{`3e4>O~vB-*G;)Q(Im0I?Ah#{>19l9K2dI
zDEZ4x34pv@$|da*3S0&SB<7uhdl#o0_vY;VKfcF4{ZPsN{wuRyJUY8#51p&n>wjY)
zuI8aGuG=GT#vwH#T8@+4i6|6?(vxZ1R+NlmD?NO!Y=7|2lXiYmqAqj&%+BV={%TmP
ztfM!y?jrWMhi-H`C$_Drt@zk&mYfHPkl}W9pthuKX09Un^wkvIZeiofb#HaI%
zHa(3v`^c84Pe15P>(*|7i?ue($$`*S_c9eUBZw
z;W~FiBVe@#;s{O%nF~Nh^YKsCCQh6(TKEcOTS%uHzh@!kZYsQe9%+(iJ86HGuN;@#AoV8|~m#*O=x%F)@n1
z>6mra^|}#`ZhmY3Ubv9cR^Z+=;xw`y_eyEz;Xf$o$w8Nhqr4}TDHET^&xTWya*nDJ
zRg0U(^mWU5Xd7HIaReOKU~r`8NGbrbst?vk+)c8grS3CV!stG%?yWK$HLUJ6FWomU
z4Taoy&Y_prbuO{$tCsFXXy-C;vO7u1xBgD3m$XeNaETPS4gJP{xNpwI#QLRQFGX;`
zm$-Uy>T0UD+YaXK7rw7IE_%|(C)L)!h_<@?`40^Hd|q*5qkn(>%3qC!FBz)%chy!^
z{NLdvn79sO0AA^2DvhF38S7AZ^6~LVBI3ovyh!kwWF=`%t~a;ze%@2}7Yq)L!!9ms*i)67Q}M>O>qdd9&8+q#q|yvA9=2lud4#
zfQ0~LhVPyY^{E`3#u|6M(?IM&0YX`S
z@9#NeQ&Xdak7+>fa1HL>B0e!O2x1&@FXe%~46v)9IJr!}a!aGY*(Erx4ssAn-{z6h67{uR%Ujk!g()C%&M0XV*hyt(Pk
zt*tf!2RMhtbrI7CT}!w(I)T-t!R8;F&qHwhn>nu5n1ab*N_=6S;UzGj)Wl-DvZ(LitJ
zvW}=)_skQ0hG#KMXLCwi3_l-qH^Y5rA@t3S6s_q%KYH9Khv!4|mKeNFH-3jH0gyW+
zVA4FHz@<>&wJ&P1e*{4Ozkf311}SpqBa7{a&zJcU2^bK*o++DZdqBs;)bIA!N(@;
z9pBsQ`jCJ2nQ8mB+k=?>vF4PNy|5|p`@jGDPJm|uMEWMnGw%OeLXz8(13x}g8vpMJ
zMJex}WL#9JqrY?^Tva!o)hS7b-p+1o=xDLI*(tkncb~QP^x&?B*oM<%7}vCd|K@E9
z4rFg%r*-%2g3DO8^XJZUT(a)&PTRM4w^b-xM;~tvKqKzqDn=!8M%|2vdl8+zEP!MM
z-NR3Q?2|S&K5lp3cANG0_d7_FMfM7u%XRb!Hvk~Qgt!}boj4rnz$MMuYW7m@4dW>
z=%UK6qONUJ%hm)yZzvG(9GcDm+#h-5VLLr=%I0ThtY^<|4;$n6(;pmxKtIz5Nhbnh
zUdUVnY;-+DJ!6!|&k55ZF;fv2+RH(>Ksw}gAEE;AvfkIbAqTidpS)D4=l(|lwB>{x
z&auDw?sW&{^yRdJbOBUqe$kHvjpt)(mU?=nFG-mmQhy7?>IV|nxrYvDb+W^h0LUE@
zFlnAp;8G|c4&Lu!2=j*+zz9Uj07jg=JJD&BQHzqq!N2kEciHd$!-W0%-;C*d)~?x=
zwYR^f>j{Ut)VBIAl;O@h;6h4Ivz@UumpI*iu)+SiY1Ur#OQ)=ewc`hWYQMb|OYz_O
z`*C~Sub*~t*SBDu{hb$Cm&fgsV?qH@;F$xFJmA_xwW<2PkMlSyFIX19vsB32EC37)i1l&G
zj^2O7M$Ql0jW^z4O>iRfScEU)?{F1vWERe)^amAeJGF+x^AuuczDIJkd0WXXqmxJg
zZC*fB%fVG_$Q0oaVu->x0*K;{Df&w`*Jvp$rRxD~3T*A%EC6mD#y)M$t%#FH0GH^4
zV(e0jWpo9Q)_|3C3)g1QZOo=JxY1d$p^*^)G)~o_v-BthX42?H27m>b1tysTaP?Zb_YZu90*H+$!DL8lz9n)1T%vWVK{{Gqbngv~e0+GH}6Vi+N{Mfdho%+
z_NTx1+vwfZ;3)hCYwPNBai+Lyib^67CS#K|P=!RlnfjcK4C7F{vVxfVsOv>`U@#&-
zinB9wHZ?QhWv(YUKacno!xluyMf@jXi(+HA{;8$U0$jHOprneWv7DWE;@rHgqZ@8s
z1%QWZ96+K#lD2z+F#!{~v?=0qafSJ$5IxJaO{p~JfKj4X1(j}tdshVbj-pH02xqbh
zUy4;Yj{-1?GIR3GDQj1zJ?&h@1@h;HM*y$^{0PEm9xuM_
zE_>j?`)nRjb_+fn;iJMOBtb%@ToPtE8~t^;rj_|}v&
z_Uo@cv5xD%9Z|qU))t3Fu1zZS+(t(Ddqq
zyY2OF`zhPK|DdY~cqAbGlGmW|{Z>w=el8HEJRhu0FNaf%>7^XSz12Krp}yuviK&TD
z!+caG$LMR^8|i(nk>8+>IJM&b27TAiZk4`dSS0SG9|4ie;#N{uz*`_+&)*v5YXoK&
zz~_=5!-QD}Yw?hnaCd7JT9V_>{nXFpSL0O18*9c
z=7e=$xeuTS*9?c>;sK72pRzp%u7QJ8XP^4SN39j`m4_R3%@x<69)URc?Ux&fFx^o(zFVMcq
zt{?Fh*<1s+#`k$u2(4s|pq#Gr$s0%#AJj(0e1v`oppE6nOTYP>?Z`(z3ah8qT99q8
z60Rn-u?&h-c$>7fHg(t}NM{N3w;$7BDb+Q#V(HP;iJ9@7D^&)dFUU~;?-A9O7NUTUu>k{iwa-9H>pIESpmAJ$_7tfrxE}U;Jl~=8V{9H+4
zem2kp5L-D5jfLRWL9c7Dd?jmdBiMNfB}AcF$;{R7%={cTQ@mr0}GaQ`gsVq{($2n1CX=BD5I4UE0jK
z(4-#3O;pKz9dYX3feKybomp`ORmsD}T3yDSh7h^4EUC{QeLxA(T+n`p>Dd4vf9FL%
z&rOT`0_<=-N%MpP2?Y`gBoug-Qy{jmi2a&swr4MF
z+H?*R>>3nH3$}=d^(yAtS$ymW#5SVr_{jaou*g|-`ing&%Nqfy6+qAotVDst9QS6C
zc9Q940fL8*J%qoYyo2R7Kytpm*BbEkgWWi+T9h~ejwu{
zd>x^08u;H9s3aHK>qtW>H422wTF4(*aC`uZV(JR85{Hur+Z8~5E#O?SBv-kI4n_gy
zkFm6PiBJ0YYygu_da_BygaXf13Os-Z??3s}l>OTOx$6m+p!kN@<4E#5G3S0ZB$;i0
z|8JhLe}3l`mpq-{`=<%a%2UspwOsN7E}?;Z**ss9KvLM)!U`|-Jf_Qb5c%HsVo*SO
zDnVq}B;&1;Bb&04gFYSojE{gAl?hd-r+wb{`lyYrh5EV>SD@6hDpUO$;_tQF8j5(c
zfRjDD_F4vzR>kq*JYXybxVhqnYk{=Ln~4K>_T+^tZHF36L0ro@qFqg^bLzQGCQ=2it1f+rJNaZUs(c%DGzUJYi*muZ|sN)b>_
zAH}50p!nF^hYr9>*>*vv6Q_}b1BL_?uexHtefhqx0Vq)-1^g%#;a-+d`gD*K-KQlu
zaq^h6j1^Cs`PU)Ku47Gg_*xo7CZENT#U~k3)X{$_L(%<%N9r5MIJ%0ErbB~IT_`NS}
zu{ZzDS$p;Eh{>K{+{f+sIdUNu(>_P0`;1NVX$6sv0JZgp%{T@D+JxCsd290y{|~;Y
z!X=86x9|^b`kwJiYO9TZTT$4ZX{y#4xM+9W@nSo6&pkE=S1bbFW76Uy>Kikamyd_De8I{>;1h`L1+`R$njtja5P4ojdC);)PZvDP`YFz(#Y6~J2>
zKw4rxw_w$B>g;KoH-IKwHE}D~@mjfp{~`quuVB@pP&RdNMimpX7SQ_gyI*9(!{^;Z
z`T)wI!-J#lAhNTg9WLgats>elqmbD|JV|*WmnDhrUlH1BO!*3$C+K(qiKEDTUflThCuq9G3+I$up#`os%sf7hMI=I&kzhp)VI>GdbNy2Z92VJbcQ#
z^GC+;9w^grb$oCu99T+BF3x8?1?!mVuK`fE`~_*IJ4*?G+!_6z@zyVW_~tRQL>J&0
zzY$Mz0%ymUZ3K>_F3ohNq0sm{S*K$aT5=pSz=SDv(MuU
zGLEb018)29%`|*}Z7Vgr_}1F1?+6YgFSzg|PUKB}Kh7vW`*6wR%<_ehqyxBqWf1tzvSfGdU;hOL154aOg@nM{Kh>
z#M@&8hHfE--|+ZpbSmp@X-;5iD`~C^j5p6a4;L~8=T7F}qllwhI@b7(Xn-S`M+b5W
zV2P*~?%s*BXNfDgY`X!r$4{TKjt)4F0JuqF8TNK{*&t&M5fY{oQ*G%g9yxY|bvD?w
z*In=P&LLJFBg~9M(*iaMHIqkNEwl0(K+b`KSJ;DxPuW#hUtxzIc-R5wARNsM;^#%?
zx1VuOo;eG6TEoF&7Au(*%(wxC%)1jOor{C#(RGB2$`_cxo8~OvMDnqdOB1vluq+3d
z((zm&E@d0HNx{sU;aKJX$}5dV+WUe~)B7Ut++#s2$;$L;i3)jg2^I1U_dB^1u%
zrH_1d-mcx9vug>C{ZC(=y8u8+bly@(+3C@$4FfdAHT>+6l6{F_-GBYXS-b1%27CQC
zv|b?2{>;M*_U?~N;_MP-%V5&V&|82wv9LIe8%U}JfwGbL+-<~}Lzh1zI8{ad4
zyV+HYgyxS}Pj8=#-34AG{%(MqC@gDY4fl@GgA`hXKe;u
z%BI!~x|EF9+=O4AIq19&@I!|a|3Lz)a+oMR#|on8A~b;I=Am&m;Ob@J(k>y=o&kJM
zLW^efMz6W@3c&qd_f)R>GdPu8UQWZkt)Lq@XM3C4@OOlOJpQtFNp!22VCAKh*3cpT?@#>5%jF3uJQGfukxD}Om^|Lq6+-3$6cdCh}o
z_qV4#ZsKQ;EZBejfeUp$_jIQ1WB1R;T_neZ0-F?w0b^`rqYH7_W~CixB0#7w_c&=|
zb$Mc35FdkRP`iLbL2-T9fwgbzH=c8r1THw&|AKd+@@UtHhmffF`fG2rkLLb@&@OZI-fBl4_=-)8+-L2Cw&?&MF5?xk=c?kSgPaIT|Rvl
zG5nb%M39_HcJ=N8KtfPYOu|8#ju$80Y0C8wc#4nl8Rz~N$~Uh@^&Ozbh0*gA9w^##
z?jw7-o>QL0?c+EiR6RY{yreA2(E+7%Ds>KZ0MqI-T-G_|>PKb#EO9Y8HFuI8d&k*H
zkI(V8kM*cL$Cs2$|M4hr*H6A-pV`x@3$TFRlSIOok`xl=sy>{CkzPsuy~G<-`uGF$
zb_lLwERvRpTa4G=-adZ+f_)DRjJnPn4v{eG-7Y8vt1(c9-7&TGHA}fX*Z(6nHEO#NhEM1C-jdd9JmEZav@IAM~)R
zt3-7|C5MuprBx6OMjN_s>{NER>D5Df)r))8IL#QP>_*RT{Go?%xLB~+dp?b>UCNpe
z?-nueXuunJV^+6{7sMu)9JA2~drH_=ZCfR_c#WpMG6gOhmV=xCtpYK`6f@)x-zW@vO2
z{m5^$FMRgXh~cx$v*IFZIgo5pKh0Qh14MIJk0ZY8
zQicA5C+fq2CO<5MDCLY&9@rgZLN|Fm-)Ty~A6v^Ug9JXc8$=AH$srTxr;jt
z#gAgjZoK(6`|~e;$=c`U39#MDk1b*Y!U60*bO*lt;E3U4alKq)UAyt5
zI>%^eYzU4hbCq}R41jcec+?Kve#k!cp-okE^Ae920EX*`vE_$SV0Z<9)B!*hfU3f=l=XLpc~o%i(+7|opPXPW3(OIf
z0>1J}3o~59|0Uegk;!Sh?VG+4j%TkwC8C+014#I&r1Ch09}RHhzNpsAOOu{nmMXHD&R`|}j#buk)HHN79s14xXn>AK#mU*J%n(Sf75kr7~YPv!!R
ze$J7+$Fx!ZJ4gwD+yU{DrU?a}3Uqovx;OyX|1!e&PEt9PZ;CkrSaZ@CJaUb~1wEZ&T?f>~7
z_C}lTqg%?LXmhMbUrtJ1z!Z2&!6S6lrt6!nz@7k}P@fL^P??{=weq}fg!nnCz$VN;
zg7RGAJZ$>jf&%G1D#>_;+Vm~J5Xy8^t`4Ad{dG6mXRg1}mM7<}Nc_G7*W3gbtqGCu
zk~Mbp0D6|Kt*6^Y;K1GY<QCb7RpK5CazgUTKU#
zae50*ioMETxILw^$`cU>(4a_=nAKJg3Jd+&5$uk+u9wPNG{Vuv!&pMEw1!&wc?H
zJ-rT$vT!jQ@&&sKfO+5H2d%ZU4e|Ao9Xznd9hgsI;k~{LutU@>d^LCmpO+W|$(Lmv
zE^emgM$xClIrfTT6V_l@MBGNi+0(444xn69r=Zff`hn(MSZz;D&$^DNz_3yt8pznF
zPp*bAlF2g96?8Cr5zEU$z61!L9veXyvfi38a4O?lQ(-}#6inJD6nFv(+;wGx{cns@^u6$S
zUrJBV#pY!hrTp=yr|c5|!@Ca^yw4-&aWdSJa^srKiex^AZlWx`|LDj00*{P$bp1bn
zVAwtaXY#Gz*15gr&2Rv2{Fyry5Ac<81x|#(Oki
zKrkzu2H@xWw+dArp#>LWQ^_r8{Yq3pb$G|vbgR@WZR?x3fsIY}9j|$vee5s)0x>mU
z3YR}Uy}PZstCwKfDI8&U*(X2t5nG&-FUb!8vt7Z%IRFm2F{P*DyQBiFm{nL=s^r=CdcNn
z=0>j)6fUC~BK^4&@+mCQ^8nQxoWe3r&v&6adGz>6oSXMr8%8(-XU{qCX~h}2^b$`#
zI0RT6A*Nvtk^ZE$qc1r=v*0la_g@P)c5nhLhQoAc3lcaRpO_l}>p5{b0Ygj?FpaNG
zz*8BKx`KL-<3BSG;Fja}ETFs@5I;OPWD}F)ZZK4kgLOEqFnn!(X%c-(xSVhrzb)UmCqqC3`wDTV;Dk3kZ=NBYDje%YOnUB&*R2zSY0#J;{aR4bG
zxd~3gb#XL7i_U>{07+h8Qr7EUm+JzOpvS;{1W0;h8v#kcxZFB@+1{a;Fqb
z+9wov0t)QIKt>L?AHc}qRAFS!k8
z*H}tRRNRzz-;s69Gff!3MZk@$x5vj3C9lFATBU<&f4jL0V2aRVeGPKS-6Nba0@#+S_$q_aZtFMd1m3>
z3HTN*x(BLNaV{u+V6t9qta|HG7>F#ms>@;;}5gWH)Oe2x9jCR_v=x%|tnZ-Y2
z32tEti|M&i749J+8gU*j&~xzMZUE#N_tltHct}LpN4|!eAi!b+B5hfwccTVS%R0{k
z2HWUwcx2SW6{h8Y8_<+S6t3$6!G!2Wf3x0lv^#t59DsAi3Rqqja%o$HzQ8SfNMf0e
zI9qtE1nrI!UFc;meK{8G5eM3z2*0U*`{U=s3B2t?fY5(T=o!+p0hm)OiI(-OIH+We
z4jef;FcdqMIp{fY0IEr7yy)2pGl=@3j*{91okseiegKVIWg5?~qf#Jo^_l}kfyY(2
zyBwj#3O^zgB5@6u04m#QV(!uGcG^?YKB2%fOo7{OyUk~C@4fdv!!vpG@DB_Stp+aQ
zlZjUI=+@8OlK(-pnvSNFeb<%X>r=ppc46Il%??)U}*@^
z_AtH}j~qV&H*LxR$b3@@g>UcWUMK;lL}SnakuG5bIRwkSY@|c
zYfC5njUY;1wcGE!gD5#u)>Op88Qnl}KzHx%bI05Zh=EZdv?25pv$>RY5>=-MPU#GF
z&cRV!!n!+;5zNgu-DqRyhw)oEhuC?|4jsDMDoe|V?+KNND0~AqW+w**5YyuZY!Oc8
z75i)m-L&g-fHMtqNH{ECBDx^wkeVt=nOmV7RHobrVw{6gaX2x-c9
zednv}`@ipv)`18;#h7w5{#gHZF=7h;Zvt}S{0RsO6awft7bq0K1JUsZR~T@obJbND
zi}p3CK84;kJTK~EGsi02VhQU?_mNO5dy8SJ6OGfE*z5BcejS%6*aeUj|WC
zIMY??RXLaXfM}=ihwpvQUohYf(yQL}cR!o3A$LH)q-jEdCrg1F_vP%yeHZ$^d9rPv
z%k>_Q)%jdr)8oy3JDctC>X_~NnKurYJ*|EL9)B`V`;{0`YT&06O>QkzyEbPI58~ZA
z;6Ch?n030EXeAcUj5y6
zdgQF_U9Uq%eA-so^)n#8B8r8bph-Dj%wx5(3~O)-*=!XB!>=AWOn~Ykzy)DQaD+HE
zHH#u(8ToArnRcB`fwIO&#z3Gogk3?#%(yAohT|w!%A0r#aSSV%bI+g*m;<40AeZh2
zH7WjKp`i&&l{)hPZWrEt&!ub#`@PwKI!iC5oervqYGhK70p_BLi=4
zD&XXDM!?HjP|y_G5{p*%q>lB;`*>4}%)4i~A+Rgy4?d(#!w#;}&m5s}bQ|Rqvxjgu
zQY2i4ZCi`OTj>UE{f^L$jgsG3(5w;zKOnoFigf3UTUsrwvr3zI#`%!Ghhz+SCcvYBgzvzcla66Sfa{=2KmJ#FjP5%9KiJzYRessK*u!>wj?1Q-UehzuK=tcc|Z4VJkF
z%CEq7+$`hXEK}dvl@2hsX1}?%OUxty@)8M_R8J_7P#~ee3zh;~IO&o}5OX5G?tXm!
zm6>P{c3HD;#_{BJ+~O#PQosTkL59t%Y}VSy`HaA2P^0wWLZi)I{8a6Q1E?U-sUejY
z_es_$=LYcJ4M=`(9l5uC;e^0U43TG1Mk)3T;0W3-(QowVbeh%#4V}or&&;tOAV&ev!
zyj8%8)vVa56Q_}NcVO{Sa2fUj3Y&6LA{*+Y>z)0;S
z%U;44khp9DTx&E+F~`<)BMvv`;kvEiqX-juOyi|MDf99PK&%|!NQp>P$+nmIfl&dxOP*aJuWK&j=AKmv
zU@li3>xzf7!;}EX9TG5Uo>1U