Jumia-Web-Scraping.Rmd

---
title: "Jumia-Web-scraping Project"
output:
  pdf_document: default
  html_document:
    df_print: paged
---

# Jumia Web-scraping Project

-Using R to scrape Jumia's website products and their discounts on `r lubridate::today()`
```{r}
# loading necessary packages
library(rvest)
library(dplyr)
library(httr)

```

## web scraping
-We start by getting the website url and reading getting its contents
```{r}

url= "https://www.jumia.co.ke/"

# reading url
page = read_html(url)

# checking response
(httr::status_code(GET(url))) # its 200
```
-Then extract all the div tags that contain the diferent categories of products e.g flash sales, clearance sales etc
```{r}
# Extracting all category tabs that contain all the products 
divs = page %>% html_nodes("div[class='col16 -pvs']") #saving to div variable

# creating an empty data frame to save the data
jumia = tibble(  "Name"= character(),      # name column
         "Price" =character(),             # price column
         "Discount" =character(),          # discount column
         "Category"= character())          # category column
```



-We then create a function that scrapes all the products from each category div.
```{r}
# creating a function that extracts all the products information in a certain category
category = function(div){ # passing a div parameter
  
  # assigning category as the div's title/header in the website
  category = div  %>%html_nodes("h2") %>% html_text()  # text in the h2 tag
  
  # if a category contains see all link
  see_all_link = div %>% html_nodes("a[class='-df -i-ctr -upp -m -mls -pvxs']")%>% html_attr("href")
  
  if (length(see_all_link)){
    
    url_ = paste0(url,see_all_link %>%  stringr::str_extract( "([^/]+)/$") )
    new_page = read_html(url_)
    products = new_page %>%   
      html_nodes("div[data-catalog='true']") %>%
      html_nodes("article")
    
    # scraping the various product attributes
    name = products %>% html_node("h3[class='name']")%>%
      html_text()
    price = products %>% html_node("div[class='prc']") %>%
      html_text()
    discount = products %>% 
      html_node("div[class='bdg _dsct _sm']") %>% html_text() 
   
    
    # scarping the bottom pages in our new page of all categories
    next_page_link = new_page %>% html_node("a[aria-label='Next Page']") %>% html_attr("href")
    
    while(!is.na(next_page_link)) {
      url_page = paste0(url_, next_page_link %>%  stringr::str_extract( "\\?([^/]+)$") ) # getting url for that page
      new_page = read_html(url_page)                                                     #reading the page contents

      products = new_page %>% html_nodes("div[data-catalog='true']") %>%  html_nodes("article")
      
        # scraping all product from this page
      name_ = products %>% html_node("h3[class='name']")%>%
        html_text()
      price_ = products %>% html_node("div[class='prc']") %>%
        html_text()
      discount_ = products %>% 
        html_node("div[class='bdg _dsct _sm']") %>% html_text() 

      name=c(name,name_); price=c(price,price_); discount=c(discount,discount_)
      
      next_page_link = new_page %>% html_node("a[aria-label='Next Page']") %>% html_attr("href")
    }
    
  }
  
  else {
    # selecting all items in that div
    items = div  %>% html_nodes("div[class='itm col']") 
    name = items %>% html_node("div[class='name']") %>%
      html_text() # all item names
    price = items %>% html_node("div[class='prc']") %>%
      html_text() # all item prices
    discount = items %>% html_node("div[class='bdg _dsct']") %>%
      html_text() #all item discounts
  }
  
 # returning a data frame of the extracted items
 return(tibble( "Name"= name,  
         "Price" = price,
         "Discount" = discount,
         "Category"=  category))

}

# running the above category function to all the category divs and appending the data to the jumia data frame
for(div in divs[c(2,4:23)]){
  jumia= bind_rows(jumia,category(div))
}

# previewing our data
head(jumia, n=10)
```

## Cleaning and saving the scraped data
-After scraping the items, we clean the data and store it in csv format.
```{r}
# converting the price and discount columns to integers

# In both columns we will replace all non numeric numbers with a blank
jumia = jumia %>% 
  mutate(Price = as.numeric(jumia$Price %>% stringr::str_sub(-8) %>% 
                              stringr::str_replace_all(pattern="[^0-9]","") )
         ,
         Discount= as.numeric(Discount %>% 
                                stringr::str_replace_all(pattern="[^0-9]","")  )) 
# previewing clean data
head(jumia)
```


```{r}
# assigning zero to NAs in discount column
jumia$Discount[is.na(jumia$Discount)] = 0

# saving as csv
readr::write_csv(jumia,"./data/jumia_data.csv")
```



# Data Analysis
```{r}
# loading required packages
library(dplyr)
library(ggplot2)

#reading data
jumia=read.csv("./data/jumia_data.csv")

# previewing the structure of our scraped data
jumia %>% skimr::skim()
```



-The scraped data contains 14,724  rows of products and 4 columns of product attributes, with 2 numeric and 2 categorical columns. 97 items do not have discounts.

> **NB**: The data use is scraped on 7th February, 2024 from Jumia Website

-Bellow is the list of the scarped categories and the count of items in each.
```{r}
# counts of items for each scraped category
jumia %>% 
  group_by(Category) %>% count() %>% arrange(desc(n))

```


-Below is a bar plot of the various categories and their average discounts on products
```{r}
# bar plot of categories vs discount

# creating plot
plot_1= jumia %>% 
  mutate(Category = strtrim(Category,19)) %>% 
  group_by(Category) %>% 
  summarize(count=n(),discount= mean(Discount, na.rm = T)) %>% 
  arrange(desc(discount)) %>% 
  ggplot(aes(x=forcats::fct_reorder(Category,desc(discount)), 
               y=discount))+
  geom_col(fill="cadetblue4") + # bar plot layer
  geom_label(aes(label = paste(round(discount,1),"%")),size=3)+ #adding discount as text labels

# formating the plot
  # adding x and y labels
  labs(title = "Top 10 Categories with Highest Discounts",
        subtitle = "BarPlot of Categories Vs Discount",
       y="Discount",
       x="Categories",
        caption="@mulei")+
  theme_bw()+
  # rotating the x labels
  theme(axis.text.x = element_text(angle = 90),
        plot.background = element_rect(fill = "lightgray"))
  
plot_1
```
-Clearance Deals have the highest discounted products, as the category has a discount of 43.7%, hence if in need of large discounts go to the clearance deals category followed by Men's shoes and Deals on Phones and accessories.


-Below is a bar plot of the top 5 categories with the lowest prices
```{r}
# bar plot of categories vs Price

# creating plot
plot_2= jumia %>% 
  group_by(Category) %>% 
  summarize(count=n(),price= mean(Price, na.rm = T)) %>% 
  filter( rank(price)<=5) %>% 
  ggplot(aes(x=forcats::fct_reorder(Category,desc(price)), 
               y=price))+
  geom_col(fill="cadetblue4") + # bar plot layer
  geom_label(aes(label = round(price,0)),size=3)+ #adding discount as text labels

# formatting the plot
  # adding x and y labels
  labs( title = "Top 5 Categories with Lowest Price Range",
       subtitle = "BarPlot of Categoriry vs Price",
       y="Price",
       x="Categories",
       caption="@mulei")+
  theme_bw()+
  # rotating the x labels
  theme(axis.text.x = element_text(angle = 90),
        plot.background = element_rect(fill = "lightgray"))

plot_2 
```
-Body shop has products with the lowest prices followed by Clearance Deals and Drink up discounts

-Bellow are the top 10 products with the highest discounts today

```{r}
# filtering the data
plot_3 = jumia %>% 
  arrange(desc(Discount), desc(Price)) %>%
  filter(row_number() <= 5) %>% 
  mutate(Name=stringr::str_sub(Name,1,25)) %>% 

  # creating the plot
ggplot( aes(x= Name,
            y= Price))+
  geom_col(fill="cadetblue4")+
  geom_label( aes(label=paste("Price:",Price, ", -",Discount,"%")),size=3)+
  labs(title = "Top 5 Products with Highets Discounts",
       subtitle = "BarPlot of Product Name Vs Discount",
       y="Discount",
       x="Product Name",
       caption = "@mulei")+
  theme_bw()+
  theme(axis.text.x = element_text(angle=90),
        plot.background = element_rect(fill = "lightgray"))

plot_3  
```


```{r}
# saving the plots
ggsave("./data/plot_1.png", plot_1, width=8, height=4)
ggsave("./data/plot_2.png", plot_2,width=7, height=4)
ggsave("./data/plot_3.png", plot_3,width=7, height=4)
```