Skip to content

Commit

Permalink
Revert all changes
Browse files Browse the repository at this point in the history
  • Loading branch information
@elpham6
elpham6 committed Apr 17, 2024
1 parent e1543a4 commit 614ddad
Show file tree
Hide file tree
Showing 8 changed files with 307 additions and 17 deletions.
150 changes: 150 additions & 0 deletions documentation/Sphinx - How to get started.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# Sphinx - How to get started?

## Prereqs

- You have proper documentation for your code. This includes doc strings.
- Make sure that your doc strings follow a standard, eg. PEP, Google, Numpy, etc.
- Highly recommended to use a linter for both your code and docs. I use Ruff.

## 1. Install Sphinx

*refer: https://www.sphinx-doc.org/en/master/usage/installation.html*

- **for Ubuntu/Debian** : `sudo apt install python3-sphinx`

## 1.1 Folder Structure (Recommended)

I like my docs as a separate folder in `src`. Therefore I create a new directory `docs`. Move here for the next step.

```
├── src
│ ├── build
│ ├── package_dir
│ │ ├── __init__.py
│ │ ├── module_1.py
│ │ ├── module_2.py
│ │ ├── subpackage_dir
│ │ │ ├── __init__.py
│ │ │ ├── module_1.py
│ │ │ ├── module_2.py
├── other directories and files not documented
│ ├── css
│ │ ├── **/*.css
│ ├── images
│ ├── js
│ ├── index.html
├── pyproject.toml
├── requirements.txt
├── package-lock.json
└── .gitignore
```

## 2. Quickstart

Run:

```
sphinx-quickstart
```

This will ask you the following:

1. **Do you wanna separate your build and source?** N, Since I recommend a folder structure with a separate docs
directory.
2. Name of Project
3. Author Name
4. Project Release
5. Project Language

If run successfully this will create a `conf.py` file, the make files and folders like `_build`, `_static`,
and `_template`.

## 3. Extract docs

Move to `src` dir.
Run:

```
sphinx-apidoc -o docs package_dir
```

Note that: here `-o` flag is the output folder, in our structure it is the `docs` directory, and `package_dir` is the
directory with all the code you want to document.

This will create `.rst` files for each Python module.

## 4. Conf.py changes

The `conf.py` file is in `docs` folder.
You can see all the details you specified in the `quickstart` here.

## 4.1 OS Syspath Changes

Add the following to the beginning of `conf.py`.

```
import os
import sys
sys.path.insert(0, os.path.abspath("../package_dir"))
```

*Note that this assumes the above-mentioned folder structure, if you have a different structure, make sure to point it
to the source code. *

## 4.3 Extensions

Sphinx has a lot of useful extensions. These should be added to the `extensions` tag. Some of the extensions I use are:

1. `sphinx.ext.autodoc` - This extension automatically takes doc strings.
2. `sphinx.ext.linkcode` (Optional) - This extension provides a link to the GitHub code. *(Note that this extension
requires other configs. Refer to Sphinx extension documentation for more details.)*
3. `sphinx.ext.viewcode` (Optional) - This extension is similar to the above extension, instead of linking the code to
GitHub, it displays the code in a static webpage.
4. `sphinx.ext.napoleon` (Optional) - Since we use Google doc-string standard, this is an essential extension.

## 4.2 Theme (Optional)

I use a Sphinx theme, which can be installed by running,

```
pip install sphinx-rtd-theme
```

Change the `html_theme` tag in the `conf.py` to `sphinx_rtd_theme`.

You can find more themes at various sources like `www.sphinx-themes.org/`,
`https://sphinxthemes.com`, etc.

## 5 Building the docs

To finally generate the docs run the following command from the `docs` directory.

```
make html
```

This will create a `build` directory, where you can find the html files. Opening the `index.html` shows you the homepage
of your docs.

## 6.1 Adding other pages (Optional)

To add other pages to your sphinx website, you just have to create `.rst` reStructuredText files in the appropriate
location and add them to your `index.rst` or to the `toctree` of a file already mentioned in `index.rst`.

For more instructions on defining document structure refer
[Defining Docuement Structure](https://www.sphinx-doc.org/en/master/usage/quickstart.html#defining-document-structure)

For instructions on how to format reStructuredText refer to
[reStructuredText Basics](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html)

## 6.2 Adding examples (Optional)

TODO

- [https://sphinx-gallery.github.io/stable/index.html]()

### References

- [https://www.sphinx-doc.org/en/master/usage/quickstart.html]()
- [https://samnicholls.net/2016/06/15/how-to-sphinx-readthedocs/]()
Binary file added src/docs/_build/doctrees/get_started.introduction.doctree
View file
Binary file not shown.
8 changes: 8 additions & 0 deletions src/docs/_build/html/_sources/get_started.introduction.rst.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
GRAG Overview
=============

GRAG provides an implementation of Retrieval-Augmented Generation that is completely open-sourced.

Retrieval-Augmented Generation
###################

131 changes: 131 additions & 0 deletions src/docs/_build/html/get_started.introduction.html
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="./">
<head>
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>GRAG Overview &mdash; GRAG 0.0.1 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=80d5e7a1" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=19f00094" />
<link rel="stylesheet" type="text/css" href="_static/sg_gallery.css?v=61a4c737" />
<link rel="stylesheet" type="text/css" href="_static/sg_gallery-binder.css?v=f4aeca0c" />
<link rel="stylesheet" type="text/css" href="_static/sg_gallery-dataframe.css?v=2082cf3c" />
<link rel="stylesheet" type="text/css" href="_static/sg_gallery-rendered-html.css?v=1277b6f3" />


<!--[if lt IE 9]>
<script src="_static/js/html5shiv.min.js"></script>
<![endif]-->

<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=d45e8c67"></script>
<script src="_static/doctools.js?v=888ff710"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Installation" href="get_started.installation.html" />
<link rel="prev" title="Get Started" href="get_started.html" />
</head>

<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >



<a href="index.html" class="icon icon-home">
GRAG
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul class="current">
<li class="toctree-l1 current"><a class="reference internal" href="get_started.html">Get Started</a><ul class="current">
<li class="toctree-l2 current"><a class="current reference internal" href="#">GRAG Overview</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#retrieval-augmented-generation">Retrieval-Augmented Generation</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="get_started.installation.html">Installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="get_started.llms.html">LLMs</a></li>
<li class="toctree-l2"><a class="reference internal" href="get_started.vectordb.html">Vector Stores</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="grag.html">GRAG</a></li>
<li class="toctree-l1"><a class="reference internal" href="auto_examples_index.html">Cookbooks</a></li>
</ul>

</div>
</div>
</nav>

<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">GRAG</a>
</nav>

<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="get_started.html">Get Started</a></li>
<li class="breadcrumb-item active">GRAG Overview</li>
<li class="wy-breadcrumbs-aside">
<a href="https://github.com/arjbingly/Capstone_5/blob/main/src/get_started.introduction.rst" class="fa fa-github"> Edit on GitHub</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">

<section id="grag-overview">
<h1>GRAG Overview<a class="headerlink" href="#grag-overview" title="Link to this heading"></a></h1>
<p>GRAG provides an implementation of Retrieval-Augmented Generation that is completely open-sourced.</p>
<section id="retrieval-augmented-generation">
<h2>Retrieval-Augmented Generation<a class="headerlink" href="#retrieval-augmented-generation" title="Link to this heading"></a></h2>
</section>
</section>


</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="get_started.html" class="btn btn-neutral float-left" title="Get Started" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="get_started.installation.html" class="btn btn-neutral float-right" title="Installation" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>

<hr/>

<div role="contentinfo">
<p>&#169; Copyright 2024, Arjun Bingly, Sanchit Vijay, Erica Pham, Kunal Inglunkar.</p>
</div>

Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.


</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>

</body>
</html>
4 changes: 2 additions & 2 deletions src/docs/conf.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@
# -- Project information -----------------------------------------------------

project = 'GRAG'
copyright = '2024, Arjun Bingly, Sanchit Vijay, Erica Pham, Kunal Inglunkar'
author = 'Arjun Bingly, Sanchit Vijay, Erica Pham, Kunal Inglunkar'
copyright = '2024, Arjun Bingly, Sanchit Vijay, Erika Pham, Kunal Inglunkar'
author = 'Arjun Bingly, Sanchit Vijay, Erika Pham, Kunal Inglunkar'

# The full version, including alpha/beta/rc tags
release = '0.0.1'
Expand Down
4 changes: 2 additions & 2 deletions src/docs/get_started.installation.rst
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Installation
===============
*Since we are under the development phase, we have not published to pypi yet.*
*Since we are just in the development phase we have not published to pypi yet.*

* ``git clone`` the repository
* ``pip install .`` from the repository
* *For Developers*: ``pip install -e .``

Further customization can be made on the config file, `src/config.ini`.
Moreover, further customization can be made on the config file, `src/config.ini`.
8 changes: 8 additions & 0 deletions src/docs/get_started.introduction.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
GRAG Overview
=============

GRAG provides an implementation of Retrieval-Augmented Generation that is completely open-sourced.

Retrieval-Augmented Generation
###################

19 changes: 6 additions & 13 deletions src/docs/get_started.vectordb.rst
View file
Original file line number Diff line number Diff line change
@@ -1,42 +1,36 @@
Vector Stores
===============

Vector store or vector database is a type of database that stores data in high-dimensional vectors.
This is a crucial component of RAG, storing embeddings for both retrieval and generation processes.
# Explain what a vector store is...

Supported Vector Stores
########################

Currently supported vectorstores are:

1. Chroma
2. DeepLake
2. Deeplake

Chroma
*******
Since Chroma is a server-client based vector database, make sure to run the server.

* To run Chroma locally, either:
1. Move to `src/scripts` then run ``source run_chroma.sh`` OR
2. Refer to `Running Chroma in ClientServer <https://docs.trychroma.com/usage-guide#running-chroma-in-clientserver-mode>`_.
* To run Chroma locally, either use move to `src/scripts` then run `source run_chroma.sh` or refer to
`Running Chroma in ClientServer <https://docs.trychroma.com/usage-guide#running-chroma-in-clientserver-mode>`_.
This by default runs on port 8000.
* If Chroma is not run locally, change ``host`` and ``port`` under ``chroma`` in `src/config.ini`, or provide the arguments
* If Chroma is not run locally, change `host` and `port` under `chroma` in `src/config.ini`, or provide the arguments
explicitly.

For non-supported vectorstores, (...)

Embeddings
###########

* By default, the embedding model is `instructor-xl`. Can be changed by changing ``embedding_type`` and ``embedding_model``
* By default, the embedding model is `instructor-xl`. Can be changed by changing `embedding_type` and `embedding_model`
in `src/config.ini` or providing the arguments explicitly.
* Any huggingface embeddings can be used.

Data Ingestion
###############

For more details on data ingestion, refer to our `cookbook<https://github.com/arjbingly/Capstone_5/blob/main/cookbook/Basic-RAG/README.md>`_.

::

client = DeepLakeClient() # Any vectordb client
Expand All @@ -47,4 +41,3 @@ For more details on data ingestion, refer to our `cookbook<https://github.com/ar


retriever.ingest(dir_path)

0 comments on commit 614ddad

Please sign in to comment.