Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update of the documentation (prepare for the old web site to be offline) #371

Merged
merged 1 commit into from
May 14, 2024
Merged

Update of the documentation (prepare for the old web site to be offline) #371

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
159 changes: 33 additions & 126 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,9 @@
</em>
<br />
<em>
<a href="https://whotracks.me/trackers.html" target="_blank" rel="noopener noreferrer">Trackers</a>
· <a href="https://whotracks.me/websites.html" target="_blank" rel="noopener noreferrer">Websites</a>
· <a href="https://whotracks.me/blog.html" target="_blank" rel="noopener noreferrer">Blog</a>
· <a href="https://whotracks.me/explorer.html" target="_blank" rel="noopener noreferrer">Explorer</a>
<a href="https://www.ghostery.com/whotracksme/trackers" target="_blank" rel="noopener noreferrer">Trackers</a>
· <a href="https://www.ghostery.com/whotracksme/websites" target="_blank" rel="noopener noreferrer">Websites</a>
· <a href="https://www.ghostery.com/whotracksme/explorer" target="_blank" rel="noopener noreferrer">Explorer</a>
</em>
</p>

Expand All @@ -29,96 +28,36 @@
<img alt="License Badge" src="https://img.shields.io/github/license/ghostery/whotracks.me?style=flat-square"></a>
</p>

This repository contains:

- data on trackers and websites as shown on [whotracks.me](https://whotracks.me/) (WTM)
- database mapping tracker domains to companies
- code to render the [whotracks.me](https://whotracks.me/) site

---

> :warning: **Upcoming changes**
>
> We are in the process of migrating the website to [ghostery.com/whotracksme](https://ghostery.com/whotracksme):
>
> * https://www.ghostery.com/whotracksme
> * https://www.ghostery.com/whotracksme/trackers
> * https://www.ghostery.com/whotracksme/websites
> * https://www.ghostery.com/whotracksme/explorer
>
> The following documentation has not been updated yet and still points to the old website.
> You can already try out the new website with the links above. If you have any feedback,
> about missing functionality or if you spot inconsistencies, feel free to create a
> [Github issue](https://github.com/whotracksme/whotracks.me/issues).
>
> Monthly data dumps will not be affected by these changes. The licensing also remains unchanged.
>
> For more information, see https://github.com/whotracksme/whotracks.me/issues/367
>
> For now, if you came from the Ghostery website and have question on the data itself
> (e.g. how to download or looking for a technical documentation of the fields),
> it is best to start here:
>
> https://github.com/whotracksme/whotracks.me/blob/master/whotracksme/data/Readme.md
>
> If it does not answer your questions, also feel free to create a
> [Github issue](https://github.com/whotracksme/whotracks.me/issues).
> It helps us to improve the missing parts of the documentation.

---

# Installation

Python 3.11 is needed to build the site. We recommend creating a
[virtualenv](http://docs.python-guide.org/en/latest/dev/virtualenvs/)
to install the dependencies, or use `pipenv` or )

to

```sh
python -m venv venv
. venv/bin/activate
```

After the initial setup, you can proceed with installing whotracks.me.
# Downloading the data

For nushell:
Each month, we release a new version of the web site. The data from the last month can be directly [accessed through the website](https://www.ghostery.com/whotracksme/explorer).

```nushell
python -m virtualenv venv
overlay use venv/bin/activate.nu
```
The raw data, from which the graphs have been computed, is also available as an open data set (updated every month). You can also
download historical data. More information on the raw data can be found [here](whotracksme/data/Readme.md).

## With Pip
WhoTracks.me also builts heavily on another open source project called [TrackerDB](https://github.com/ghostery/trackerdb);
all meta data (e.g. company descriptions) is maintained there.

```sh
$ python -m pip install git+https://github.com/ghostery/whotracks.me.git
```

## From source
# Using the data

After cloning the repository:
You can directly use the [raw data](whotracksme/data/Readme.md), which are all text files. As an alternative, you an also
download it locally and use the Python API:

```sh
$ python -m pip install -r requirements.txt
$ python -m pip install -e .
```
python3.11 -m venv venv
. venv/bin/activate
pip install git+https://github.com/ghostery/whotracks.me.git
```

That’s all you need to get started\!

# Downloading the data

Each month, we release a new version of the web site. The raw data, from which the
graphs have been computed, are also available as an open data set (updated every month).

The data from month can be also directly [accessed through the website](https://whotracks.me/explorer.html).

More information on the raw data can be found in `whotracksme/data/Readme.md`.
... or if you have locally checked it out:

# Using the data
```
python3.11 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
```

To get started with the data, everything you need can be found in
`whotracksme.data`:
The Python API can now be accessed as follows (make sure you have already downloaded data):

```python
from whotracksme.data.loader import DataSource
Expand All @@ -131,7 +70,7 @@ data.companies
data.sites
```

A whitepaper for whotracks.me is available at https://arxiv.org/abs/1804.08959, and here's a BibTeX entry that you can use to cite it in a publication:
A whitepaper for WhoTracks.me is available at https://arxiv.org/abs/1804.08959, and here's a BibTeX entry that you can use to cite it in a publication:

```
@misc{whotracksme,
Expand All @@ -144,56 +83,24 @@ A whitepaper for whotracks.me is available at https://arxiv.org/abs/1804.08959,
}
```

# Building the site

Building the site requires a few extra dependencies, not installed by
default to not make the installation heavier than it needs to be. You
will need to install `whotracksme` from the repository, because not all
assets are packaged with `whotracksme` released on pypi:

```sh
$ python -m pip install -r requirements-dev.txt
$ python -m pip install -e '.[dev]'
```

Once this is done, you will have access to a `whotracksme` entry point
that can be used this way:

```sh
$ whotracksme website [serve]
```

The `serve` part is optional and can be used while making changes on the
website.

All generated artifacts can be found in the `_site/` folder.

> If you debug the website generator, the parallel execution can be
> disabled by setting the environment variable DEBUG=1.

## Tests

To run tests, you will need `pytest`, or simply install `whotracksme`
with the `dev` extra:

```sh
$ python -m pip install -e '.[dev]'
$ pytest
```

# Contributing

We are happy to take contributions on:
We rely on contributions for the community to keep the quality of this project high. If you want, you can support us in multiple ways:
* Do you see inconsistencies in the data? Please open a Github issue [here](https://github.com/whotracksme/whotracks.me/issues). We will have a look!
* Do you see wrong company descriptions? Did we put something in the category? Please check out the [TrackerDB project](https://github.com/ghostery/trackerdb), where all the meta data is kept, and open an [issue](https://github.com/ghostery/trackerdb/issues), or send us a pull request.
* Do you have any feedback on the [WhoTracks.me homepage](https://www.ghostery.com/whotracksme) or about the documentation? Please, let us know, so we can improve.

- Guest articles for our blog in the topics of tracking, privacy and security. Feel free to use the data in this repository if you need inspiration.
- Feature requests that are doable using the WTM database.
- Curating our database of tracker profiles. Open an issue if you spot anything odd.
You can also contact us via email at [[email protected]](mailto:[email protected])

# Right to Amend

Please read our [Guideline for 3rd parties](https://github.com/ghostery/whotracks.me/blob/master/RIGHT_TO_AMEND.md) wanting to suggest
corrections to their data.

# Local builds

[Readme on local builds](docs/local-build.md) (this is mostly relevant for the maintainer of this project)

# License

The content of this project itself is licensed under the [Creative
Expand Down
40 changes: 40 additions & 0 deletions docs/local-build.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Local development

## Building the website (the static HTML based version) and the internal API

The code to build the website on https://www.ghostery.com/whotracksme is not public;
but for local testing, you can still use the code for the [previous version](https://web.archive.org/web/20240501140903/https://whotracks.me/).

Python 3.11 is needed to build the site (Python 3.12 is currently not supported):

```sh
python3.11 -m venv venv
. venv/bin/activate
pip -r requirements-dev.txt
pip install -e '.[dev]'
```

If you have not done so, make sure that you have downloaded data (see [Data Readme](../whotracksme/data/Readme.md)).

```
whotracksme website
```

It will generate static HTML files in the `_site` directory. Plus, it will also create a JSON files
in the `_site/api/` directory. Use them at your own risk, since the format is expected to change over
time. If there is interest to stabilize the API files, let us know. Currently, it is only used internally
within Ghostery to power the new website.

> Hint: if you debug the website generator, the parallel execution can be
> disabled by setting the environment variable DEBUG=1.

## Tests

To run the unit tests:

```sh
python3.11 -m venv venv
. venv/bin/activate
python -m pip install -e '.[dev]'
pytest
```
Loading
Loading