Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Backups #118

Open
3 of 4 tasks
lemoer opened this issue Jan 31, 2021 · 20 comments
Open
3 of 4 tasks

Backups #118

lemoer opened this issue Jan 31, 2021 · 20 comments
Labels
enhancement question

Comments

@lemoer
Copy link
Contributor

lemoer commented Jan 31, 2021 •
edited by 1977er
Loading

Things to Backup:

  • Wiki
  • Grafana (how?)
  • Influx?
  • Zabbix
@1977er
Copy link
Member

1977er commented Jan 31, 2021 •
edited
Loading

Wir brauchen einen Ort, an dem wir kostengünstig ein paar TB abladen können.

  • 32G für die web-VM inkl. Wiki.
    • 600M für das Wiki alleine
  • 32G + 256G für die harvester-VM inkl. Influx, Grafana und ggf noch Prometheus
    • 131G InfluxDB
    • 95G Prometheus (abgekündigt)
    • 35M Grafana (einfache SQLite DB)
  • 128G für die monitor-VM (inkl. Zabbix-MySQL, Netbox, Loki)
    • 17G Zabbix DB
    • 21G Loki (wird das noch genutzt?)
    • 1G Netbox (Postgresql)

@CodeFetch
Copy link
Contributor

Der Hypervisor von sn08 hat Platz ;-)

@1977er
Copy link
Member

1977er commented Feb 1, 2021

@CodeFetch:
Ist vermutlich besser, das in eine VM zu packen, um es potentiell von anderen Kundendaten wegzukapseln. Hast Du da noch Legacy IP-Adressen übrig oder wollen wir das mit v6-only machen?

@AiyionPrime
Copy link
Member

AiyionPrime commented Feb 20, 2021 •
edited
Loading

@CodeFetch v6, oder v4, das ist hier die Frage :)

@AiyionPrime
Copy link
Member

AiyionPrime commented Feb 25, 2021 •
edited
Loading

Ich hätte gerne backups vom gitolite.
Was das ganze etwas schwieriger macht, weil die verschlüsselt sein sollten.

  • 3-5GB für gitolite (ein unkomprimierter snapshot sind derzeit etwa 750MB)

auf ns1 /home/git

  • gerne vollständig
  • verschlüsselt

@lemoer
Copy link
Contributor Author

lemoer commented Feb 26, 2021 via email

@AiyionPrime
Copy link
Member

https://github.com/arillso/ansible.restic

lemoer added a commit that referenced this issue Feb 27, 2021
@lemoer
all: Add new role ffh.restic.
8433023 
For most of the nodes, this only backups /etc/. Some nodes specify
the restic_backup_paths variable in their host_vars and backup additional
paths.

Currently this is not rolled out yet, as @1997er has to create the
backup machine first. However, once he is finished, he can adjust the
remaining variables (marked with "TODO") in group_vars/all/backups.yml.
Then he can roll all playbooks with "-t restic" and things should work
out smoothly. Let's see...

Some discussion about this can be found in #118.
@lemoer lemoer mentioned this issue Feb 27, 2021
Merged
lemoer added a commit that referenced this issue Feb 27, 2021
@lemoer
all: Add new role ffh.restic.
98f3e47 
For most of the nodes, this only backups /etc/. Some nodes specify
the restic_backup_paths variable in their host_vars and backup additional
paths.

Currently this is not rolled out yet, as @1997er has to create the
backup machine first. However, once he is finished, he can adjust the
remaining variables (marked with "TODO") in group_vars/all/backups.yml.
Then he can roll all playbooks with "-t restic" and things should work
out smoothly. Let's see...

Some discussion about this can be found in #118.
@lemoer
Copy link
Contributor Author

lemoer commented Feb 27, 2021

I pushed my backup role in #176. The backup host is not available yet, therefore I created only a PR and did not merge it onto master directly. @1977er wants to deploy the backup host tomorrow. Then he can use #176 to install some initial backups.

#176 covers:

  • /etc/ on all hosts
  • /var/www/ on web (backup of wiki)
  • /home/git/ on ns1 (backup of gitolite)

Further stuff can be added later.

@1977er 1977er closed this as completed Mar 1, 2021
@lemoer lemoer reopened this Apr 11, 2021
@lemoer
Copy link
Contributor Author

lemoer commented Apr 11, 2021

@1977er Why did you close this?

lemoer added a commit that referenced this issue Apr 11, 2021
@lemoer
harvester: backup grafana (#118)
f016dc2
@lemoer
Copy link
Contributor Author

lemoer commented Apr 11, 2021 •
edited
Loading

I just pushed (and rolled) a commit to backup grafana. This currently takes ~110 MB of storage.

@lemoer
Copy link
Contributor Author

lemoer commented Apr 11, 2021 •
edited
Loading

Maybe we should discuss on Thursday what's left here? Or so? @1977er @AiyionPrime @CodeFetch

@1977er
Copy link
Member

1977er commented May 19, 2021

The only thing thats left open is whether we should/could backup influx.

@lemoer
Copy link
Contributor Author

lemoer commented Sep 10, 2021

Zabbix should definitely also be backed up.

@lemoer
Copy link
Contributor Author

lemoer commented Sep 14, 2021

Regarding zabbix: I looked into the size of the mysql tables:

+--------------------+----------------------------------------------------+-----------+
| Database           | Table                                              | Size (MB) |
+--------------------+----------------------------------------------------+-----------+
| zabbix             | history_uint                                       |     32686 |
| zabbix             | history                                            |      5075 |
| zabbix             | trends_uint                                        |      1423 |
| zabbix             | history_text                                       |       386 |
| zabbix             | trends                                             |       269 |
| zabbix             | history_str                                        |        20 |
| zabbix             | alerts                                             |         8 |
| zabbix             | items                                              |         5 |
| zabbix             | events                                             |         4 |
...

@lemoer
Copy link
Contributor Author

lemoer commented Sep 14, 2021

CC: @1977er Can we just leave out the tables history_uint, history, trends_uint, history_text, trends for the backup?

@1977er
Copy link
Member

1977er commented Dec 25, 2021

Spitting out a zabbix mysqldump without the forementioned big tables. Let restic backup the sql file. See zabbix-sql role.

@lemoer
Copy link
Contributor Author

lemoer commented Dec 27, 2021

Can we consider it done?

@lemoer
Copy link
Contributor Author

lemoer commented Dec 27, 2021

Yes. @1977er said in mumble, that it's done. So only influx is left now.

@1977er
Copy link
Member

1977er commented Nov 1, 2023

Influx is quite big. We could

  1. reduce the retention time from 6mo to 1w
  2. skip the data at all
  3. buy new hardware every now and then

I prefer 1) or 2).

@1977er
Copy link
Member

1977er commented Feb 7, 2024

I will go with 2) and close this issue on thursday if noone raises an objection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lemoer @1977er @CodeFetch @AiyionPrime