Gitea Down Because pg become read-only

[east4ming]

Describe the bug

After installing in high availability mode, gitea was rebooted, and after the reboot, gitea went down because the init container configure-gitea failed to run.

• I have read the documentation
• I have searched open issues

To reproduce

Steps to reproduce the behavior:

1. Install Homelab
2. Stop gitea
3. See error log

Expected behavior

Please guide me step by step on how to troubleshoot this problem.

Logs and screenshots

init container configure-gitea logs: Command error: pq: cannot execute UPDATE in a read-only transaction

==== BEGIN GITEA CONFIGURATION ====
2024/11/05 07:47:40 cmd/migrate.go:33:runMigrate() [I] AppPath: /usr/local/bin/gitea
2024/11/05 07:47:40 cmd/migrate.go:34:runMigrate() [I] AppWorkPath: /data
2024/11/05 07:47:40 cmd/migrate.go:35:runMigrate() [I] Custom path: /data/gitea
2024/11/05 07:47:40 cmd/migrate.go:36:runMigrate() [I] Log path: /data/log
2024/11/05 07:47:40 cmd/migrate.go:37:runMigrate() [I] Configuration file: /data/gitea/conf/app.ini
2024/11/05 07:47:40 [email protected]/command.go:267:Run() [I] PING DATABASE postgres
2024/11/05 07:47:40 [email protected]/command.go:267:Run() [W] Table oauth2_application Column confidential_client db default is true, struct default is TRUE
2024/11/05 07:47:40 [email protected]/command.go:267:Run() [W] Table system_setting Column version db default is , struct default is 1
2024/11/05 07:47:40 [email protected]/command.go:267:Run() [W] Table user Column max_repo_creation db default is '-1', struct default is -1
2024/11/05 07:47:41 [email protected]/command.go:267:Run() [W] Table label Column archived_unix db default is , struct default is NULL
Wait for redis to become avialable...
gitea-redis-cluster-headless.gitea.svc.cluster.local (10.0.0.21:6379) open
Admin account 'gitea_admin' already exist. Running update to sync password...
Command error: pq: cannot execute UPDATE in a read-only transaction

[east4ming]

The problem is resolved.
Here's my workaround, for reference, since I don't know anything about postgres. So I didn't do any postgres sql related operations. Here's how it works.

1. argocd disable auto-sync for argocd and gitea
2. scale down gitea
3. scale down gitea-postgresql-ha-pgpool
4. scale down gitea-postgresql-ha-postgresql
5. scale up gitea-postgresql-ha-postgresql replicas to 1
6. scale up gitea-postgresql-ha-postgresql replicas to 2, then 3
7. scale up gitea-postgresql-ha-postgresql
8. scale up gitea
9. RESOLVED.

[khuedoan]

Thank you for posting the solution! I'll close this issue as resolved.

[east4ming]

@khuedoan The same problem occurred again today.
To summarize these two issues, here's the overview:

1. rook-ceph or k8s node reboots. The rook-ceph or k8s node reboots and other behaviors that cause the rook-ceph storage or cluster state to be abnormal.
2. rook-ceph or k8s cluster state can be automatically restored to normal.
3. but gitea's pg statefulset is affected, becoming a read-only system, and is not able to heal itself once the underlying system is back to normal.
4. At the moment I can only intervene manually to recover.

Given this, I think a possible reason is that the bitnami pg used by gitea is not robust enough. Could you consider replacing it with a better k8s pg implementation, such as CloudNativePG?

Thanks!

[khuedoan]

Thanks for following up on the issue. I'll look into it.

For Gitea specifically, I'm considering switching to single-node mode with SQLite because I don't need HA for Gitea that much. With fewer components involved, there's less things that could go wrong (the tradeoff is accepting downtime during weekly upgrades, but I've scheduled those for when I'm asleep)

[east4ming]

Thanks for following up on the issue. I'll look into it.

For Gitea specifically, I'm considering switching to single-node mode with SQLite because I don't need HA for Gitea that much. With fewer components involved, there's less things that could go wrong (the tradeoff is accepting downtime during weekly upgrades, but I've scheduled those for when I'm asleep)

I agree.