feat: add NAS backup, verification, and DR scripts
Some checks failed
Deploy to Production / Build and Deploy (push) Has been cancelled
Some checks failed
Deploy to Production / Build and Deploy (push) Has been cancelled
This commit is contained in:
@@ -1,102 +1,319 @@
|
|||||||
# Playbook de Sauvegarde Complète & Reprise d'Activité (Disaster Recovery)
|
# Disaster Recovery — Wordly.art
|
||||||
> Gestion des pannes matérielles, sauvegarde de Nginx Proxy Manager (NPM) et transfert distant (sans NAS).
|
## Guide opérationnel complet
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🎯 Objectif
|
## Architecture
|
||||||
Ce document explique comment automatiser la sauvegarde et restaurer l'intégralité de la plateforme SaaS **Wordly.art** (Base de données, fichier de configuration `.env` contenant vos secrets, et configurations de routage SSL/Proxy de **Nginx Proxy Manager**) sur un nouveau serveur en cas de crash du serveur principal.
|
|
||||||
|
|
||||||
---
|
```
|
||||||
|
[ Internet ]
|
||||||
## ⚙️ 1. Variables de configuration dans le `.env`
|
│
|
||||||
|
▼ (80/443)
|
||||||
Pour activer les options de reprise d'activité, ajoutez ces variables dans votre fichier `.env` de production :
|
┌────────────────────────────────────┐
|
||||||
|
│ NPM dédié : 192.168.1.184 │ ← STABLE (ne tombe pas)
|
||||||
```ini
|
│ Interface admin : :81 │
|
||||||
# ============== Configuration Disaster Recovery (DR) ==============
|
└────────────┬───────────────────────┘
|
||||||
# Choix de la destination : LOCAL, NAS, ou SCP
|
│ Forward Hostname → IP du serveur actif
|
||||||
BACKUP_DEST_TYPE=LOCAL
|
▼
|
||||||
# Chemin local ou point de montage (ex: /mnt/nas-backups/wordly)
|
┌────────────────────────────────────┐
|
||||||
BACKUP_DEST_PATH=/var/backups/wordly
|
│ Serveur APP : 192.168.1.151 │ ← PEUT CRASHER
|
||||||
|
│ Docker: postgres, redis, │
|
||||||
# Configuration SSH/SCP (requis uniquement si BACKUP_DEST_TYPE=SCP)
|
│ backend:8001, frontend:3000│
|
||||||
SCP_HOST=192.168.1.200
|
└────────────┬───────────────────────┘
|
||||||
SCP_USER=backup_user
|
│ rsync SSH toutes les 6h (cron)
|
||||||
SCP_KEY_PATH=/root/.ssh/id_rsa
|
▼
|
||||||
SCP_PORT=22
|
┌────────────────────────────────────────────────┐
|
||||||
SCP_DEST_PATH=/var/backups/wordly_saas
|
│ NAS Synology : 192.168.1.146 │ ← SOURCE DE VÉRITÉ
|
||||||
|
│ Chemin réel : /volume1/backups/wordly │
|
||||||
# Configurations des dossiers de Nginx Proxy Manager (NPM)
|
│ Accès : SSH key (wordly-backup@nas) │
|
||||||
# Laissez vide si NPM tourne sur une autre machine et n'est pas géré ici.
|
│ Pas de montage CIFS — rsync direct │
|
||||||
NPM_DATA_DIR=/opt/npm/data
|
└────────────┬───────────────────────────────────┘
|
||||||
NPM_LETSENCRYPT_DIR=/opt/npm/letsencrypt
|
│ (en cas de crash de .151)
|
||||||
|
▼
|
||||||
|
┌────────────────────────────────────┐
|
||||||
|
│ Serveur SECOURS : 192.168.1.98 │ ← Docker déjà installé
|
||||||
|
│ Restaure via rsync SSH depuis NAS│
|
||||||
|
│ → NPM redirigé automatiquement │
|
||||||
|
└────────────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Pourquoi rsync SSH et pas CIFS/SMB ?**
|
||||||
|
- Pas de montage à gérer, pas de `/etc/fstab` à configurer
|
||||||
|
- Fonctionne même si le NAS redémarre (pas de montage stale)
|
||||||
|
- Chemin exact `/volume1/backups/wordly` utilisable directement
|
||||||
|
- SSH chiffré, clé sans mot de passe pour l'automatisation
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🛠️ 2. Comment configurer la sauvegarde à distance (Mode SCP)
|
## RPO / RTO
|
||||||
|
|
||||||
Si vous n'avez pas de NAS, le mode **SCP** permet d'envoyer chaque nuit l'archive complète vers une autre machine ou ordinateur de votre réseau local (ex: `192.168.1.200`).
|
| Scénario | Données perdues max | Temps de remise en route | Procédure |
|
||||||
|
|----------|--------------------|--------------------------|-|
|
||||||
|
| Container crashe | 0 | ~30s | Autorestart Docker |
|
||||||
|
| Process PostgreSQL crashe | 0–5s | ~1 min | Autorestart + WAL |
|
||||||
|
| Corruption DB partielle | 0–6h | ~5 min | Restore depuis NAS |
|
||||||
|
| Serveur .151 mort | 0–6h | **~25 min** | Restore NAS sur .98 + NPM auto |
|
||||||
|
| Erreur humaine (DROP) | 0–6h | ~5 min | Restore snapshot précédent |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Ce qui est sauvegardé
|
||||||
|
|
||||||
|
| Composant | Sauvegardé | Fréquence |
|
||||||
|
|-----------|-----------|-----------|
|
||||||
|
| PostgreSQL `pg_dump` | ✅ | Toutes les 6h |
|
||||||
|
| `.env` (secrets, clés API, Stripe...) | ✅ | Dans chaque archive DR |
|
||||||
|
| `docker-compose.yml` | ✅ | Dans chaque archive DR |
|
||||||
|
| Dossier `docker/` (configs) | ✅ | Dans chaque archive DR |
|
||||||
|
| Redis | ❌ | Cache — sessions perdues à la restore (reconnexion users) |
|
||||||
|
| Config NPM | ❌ | NPM sur .184 (stable). Seul Forward Host change via API. |
|
||||||
|
| Métriques Prometheus | ❌ | Non critique, repart de zéro |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. SETUP INITIAL (une seule fois sur .151)
|
||||||
|
|
||||||
|
### Étape 1 : Créer le compte sur le NAS Synology
|
||||||
|
|
||||||
|
**Connectez-vous à l'interface DSM : `http://192.168.1.146:5000`**
|
||||||
|
|
||||||
|
#### 1a. Créer l'utilisateur dédié
|
||||||
|
|
||||||
|
```
|
||||||
|
DSM → Panneau de configuration → Utilisateurs et groupes → Créer
|
||||||
|
Nom d'utilisateur : wordly-backup
|
||||||
|
Mot de passe : [choisissez un mot de passe fort]
|
||||||
|
☑ L'utilisateur ne peut pas changer son mot de passe
|
||||||
|
→ Suivant
|
||||||
|
|
||||||
|
Permissions sur les dossiers partagés :
|
||||||
|
backups → ☑ Lecture/Écriture
|
||||||
|
→ Suivant → Terminer
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 1b. Activer SSH sur le NAS
|
||||||
|
|
||||||
|
```
|
||||||
|
DSM → Panneau de configuration → Terminal et SNMP
|
||||||
|
☑ Activer le service SSH
|
||||||
|
Port : 22 (ou autre si vous avez changé)
|
||||||
|
→ Appliquer
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 1c. Créer le dossier wordly sur le NAS
|
||||||
|
|
||||||
### Étape A : Générer une clé SSH sur le serveur principal
|
|
||||||
Sur le serveur applicatif (`192.168.1.151`), si vous n'avez pas de clé SSH :
|
|
||||||
```bash
|
```bash
|
||||||
sudo ssh-keygen -t rsa -b 4096 -N "" -f /root/.ssh/id_rsa
|
# Depuis votre poste (ou n'importe quelle machine sur le réseau) :
|
||||||
```
|
ssh admin@192.168.1.146
|
||||||
|
mkdir -p /volume1/backups/wordly/snapshots
|
||||||
### Étape B : Autoriser la connexion sur la machine de backup
|
mkdir -p /volume1/backups/wordly/scripts
|
||||||
Copiez la clé publique sur votre machine de sauvegarde (`192.168.1.200`) :
|
chown -R wordly-backup:users /volume1/backups/wordly
|
||||||
```bash
|
exit
|
||||||
sudo ssh-copy-id -i /root/.ssh/id_rsa.pub backup_user@192.168.1.200
|
|
||||||
```
|
|
||||||
*Vérification* : Exécutez `sudo ssh -i /root/.ssh/id_rsa backup_user@192.168.1.200` depuis le serveur principal. Vous devez vous connecter **sans saisir de mot de passe**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📅 3. Automatisation quotidienne
|
|
||||||
|
|
||||||
Ajoutez le script à votre crontab pour qu'il s'exécute automatiquement chaque nuit à 03h30 :
|
|
||||||
```bash
|
|
||||||
sudo crontab -e
|
|
||||||
```
|
|
||||||
Ajoutez cette ligne tout à la fin :
|
|
||||||
```cron
|
|
||||||
30 3 * * * /opt/wordly/scripts/disaster-recovery.sh --backup >> /var/log/wordly-dr-backup.log 2>&1
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🚨 4. Procédure de restauration sur un nouveau serveur (Failover)
|
### Étape 2 : Configurer les variables dans `.env` sur `.151`
|
||||||
|
|
||||||
Si le serveur principal crashe complètement et que vous devez remonter l'infrastructure sur un serveur de secours (ex: `192.168.1.152`) :
|
|
||||||
|
|
||||||
### Étape 4.1 : Récupérer l'archive de sauvegarde
|
|
||||||
Récupérez le dernier fichier `wordly_dr_TIMESTAMP.tar.gz` depuis votre stockage de backup (NAS, machine de backup distante via SCP, ou clé USB).
|
|
||||||
|
|
||||||
### Étape 4.2 : Installer Docker sur le nouveau serveur
|
|
||||||
```bash
|
```bash
|
||||||
curl -fsSL https://get.docker.com | sh
|
# ── NAS SSH ───────────────────────────────────
|
||||||
sudo usermod -aG docker $USER && newgrp docker
|
NAS_HOST=192.168.1.146
|
||||||
|
NAS_USER=wordly-backup
|
||||||
|
NAS_PATH=/volume1/backups/wordly
|
||||||
|
NAS_SSH_PORT=22
|
||||||
|
NAS_SSH_KEY=/root/.ssh/wordly_nas_key
|
||||||
|
|
||||||
|
# ── Alertes Telegram ──────────────────────────
|
||||||
|
TELEGRAM_BOT_TOKEN= # Voir section "Créer un bot Telegram" ci-dessous
|
||||||
|
TELEGRAM_CHAT_ID= # Votre chat ID personnel
|
||||||
|
|
||||||
|
# ── NPM Failover API ──────────────────────────
|
||||||
|
NPM_API_URL=http://192.168.1.184:81/api
|
||||||
|
NPM_ADMIN_EMAIL=admin@wordly.art
|
||||||
|
NPM_ADMIN_PASSWORD=VotreMotDePasseNPM
|
||||||
|
NPM_PROXY_HOST_DOMAIN=wordly.art
|
||||||
|
|
||||||
|
# ── Rétention ────────────────────────────────
|
||||||
|
DAILY_RETENTION=7
|
||||||
|
WEEKLY_RETENTION=4
|
||||||
|
MONTHLY_RETENTION=6
|
||||||
|
DR_RETENTION_DAYS=30
|
||||||
```
|
```
|
||||||
|
|
||||||
### Étape 4.3 : Lancer la restauration automatique
|
---
|
||||||
1. Créez le dossier de destination et placez-vous dedans :
|
|
||||||
```bash
|
|
||||||
sudo mkdir -p /opt/wordly
|
|
||||||
cd /opt/wordly
|
|
||||||
```
|
|
||||||
2. Lancez le script de restauration à partir de l'archive (le script va extraire le `.env`, copier le `docker-compose.yml`, restaurer les configurations et certificats SSL de NPM, démarrer Docker et réinjecter les données de la base de données) :
|
|
||||||
```bash
|
|
||||||
# Remplacez par le nom ou le chemin exact de votre archive
|
|
||||||
bash /chemin/vers/votre/archive/scripts/disaster-recovery.sh --restore /chemin/vers/votre/archive/wordly_dr_20260607_033000.tar.gz
|
|
||||||
```
|
|
||||||
3. Validez l'action en saisissant `RESTORE-ALL` lorsque le script vous le demande.
|
|
||||||
|
|
||||||
### Étape 4.4 : Redirection du trafic réseau
|
### Étape 3 : Créer un bot Telegram (5 minutes)
|
||||||
Puisque le serveur a changé d'adresse IP (de `192.168.1.151` à `192.168.1.152`) :
|
|
||||||
|
|
||||||
#### Cas A : Si NPM tournait sur le serveur qui a crashé
|
1. Ouvrir Telegram → chercher **@BotFather**
|
||||||
Le script a restauré NPM sur la nouvelle machine. Vous devez simplement aller sur le routeur de votre box internet et modifier la redirection des ports **80** et **443** (Port Forwarding) pour qu'ils pointent vers la nouvelle IP `192.168.1.152` au lieu de `192.168.1.151`.
|
2. Envoyer `/newbot`
|
||||||
|
3. Nom : `Wordly Monitoring` / Username : `wordly_monitor_bot`
|
||||||
|
4. Copier le token → `TELEGRAM_BOT_TOKEN`
|
||||||
|
5. Envoyer un message à votre bot
|
||||||
|
6. Aller sur `https://api.telegram.org/bot<TOKEN>/getUpdates`
|
||||||
|
7. Copier le `chat.id` → `TELEGRAM_CHAT_ID`
|
||||||
|
|
||||||
#### Cas B : Si NPM tourne sur une machine externe dédiée
|
---
|
||||||
Connectez-vous à l'interface web de votre NPM (http://IP_NPM:81), modifiez les Proxy Hosts de `wordly.art` et changez le champ **Forward Hostname/IP** pour remplacer `192.168.1.151` par la nouvelle IP `192.168.1.152`.
|
|
||||||
|
### Étape 4 : Configurer SSH sans mot de passe vers le NAS
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Sur le serveur .151 (en root)
|
||||||
|
sudo bash scripts/setup-nas.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Ce script :
|
||||||
|
- Génère une clé SSH dédiée : `/root/.ssh/wordly_nas_key`
|
||||||
|
- La copie sur le NAS (**mot de passe demandé une seule fois**)
|
||||||
|
- Teste la connexion sans mot de passe
|
||||||
|
- Crée la structure de dossiers sur le NAS
|
||||||
|
- Configure `~/.ssh/config` avec l'alias `wordly-nas`
|
||||||
|
- Copie les scripts sur le NAS (disponibles depuis `.98` pour la restauration)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Étape 5 : Tester le premier backup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/backup-to-nas.sh --full
|
||||||
|
|
||||||
|
# Vérifier que l'archive est bien arrivée sur le NAS
|
||||||
|
bash scripts/backup-to-nas.sh --list
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Étape 6 : Tester la vérification automatique
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/verify-backups.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Étape 7 : Tester le failover NPM (sans rien changer)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/npm-failover.sh --dry-run --target-ip 192.168.1.98
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Étape 8 : Activer les crons
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/install-crontab.sh
|
||||||
|
crontab -l # Vérifier
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. VÉRIFICATION QUOTIDIENNE (automatique)
|
||||||
|
|
||||||
|
```
|
||||||
|
0 */6 * * * backup-to-nas.sh → Snapshot DB + archive → NAS via rsync SSH
|
||||||
|
30 */6 * * * verify-backups.sh → 8 vérifications + alerte Telegram si erreur
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. RESTAURATION D'URGENCE (quand .151 est mort)
|
||||||
|
|
||||||
|
> **Durée estimée : 20–25 minutes**
|
||||||
|
|
||||||
|
### Sur le serveur de secours `192.168.1.98`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Installer les prérequis (Docker déjà installé)
|
||||||
|
apt-get install -y rsync jq
|
||||||
|
|
||||||
|
# 2. Récupérer la clé SSH depuis le NAS (ou depuis une autre source sécurisée)
|
||||||
|
# Option A : copier la clé depuis un endroit sûr (gestionnaire de mots de passe, etc.)
|
||||||
|
mkdir -p /root/.ssh && chmod 700 /root/.ssh
|
||||||
|
# collez le contenu de /root/.ssh/wordly_nas_key ici
|
||||||
|
nano /root/.ssh/wordly_nas_key
|
||||||
|
chmod 600 /root/.ssh/wordly_nas_key
|
||||||
|
|
||||||
|
# 3. Tester la connexion NAS
|
||||||
|
ssh -i /root/.ssh/wordly_nas_key wordly-backup@192.168.1.146 "echo OK"
|
||||||
|
|
||||||
|
# 4. Voir les archives disponibles
|
||||||
|
ssh -i /root/.ssh/wordly_nas_key wordly-backup@192.168.1.146 \
|
||||||
|
"ls -lht /volume1/backups/wordly/snapshots/ | head -10"
|
||||||
|
|
||||||
|
# 5. Télécharger la dernière archive depuis le NAS
|
||||||
|
rsync -az \
|
||||||
|
-e "ssh -i /root/.ssh/wordly_nas_key" \
|
||||||
|
wordly-backup@192.168.1.146:/volume1/backups/wordly/snapshots/wordly_dr_TIMESTAMP.tar.gz \
|
||||||
|
/tmp/
|
||||||
|
|
||||||
|
# 6. Télécharger les scripts de restauration depuis le NAS
|
||||||
|
rsync -az \
|
||||||
|
-e "ssh -i /root/.ssh/wordly_nas_key" \
|
||||||
|
wordly-backup@192.168.1.146:/volume1/backups/wordly/scripts/ \
|
||||||
|
/opt/wordly/scripts/
|
||||||
|
|
||||||
|
# 7. Lancer la restauration complète
|
||||||
|
bash /opt/wordly/scripts/disaster-recovery.sh \
|
||||||
|
--restore /tmp/wordly_dr_TIMESTAMP.tar.gz
|
||||||
|
```
|
||||||
|
|
||||||
|
**Le script fait automatiquement :**
|
||||||
|
1. Extrait `.env`, `docker-compose.yml`, configs Docker
|
||||||
|
2. Lance tous les containers Docker
|
||||||
|
3. Attend que PostgreSQL soit healthy
|
||||||
|
4. Restaure le dump SQL
|
||||||
|
5. Health check sur `http://localhost:8001/health` (max 180s)
|
||||||
|
6. **Si OK → appelle NPM API → bascule le trafic vers `192.168.1.98`**
|
||||||
|
7. **Alerte Telegram : "✅ Wordly.art DR COMPLET"**
|
||||||
|
|
||||||
|
**Si NPM failover automatique échoue (dernier recours) :**
|
||||||
|
```
|
||||||
|
http://192.168.1.184:81 → Proxy Hosts → wordly.art → Edit
|
||||||
|
Forward Hostname : 192.168.1.98
|
||||||
|
→ Save
|
||||||
|
# Changement immédiat, 0 redémarrage nécessaire
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. CONSERVATION DE LA CLÉ SSH NAS
|
||||||
|
|
||||||
|
> [!IMPORTANT]
|
||||||
|
> La clé `/root/.ssh/wordly_nas_key` est **critique** pour la restauration depuis `.98`.
|
||||||
|
> Conservez-la dans au minimum 2 endroits sécurisés :
|
||||||
|
> - Gestionnaire de mots de passe (Bitwarden, 1Password, etc.)
|
||||||
|
> - Coffre-fort KeePass chiffré sur un support physique
|
||||||
|
>
|
||||||
|
> Sans cette clé, vous ne pouvez pas accéder aux archives sur le NAS depuis `.98`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. SCRIPTS DE RÉFÉRENCE
|
||||||
|
|
||||||
|
| Script | Usage | Déclenchement |
|
||||||
|
|--------|-------|---------------|
|
||||||
|
| `setup-nas.sh` | Configure SSH → NAS, génère clé, copie scripts | **Once** (root requis) |
|
||||||
|
| `backup-to-nas.sh` | pg_dump + archive DR → NAS via rsync SSH | Cron toutes les 6h |
|
||||||
|
| `backup-to-nas.sh --list` | Lister les archives disponibles sur le NAS | Manuel |
|
||||||
|
| `verify-backups.sh` | 8 checks intégrité + Telegram | Cron toutes les 6h+30m |
|
||||||
|
| `disaster-recovery.sh --backup` | Archive DR → NAS | Inclus dans backup-to-nas |
|
||||||
|
| `disaster-recovery.sh --restore <archive>` | Restauration complète | **Urgence** |
|
||||||
|
| `npm-failover.sh --target-ip <IP>` | Bascule NPM vers une IP | Appelé automatiquement |
|
||||||
|
| `npm-failover.sh --dry-run --target-ip <IP>` | Test sans modifier NPM | Test initial |
|
||||||
|
| `install-crontab.sh` | Installe les crons | **Once** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. LOGS
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Logs backup (sur .151)
|
||||||
|
tail -f /var/log/wordly-backup.log
|
||||||
|
|
||||||
|
# Logs vérification (sur .151)
|
||||||
|
tail -f /var/log/wordly-verify.log
|
||||||
|
|
||||||
|
# Logs Docker (sur le serveur actif)
|
||||||
|
docker compose logs -f backend
|
||||||
|
docker compose logs -f postgres
|
||||||
|
```
|
||||||
|
|||||||
@@ -1,287 +1,356 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# ============================================
|
# ==============================================================================
|
||||||
# Wordly.art - PostgreSQL Backup to NAS
|
# Wordly.art - PostgreSQL Backup vers NAS Synology via SSH/rsync
|
||||||
# ============================================
|
# ==============================================================================
|
||||||
# CRON: Run daily at 03:00
|
# Sauvegarde la base PostgreSQL et l'archive DR sur le NAS via SSH/rsync.
|
||||||
# 0 3 * * * /opt/wordly/scripts/backup-to-nas.sh >> /var/log/wordly-backup.log 2>&1
|
# Pas de montage CIFS — rsync SSH direct sur /volume1/backups/wordly.
|
||||||
#
|
#
|
||||||
# Usage:
|
# CRON (installé par install-crontab.sh) :
|
||||||
# ./backup-to-nas.sh # Default: daily backup
|
# 0 */6 * * * bash /opt/wordly/scripts/backup-to-nas.sh >> /var/log/wordly-backup.log 2>&1
|
||||||
# ./backup-to-nas.sh --full # Full backup with upload cleanup
|
#
|
||||||
# ./backup-to-nas.sh --restore FILE # Restore from specific backup
|
# Usage :
|
||||||
# ============================================
|
# ./backup-to-nas.sh # Backup complet → NAS
|
||||||
|
# ./backup-to-nas.sh --full # Identique (alias explicite)
|
||||||
|
# ./backup-to-nas.sh --list # Lister les archives disponibles sur le NAS
|
||||||
|
# ==============================================================================
|
||||||
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
# ===========================================
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
# CONFIGURATION - MODIFY THESE VALUES
|
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||||
# ===========================================
|
|
||||||
# NAS settings (SMB/CIFS or NFS mount point)
|
|
||||||
NAS_BACKUP_DIR="/mnt/nas-backups/wordly"
|
|
||||||
|
|
||||||
# Docker container name for PostgreSQL
|
# ==============================================================================
|
||||||
POSTGRES_CONTAINER="wordly-postgres"
|
# CHARGER LE .env
|
||||||
POSTGRES_USER="translate"
|
# ==============================================================================
|
||||||
POSTGRES_DB="translate_db"
|
ENV_FILE="${PROJECT_ROOT}/.env"
|
||||||
POSTGRES_PASSWORD="yLLgkEvt6mvzGDdoqtQvI1vEgMmR-W75ZTPW5StaIAU"
|
if [ -f "${ENV_FILE}" ]; then
|
||||||
|
set -a
|
||||||
|
source "${ENV_FILE}"
|
||||||
|
set +a
|
||||||
|
else
|
||||||
|
echo "ERROR: .env introuvable : ${ENV_FILE}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
# Backup retention
|
# ==============================================================================
|
||||||
DAILY_RETENTION=7 # Keep 7 daily backups
|
# CONFIGURATION (depuis .env)
|
||||||
WEEKLY_RETENTION=4 # Keep 4 weekly backups
|
# ==============================================================================
|
||||||
MONTHLY_RETENTION=6 # Keep 6 monthly backups
|
|
||||||
|
|
||||||
# Notification (optional - leave empty to disable)
|
# NAS SSH
|
||||||
NOTIFICATION_WEBHOOK="" # Slack/Discord webhook URL
|
NAS_HOST="${NAS_HOST:-192.168.1.146}"
|
||||||
|
NAS_USER="${NAS_USER:-wordly-backup}"
|
||||||
|
NAS_PATH="${NAS_PATH:-/volume1/backups/wordly}"
|
||||||
|
NAS_SSH_PORT="${NAS_SSH_PORT:-22}"
|
||||||
|
NAS_SSH_KEY="${NAS_SSH_KEY:-/root/.ssh/wordly_nas_key}"
|
||||||
|
|
||||||
# ===========================================
|
# PostgreSQL
|
||||||
|
POSTGRES_CONTAINER="${POSTGRES_CONTAINER:-wordly-postgres}"
|
||||||
|
POSTGRES_USER="${POSTGRES_USER:-translate}"
|
||||||
|
POSTGRES_DB="${POSTGRES_DB:-translate_db}"
|
||||||
|
POSTGRES_PASSWORD="${POSTGRES_PASSWORD:?POSTGRES_PASSWORD doit être défini dans .env}"
|
||||||
|
|
||||||
|
# Rétention sur le NAS (nombre d'archives à garder)
|
||||||
|
DAILY_RETENTION=${DAILY_RETENTION:-7}
|
||||||
|
WEEKLY_RETENTION=${WEEKLY_RETENTION:-4}
|
||||||
|
MONTHLY_RETENTION=${MONTHLY_RETENTION:-6}
|
||||||
|
|
||||||
|
# Telegram
|
||||||
|
TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN:-}"
|
||||||
|
TELEGRAM_CHAT_ID="${TELEGRAM_CHAT_ID:-}"
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
# INTERNALS
|
# INTERNALS
|
||||||
# ===========================================
|
# ==============================================================================
|
||||||
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
|
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
|
||||||
DATE_ONLY=$(date +"%Y-%m-%d")
|
DAY_OF_WEEK=$(date +"%u") # 1=Lun, 7=Dim
|
||||||
DAY_OF_WEEK=$(date +"%u") # 1=Mon, 7=Sun
|
|
||||||
DAY_OF_MONTH=$(date +"%d")
|
DAY_OF_MONTH=$(date +"%d")
|
||||||
BACKUP_NAME="wordly_db_${TIMESTAMP}.sql.gz"
|
SNAPSHOT_NAME="wordly_dr_${TIMESTAMP}.tar.gz"
|
||||||
BACKUP_PATH="${NAS_BACKUP_DIR}/${BACKUP_NAME}"
|
LOCAL_TMP="/tmp/wordly_backup_${TIMESTAMP}"
|
||||||
LOG_PREFIX="[Wordly Backup ${TIMESTAMP}]"
|
SSH_CMD="ssh -i ${NAS_SSH_KEY} -p ${NAS_SSH_PORT} -o BatchMode=yes -o ConnectTimeout=10"
|
||||||
|
RSYNC_CMD="rsync -az -e 'ssh -i ${NAS_SSH_KEY} -p ${NAS_SSH_PORT} -o BatchMode=yes -o ConnectTimeout=10'"
|
||||||
|
|
||||||
# Colors for terminal output
|
|
||||||
RED='\033[0;31m'
|
RED='\033[0;31m'
|
||||||
GREEN='\033[0;32m'
|
GREEN='\033[0;32m'
|
||||||
YELLOW='\033[1;33m'
|
YELLOW='\033[1;33m'
|
||||||
NC='\033[0m'
|
NC='\033[0m'
|
||||||
|
LOG_PREFIX="[Backup ${TIMESTAMP}]"
|
||||||
|
|
||||||
# ===========================================
|
log() { echo "${LOG_PREFIX} $1"; }
|
||||||
# FUNCTIONS
|
log_success() { echo -e "${LOG_PREFIX} ${GREEN}✅ $1${NC}"; }
|
||||||
# ===========================================
|
log_error() { echo -e "${LOG_PREFIX} ${RED}❌ ERROR: $1${NC}"; }
|
||||||
|
log_warning() { echo -e "${LOG_PREFIX} ${YELLOW}⚠️ $1${NC}"; }
|
||||||
|
|
||||||
log() {
|
# ==============================================================================
|
||||||
echo "${LOG_PREFIX} $1"
|
# TELEGRAM
|
||||||
}
|
# ==============================================================================
|
||||||
|
send_telegram() {
|
||||||
log_success() {
|
|
||||||
echo -e "${LOG_PREFIX} ${GREEN}$1${NC}"
|
|
||||||
}
|
|
||||||
|
|
||||||
log_error() {
|
|
||||||
echo -e "${LOG_PREFIX} ${RED}ERROR: $1${NC}"
|
|
||||||
}
|
|
||||||
|
|
||||||
log_warning() {
|
|
||||||
echo -e "${LOG_PREFIX} ${YELLOW}WARNING: $1${NC}"
|
|
||||||
}
|
|
||||||
|
|
||||||
send_notification() {
|
|
||||||
local message="$1"
|
local message="$1"
|
||||||
if [ -n "${NOTIFICATION_WEBHOOK}" ]; then
|
if [ -n "${TELEGRAM_BOT_TOKEN}" ] && [ -n "${TELEGRAM_CHAT_ID}" ]; then
|
||||||
curl -s -X POST "${NOTIFICATION_WEBHOOK}" \
|
curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
|
||||||
-H "Content-Type: application/json" \
|
-d "chat_id=${TELEGRAM_CHAT_ID}" \
|
||||||
-d "{\"text\": \"${message}\"}" > /dev/null 2>&1 || true
|
-d "text=${message}" \
|
||||||
|
-d "parse_mode=Markdown" \
|
||||||
|
>/dev/null 2>&1 || true
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# PRÉREQUIS
|
||||||
|
# ==============================================================================
|
||||||
check_prerequisites() {
|
check_prerequisites() {
|
||||||
# Check NAS mount
|
log "Vérification des prérequis..."
|
||||||
if [ ! -d "${NAS_BACKUP_DIR}" ]; then
|
|
||||||
log_error "NAS backup directory not found: ${NAS_BACKUP_DIR}"
|
|
||||||
log "Attempting to mount NAS..."
|
|
||||||
|
|
||||||
# Try to mount if configured via /etc/fstab
|
# Clé SSH
|
||||||
mount "${NAS_BACKUP_DIR}" 2>/dev/null || true
|
if [ ! -f "${NAS_SSH_KEY}" ]; then
|
||||||
|
log_error "Clé SSH introuvable : ${NAS_SSH_KEY}"
|
||||||
if [ ! -d "${NAS_BACKUP_DIR}" ]; then
|
log_error "Lancez d'abord : sudo bash scripts/setup-nas.sh"
|
||||||
log_error "Cannot mount NAS. Aborting."
|
|
||||||
send_notification "Wordly Backup FAILED: NAS not mounted at ${NAS_BACKUP_DIR}"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check Docker
|
|
||||||
if ! docker ps --format '{{.Names}}' | grep -q "${POSTGRES_CONTAINER}"; then
|
|
||||||
log_error "PostgreSQL container '${POSTGRES_CONTAINER}' is not running."
|
|
||||||
send_notification "Wordly Backup FAILED: PostgreSQL container not running"
|
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
log_success "Prerequisites OK"
|
# Connectivité SSH vers le NAS
|
||||||
}
|
if ! ${SSH_CMD} "${NAS_USER}@${NAS_HOST}" "echo OK" >/dev/null 2>&1; then
|
||||||
|
log_error "Impossible de se connecter au NAS ${NAS_HOST} via SSH."
|
||||||
create_backup() {
|
log_error "Vérifiez : ssh -i ${NAS_SSH_KEY} ${NAS_USER}@${NAS_HOST}"
|
||||||
log "Starting backup of '${POSTGRES_DB}'..."
|
send_telegram "🚨 *Wordly Backup ÉCHOUÉ*
|
||||||
|
NAS inaccessible : ${NAS_HOST}
|
||||||
# Create backup directory structure
|
Date : $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
mkdir -p "${NAS_BACKUP_DIR}/daily"
|
|
||||||
mkdir -p "${NAS_BACKUP_DIR}/weekly"
|
|
||||||
mkdir -p "${NAS_BACKUP_DIR}/monthly"
|
|
||||||
|
|
||||||
# Run pg_dump inside Docker container
|
|
||||||
docker exec "${POSTGRES_CONTAINER}" pg_dump \
|
|
||||||
-U "${POSTGRES_USER}" \
|
|
||||||
-d "${POSTGRES_DB}" \
|
|
||||||
--format=custom \
|
|
||||||
--compress=9 \
|
|
||||||
--no-owner \
|
|
||||||
--no-acl \
|
|
||||||
2>/dev/null | gzip > "${NAS_BACKUP_DIR}/daily/${BACKUP_NAME}"
|
|
||||||
|
|
||||||
local backup_size=$(du -h "${NAS_BACKUP_DIR}/daily/${BACKUP_NAME}" | cut -f1)
|
|
||||||
|
|
||||||
if [ -f "${NAS_BACKUP_DIR}/daily/${BACKUP_NAME}" ]; then
|
|
||||||
log_success "Backup created: ${BACKUP_NAME} (${backup_size})"
|
|
||||||
|
|
||||||
# Copy to weekly/monthly if applicable
|
|
||||||
if [ "${DAY_OF_WEEK}" = "7" ]; then
|
|
||||||
cp "${NAS_BACKUP_DIR}/daily/${BACKUP_NAME}" "${NAS_BACKUP_DIR}/weekly/"
|
|
||||||
log "Weekly backup copied"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ "${DAY_OF_MONTH}" = "01" ]; then
|
|
||||||
cp "${NAS_BACKUP_DIR}/daily/${BACKUP_NAME}" "${NAS_BACKUP_DIR}/monthly/"
|
|
||||||
log "Monthly backup copied"
|
|
||||||
fi
|
|
||||||
|
|
||||||
send_notification "Wordly Backup SUCCESS: ${BACKUP_NAME} (${backup_size})"
|
|
||||||
else
|
|
||||||
log_error "Backup file was not created!"
|
|
||||||
send_notification "Wordly Backup FAILED: pg_dump produced no output"
|
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
log_success "NAS SSH : OK"
|
||||||
|
|
||||||
|
# Docker + container PostgreSQL
|
||||||
|
if ! docker ps --format '{{.Names}}' 2>/dev/null | grep -q "^${POSTGRES_CONTAINER}$"; then
|
||||||
|
log_error "Container PostgreSQL '${POSTGRES_CONTAINER}' n'est pas en cours d'exécution !"
|
||||||
|
send_telegram "🚨 *Wordly Backup ÉCHOUÉ*
|
||||||
|
PostgreSQL container non trouvé
|
||||||
|
Date : $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
log_success "PostgreSQL container : OK"
|
||||||
}
|
}
|
||||||
|
|
||||||
cleanup_old_backups() {
|
# ==============================================================================
|
||||||
log "Cleaning up old backups..."
|
# BACKUP POSTGRESQL
|
||||||
|
# ==============================================================================
|
||||||
|
backup_postgres() {
|
||||||
|
log "Dump PostgreSQL de '${POSTGRES_DB}'..."
|
||||||
|
mkdir -p "${LOCAL_TMP}"
|
||||||
|
|
||||||
# Daily: keep last N days
|
local dump_file="${LOCAL_TMP}/db_${TIMESTAMP}.dump.gz"
|
||||||
local daily_count=$(ls -1 "${NAS_BACKUP_DIR}/daily/" 2>/dev/null | wc -l)
|
|
||||||
if [ "${daily_count}" -gt "${DAILY_RETENTION}" ]; then
|
|
||||||
ls -1t "${NAS_BACKUP_DIR}/daily/" | tail -n +$((DAILY_RETENTION + 1)) | while read -r f; do
|
|
||||||
rm -f "${NAS_BACKUP_DIR}/daily/${f}"
|
|
||||||
log " Deleted daily: ${f}"
|
|
||||||
done
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Weekly: keep last N weeks
|
if ! docker exec \
|
||||||
local weekly_count=$(ls -1 "${NAS_BACKUP_DIR}/weekly/" 2>/dev/null | wc -l)
|
-e PGPASSWORD="${POSTGRES_PASSWORD}" \
|
||||||
if [ "${weekly_count}" -gt "${WEEKLY_RETENTION}" ]; then
|
"${POSTGRES_CONTAINER}" \
|
||||||
ls -1t "${NAS_BACKUP_DIR}/weekly/" | tail -n +$((WEEKLY_RETENTION + 1)) | while read -r f; do
|
pg_dump \
|
||||||
rm -f "${NAS_BACKUP_DIR}/weekly/${f}"
|
-U "${POSTGRES_USER}" \
|
||||||
log " Deleted weekly: ${f}"
|
-d "${POSTGRES_DB}" \
|
||||||
done
|
--format=custom \
|
||||||
fi
|
--no-owner \
|
||||||
|
--no-acl \
|
||||||
# Monthly: keep last N months
|
2>/dev/null | gzip > "${dump_file}"; then
|
||||||
local monthly_count=$(ls -1 "${NAS_BACKUP_DIR}/monthly/" 2>/dev/null | wc -l)
|
log_error "pg_dump a échoué !"
|
||||||
if [ "${monthly_count}" -gt "${MONTHLY_RETENTION}" ]; then
|
send_telegram "🚨 *Wordly Backup ÉCHOUÉ*
|
||||||
ls -1t "${NAS_BACKUP_DIR}/monthly/" | tail -n +$((MONTHLY_RETENTION + 1)) | while read -r f; do
|
pg_dump error sur ${POSTGRES_DB}
|
||||||
rm -f "${NAS_BACKUP_DIR}/monthly/${f}"
|
Date : $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
log " Deleted monthly: ${f}"
|
rm -rf "${LOCAL_TMP}"
|
||||||
done
|
|
||||||
fi
|
|
||||||
|
|
||||||
log_success "Cleanup done"
|
|
||||||
}
|
|
||||||
|
|
||||||
verify_backup() {
|
|
||||||
log "Verifying backup integrity..."
|
|
||||||
|
|
||||||
if gzip -t "${NAS_BACKUP_DIR}/daily/${BACKUP_NAME}" 2>/dev/null; then
|
|
||||||
log_success "Backup integrity OK"
|
|
||||||
else
|
|
||||||
log_error "Backup integrity check FAILED!"
|
|
||||||
send_notification "Wordly Backup WARNING: Integrity check failed for ${BACKUP_NAME}"
|
|
||||||
# Don't delete - let admin investigate
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
restore_backup() {
|
|
||||||
local backup_file="$1"
|
|
||||||
|
|
||||||
if [ -z "${backup_file}" ]; then
|
|
||||||
log_error "Usage: $0 --restore <backup_file>"
|
|
||||||
echo ""
|
|
||||||
echo "Available backups:"
|
|
||||||
echo "=== Daily ==="
|
|
||||||
ls -lht "${NAS_BACKUP_DIR}/daily/" 2>/dev/null || echo " (none)"
|
|
||||||
echo "=== Weekly ==="
|
|
||||||
ls -lht "${NAS_BACKUP_DIR}/weekly/" 2>/dev/null || echo " (none)"
|
|
||||||
echo "=== Monthly ==="
|
|
||||||
ls -lht "${NAS_BACKUP_DIR}/monthly/" 2>/dev/null || echo " (none)"
|
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Find the file
|
# Vérification taille
|
||||||
local full_path=""
|
local size_bytes
|
||||||
for dir in daily weekly monthly; do
|
size_bytes=$(stat -c %s "${dump_file}" 2>/dev/null || stat -f %z "${dump_file}")
|
||||||
if [ -f "${NAS_BACKUP_DIR}/${dir}/${backup_file}" ]; then
|
local min_bytes=$((1024 * 1024)) # 1MB minimum
|
||||||
full_path="${NAS_BACKUP_DIR}/${dir}/${backup_file}"
|
|
||||||
break
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
if [ -z "${full_path}" ]; then
|
if [ "${size_bytes}" -lt "${min_bytes}" ]; then
|
||||||
log_error "Backup file not found: ${backup_file}"
|
log_error "Dump trop petit ($(numfmt --to=iec ${size_bytes})) — base de données vide ?"
|
||||||
|
send_telegram "🚨 *Wordly Backup ÉCHOUÉ*
|
||||||
|
Dump PostgreSQL trop petit : $(numfmt --to=iec ${size_bytes})
|
||||||
|
Date : $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
rm -rf "${LOCAL_TMP}"
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo ""
|
log_success "Dump PostgreSQL : $(numfmt --to=iec ${size_bytes})"
|
||||||
log_warning "RESTORE MODE - This will OVERWRITE the current database!"
|
echo "${dump_file}"
|
||||||
echo " File: ${full_path}"
|
|
||||||
echo " Database: ${POSTGRES_DB}"
|
|
||||||
echo ""
|
|
||||||
read -p "Are you sure? Type 'YES' to confirm: " confirm
|
|
||||||
|
|
||||||
if [ "${confirm}" != "YES" ]; then
|
|
||||||
log "Restore cancelled."
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
log "Restoring from ${full_path}..."
|
|
||||||
|
|
||||||
# Create a safety backup first
|
|
||||||
log "Creating safety backup before restore..."
|
|
||||||
SAFETY_NAME="wordly_db_pre_restore_${TIMESTAMP}.sql.gz"
|
|
||||||
docker exec "${POSTGRES_CONTAINER}" pg_dump \
|
|
||||||
-U "${POSTGRES_USER}" \
|
|
||||||
-d "${POSTGRES_DB}" \
|
|
||||||
--format=custom \
|
|
||||||
--compress=9 \
|
|
||||||
2>/dev/null | gzip > "${NAS_BACKUP_DIR}/daily/${SAFETY_NAME}"
|
|
||||||
log "Safety backup: ${SAFETY_NAME}"
|
|
||||||
|
|
||||||
# Restore
|
|
||||||
gunzip -c "${full_path}" | docker exec -i "${POSTGRES_CONTAINER}" \
|
|
||||||
pg_restore \
|
|
||||||
-U "${POSTGRES_USER}" \
|
|
||||||
-d "${POSTGRES_DB}" \
|
|
||||||
--clean \
|
|
||||||
--if-exists \
|
|
||||||
--no-owner \
|
|
||||||
--no-acl \
|
|
||||||
2>/dev/null || true
|
|
||||||
|
|
||||||
log_success "Restore completed!"
|
|
||||||
log_warning "Restart backend: docker restart wordly-backend"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# ===========================================
|
# ==============================================================================
|
||||||
|
# CRÉER L'ARCHIVE DR (dump + .env + docker-compose + configs)
|
||||||
|
# ==============================================================================
|
||||||
|
create_dr_archive() {
|
||||||
|
local dump_file="$1"
|
||||||
|
log "Construction de l'archive DR..."
|
||||||
|
|
||||||
|
# Copier les fichiers de config
|
||||||
|
[ -f "${PROJECT_ROOT}/.env" ] && cp "${PROJECT_ROOT}/.env" "${LOCAL_TMP}/.env.production"
|
||||||
|
[ -f "${PROJECT_ROOT}/docker-compose.yml" ] && cp "${PROJECT_ROOT}/docker-compose.yml" "${LOCAL_TMP}/"
|
||||||
|
[ -d "${PROJECT_ROOT}/docker" ] && cp -r "${PROJECT_ROOT}/docker" "${LOCAL_TMP}/"
|
||||||
|
|
||||||
|
# Compresser
|
||||||
|
local archive_path="/tmp/${SNAPSHOT_NAME}"
|
||||||
|
tar -czf "${archive_path}" -C "${LOCAL_TMP}" .
|
||||||
|
rm -rf "${LOCAL_TMP}"
|
||||||
|
|
||||||
|
# Vérification intégrité
|
||||||
|
if ! gzip -t "${archive_path}" 2>/dev/null; then
|
||||||
|
log_error "Archive DR corrompue !"
|
||||||
|
rm -f "${archive_path}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
local size
|
||||||
|
size=$(du -h "${archive_path}" | cut -f1)
|
||||||
|
log_success "Archive DR créée : ${SNAPSHOT_NAME} (${size})"
|
||||||
|
echo "${archive_path}|${size}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# ENVOYER SUR LE NAS VIA SCP/rsync SSH
|
||||||
|
# ==============================================================================
|
||||||
|
push_to_nas() {
|
||||||
|
local archive_path="$1"
|
||||||
|
local size="$2"
|
||||||
|
|
||||||
|
log "Transfert vers le NAS via rsync SSH..."
|
||||||
|
log " Source : ${archive_path}"
|
||||||
|
log " Dest : ${NAS_USER}@${NAS_HOST}:${NAS_PATH}/snapshots/${SNAPSHOT_NAME}"
|
||||||
|
|
||||||
|
# Dossier quotidien/hebdo/mensuel sur le NAS
|
||||||
|
local nas_dest="${NAS_PATH}/snapshots"
|
||||||
|
|
||||||
|
# Transfer principal
|
||||||
|
if ! rsync -az \
|
||||||
|
-e "ssh -i ${NAS_SSH_KEY} -p ${NAS_SSH_PORT} -o BatchMode=yes -o ConnectTimeout=30" \
|
||||||
|
"${archive_path}" \
|
||||||
|
"${NAS_USER}@${NAS_HOST}:${nas_dest}/${SNAPSHOT_NAME}"; then
|
||||||
|
log_error "rsync vers le NAS a échoué !"
|
||||||
|
send_telegram "🚨 *Wordly Backup ÉCHOUÉ*
|
||||||
|
rsync SSH vers ${NAS_HOST} a échoué
|
||||||
|
Fichier local conservé : ${archive_path}
|
||||||
|
Date : $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
# Garder le fichier local comme fallback
|
||||||
|
mkdir -p "${PROJECT_ROOT}/backups/emergency"
|
||||||
|
mv "${archive_path}" "${PROJECT_ROOT}/backups/emergency/${SNAPSHOT_NAME}"
|
||||||
|
log_warning "Archive conservée localement : ${PROJECT_ROOT}/backups/emergency/${SNAPSHOT_NAME}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
log_success "Archive transférée sur le NAS : ${nas_dest}/${SNAPSHOT_NAME}"
|
||||||
|
|
||||||
|
# Copie hebdomadaire (dimanche)
|
||||||
|
if [ "${DAY_OF_WEEK}" = "7" ]; then
|
||||||
|
${SSH_CMD} "${NAS_USER}@${NAS_HOST}" \
|
||||||
|
"cp ${nas_dest}/${SNAPSHOT_NAME} ${NAS_PATH}/snapshots/weekly_${SNAPSHOT_NAME}" 2>/dev/null || true
|
||||||
|
log "Archive hebdomadaire copiée."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Copie mensuelle (1er du mois)
|
||||||
|
if [ "${DAY_OF_MONTH}" = "01" ]; then
|
||||||
|
${SSH_CMD} "${NAS_USER}@${NAS_HOST}" \
|
||||||
|
"cp ${nas_dest}/${SNAPSHOT_NAME} ${NAS_PATH}/snapshots/monthly_${SNAPSHOT_NAME}" 2>/dev/null || true
|
||||||
|
log "Archive mensuelle copiée."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Nettoyage local
|
||||||
|
rm -f "${archive_path}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# ROTATION DES ARCHIVES SUR LE NAS
|
||||||
|
# ==============================================================================
|
||||||
|
cleanup_nas() {
|
||||||
|
log "Rotation des archives sur le NAS (conservation : ${DAILY_RETENTION} jours)..."
|
||||||
|
|
||||||
|
# Supprimer les archives wordly_dr_* plus vieilles que DAILY_RETENTION
|
||||||
|
${SSH_CMD} "${NAS_USER}@${NAS_HOST}" \
|
||||||
|
"find ${NAS_PATH}/snapshots -name 'wordly_dr_*.tar.gz' -mtime +${DAILY_RETENTION} -delete 2>/dev/null; \
|
||||||
|
find ${NAS_PATH}/snapshots -name 'weekly_*.tar.gz' | sort -r | tail -n +$((WEEKLY_RETENTION + 1)) | xargs rm -f 2>/dev/null; \
|
||||||
|
find ${NAS_PATH}/snapshots -name 'monthly_*.tar.gz' | sort -r | tail -n +$((MONTHLY_RETENTION + 1)) | xargs rm -f 2>/dev/null; \
|
||||||
|
echo OK" | grep -q "OK"
|
||||||
|
|
||||||
|
log_success "Rotation des archives OK"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# SYNCHRONISER LES SCRIPTS SUR LE NAS (pour restauration depuis .98)
|
||||||
|
# ==============================================================================
|
||||||
|
sync_scripts() {
|
||||||
|
rsync -az \
|
||||||
|
-e "ssh -i ${NAS_SSH_KEY} -p ${NAS_SSH_PORT} -o BatchMode=yes" \
|
||||||
|
--exclude="__pycache__" \
|
||||||
|
--exclude="*.pyc" \
|
||||||
|
"${SCRIPT_DIR}/" \
|
||||||
|
"${NAS_USER}@${NAS_HOST}:${NAS_PATH}/scripts/" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# LISTER LES ARCHIVES DISPONIBLES
|
||||||
|
# ==============================================================================
|
||||||
|
list_archives() {
|
||||||
|
log "Archives disponibles sur le NAS :"
|
||||||
|
${SSH_CMD} "${NAS_USER}@${NAS_HOST}" \
|
||||||
|
"ls -lht ${NAS_PATH}/snapshots/wordly_dr_*.tar.gz 2>/dev/null || echo '(aucune archive)'"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
# MAIN
|
# MAIN
|
||||||
# ===========================================
|
# ==============================================================================
|
||||||
|
main() {
|
||||||
|
case "${1:-}" in
|
||||||
|
--list)
|
||||||
|
ENV_FILE="${PROJECT_ROOT}/.env"
|
||||||
|
[ -f "${ENV_FILE}" ] && { set -a; source "${ENV_FILE}"; set +a; }
|
||||||
|
list_archives
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
--full|*)
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
case "${1:-}" in
|
echo ""
|
||||||
--restore)
|
echo "================================================================="
|
||||||
restore_backup "${2:-}"
|
echo " Wordly.art — Backup → NAS Synology 192.168.1.146"
|
||||||
;;
|
echo " DB : ${POSTGRES_DB}"
|
||||||
--full)
|
echo " NAS : ${NAS_USER}@${NAS_HOST}:${NAS_PATH}"
|
||||||
check_prerequisites
|
echo " $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
create_backup
|
echo "================================================================="
|
||||||
verify_backup
|
echo ""
|
||||||
cleanup_old_backups
|
|
||||||
log_success "Full backup cycle complete!"
|
check_prerequisites
|
||||||
;;
|
|
||||||
*)
|
# 1. Dump PostgreSQL
|
||||||
check_prerequisites
|
local dump_file
|
||||||
create_backup
|
dump_file=$(backup_postgres)
|
||||||
verify_backup
|
|
||||||
cleanup_old_backups
|
# 2. Créer l'archive DR
|
||||||
log_success "Backup complete!"
|
local archive_info
|
||||||
;;
|
archive_info=$(create_dr_archive "${dump_file}")
|
||||||
esac
|
local archive_path="${archive_info%%|*}"
|
||||||
|
local archive_size="${archive_info##*|}"
|
||||||
|
|
||||||
|
# 3. Envoyer sur le NAS via rsync SSH
|
||||||
|
push_to_nas "${archive_path}" "${archive_size}"
|
||||||
|
|
||||||
|
# 4. Rotation
|
||||||
|
cleanup_nas
|
||||||
|
|
||||||
|
# 5. Sync scripts
|
||||||
|
sync_scripts
|
||||||
|
|
||||||
|
# 6. Notification Telegram
|
||||||
|
send_telegram "✅ *Wordly.art Backup OK*
|
||||||
|
Archive : \`${SNAPSHOT_NAME}\`
|
||||||
|
Taille : ${archive_size}
|
||||||
|
NAS : \`${NAS_PATH}/snapshots/\`
|
||||||
|
Date : $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
log_success "================================================================="
|
||||||
|
log_success "Backup complet terminé !"
|
||||||
|
log_success " Archive : ${NAS_PATH}/snapshots/${SNAPSHOT_NAME}"
|
||||||
|
log_success " Lister : bash scripts/backup-to-nas.sh --list"
|
||||||
|
log_success "================================================================="
|
||||||
|
echo ""
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
|
|||||||
@@ -1,9 +1,16 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# ==============================================================================
|
# ==============================================================================
|
||||||
# Wordly.art - Disaster Recovery (DR) Backup & Restore Playbook (V2)
|
# Wordly.art - Disaster Recovery (DR) Backup & Restore Playbook (V3)
|
||||||
# ==============================================================================
|
# ==============================================================================
|
||||||
# Packages app configs (.env, docker-compose), database backups, and NPM
|
# Archives app configs (.env, docker-compose), database backup, and exports
|
||||||
# configs, and exports them to LOCAL, NAS, or remote SCP storage.
|
# to the NAS at 192.168.1.146.
|
||||||
|
#
|
||||||
|
# On RESTORE: deploys app on the new server and automatically updates NPM
|
||||||
|
# (192.168.1.184) to reroute traffic via API — no manual intervention needed.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./disaster-recovery.sh --backup # Create DR archive → NAS
|
||||||
|
# ./disaster-recovery.sh --restore <archive> # Restore on THIS machine
|
||||||
# ==============================================================================
|
# ==============================================================================
|
||||||
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
@@ -31,49 +38,54 @@ if [ -f "${ENV_FILE}" ]; then
|
|||||||
set +a
|
set +a
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Config Defaults & Type Resolution
|
# NAS SSH (même config que backup-to-nas.sh)
|
||||||
BACKUP_DEST_TYPE="${BACKUP_DEST_TYPE:-LOCAL}" # LOCAL, NAS, SCP
|
NAS_HOST="${NAS_HOST:-192.168.1.146}"
|
||||||
BACKUP_DEST_PATH="${BACKUP_DEST_PATH:-${PROJECT_ROOT}/backups}"
|
NAS_USER="${NAS_USER:-wordly-backup}"
|
||||||
DR_RETENTION_DAYS=${DR_RETENTION_DAYS:-14}
|
NAS_PATH="${NAS_PATH:-/volume1/backups/wordly}"
|
||||||
|
NAS_SSH_PORT="${NAS_SSH_PORT:-22}"
|
||||||
|
NAS_SSH_KEY="${NAS_SSH_KEY:-/root/.ssh/wordly_nas_key}"
|
||||||
|
BACKUP_DEST_PATH="${NAS_PATH}/snapshots"
|
||||||
|
DR_RETENTION_DAYS=${DR_RETENTION_DAYS:-30}
|
||||||
|
|
||||||
# SCP Configuration
|
# IP of THIS server (used during restore to configure NPM failover)
|
||||||
SCP_HOST="${SCP_HOST:-}"
|
SERVER_IP="${SERVER_IP:-}"
|
||||||
SCP_USER="${SCP_USER:-}"
|
|
||||||
SCP_KEY_PATH="${SCP_KEY_PATH:-~/.ssh/id_rsa}"
|
|
||||||
SCP_PORT="${SCP_PORT:-22}"
|
|
||||||
SCP_DEST_PATH="${SCP_DEST_PATH:-/var/backups/wordly}"
|
|
||||||
|
|
||||||
# NPM Configuration directories
|
# Telegram
|
||||||
NPM_DATA_DIR="${NPM_DATA_DIR:-}"
|
TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN:-}"
|
||||||
NPM_LETSENCRYPT_DIR="${NPM_LETSENCRYPT_DIR:-}"
|
TELEGRAM_CHAT_ID="${TELEGRAM_CHAT_ID:-}"
|
||||||
|
|
||||||
# ==============================================================================
|
# ==============================================================================
|
||||||
# DESTINATION PREPARATION
|
# SEND TELEGRAM NOTIFICATION
|
||||||
|
# ==============================================================================
|
||||||
|
send_telegram() {
|
||||||
|
local message="$1"
|
||||||
|
if [ -n "${TELEGRAM_BOT_TOKEN}" ] && [ -n "${TELEGRAM_CHAT_ID}" ]; then
|
||||||
|
curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
|
||||||
|
-d "chat_id=${TELEGRAM_CHAT_ID}" \
|
||||||
|
-d "text=${message}" \
|
||||||
|
-d "parse_mode=Markdown" \
|
||||||
|
>/dev/null 2>&1 || true
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# DESTINATION PREPARATION (backup mode)
|
||||||
# ==============================================================================
|
# ==============================================================================
|
||||||
prepare_destination() {
|
prepare_destination() {
|
||||||
if [ "${BACKUP_DEST_TYPE}" = "NAS" ] || [ "${BACKUP_DEST_TYPE}" = "LOCAL" ]; then
|
local ssh_cmd="ssh -i ${NAS_SSH_KEY} -p ${NAS_SSH_PORT} -o BatchMode=yes -o ConnectTimeout=10"
|
||||||
if [ ! -d "${BACKUP_DEST_PATH}" ]; then
|
|
||||||
mkdir -p "${BACKUP_DEST_PATH}" 2>/dev/null || true
|
log "Vérification de la connectivité SSH vers le NAS ${NAS_HOST}..."
|
||||||
fi
|
if ! ${ssh_cmd} "${NAS_USER}@${NAS_HOST}" "echo OK" >/dev/null 2>&1; then
|
||||||
|
log_error "Impossible de joindre le NAS ${NAS_HOST} via SSH."
|
||||||
if [ ! -w "${BACKUP_DEST_PATH}" ]; then
|
log_error "Lancez d'abord : sudo bash scripts/setup-nas.sh"
|
||||||
log_warning "Backup destination path '${BACKUP_DEST_PATH}' is not writable. Falling back to local backups."
|
exit 1
|
||||||
BACKUP_DEST_PATH="${PROJECT_ROOT}/backups"
|
|
||||||
BACKUP_DEST_TYPE="LOCAL"
|
|
||||||
mkdir -p "${BACKUP_DEST_PATH}/dr"
|
|
||||||
fi
|
|
||||||
DR_LOCAL_DIR="${BACKUP_DEST_PATH}/dr"
|
|
||||||
mkdir -p "${DR_LOCAL_DIR}"
|
|
||||||
elif [ "${BACKUP_DEST_TYPE}" = "SCP" ]; then
|
|
||||||
if [ -z "${SCP_HOST}" ] || [ -z "${SCP_USER}" ]; then
|
|
||||||
log_error "SCP backup selected but SCP_HOST or SCP_USER is not configured in .env."
|
|
||||||
log_warning "Falling back to LOCAL backup directory."
|
|
||||||
BACKUP_DEST_TYPE="LOCAL"
|
|
||||||
BACKUP_DEST_PATH="${PROJECT_ROOT}/backups"
|
|
||||||
DR_LOCAL_DIR="${BACKUP_DEST_PATH}/dr"
|
|
||||||
mkdir -p "${DR_LOCAL_DIR}"
|
|
||||||
fi
|
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# S'assurer que le dossier snapshots existe sur le NAS
|
||||||
|
${ssh_cmd} "${NAS_USER}@${NAS_HOST}" \
|
||||||
|
"mkdir -p ${NAS_PATH}/snapshots" 2>/dev/null || true
|
||||||
|
|
||||||
|
log_success "NAS SSH OK — Destination : ${NAS_USER}@${NAS_HOST}:${NAS_PATH}/snapshots"
|
||||||
}
|
}
|
||||||
|
|
||||||
# ==============================================================================
|
# ==============================================================================
|
||||||
@@ -127,22 +139,11 @@ perform_backup() {
|
|||||||
mkdir -p "${packing_dir}/db_backup"
|
mkdir -p "${packing_dir}/db_backup"
|
||||||
cp "${latest_db_backup}" "${packing_dir}/db_backup/"
|
cp "${latest_db_backup}" "${packing_dir}/db_backup/"
|
||||||
|
|
||||||
# 5. Pack Nginx Proxy Manager (NPM) configs if configured
|
# 5. Note: NPM config is NOT backed up here.
|
||||||
local has_npm_data=false
|
# NPM runs on its own dedicated server (192.168.1.184) and is stable.
|
||||||
if [ -n "${NPM_DATA_DIR}" ] && [ -d "${NPM_DATA_DIR}" ]; then
|
# Only the forward_host IP needs to change during failover, which is
|
||||||
log "Packaging Nginx Proxy Manager /data directory..."
|
# done automatically via the NPM API by npm-failover.sh during restore.
|
||||||
cp -r "${NPM_DATA_DIR}" "${packing_dir}/npm_data"
|
log "NPM is on dedicated server 192.168.1.184 — no NPM config to backup."
|
||||||
has_npm_data=true
|
|
||||||
fi
|
|
||||||
if [ -n "${NPM_LETSENCRYPT_DIR}" ] && [ -d "${NPM_LETSENCRYPT_DIR}" ]; then
|
|
||||||
log "Packaging Nginx Proxy Manager /etc/letsencrypt directory..."
|
|
||||||
cp -r "${NPM_LETSENCRYPT_DIR}" "${packing_dir}/npm_letsencrypt"
|
|
||||||
has_npm_data=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ "${has_npm_data}" = "false" ]; then
|
|
||||||
log_warning "NPM directories (NPM_DATA_DIR / NPM_LETSENCRYPT_DIR) not configured or not found. Skipping NPM config packaging."
|
|
||||||
fi
|
|
||||||
|
|
||||||
# 6. Compress DR Archive
|
# 6. Compress DR Archive
|
||||||
local dr_archive_name="wordly_dr_${TIMESTAMP}.tar.gz"
|
local dr_archive_name="wordly_dr_${TIMESTAMP}.tar.gz"
|
||||||
@@ -160,44 +161,47 @@ perform_backup() {
|
|||||||
local size
|
local size
|
||||||
size=$(du -h "${local_archive_path}" | cut -f1)
|
size=$(du -h "${local_archive_path}" | cut -f1)
|
||||||
|
|
||||||
# 7. Route to Destination
|
# 7. Envoyer l'archive sur le NAS via rsync SSH
|
||||||
if [ "${BACKUP_DEST_TYPE}" = "LOCAL" ] || [ "${BACKUP_DEST_TYPE}" = "NAS" ]; then
|
local ssh_cmd="ssh -i ${NAS_SSH_KEY} -p ${NAS_SSH_PORT} -o BatchMode=yes -o ConnectTimeout=30"
|
||||||
local dest_path="${DR_LOCAL_DIR}/${dr_archive_name}"
|
local dest_path="${BACKUP_DEST_PATH}/${dr_archive_name}"
|
||||||
mv "${local_archive_path}" "${dest_path}"
|
|
||||||
log_success "DR archive created successfully (${size}) at: ${dest_path}"
|
|
||||||
|
|
||||||
# Retention
|
|
||||||
log "Applying retention policy (pruning files older than ${DR_RETENTION_DAYS} days)..."
|
|
||||||
find "${DR_LOCAL_DIR}" -name "wordly_dr_*.tar.gz" -mtime +"${DR_RETENTION_DAYS}" -exec rm -f {} \;
|
|
||||||
|
|
||||||
elif [ "${BACKUP_DEST_TYPE}" = "SCP" ]; then
|
|
||||||
log "Transferring DR archive to remote server via SCP (${SCP_USER}@${SCP_HOST}:${SCP_PORT})..."
|
|
||||||
|
|
||||||
# Test connection & Create remote directory if not exists
|
|
||||||
if ! ssh -p "${SCP_PORT}" -i "${SCP_KEY_PATH}" -o ConnectTimeout=5 -o StrictHostKeyChecking=no "${SCP_USER}@${SCP_HOST}" "mkdir -p ${SCP_DEST_PATH}" 2>/dev/null; then
|
|
||||||
log_error "SSH connection to ${SCP_USER}@${SCP_HOST} failed. Saving archive locally instead."
|
|
||||||
mkdir -p "${PROJECT_ROOT}/backups/dr"
|
|
||||||
mv "${local_archive_path}" "${PROJECT_ROOT}/backups/dr/${dr_archive_name}"
|
|
||||||
log_warning "DR backup saved locally at: ${PROJECT_ROOT}/backups/dr/${dr_archive_name}"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# SCP copy
|
log "Transfert de l'archive DR vers le NAS via rsync SSH..."
|
||||||
if scp -P "${SCP_PORT}" -i "${SCP_KEY_PATH}" -o StrictHostKeyChecking=no "${local_archive_path}" "${SCP_USER}@${SCP_HOST}:${SCP_DEST_PATH}/${dr_archive_name}"; then
|
if ! rsync -az \
|
||||||
log_success "DR archive transferred successfully to ${SCP_USER}@${SCP_HOST}:${SCP_DEST_PATH}/${dr_archive_name}"
|
-e "ssh -i ${NAS_SSH_KEY} -p ${NAS_SSH_PORT} -o BatchMode=yes -o ConnectTimeout=30" \
|
||||||
rm -f "${local_archive_path}"
|
"${local_archive_path}" \
|
||||||
|
"${NAS_USER}@${NAS_HOST}:${BACKUP_DEST_PATH}/${dr_archive_name}"; then
|
||||||
# Remote retention prune
|
log_error "rsync SSH vers le NAS a échoué !"
|
||||||
log "Applying remote retention policy on backup server..."
|
log_warning "Archive conservée localement : ${local_archive_path}"
|
||||||
ssh -p "${SCP_PORT}" -i "${SCP_KEY_PATH}" -o StrictHostKeyChecking=no "${SCP_USER}@${SCP_HOST}" \
|
send_telegram "🚨 *Wordly DR Backup FAILED*
|
||||||
"find ${SCP_DEST_PATH} -name 'wordly_dr_*.tar.gz' -mtime +${DR_RETENTION_DAYS} -exec rm -f {} \;" || true
|
rsync NAS échoué : ${NAS_HOST}
|
||||||
else
|
Fichier local : ${local_archive_path}
|
||||||
log_error "SCP file transfer failed. Retaining local backup."
|
Date: $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
mkdir -p "${PROJECT_ROOT}/backups/dr"
|
exit 1
|
||||||
mv "${local_archive_path}" "${PROJECT_ROOT}/backups/dr/${dr_archive_name}"
|
|
||||||
fi
|
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
rm -f "${local_archive_path}"
|
||||||
|
log_success "Archive DR transférée (${size}) → ${NAS_USER}@${NAS_HOST}:${dest_path}"
|
||||||
|
|
||||||
|
# Retention policy sur le NAS
|
||||||
|
log "Rotation des archives (>${DR_RETENTION_DAYS} jours) sur le NAS..."
|
||||||
|
${ssh_cmd} "${NAS_USER}@${NAS_HOST}" \
|
||||||
|
"find ${BACKUP_DEST_PATH} -name 'wordly_dr_*.tar.gz' -mtime +${DR_RETENTION_DAYS} -delete 2>/dev/null; echo OK" | grep -q "OK" || true
|
||||||
|
|
||||||
|
# Sync scripts
|
||||||
|
if command -v rsync &>/dev/null; then
|
||||||
|
rsync -az \
|
||||||
|
-e "ssh -i ${NAS_SSH_KEY} -p ${NAS_SSH_PORT} -o BatchMode=yes" \
|
||||||
|
--exclude="__pycache__" \
|
||||||
|
"${SCRIPT_DIR}/" \
|
||||||
|
"${NAS_USER}@${NAS_HOST}:${NAS_PATH}/scripts/" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
send_telegram "✅ *Wordly.art DR Backup OK*
|
||||||
|
Archive: \`${dr_archive_name}\`
|
||||||
|
Taille: ${size}
|
||||||
|
NAS: \`${dest_path}\`
|
||||||
|
Date: $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
|
||||||
log_success "Disaster Recovery backup complete."
|
log_success "Disaster Recovery backup complete."
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -250,22 +254,7 @@ perform_restore() {
|
|||||||
source "${PROJECT_ROOT}/.env"
|
source "${PROJECT_ROOT}/.env"
|
||||||
set +a
|
set +a
|
||||||
|
|
||||||
# Restore NPM configs to their target directories if present in the package
|
log_success "Docker configurations and env keys restored."
|
||||||
if [ -d "${PROJECT_ROOT}/npm_data" ] && [ -n "${NPM_DATA_DIR}" ]; then
|
|
||||||
log "Restoring NPM /data directory..."
|
|
||||||
mkdir -p "$(dirname "${NPM_DATA_DIR}")"
|
|
||||||
rm -rf "${NPM_DATA_DIR}"
|
|
||||||
mv "${PROJECT_ROOT}/npm_data" "${NPM_DATA_DIR}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ -d "${PROJECT_ROOT}/npm_letsencrypt" ] && [ -n "${NPM_LETSENCRYPT_DIR}" ]; then
|
|
||||||
log "Restoring NPM /etc/letsencrypt directory..."
|
|
||||||
mkdir -p "$(dirname "${NPM_LETSENCRYPT_DIR}")"
|
|
||||||
rm -rf "${NPM_LETSENCRYPT_DIR}"
|
|
||||||
mv "${PROJECT_ROOT}/npm_letsencrypt" "${NPM_LETSENCRYPT_DIR}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
log_success "Docker configurations, env keys, and NPM configurations restored."
|
|
||||||
|
|
||||||
# Boot Docker Compose Services
|
# Boot Docker Compose Services
|
||||||
log "Spinning up Docker containers (database, redis, backend, frontend, NPM if configured)..."
|
log "Spinning up Docker containers (database, redis, backend, frontend, NPM if configured)..."
|
||||||
@@ -328,10 +317,63 @@ perform_restore() {
|
|||||||
log "Restarting application backend..."
|
log "Restarting application backend..."
|
||||||
${compose_cmd} restart backend
|
${compose_cmd} restart backend
|
||||||
|
|
||||||
|
# HTTP Health check (wait up to 3 minutes)
|
||||||
|
log "Waiting for application health check (max 180s)..."
|
||||||
|
local app_url="http://localhost:8001/health"
|
||||||
|
local health_ok=false
|
||||||
|
for i in $(seq 1 36); do
|
||||||
|
local http_code
|
||||||
|
http_code=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 3 --max-time 5 "${app_url}" 2>/dev/null || echo "000")
|
||||||
|
if [ "${http_code}" = "200" ]; then
|
||||||
|
health_ok=true
|
||||||
|
log_success "App is healthy (HTTP 200) after $((i * 5))s"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
echo " Health check attempt ${i}/36... (HTTP ${http_code})"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ "${health_ok}" = "false" ]; then
|
||||||
|
log_error "App did NOT become healthy within 180s!"
|
||||||
|
log_error "NPM failover will NOT be triggered automatically."
|
||||||
|
log_error "Investigate: docker compose logs backend"
|
||||||
|
send_telegram "🚨 *Wordly.art DR FAILED — App unhealthy*
|
||||||
|
Serveur: \`$(hostname -I | awk '{print $1}')\`
|
||||||
|
Date: $(date '+%Y-%m-%d %H:%M:%S')
|
||||||
|
Action: vérifiez les logs Docker"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# NPM AUTOMATIC FAILOVER
|
||||||
|
# ==============================================================================
|
||||||
|
log "App is healthy. Triggering NPM failover..."
|
||||||
|
local this_server_ip
|
||||||
|
this_server_ip="${SERVER_IP:-$(hostname -I | awk '{print $1}')}"
|
||||||
|
|
||||||
|
if bash "${SCRIPT_DIR}/npm-failover.sh" --target-ip "${this_server_ip}"; then
|
||||||
|
log_success "NPM now routes traffic to this server (${this_server_ip})"
|
||||||
|
send_telegram "✅ *Wordly.art DR COMPLET*
|
||||||
|
Serveur actif: \`${this_server_ip}\`
|
||||||
|
NPM redirigé automatiquement
|
||||||
|
Date: $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
else
|
||||||
|
log_error "NPM failover script FAILED."
|
||||||
|
log_warning "Manual failover required:"
|
||||||
|
log_warning " → Go to http://192.168.1.184:81"
|
||||||
|
log_warning " → Edit proxy host for ${NPM_PROXY_HOST_DOMAIN:-wordly.art}"
|
||||||
|
log_warning " → Change Forward Hostname to: ${this_server_ip}"
|
||||||
|
send_telegram "⚠️ *Wordly.art DR — NPM manuel requis*
|
||||||
|
App OK sur: \`${this_server_ip}\`
|
||||||
|
NPM failover automatique a échoué
|
||||||
|
Action: http://192.168.1.184:81 → modifier Forward Host"
|
||||||
|
fi
|
||||||
|
|
||||||
log_success "=========================================================================="
|
log_success "=========================================================================="
|
||||||
log_success "DISASTER RECOVERY SYSTEM RESTORE COMPLETE!"
|
log_success "DISASTER RECOVERY SYSTEM RESTORE COMPLETE!"
|
||||||
log_success "=========================================================================="
|
log_success "=========================================================================="
|
||||||
log "Your application and reverse-proxy routes are restored."
|
log_success " App: http://${this_server_ip}:8001/health"
|
||||||
|
log_success " NPM: http://192.168.1.184:81"
|
||||||
echo ""
|
echo ""
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
73
scripts/install-crontab.sh
Normal file
73
scripts/install-crontab.sh
Normal file
@@ -0,0 +1,73 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# ==============================================================================
|
||||||
|
# Wordly.art - Install Backup Crontab
|
||||||
|
# ==============================================================================
|
||||||
|
# Run ONCE to install all scheduled backup tasks.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bash scripts/install-crontab.sh
|
||||||
|
# ==============================================================================
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
CRONTAB_FILE="${SCRIPT_DIR}/crontab.wordly"
|
||||||
|
LOG_DIR="/var/log"
|
||||||
|
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
NC='\033[0m'
|
||||||
|
|
||||||
|
log() { echo "[Crontab] $1"; }
|
||||||
|
log_success() { echo -e "[Crontab] ${GREEN}✅ $1${NC}"; }
|
||||||
|
log_warning() { echo -e "[Crontab] ${YELLOW}⚠️ $1${NC}"; }
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 1. Create the crontab file
|
||||||
|
# ==============================================================================
|
||||||
|
cat > "${CRONTAB_FILE}" <<EOF
|
||||||
|
# ==============================================================================
|
||||||
|
# Wordly.art — Backup & Disaster Recovery Crontab
|
||||||
|
# Installed by: bash scripts/install-crontab.sh
|
||||||
|
# ==============================================================================
|
||||||
|
|
||||||
|
# Backup to NAS every 6 hours (00:00, 06:00, 12:00, 18:00)
|
||||||
|
0 */6 * * * bash ${SCRIPT_DIR}/backup-to-nas.sh >> ${LOG_DIR}/wordly-backup.log 2>&1
|
||||||
|
|
||||||
|
# Verify backup integrity 30 minutes after each backup
|
||||||
|
30 */6 * * * bash ${SCRIPT_DIR}/verify-backups.sh >> ${LOG_DIR}/wordly-verify.log 2>&1
|
||||||
|
|
||||||
|
# Rotate logs weekly (keep last 30 days)
|
||||||
|
0 4 * * 0 find ${LOG_DIR} -name "wordly-*.log" -mtime +30 -delete 2>/dev/null || true
|
||||||
|
EOF
|
||||||
|
|
||||||
|
log_success "Crontab file created: ${CRONTAB_FILE}"
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 2. Install crontab for current user
|
||||||
|
# ==============================================================================
|
||||||
|
log "Installing crontab..."
|
||||||
|
|
||||||
|
# Preserve existing crontab (if any), append new entries
|
||||||
|
EXISTING_CRON=$(crontab -l 2>/dev/null || true)
|
||||||
|
|
||||||
|
# Remove any existing wordly entries (to avoid duplicates on re-run)
|
||||||
|
EXISTING_CRON_CLEAN=$(echo "${EXISTING_CRON}" | grep -v "wordly" | grep -v "backup-to-nas" | grep -v "verify-backups" || true)
|
||||||
|
|
||||||
|
# Combine
|
||||||
|
NEW_CRON=$(printf "%s\n%s\n" "${EXISTING_CRON_CLEAN}" "$(cat "${CRONTAB_FILE}")")
|
||||||
|
echo "${NEW_CRON}" | crontab -
|
||||||
|
|
||||||
|
log_success "Crontab installed!"
|
||||||
|
echo ""
|
||||||
|
log "Current crontab:"
|
||||||
|
crontab -l | grep -E "wordly|backup|verify" | sed 's/^/ /'
|
||||||
|
echo ""
|
||||||
|
log_success "Scheduled jobs:"
|
||||||
|
log_success " Every 6h (00:00/06:00/12:00/18:00) → backup-to-nas.sh"
|
||||||
|
log_success " Every 6h+30min → verify-backups.sh"
|
||||||
|
log_success " Every Sunday at 04:00 → log rotation"
|
||||||
|
echo ""
|
||||||
|
log_warning "Logs will be written to:"
|
||||||
|
log_warning " ${LOG_DIR}/wordly-backup.log"
|
||||||
|
log_warning " ${LOG_DIR}/wordly-verify.log"
|
||||||
325
scripts/npm-failover.sh
Normal file
325
scripts/npm-failover.sh
Normal file
@@ -0,0 +1,325 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# ==============================================================================
|
||||||
|
# Wordly.art - NPM Failover via API
|
||||||
|
# ==============================================================================
|
||||||
|
# Automatically updates Nginx Proxy Manager's forward host via its REST API.
|
||||||
|
# Called by disaster-recovery.sh after a successful health check on the new server.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./npm-failover.sh --target-ip 192.168.1.98 # Switch to new server
|
||||||
|
# ./npm-failover.sh --target-ip 192.168.1.151 # Rollback to original server
|
||||||
|
# ./npm-failover.sh --dry-run --target-ip 192.168.1.98 # Test without modifying NPM
|
||||||
|
# ==============================================================================
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||||
|
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
|
||||||
|
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m'
|
||||||
|
|
||||||
|
log() { echo -e "[NPM-Failover ${TIMESTAMP}] $1"; }
|
||||||
|
log_success() { echo -e "[NPM-Failover ${TIMESTAMP}] ${GREEN}$1${NC}"; }
|
||||||
|
log_warning() { echo -e "[NPM-Failover ${TIMESTAMP}] ${YELLOW}WARNING: $1${NC}"; }
|
||||||
|
log_error() { echo -e "[NPM-Failover ${TIMESTAMP}] ${RED}ERROR: $1${NC}"; }
|
||||||
|
log_info() { echo -e "[NPM-Failover ${TIMESTAMP}] ${BLUE}$1${NC}"; }
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 1. LOAD CONFIGURATION FROM .env
|
||||||
|
# ==============================================================================
|
||||||
|
ENV_FILE="${PROJECT_ROOT}/.env"
|
||||||
|
if [ -f "${ENV_FILE}" ]; then
|
||||||
|
set -a
|
||||||
|
source "${ENV_FILE}"
|
||||||
|
set +a
|
||||||
|
fi
|
||||||
|
|
||||||
|
NPM_API_URL="${NPM_API_URL:-}"
|
||||||
|
NPM_ADMIN_EMAIL="${NPM_ADMIN_EMAIL:-}"
|
||||||
|
NPM_ADMIN_PASSWORD="${NPM_ADMIN_PASSWORD:-}"
|
||||||
|
NPM_PROXY_HOST_DOMAIN="${NPM_PROXY_HOST_DOMAIN:-wordly.art}"
|
||||||
|
TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN:-}"
|
||||||
|
TELEGRAM_CHAT_ID="${TELEGRAM_CHAT_ID:-}"
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 2. ARGUMENT PARSING
|
||||||
|
# ==============================================================================
|
||||||
|
TARGET_IP=""
|
||||||
|
DRY_RUN=false
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--target-ip)
|
||||||
|
TARGET_IP="$2"
|
||||||
|
shift 2
|
||||||
|
;;
|
||||||
|
--dry-run)
|
||||||
|
DRY_RUN=true
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
log_error "Unknown argument: $1"
|
||||||
|
echo "Usage: $0 --target-ip <IP> [--dry-run]"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 3. VALIDATION
|
||||||
|
# ==============================================================================
|
||||||
|
validate_config() {
|
||||||
|
local errors=0
|
||||||
|
|
||||||
|
if [ -z "${TARGET_IP}" ]; then
|
||||||
|
log_error "--target-ip is required."
|
||||||
|
errors=$((errors + 1))
|
||||||
|
fi
|
||||||
|
if [ -z "${NPM_API_URL}" ]; then
|
||||||
|
log_error "NPM_API_URL is not set in .env (example: http://192.168.1.184:81/api)"
|
||||||
|
errors=$((errors + 1))
|
||||||
|
fi
|
||||||
|
if [ -z "${NPM_ADMIN_EMAIL}" ]; then
|
||||||
|
log_error "NPM_ADMIN_EMAIL is not set in .env"
|
||||||
|
errors=$((errors + 1))
|
||||||
|
fi
|
||||||
|
if [ -z "${NPM_ADMIN_PASSWORD}" ]; then
|
||||||
|
log_error "NPM_ADMIN_PASSWORD is not set in .env"
|
||||||
|
errors=$((errors + 1))
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! command -v curl &>/dev/null; then
|
||||||
|
log_error "curl is not installed. Required for NPM API calls."
|
||||||
|
errors=$((errors + 1))
|
||||||
|
fi
|
||||||
|
if ! command -v jq &>/dev/null; then
|
||||||
|
log_error "jq is not installed. Required for JSON parsing. Install: apt-get install jq"
|
||||||
|
errors=$((errors + 1))
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "${errors}" -gt 0 ]; then
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 4. TELEGRAM NOTIFICATION
|
||||||
|
# ==============================================================================
|
||||||
|
send_telegram() {
|
||||||
|
local message="$1"
|
||||||
|
if [ -n "${TELEGRAM_BOT_TOKEN}" ] && [ -n "${TELEGRAM_CHAT_ID}" ]; then
|
||||||
|
curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
|
||||||
|
-d "chat_id=${TELEGRAM_CHAT_ID}" \
|
||||||
|
-d "text=${message}" \
|
||||||
|
-d "parse_mode=Markdown" \
|
||||||
|
>/dev/null 2>&1 || true
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 5. NPM API AUTHENTICATION
|
||||||
|
# ==============================================================================
|
||||||
|
npm_authenticate() {
|
||||||
|
log "Authenticating with NPM API at ${NPM_API_URL}..."
|
||||||
|
|
||||||
|
local response
|
||||||
|
response=$(curl -s -w "\n%{http_code}" \
|
||||||
|
-X POST "${NPM_API_URL}/tokens" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "{\"identity\": \"${NPM_ADMIN_EMAIL}\", \"secret\": \"${NPM_ADMIN_PASSWORD}\"}" \
|
||||||
|
--connect-timeout 10 \
|
||||||
|
--max-time 15)
|
||||||
|
|
||||||
|
local http_code
|
||||||
|
http_code=$(echo "${response}" | tail -n1)
|
||||||
|
local body
|
||||||
|
body=$(echo "${response}" | head -n-1)
|
||||||
|
|
||||||
|
if [ "${http_code}" != "200" ]; then
|
||||||
|
log_error "NPM authentication failed (HTTP ${http_code}). Check NPM_ADMIN_EMAIL and NPM_ADMIN_PASSWORD."
|
||||||
|
log_error "Response: ${body}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
local token
|
||||||
|
token=$(echo "${body}" | jq -r '.token // empty')
|
||||||
|
if [ -z "${token}" ]; then
|
||||||
|
log_error "Could not extract token from NPM response."
|
||||||
|
log_error "Response: ${body}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
log_success "NPM authentication successful."
|
||||||
|
echo "${token}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 6. FIND PROXY HOST BY DOMAIN
|
||||||
|
# ==============================================================================
|
||||||
|
npm_find_proxy_host() {
|
||||||
|
local token="$1"
|
||||||
|
log "Looking up proxy host for domain: ${NPM_PROXY_HOST_DOMAIN}..."
|
||||||
|
|
||||||
|
local response
|
||||||
|
response=$(curl -s -w "\n%{http_code}" \
|
||||||
|
-X GET "${NPM_API_URL}/nginx/proxy-hosts?expand=domain_names" \
|
||||||
|
-H "Authorization: Bearer ${token}" \
|
||||||
|
--connect-timeout 10 \
|
||||||
|
--max-time 15)
|
||||||
|
|
||||||
|
local http_code
|
||||||
|
http_code=$(echo "${response}" | tail -n1)
|
||||||
|
local body
|
||||||
|
body=$(echo "${response}" | head -n-1)
|
||||||
|
|
||||||
|
if [ "${http_code}" != "200" ]; then
|
||||||
|
log_error "Failed to retrieve proxy hosts (HTTP ${http_code})"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Find the proxy host ID matching our domain
|
||||||
|
local host_id
|
||||||
|
host_id=$(echo "${body}" | jq -r \
|
||||||
|
--arg domain "${NPM_PROXY_HOST_DOMAIN}" \
|
||||||
|
'.[] | select(.domain_names[] == $domain) | .id' | head -n1)
|
||||||
|
|
||||||
|
if [ -z "${host_id}" ]; then
|
||||||
|
log_error "No proxy host found for domain '${NPM_PROXY_HOST_DOMAIN}' in NPM."
|
||||||
|
log_error "Available domains:"
|
||||||
|
echo "${body}" | jq -r '.[].domain_names[]' | sed 's/^/ - /' >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
log_success "Found proxy host ID: ${host_id} for ${NPM_PROXY_HOST_DOMAIN}"
|
||||||
|
|
||||||
|
# Also retrieve current forward_host for logging
|
||||||
|
local current_host
|
||||||
|
current_host=$(echo "${body}" | jq -r \
|
||||||
|
--arg domain "${NPM_PROXY_HOST_DOMAIN}" \
|
||||||
|
'.[] | select(.domain_names[] == $domain) | .forward_host' | head -n1)
|
||||||
|
log_info "Current forward host: ${current_host}"
|
||||||
|
|
||||||
|
echo "${host_id}|${current_host}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 7. UPDATE PROXY HOST FORWARD IP
|
||||||
|
# ==============================================================================
|
||||||
|
npm_update_proxy_host() {
|
||||||
|
local token="$1"
|
||||||
|
local host_id="$2"
|
||||||
|
local new_ip="$3"
|
||||||
|
|
||||||
|
log "Updating proxy host ${host_id} → forward to ${new_ip}..."
|
||||||
|
|
||||||
|
# First, get the full current configuration to preserve all existing settings
|
||||||
|
local current_config
|
||||||
|
current_config=$(curl -s \
|
||||||
|
-X GET "${NPM_API_URL}/nginx/proxy-hosts/${host_id}" \
|
||||||
|
-H "Authorization: Bearer ${token}" \
|
||||||
|
--connect-timeout 10 \
|
||||||
|
--max-time 15)
|
||||||
|
|
||||||
|
# Build the update payload preserving existing config, only changing forward_host
|
||||||
|
local update_payload
|
||||||
|
update_payload=$(echo "${current_config}" | jq \
|
||||||
|
--arg new_ip "${new_ip}" \
|
||||||
|
'. + {"forward_host": $new_ip}')
|
||||||
|
|
||||||
|
if [ "${DRY_RUN}" = "true" ]; then
|
||||||
|
log_warning "[DRY RUN] Would send PUT to ${NPM_API_URL}/nginx/proxy-hosts/${host_id}"
|
||||||
|
log_warning "[DRY RUN] Payload: ${update_payload}"
|
||||||
|
log_success "[DRY RUN] NPM failover simulation complete — no changes made."
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
local response
|
||||||
|
response=$(curl -s -w "\n%{http_code}" \
|
||||||
|
-X PUT "${NPM_API_URL}/nginx/proxy-hosts/${host_id}" \
|
||||||
|
-H "Authorization: Bearer ${token}" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "${update_payload}" \
|
||||||
|
--connect-timeout 10 \
|
||||||
|
--max-time 15)
|
||||||
|
|
||||||
|
local http_code
|
||||||
|
http_code=$(echo "${response}" | tail -n1)
|
||||||
|
local body
|
||||||
|
body=$(echo "${response}" | head -n-1)
|
||||||
|
|
||||||
|
if [ "${http_code}" != "200" ]; then
|
||||||
|
log_error "Failed to update proxy host (HTTP ${http_code})"
|
||||||
|
log_error "Response: ${body}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Verify the change was applied
|
||||||
|
local confirmed_host
|
||||||
|
confirmed_host=$(echo "${body}" | jq -r '.forward_host // empty')
|
||||||
|
if [ "${confirmed_host}" != "${new_ip}" ]; then
|
||||||
|
log_error "NPM accepted the request but the forward_host is '${confirmed_host}', expected '${new_ip}'."
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
log_success "NPM proxy host updated successfully: ${NPM_PROXY_HOST_DOMAIN} → ${new_ip}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 8. MAIN
|
||||||
|
# ==============================================================================
|
||||||
|
main() {
|
||||||
|
echo ""
|
||||||
|
echo "========================================================="
|
||||||
|
echo " Wordly.art — NPM Failover"
|
||||||
|
echo " Target IP : ${TARGET_IP:-NOT SET}"
|
||||||
|
echo " NPM API : ${NPM_API_URL:-NOT SET}"
|
||||||
|
echo " Domain : ${NPM_PROXY_HOST_DOMAIN}"
|
||||||
|
echo " Dry Run : ${DRY_RUN}"
|
||||||
|
echo "========================================================="
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
validate_config
|
||||||
|
|
||||||
|
# Step 1: Authenticate
|
||||||
|
local token
|
||||||
|
token=$(npm_authenticate)
|
||||||
|
|
||||||
|
# Step 2: Find proxy host ID and current IP
|
||||||
|
local host_info
|
||||||
|
host_info=$(npm_find_proxy_host "${token}")
|
||||||
|
local host_id="${host_info%%|*}"
|
||||||
|
local current_ip="${host_info##*|}"
|
||||||
|
|
||||||
|
if [ "${current_ip}" = "${TARGET_IP}" ]; then
|
||||||
|
log_warning "NPM already points to ${TARGET_IP}. No change needed."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Step 3: Update forward host
|
||||||
|
npm_update_proxy_host "${token}" "${host_id}" "${TARGET_IP}"
|
||||||
|
|
||||||
|
# Step 4: Notify
|
||||||
|
if [ "${DRY_RUN}" = "false" ]; then
|
||||||
|
local msg="🔀 *Wordly.art NPM Failover*
|
||||||
|
Domaine : \`${NPM_PROXY_HOST_DOMAIN}\`
|
||||||
|
Ancien serveur : \`${current_ip}\`
|
||||||
|
Nouveau serveur : \`${TARGET_IP}\`
|
||||||
|
Heure : $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
send_telegram "${msg}"
|
||||||
|
log_success "Telegram notification sent."
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
log_success "========================================================="
|
||||||
|
log_success "NPM Failover COMPLETE"
|
||||||
|
log_success " ${NPM_PROXY_HOST_DOMAIN} now routes to → ${TARGET_IP}"
|
||||||
|
log_success "========================================================="
|
||||||
|
echo ""
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
264
scripts/setup-nas.sh
Normal file
264
scripts/setup-nas.sh
Normal file
@@ -0,0 +1,264 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# ==============================================================================
|
||||||
|
# Wordly.art - NAS Setup via SSH/rsync
|
||||||
|
# ==============================================================================
|
||||||
|
# Configure l'accès SSH sans mot de passe au NAS Synology.
|
||||||
|
# Remplace l'approche CIFS/SMB par rsync sur SSH :
|
||||||
|
# - Pas de montage à gérer, pas de fstab
|
||||||
|
# - Path exact /volume1/backups/wordly utilisable directement
|
||||||
|
# - SSH chiffré, robuste aux redémarrages NAS
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# sudo bash scripts/setup-nas.sh
|
||||||
|
#
|
||||||
|
# Prérequis côté NAS Synology (voir DISASTER_RECOVERY.md section 1) :
|
||||||
|
# 1. Compte 'wordly-backup' créé dans DSM → Utilisateurs et groupes
|
||||||
|
# 2. Accès R/W sur le dossier 'backups'
|
||||||
|
# 3. SSH activé dans DSM → Terminal et SNMP
|
||||||
|
# 4. Dossier /volume1/backups/wordly créé et chown wordly-backup
|
||||||
|
# ==============================================================================
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||||
|
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m'
|
||||||
|
|
||||||
|
log() { echo -e "[NAS-Setup] $1"; }
|
||||||
|
log_success() { echo -e "[NAS-Setup] ${GREEN}✅ $1${NC}"; }
|
||||||
|
log_warning() { echo -e "[NAS-Setup] ${YELLOW}⚠️ $1${NC}"; }
|
||||||
|
log_error() { echo -e "[NAS-Setup] ${RED}❌ $1${NC}"; }
|
||||||
|
log_info() { echo -e "[NAS-Setup] ${BLUE}ℹ️ $1${NC}"; }
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 1. ROOT CHECK
|
||||||
|
# ==============================================================================
|
||||||
|
if [ "$EUID" -ne 0 ]; then
|
||||||
|
log_error "Ce script doit être exécuté en root : sudo bash $0"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 2. CHARGER LE .env
|
||||||
|
# ==============================================================================
|
||||||
|
ENV_FILE="${PROJECT_ROOT}/.env"
|
||||||
|
if [ -f "${ENV_FILE}" ]; then
|
||||||
|
set -a
|
||||||
|
source "${ENV_FILE}"
|
||||||
|
set +a
|
||||||
|
fi
|
||||||
|
|
||||||
|
NAS_HOST="${NAS_HOST:-192.168.1.146}"
|
||||||
|
NAS_USER="${NAS_USER:-wordly-backup}"
|
||||||
|
NAS_PATH="${NAS_PATH:-/volume1/backups/wordly}"
|
||||||
|
NAS_SSH_PORT="${NAS_SSH_PORT:-22}"
|
||||||
|
NAS_SSH_KEY="${NAS_SSH_KEY:-/root/.ssh/wordly_nas_key}"
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 3. VÉRIFIER QUE SSH EST DISPO SUR LE NAS
|
||||||
|
# ==============================================================================
|
||||||
|
check_nas_reachable() {
|
||||||
|
log "Vérification de la connectivité SSH vers ${NAS_HOST}:${NAS_SSH_PORT}..."
|
||||||
|
if ! nc -z -w5 "${NAS_HOST}" "${NAS_SSH_PORT}" 2>/dev/null; then
|
||||||
|
log_error "Impossible de joindre ${NAS_HOST} sur le port ${NAS_SSH_PORT}."
|
||||||
|
log_error "Vérifiez que SSH est activé dans DSM → Panneau de configuration → Terminal et SNMP."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
log_success "NAS ${NAS_HOST}:${NAS_SSH_PORT} est joignable"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 4. GÉNÉRER LA CLÉ SSH DÉDIÉE (si elle n'existe pas)
|
||||||
|
# ==============================================================================
|
||||||
|
generate_ssh_key() {
|
||||||
|
if [ -f "${NAS_SSH_KEY}" ]; then
|
||||||
|
log_info "Clé SSH déjà existante : ${NAS_SSH_KEY}"
|
||||||
|
log_info "Si vous voulez la regénérer : rm ${NAS_SSH_KEY} ${NAS_SSH_KEY}.pub"
|
||||||
|
else
|
||||||
|
log "Génération de la clé SSH dédiée aux backups NAS..."
|
||||||
|
ssh-keygen -t ed25519 \
|
||||||
|
-C "wordly-backup@$(hostname)-$(date +%Y%m%d)" \
|
||||||
|
-f "${NAS_SSH_KEY}" \
|
||||||
|
-N ""
|
||||||
|
chmod 600 "${NAS_SSH_KEY}"
|
||||||
|
log_success "Clé SSH générée : ${NAS_SSH_KEY}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
log_info "Clé publique à autoriser sur le NAS :"
|
||||||
|
cat "${NAS_SSH_KEY}.pub"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 5. COPIER LA CLÉ PUBLIQUE SUR LE NAS
|
||||||
|
# ==============================================================================
|
||||||
|
install_ssh_key() {
|
||||||
|
log "Copie de la clé publique sur le NAS ${NAS_USER}@${NAS_HOST}..."
|
||||||
|
log_warning "Le mot de passe du compte '${NAS_USER}' sur le NAS vous sera demandé UNE SEULE FOIS."
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
if ssh-copy-id \
|
||||||
|
-i "${NAS_SSH_KEY}.pub" \
|
||||||
|
-p "${NAS_SSH_PORT}" \
|
||||||
|
-o StrictHostKeyChecking=accept-new \
|
||||||
|
"${NAS_USER}@${NAS_HOST}"; then
|
||||||
|
log_success "Clé publique installée sur le NAS — plus de mot de passe requis"
|
||||||
|
else
|
||||||
|
log_error "Échec de ssh-copy-id. Avez-vous bien créé le compte '${NAS_USER}' sur le Synology ?"
|
||||||
|
log_error "Vérifiez aussi que SSH est activé dans DSM."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 6. TESTER LA CONNEXION SANS MOT DE PASSE
|
||||||
|
# ==============================================================================
|
||||||
|
test_ssh_connection() {
|
||||||
|
log "Test de la connexion SSH sans mot de passe..."
|
||||||
|
local result
|
||||||
|
if result=$(ssh \
|
||||||
|
-i "${NAS_SSH_KEY}" \
|
||||||
|
-p "${NAS_SSH_PORT}" \
|
||||||
|
-o StrictHostKeyChecking=accept-new \
|
||||||
|
-o ConnectTimeout=10 \
|
||||||
|
-o BatchMode=yes \
|
||||||
|
"${NAS_USER}@${NAS_HOST}" \
|
||||||
|
"echo OK" 2>/dev/null); then
|
||||||
|
if [ "${result}" = "OK" ]; then
|
||||||
|
log_success "Connexion SSH sans mot de passe : OK"
|
||||||
|
else
|
||||||
|
log_error "Connexion établie mais réponse inattendue : ${result}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
log_error "Connexion SSH sans mot de passe ÉCHOUÉE."
|
||||||
|
log_error "Vérifiez que la clé a bien été copiée avec ssh-copy-id."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 7. CRÉER LA STRUCTURE DE DOSSIERS SUR LE NAS
|
||||||
|
# ==============================================================================
|
||||||
|
create_nas_directories() {
|
||||||
|
log "Création de la structure de dossiers sur le NAS : ${NAS_PATH}..."
|
||||||
|
ssh \
|
||||||
|
-i "${NAS_SSH_KEY}" \
|
||||||
|
-p "${NAS_SSH_PORT}" \
|
||||||
|
-o BatchMode=yes \
|
||||||
|
"${NAS_USER}@${NAS_HOST}" \
|
||||||
|
"mkdir -p ${NAS_PATH}/snapshots ${NAS_PATH}/scripts && echo OK"
|
||||||
|
log_success "Structure créée sur le NAS :"
|
||||||
|
log_success " ${NAS_PATH}/snapshots/ → archives DR"
|
||||||
|
log_success " ${NAS_PATH}/scripts/ → scripts de restauration"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 8. TESTER L'ÉCRITURE SUR LE NAS
|
||||||
|
# ==============================================================================
|
||||||
|
test_nas_write() {
|
||||||
|
log "Test d'écriture sur le NAS..."
|
||||||
|
local test_file="${NAS_PATH}/.write_test_$$"
|
||||||
|
if ssh \
|
||||||
|
-i "${NAS_SSH_KEY}" \
|
||||||
|
-p "${NAS_SSH_PORT}" \
|
||||||
|
-o BatchMode=yes \
|
||||||
|
"${NAS_USER}@${NAS_HOST}" \
|
||||||
|
"touch ${test_file} && rm -f ${test_file} && echo OK" | grep -q "OK"; then
|
||||||
|
log_success "Écriture sur le NAS : OK"
|
||||||
|
else
|
||||||
|
log_error "Le NAS est accessible mais pas accessible en écriture !"
|
||||||
|
log_error "Vérifiez les permissions du dossier '${NAS_PATH}' pour l'utilisateur '${NAS_USER}'."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 9. CRÉER LE FICHIER DE CONFIG SSH (~/.ssh/config)
|
||||||
|
# ==============================================================================
|
||||||
|
configure_ssh_config() {
|
||||||
|
local ssh_config="/root/.ssh/config"
|
||||||
|
local host_entry="
|
||||||
|
# Wordly.art — NAS Synology Backup
|
||||||
|
Host wordly-nas
|
||||||
|
HostName ${NAS_HOST}
|
||||||
|
User ${NAS_USER}
|
||||||
|
Port ${NAS_SSH_PORT}
|
||||||
|
IdentityFile ${NAS_SSH_KEY}
|
||||||
|
StrictHostKeyChecking accept-new
|
||||||
|
ConnectTimeout 10
|
||||||
|
ServerAliveInterval 30
|
||||||
|
BatchMode yes"
|
||||||
|
|
||||||
|
if grep -q "wordly-nas" "${ssh_config}" 2>/dev/null; then
|
||||||
|
log_warning "Entrée 'wordly-nas' déjà dans ${ssh_config} — ignorée."
|
||||||
|
else
|
||||||
|
mkdir -p /root/.ssh
|
||||||
|
chmod 700 /root/.ssh
|
||||||
|
echo "${host_entry}" >> "${ssh_config}"
|
||||||
|
chmod 600 "${ssh_config}"
|
||||||
|
log_success "Config SSH ajoutée : ${ssh_config}"
|
||||||
|
log_info "Vous pouvez maintenant utiliser : ssh wordly-nas"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 10. COPIER LES SCRIPTS SUR LE NAS (disponibles depuis n'importe quel serveur)
|
||||||
|
# ==============================================================================
|
||||||
|
sync_scripts_to_nas() {
|
||||||
|
log "Synchronisation des scripts sur le NAS (pour restauration depuis .98)..."
|
||||||
|
if rsync -az \
|
||||||
|
-e "ssh -i ${NAS_SSH_KEY} -p ${NAS_SSH_PORT} -o BatchMode=yes" \
|
||||||
|
--exclude="__pycache__" \
|
||||||
|
--exclude="*.pyc" \
|
||||||
|
"${SCRIPT_DIR}/" \
|
||||||
|
"${NAS_USER}@${NAS_HOST}:${NAS_PATH}/scripts/"; then
|
||||||
|
log_success "Scripts synchronisés sur le NAS : ${NAS_PATH}/scripts/"
|
||||||
|
else
|
||||||
|
log_warning "rsync a échoué — vérifiez que rsync est installé sur le NAS Synology."
|
||||||
|
log_warning "Essayez : Synology Package Center → installer 'Rsync Server'"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 11. MAIN
|
||||||
|
# ==============================================================================
|
||||||
|
main() {
|
||||||
|
echo ""
|
||||||
|
echo "================================================================="
|
||||||
|
echo " Wordly.art — Setup NAS via SSH/rsync"
|
||||||
|
echo " NAS : ${NAS_USER}@${NAS_HOST}:${NAS_SSH_PORT}"
|
||||||
|
echo " Path : ${NAS_PATH}"
|
||||||
|
echo " Clé : ${NAS_SSH_KEY}"
|
||||||
|
echo "================================================================="
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
check_nas_reachable
|
||||||
|
generate_ssh_key
|
||||||
|
install_ssh_key
|
||||||
|
test_ssh_connection
|
||||||
|
create_nas_directories
|
||||||
|
test_nas_write
|
||||||
|
configure_ssh_config
|
||||||
|
sync_scripts_to_nas
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
log_success "================================================================="
|
||||||
|
log_success "Setup NAS COMPLET"
|
||||||
|
log_success ""
|
||||||
|
log_success " ✅ Clé SSH : ${NAS_SSH_KEY}"
|
||||||
|
log_success " ✅ Connexion sans mot de passe : wordly-nas"
|
||||||
|
log_success " ✅ Dossiers créés sur le NAS"
|
||||||
|
log_success " ✅ Scripts disponibles pour DR depuis n'importe quel serveur"
|
||||||
|
log_success ""
|
||||||
|
log_success " Test rapide : ssh wordly-nas 'ls ${NAS_PATH}/'"
|
||||||
|
log_success " Étape suivante : bash scripts/backup-to-nas.sh --full"
|
||||||
|
log_success "================================================================="
|
||||||
|
echo ""
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
370
scripts/verify-backups.sh
Normal file
370
scripts/verify-backups.sh
Normal file
@@ -0,0 +1,370 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# ==============================================================================
|
||||||
|
# Wordly.art - Backup Verification & Telegram Alerts
|
||||||
|
# ==============================================================================
|
||||||
|
# Runs after every backup to validate integrity and alert on failure.
|
||||||
|
# CRON: 30 */6 * * * (30 minutes after each backup)
|
||||||
|
#
|
||||||
|
# Checks:
|
||||||
|
# - Recent snapshot exists (< 8h)
|
||||||
|
# - Snapshot size > 1MB (not empty)
|
||||||
|
# - Snapshot gzip integrity
|
||||||
|
# - PostgreSQL is responding
|
||||||
|
# - DB contains data (COUNT > 0)
|
||||||
|
# - NAS is mounted and writable
|
||||||
|
# - Disk usage < 85%
|
||||||
|
# - App HTTP health check
|
||||||
|
# ==============================================================================
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||||
|
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
|
||||||
|
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
NC='\033[0m'
|
||||||
|
|
||||||
|
log() { echo "[Verify ${TIMESTAMP}] $1"; }
|
||||||
|
log_success() { echo -e "[Verify ${TIMESTAMP}] ${GREEN}✅ $1${NC}"; }
|
||||||
|
log_warning() { echo -e "[Verify ${TIMESTAMP}] ${YELLOW}⚠️ WARNING: $1${NC}"; }
|
||||||
|
log_error() { echo -e "[Verify ${TIMESTAMP}] ${RED}❌ ERROR: $1${NC}"; }
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 1. LOAD CONFIGURATION
|
||||||
|
# ==============================================================================
|
||||||
|
ENV_FILE="${PROJECT_ROOT}/.env"
|
||||||
|
if [ -f "${ENV_FILE}" ]; then
|
||||||
|
set -a
|
||||||
|
source "${ENV_FILE}"
|
||||||
|
set +a
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Directories
|
||||||
|
NAS_MOUNT="${NAS_MOUNT:-/mnt/nas-wordly}"
|
||||||
|
LOCAL_BACKUP_DIR="${BACKUP_DIR:-/opt/wordly/backups}"
|
||||||
|
|
||||||
|
# PostgreSQL
|
||||||
|
POSTGRES_CONTAINER="${POSTGRES_CONTAINER:-wordly-postgres}"
|
||||||
|
POSTGRES_USER="${POSTGRES_USER:-translate}"
|
||||||
|
POSTGRES_DB="${POSTGRES_DB:-translate_db}"
|
||||||
|
POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-}"
|
||||||
|
|
||||||
|
# App health check
|
||||||
|
APP_HEALTH_URL="${APP_HEALTH_URL:-http://localhost:8001/health}"
|
||||||
|
|
||||||
|
# Thresholds
|
||||||
|
MAX_SNAPSHOT_AGE_HOURS=8
|
||||||
|
MIN_SNAPSHOT_SIZE_MB=1
|
||||||
|
MAX_DISK_USAGE_PERCENT=85
|
||||||
|
|
||||||
|
# Telegram
|
||||||
|
TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN:-}"
|
||||||
|
TELEGRAM_CHAT_ID="${TELEGRAM_CHAT_ID:-}"
|
||||||
|
|
||||||
|
# Track failures
|
||||||
|
FAILURES=0
|
||||||
|
WARNINGS=0
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 2. TELEGRAM
|
||||||
|
# ==============================================================================
|
||||||
|
send_telegram() {
|
||||||
|
local message="$1"
|
||||||
|
if [ -n "${TELEGRAM_BOT_TOKEN}" ] && [ -n "${TELEGRAM_CHAT_ID}" ]; then
|
||||||
|
curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
|
||||||
|
-d "chat_id=${TELEGRAM_CHAT_ID}" \
|
||||||
|
-d "text=${message}" \
|
||||||
|
-d "parse_mode=Markdown" \
|
||||||
|
>/dev/null 2>&1 || true
|
||||||
|
else
|
||||||
|
log_warning "Telegram not configured (TELEGRAM_BOT_TOKEN or TELEGRAM_CHAT_ID missing)"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 3. CHECK FUNCTIONS
|
||||||
|
# ==============================================================================
|
||||||
|
|
||||||
|
check_recent_snapshot() {
|
||||||
|
log "Check 1/8: Recent snapshot exists (< ${MAX_SNAPSHOT_AGE_HOURS}h)..."
|
||||||
|
|
||||||
|
# Look in both NAS and local backup directories
|
||||||
|
local search_dirs=("${LOCAL_BACKUP_DIR}/daily")
|
||||||
|
if mountpoint -q "${NAS_MOUNT}" 2>/dev/null; then
|
||||||
|
search_dirs+=("${NAS_MOUNT}/snapshots")
|
||||||
|
fi
|
||||||
|
|
||||||
|
local newest_snapshot=""
|
||||||
|
for dir in "${search_dirs[@]}"; do
|
||||||
|
if [ -d "${dir}" ]; then
|
||||||
|
local candidate
|
||||||
|
candidate=$(ls -t "${dir}"/*.gz 2>/dev/null | head -n1 || true)
|
||||||
|
if [ -n "${candidate}" ]; then
|
||||||
|
newest_snapshot="${candidate}"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "${newest_snapshot}" ]; then
|
||||||
|
log_error "No snapshot found in backup directories!"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check age
|
||||||
|
local snapshot_time
|
||||||
|
snapshot_time=$(stat -c %Y "${newest_snapshot}" 2>/dev/null || stat -f %m "${newest_snapshot}" 2>/dev/null)
|
||||||
|
local now
|
||||||
|
now=$(date +%s)
|
||||||
|
local age_hours=$(( (now - snapshot_time) / 3600 ))
|
||||||
|
|
||||||
|
if [ "${age_hours}" -ge "${MAX_SNAPSHOT_AGE_HOURS}" ]; then
|
||||||
|
log_error "Newest snapshot is ${age_hours}h old (max: ${MAX_SNAPSHOT_AGE_HOURS}h): $(basename "${newest_snapshot}")"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
else
|
||||||
|
log_success "Snapshot found: $(basename "${newest_snapshot}") (${age_hours}h old)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "${newest_snapshot}"
|
||||||
|
}
|
||||||
|
|
||||||
|
check_snapshot_size() {
|
||||||
|
local snapshot_path="$1"
|
||||||
|
log "Check 2/8: Snapshot size > ${MIN_SNAPSHOT_SIZE_MB}MB..."
|
||||||
|
|
||||||
|
if [ -z "${snapshot_path}" ] || [ ! -f "${snapshot_path}" ]; then
|
||||||
|
log_warning "No snapshot to size-check."
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
local size_bytes
|
||||||
|
size_bytes=$(stat -c %s "${snapshot_path}" 2>/dev/null || stat -f %z "${snapshot_path}" 2>/dev/null)
|
||||||
|
local min_bytes=$((MIN_SNAPSHOT_SIZE_MB * 1024 * 1024))
|
||||||
|
|
||||||
|
if [ "${size_bytes}" -lt "${min_bytes}" ]; then
|
||||||
|
log_error "Snapshot size is $(numfmt --to=iec ${size_bytes}) which is below minimum ${MIN_SNAPSHOT_SIZE_MB}MB — likely empty dump!"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
else
|
||||||
|
log_success "Snapshot size: $(numfmt --to=iec ${size_bytes})"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
check_snapshot_integrity() {
|
||||||
|
local snapshot_path="$1"
|
||||||
|
log "Check 3/8: Snapshot gzip integrity..."
|
||||||
|
|
||||||
|
if [ -z "${snapshot_path}" ] || [ ! -f "${snapshot_path}" ]; then
|
||||||
|
log_warning "No snapshot to integrity-check."
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
if gzip -t "${snapshot_path}" 2>/dev/null; then
|
||||||
|
log_success "Snapshot gzip integrity OK"
|
||||||
|
else
|
||||||
|
log_error "Snapshot is CORRUPTED: $(basename "${snapshot_path}")"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
check_postgres_running() {
|
||||||
|
log "Check 4/8: PostgreSQL container is running and healthy..."
|
||||||
|
|
||||||
|
if ! command -v docker &>/dev/null; then
|
||||||
|
log_warning "Docker not found — skipping PostgreSQL check."
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! docker ps --format '{{.Names}}' 2>/dev/null | grep -q "^${POSTGRES_CONTAINER}$"; then
|
||||||
|
log_error "PostgreSQL container '${POSTGRES_CONTAINER}' is NOT running!"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
local health
|
||||||
|
health=$(docker inspect --format='{{.State.Health.Status}}' "${POSTGRES_CONTAINER}" 2>/dev/null || echo "unknown")
|
||||||
|
if [ "${health}" = "healthy" ]; then
|
||||||
|
log_success "PostgreSQL container is healthy"
|
||||||
|
elif [ "${health}" = "unknown" ]; then
|
||||||
|
log_warning "PostgreSQL health status unknown (no healthcheck configured?)"
|
||||||
|
WARNINGS=$((WARNINGS + 1))
|
||||||
|
else
|
||||||
|
log_error "PostgreSQL container health status: ${health}"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
check_db_has_data() {
|
||||||
|
log "Check 5/8: Database contains data (COUNT > 0)..."
|
||||||
|
|
||||||
|
if ! command -v docker &>/dev/null; then
|
||||||
|
log_warning "Docker not found — skipping DB data check."
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! docker ps --format '{{.Names}}' 2>/dev/null | grep -q "^${POSTGRES_CONTAINER}$"; then
|
||||||
|
log_warning "PostgreSQL container not running — skipping data check."
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Count rows across key tables (gracefully handle missing tables)
|
||||||
|
local count
|
||||||
|
count=$(docker exec -e PGPASSWORD="${POSTGRES_PASSWORD}" "${POSTGRES_CONTAINER}" \
|
||||||
|
psql -U "${POSTGRES_USER}" -d "${POSTGRES_DB}" -t -A \
|
||||||
|
-c "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = 'public';" \
|
||||||
|
2>/dev/null || echo "0")
|
||||||
|
|
||||||
|
count=$(echo "${count}" | tr -d '[:space:]')
|
||||||
|
|
||||||
|
if [ "${count}" = "0" ] || [ -z "${count}" ]; then
|
||||||
|
log_error "Database appears to be empty (no public tables found)!"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
else
|
||||||
|
log_success "Database has ${count} tables in public schema"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
check_nas_mounted() {
|
||||||
|
log "Check 6/8: NAS is mounted and writable at ${NAS_MOUNT}..."
|
||||||
|
|
||||||
|
if ! mountpoint -q "${NAS_MOUNT}" 2>/dev/null; then
|
||||||
|
log_error "NAS is NOT mounted at ${NAS_MOUNT}!"
|
||||||
|
log "Attempting emergency remount..."
|
||||||
|
mount "${NAS_MOUNT}" 2>/dev/null || true
|
||||||
|
|
||||||
|
if ! mountpoint -q "${NAS_MOUNT}" 2>/dev/null; then
|
||||||
|
log_error "Emergency remount FAILED. NAS is unavailable."
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
log_warning "NAS remounted successfully (was temporarily unmounted)."
|
||||||
|
WARNINGS=$((WARNINGS + 1))
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Test write access
|
||||||
|
local test_file="${NAS_MOUNT}/.write_test_${TIMESTAMP}"
|
||||||
|
if touch "${test_file}" 2>/dev/null && rm -f "${test_file}" 2>/dev/null; then
|
||||||
|
log_success "NAS is mounted and writable"
|
||||||
|
else
|
||||||
|
log_error "NAS is mounted but NOT writable!"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
check_disk_space() {
|
||||||
|
log "Check 7/8: Disk usage < ${MAX_DISK_USAGE_PERCENT}%..."
|
||||||
|
|
||||||
|
# Check NAS disk if mounted
|
||||||
|
if mountpoint -q "${NAS_MOUNT}" 2>/dev/null; then
|
||||||
|
local nas_usage
|
||||||
|
nas_usage=$(df "${NAS_MOUNT}" | awk 'NR==2 {gsub(/%/,""); print $5}')
|
||||||
|
if [ "${nas_usage}" -ge "${MAX_DISK_USAGE_PERCENT}" ]; then
|
||||||
|
log_error "NAS disk usage is ${nas_usage}% (threshold: ${MAX_DISK_USAGE_PERCENT}%)"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
else
|
||||||
|
log_success "NAS disk usage: ${nas_usage}%"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check local disk
|
||||||
|
local local_usage
|
||||||
|
local_usage=$(df /opt 2>/dev/null | awk 'NR==2 {gsub(/%/,""); print $5}' || df / | awk 'NR==2 {gsub(/%/,""); print $5}')
|
||||||
|
if [ "${local_usage}" -ge "${MAX_DISK_USAGE_PERCENT}" ]; then
|
||||||
|
log_error "Local disk usage is ${local_usage}% (threshold: ${MAX_DISK_USAGE_PERCENT}%)"
|
||||||
|
WARNINGS=$((WARNINGS + 1))
|
||||||
|
else
|
||||||
|
log_success "Local disk usage: ${local_usage}%"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
check_app_health() {
|
||||||
|
log "Check 8/8: App HTTP health check at ${APP_HEALTH_URL}..."
|
||||||
|
|
||||||
|
if ! command -v curl &>/dev/null; then
|
||||||
|
log_warning "curl not found — skipping HTTP health check."
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
local http_code
|
||||||
|
http_code=$(curl -s -o /dev/null -w "%{http_code}" \
|
||||||
|
--connect-timeout 5 \
|
||||||
|
--max-time 10 \
|
||||||
|
"${APP_HEALTH_URL}" 2>/dev/null || echo "000")
|
||||||
|
|
||||||
|
if [ "${http_code}" = "200" ]; then
|
||||||
|
log_success "App health check passed (HTTP ${http_code})"
|
||||||
|
elif [ "${http_code}" = "000" ]; then
|
||||||
|
log_error "App is unreachable (connection timeout)"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
else
|
||||||
|
log_error "App health check returned HTTP ${http_code}"
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 4. MAIN
|
||||||
|
# ==============================================================================
|
||||||
|
main() {
|
||||||
|
echo ""
|
||||||
|
echo "========================================================="
|
||||||
|
echo " Wordly.art — Backup Verification"
|
||||||
|
echo " $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
echo "========================================================="
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Run all checks
|
||||||
|
local newest_snapshot
|
||||||
|
newest_snapshot=$(check_recent_snapshot)
|
||||||
|
|
||||||
|
check_snapshot_size "${newest_snapshot}"
|
||||||
|
check_snapshot_integrity "${newest_snapshot}"
|
||||||
|
check_postgres_running
|
||||||
|
check_db_has_data
|
||||||
|
check_nas_mounted
|
||||||
|
check_disk_space
|
||||||
|
check_app_health
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "========================================================="
|
||||||
|
echo " Results: ${FAILURES} failure(s), ${WARNINGS} warning(s)"
|
||||||
|
echo "========================================================="
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Send Telegram report
|
||||||
|
if [ "${FAILURES}" -gt 0 ]; then
|
||||||
|
local msg="🚨 *Wordly.art — Backup Verification FAILED*
|
||||||
|
Date: $(date '+%Y-%m-%d %H:%M:%S')
|
||||||
|
Failures: ${FAILURES}
|
||||||
|
Warnings: ${WARNINGS}
|
||||||
|
|
||||||
|
Check logs on 192.168.1.151:
|
||||||
|
\`cat /var/log/wordly-verify.log\`"
|
||||||
|
send_telegram "${msg}"
|
||||||
|
log_error "Verification FAILED with ${FAILURES} error(s). Telegram alert sent."
|
||||||
|
exit 1
|
||||||
|
elif [ "${WARNINGS}" -gt 0 ]; then
|
||||||
|
local msg="⚠️ *Wordly.art — Backup Verification passed with warnings*
|
||||||
|
Date: $(date '+%Y-%m-%d %H:%M:%S')
|
||||||
|
Failures: 0
|
||||||
|
Warnings: ${WARNINGS}"
|
||||||
|
send_telegram "${msg}"
|
||||||
|
log_warning "Verification passed with ${WARNINGS} warning(s)."
|
||||||
|
else
|
||||||
|
# Only send success alert once per day (at 06:30)
|
||||||
|
local hour
|
||||||
|
hour=$(date +%H)
|
||||||
|
if [ "${hour}" = "06" ]; then
|
||||||
|
local msg="✅ *Wordly.art — Daily backup check OK*
|
||||||
|
Date: $(date '+%Y-%m-%d %H:%M:%S')
|
||||||
|
All 8 checks passed."
|
||||||
|
send_telegram "${msg}"
|
||||||
|
fi
|
||||||
|
log_success "All checks passed."
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
Reference in New Issue
Block a user