Helm & Kubernetes

This guide explains how to deploy Rafiki using Helm charts on a Kubernetes cluster. Helm is a package manager for Kubernetes that allows you to define, install, and upgrade complex Kubernetes applications through Helm charts.

Rafiki uses the following key infrastructure components:

• PostgreSQL: Used for storing application data, grants, and Open Payments resources
• Redis: Used for caching and session management
• TigerBeetle (recommended): High-performance accounting database for financial transaction processing and ledger management

Prerequisites

Recommended software version

We recommend using the latest vendor supported version for each of the software dependencies listed in this section.

Before you begin, ensure you have the following:

• Kubernetes cluster deployed
• kubectl installed and configured to access your cluster
• Helm installed

Install Rafiki using Helm

Add the Interledger Helm repository

Add the official Interledger Helm repository which contains the Rafiki charts:

helm repo add interledger https://interledger.github.io/charts/interledger
helm repo update

Verify available charts

You can list the available Rafiki charts:

helm search repo interledger/rafiki

This will show you the available chart versions for the different Rafiki components:

• interledger/rafiki-auth
• interledger/rafiki-backend
• interledger/rafiki-frontend

Create values files

Create separate values files for each Rafiki service to customize your deployment. You can start with minimal configurations and add more settings as needed.

Basic auth service configuration

Create auth-values.yaml:

# Minimal auth service configuration
nodeEnv: production
logLevel: info

# Database configuration (update with your credentials)
databaseUrl:
  secretKeyRef:
    name: rafiki-secrets
    key: DATABASE_URL

# Redis configuration
redisUrl:
  secretKeyRef:
    name: rafiki-secrets
    key: REDIS_URL

# Identity server configuration
identityServer:
  domain: 'https://your-domain.com/idp'
  serverSecret:
    secretKeyRef:
      name: rafiki-secrets
      key: IDENTITY_SERVER_SECRET

# Cookie signing key
cookieKey:
  secretKeyRef:
    name: rafiki-secrets
    key: COOKIE_KEY

# Service configuration
service:
  - name: service
    type: ClusterIP
    ports:
      - port: 3006
        targetPort: 3006
        protocol: TCP
        name: auth
      - port: 3003
        targetPort: 3003
        protocol: TCP
        name: admin
      - port: 3007
        targetPort: 3007
        protocol: TCP
        name: introspection
      - port: 3009
        targetPort: 3009
        protocol: TCP
        name: interaction

Basic backend service configuration

Create backend-values.yaml:

# Minimal backend service configuration
nodeEnv: production
logLevel: info
instanceName: 'your-rafiki-instance'

# Database configuration
databaseUrl:
  secretKeyRef:
    name: rafiki-secrets
    key: DATABASE_URL

# Redis configuration
redisUrl:
  secretKeyRef:
    name: rafiki-secrets
    key: REDIS_URL

# Auth service URLs
auth:
  grantUrl: http://rafiki-auth-service.default:3006
  introspectionUrl: http://rafiki-auth-service.default:3007

# ILP configuration
ilp:
  host: 'https://your-domain.com'
  address: test.your-instance
  connector: 'http://rafiki-backend-service.default:3002'
  streamSecret:
    secretKeyRef:
      name: rafiki-secrets
      key: STREAM_SECRET

# Webhook configuration
webhook:
  url: https://your-wallet.com/webhooks/rafiki

# Key configuration
key:
  id: your-unique-key-id
  pvk: '' # Will auto-generate if empty

# Webhook signature secret
webhookSignatureSecret:
  secretKeyRef:
    name: rafiki-secrets
    key: SIGNATURE_SECRET

# Service configuration
service:
  - name: service
    type: ClusterIP
    ports:
      - port: 3000
        targetPort: 3000
        protocol: TCP
        name: open-payments
      - port: 3001
        targetPort: 3001
        protocol: TCP
        name: admin
      - port: 3002
        targetPort: 3002
        protocol: TCP
        name: connector

Basic frontend service configuration

Create frontend-values.yaml:

# Minimal frontend service configuration
nodeEnv: production
logLevel: info

# Service URLs
serviceUrls:
  GRAPHQL_URL: http://rafiki-backend-service.default:3001/graphql
  OPEN_PAYMENTS_URL: https://your-domain.com/

# Port configuration
port: 3010

# Service configuration
service:
  - name: service
    type: ClusterIP
    ports:
      - port: 3010
        targetPort: 3010
        protocol: TCP
        name: http

# Kratos configuration (if using authentication)
kratos:
  enabled: false

Create Kubernetes secrets

Before installing the services, create the required secrets:

kubectl create secret generic rafiki-secrets \
  --from-literal=DATABASE_URL="postgresql://username:password@rafiki-postgresql:5432/rafiki" \
  --from-literal=REDIS_URL="redis://rafiki-redis:6379" \
  --from-literal=IDENTITY_SERVER_SECRET="your-identity-server-secret" \
  --from-literal=COOKIE_KEY="your-cookie-signing-key" \
  --from-literal=STREAM_SECRET="your-stream-secret" \
  --from-literal=SIGNATURE_SECRET="your-webhook-signature-secret"

Install the services

Install each Rafiki service using the appropriate values file:

# Install auth service
helm install rafiki-auth interledger/rafiki-auth -f auth-values.yaml

# Install backend service
helm install rafiki-backend interledger/rafiki-backend -f backend-values.yaml

# Install frontend service
helm install rafiki-frontend interledger/rafiki-frontend -f frontend-values.yaml

If you want to install to a specific namespace:

kubectl create namespace rafiki

helm install rafiki-auth interledger/rafiki-auth -f auth-values.yaml -n rafiki
helm install rafiki-backend interledger/rafiki-backend -f backend-values.yaml -n rafiki
helm install rafiki-frontend interledger/rafiki-frontend -f frontend-values.yaml -n rafiki

Verify the deployment

Check the status of your deployments:

# Check Helm releases
helm list

# Check running pods
kubectl get pods

# Check deployed services
kubectl get services

# Check logs for any issues
kubectl logs -l app.kubernetes.io/name=rafiki-auth
kubectl logs -l app.kubernetes.io/name=rafiki-backend
kubectl logs -l app.kubernetes.io/name=rafiki-frontend

Configure environment variables

Each Rafiki service can be configured via environment variables through Helm values. Below are the complete environment variable configurations for each service:

Auth service

The Rafiki auth service is responsible for handling Open Payments authorization and grant management. It implements the Open Payments authorization server specification and manages access tokens, grants, and interactions. The auth service connects to a PostgreSQL database to store grant-related data and uses Redis for session management.

Ports exposed:

• 3003 (admin) - Auth Admin API for managing grants and access tokens
• 3006 (auth) - Open Payments authorization server endpoint
• 3007 (introspection) - Access token introspection endpoint
• 3009 (interaction) - User interaction endpoint for grant flows

Running Rafiki behind a proxy

If you plan to run your Rafiki instance behind a proxy, you must set the trustProxy variable to true

Required values

Helm variable	Default	Description
auth.postgresql.host, auth.postgresql.port, auth.postgresql.username, auth.postgresql.database, auth.postgresql.password	postgresql://postgres:password@localhost:5432/auth_development	The URL of the Postgres database storing your Open Payments grant data. For Helm, these components are provided individually.
auth.server.domain	undefined	The public endpoint for your Rafiki instance’s public Open Payments routes.
auth.cookieKey	undefined	The koa KeyGrip key that is used to sign cookies for an interaction session.
auth.identityServer.domain	undefined	The URL of your IdP’s server, used by the authorization server to inform an Open Payments client of where to redirect the end-user to start interactions.
auth.identityServer.secret	undefined	A shared secret between the authorization server and the IdP server; the authorization server will use the secret to secure its IdP-related endpoints. When the IdP server sends requests to the authorization server, the IdP server must provide the secret via an x-idp-secret header.
auth.redis.host, auth.redis.port	redis://127.0.0.1:6379	The connection URL for Redis. For Helm, these components are provided individually.

Conditionally required values

Helm value name	Default	Description
auth.trustProxy	false	Must be set to true when running Rafiki behind a proxy. When true, the X-Forwarded-Proto header is used to determine if connections are secure.

Optional values

Helm value name	Default	Description
auth.accessToken.deletionDays	30	The days until expired and/or revoked access tokens are deleted.
auth.accessToken.expirySeconds	600 (10 minutes)	The expiry time, in seconds, for access tokens.
auth.adminApi.signatureVersion	1	The version of the request signing algorithm used to generate signatures.
auth.adminAPI.signatureTtlSeconds	30	The TTL, in seconds, for which a request’s signature will be valid.
auth.port.admin	3003	The port of your Rafiki Auth Admin API server.
auth.port.auth	3006	The port of your Open Payments authorization server.
auth.workers.cleanup	1	The number of workers processing expired or revoked access tokens.
auth.enableManualMigrations	false	When true, you must run the auth Postgres database manually with the command npm run knex – migrate:latest –envproduction
auth.interaction.incomingPayment	false	When true, incoming Open Payments grant requests are interactive
auth.interactionExpirySeconds	600 (10 minutes)	The time, in seconds, for which a user can interact with a grant request before the request expires.
auth.port.interaction	3009	The port number of your Open Payments interaction-related APIs.
auth.port.introspection	3007	The port of your Open Payments access token introspection server.
auth.interaction.listAll	true	When true, grant requests that include a list-all action will require interaction. In these requests, the client asks to list resources that it did not create.
auth.logLevel	info	Pino log level
auth.nodeEnv	development	The type of node environment: development, test, or production.
auth.interaction.quote	false	When true, quote grants are interactive.
auth.redis.tlsCaFile	''	Redis TLS config
auth.redis.tlsCertFile	''	Redis TLS config
auth.redis.tlsKeyFile	''	Redis TLS config
auth.grant.waitSeconds	5	The wait time, in seconds, included in a grant request response (grant.continue).

Backend service

The Rafiki backend service handles the core Open Payments functionality and business logic. It exposes the Open Payments resource server APIs for wallet addresses, incoming payments, outgoing payments, and quotes. The backend also includes an ILP connector for sending and receiving Interledger packets. It connects to PostgreSQL for storing Open Payments resources, Redis for caching, and optionally TigerBeetle for high-performance accounting.

Ports exposed:

• 3000 (openPayments) - Open Payments resource server API
• 3001 (admin) - Backend Admin API and GraphQL endpoint
• 3002 (connector) - ILP connector for sending and receiving packets
• 3005 (autoPeering) - Auto-peering service (when enabled)

TigerBeetle vs PostgreSQL for accounting

TigerBeetle is strongly recommended for production deployments due to its high-performance accounting capabilities. If you prefer to use PostgreSQL as the accounting database, set useTigerbeetle to false.

Required values

Helm value name	Default	Description
backend.serviceUrls.AUTH_SERVER_GRANT_URL	undefined	The endpoint on your Open Payments authorization server to grant a request.
backend.serviceUrls.AUTH_SERVER_INTROSPECTION_URL	undefined	The endpoint on your Open Payments authorization server to introspect an access token.
backend.postgresql.host, backend.postgresql.port, backend.postgresql.username, backend.postgresql.database, backend.postgresql.password	postgresql://postgres:password@localhost:5432/development	The Postgres database URL of the database storing your resource data. For Helm, these components are provided individually.
backend.serviceUrls.EXCHANGE_RATES_URL	undefined	The endpoint your Rafiki instance uses to request exchange rates.
backend.ilp.address	undefined	The ILP address of your Rafiki instance.
backend.ilp.connectorUrl	undefined	The ILP connector address where ILP packets are received.
backend.key.id	undefined	Your Rafiki instance’s client key ID.
backend.serviceUrls.OPEN_PAYMENTS_URL	undefined	The public endpoint of your Open Payments resource server.
backend.redis.host, backend.redis.port	redis://127.0.0.1:6379	The Redis URL of the database handling ILP packet data. For Helm, these components are provided individually.
backend.use.tigerbeetle	true	When true, a TigerBeetle database is used for accounting. When false, a Postgres database is used.
backend.serviceUrls.WEBHOOK_URL	undefined	Your endpoint that consumes webhook events.

Conditionally required values

Helm value name	Default	Description
backend.instance.name	undefined	Your Rafiki instance’s name used to communicate for auto-peering and/or telemetry. Required when auto-peering and/or telemetry is enabled
backend.trustProxy	false	Must be set to true when running Rafiki behind a proxy. When true, the X-Forwarded-Proto header is used to determine if connections are secure.

Optional values

Helm value name	Default	Description
backend.port.admin	3001	The port of your Backend Auth API server.
backend.autoPeering.serverPort	3005	If auto-peering is enabled, the server will use this port.
backend.port.connector	3002	The port of the ILP connector for sending packets via ILP over HTTP.
backend.enable.autoPeering	false	When true, auto-peering is enabled.
backend.enableManualMigrations	false	When true, you must run the database manually with the command npm run knex – migrate:latest –env production
backend.enable.spspPaymentPointers	true	When true, the SPSP route is enabled.
backend.lifetime.exchangeRate	15_000	The time, in milliseconds, the exchange rates you provide via the EXCHANGE_RATES_URL are valid.
backend.idempotency.keyLockMs	2000	The TTL, in milliseconds, for idempotencyKey concurrency lock on GraphQL mutations on the Backend Admin API.
backend.idempotency.keyTTL	86400000 (24 hours)	The TTL, in milliseconds, for idempotencyKey on GraphQL mutations on the Backend Admin API.
backend.incomingPayment.expiryMaxMs	2592000000 (30 days)	The maximum into the future, in milliseconds, incoming payments expiry can be set to on creation.
backend.workerIdle	200	The time, in milliseconds, that workers will wait until checking an empty queue again.
backend.workers.incomingPayment	1	The number of workers processing incoming payment requests.
backend.logLevel	info	Pino log level
backend.nodeEnv	development	The type of node environment: development, test, or production.
backend.port.openPayments	3003	The port of your Open Payments resource server.
backend.workers.outgoingPayment	4	The number of workers processing outgoing payment requests.
backend.key.file	undefined	The path to your Rafiki instance’s client private key.
backend.lifetime.quote	5 * 60_000 (5 minutes)	The time, in milliseconds, an Open Payments quote is valid for.
backend.redis.tlsCaFile	''	Redis TLS config
backend.redis.tlsCertFile	''	Redis TLS config
backend.redis.tlsKeyFile	''	Redis TLS config
backend.quoteSignatureSecret	undefined	The secret to generate request header signatures for webhook event requests.
backend.signatureVersion	1	The version number to generate request header signatures for webhook events.
backend.ilp.slippage	0.01 (1%)	The accepted ILP rate fluctuation.
backend.ilp.streamSecret	undefined	The seed secret to generate shared STREAM secrets.
backend.walletAddress.deactivationPaymentGratePeriodMs	86400000 (24 hours)	The time into the future, in milliseconds, to set expiration of Open Payments incoming payments when deactivating a wallet address.
backend.walletAddress.lookupTimeoutMs	1500	The time, in milliseconds, you have to create a missing wallet address before timeout.
backend.walletAddress.pollingFrequencyMs	100	The frequency of polling while waiting for you to create a missing wallet address.
backend.serviceUrls.WALLET_ADDRESS_URL	http://127.0.0.1:3001/.well-known/pay	Your Rafiki instance’s internal wallet address.
backend.workers.walletAddress	1	The number of workers processing wallet address requests.
backend.webhookMaxRetry	10	The maximum number of times your Rafiki instance’s backend retries sending a certain webhook event to your configured WEBHOOK_URL.
backend.lifetime.webhook	2000 (2 seconds)	The time, in milliseconds, that your Rafiki instance will wait for a 200 response from your webhook endpoint. If a 200 response is not received, Rafiki will time out and try to send the webhook event again.
backend.workers.webhook	1	The number of workers processing webhook events.
backend.withdrawalThrottleDelay	undefined	The delay in liquidity withdrawal processing.

Frontend service

The Rafiki frontend service provides an administrative web interface for managing your Rafiki instance. It offers a user-friendly dashboard for monitoring and managing wallet addresses, payments, peers, and other Open Payments resources. The frontend communicates with the Backend Admin API to perform administrative operations.

Ports exposed:

• 3010 (http) - Rafiki Admin web interface

Authentication

The frontend can optionally integrate with Ory Kratos for user authentication. When authentication is disabled (kratos.enabled: false), the admin interface is accessible without login credentials.

Required Values

Helm value name	Default	Description
frontend.serviceUrls.GRAPHQL_URL	undefined	URL for Rafiki’s GraphQL Auth Admin API
frontend.serviceUrls.OPEN_PAYMENTS_URL	undefined	Your Open Payments API endpoint
frontend.port	undefined	Port from which to host the Rafiki Remix app

Conditionally required values

The following values are required only when frontend.authEnabled is set to true.

Helm value name	Default	Description
frontend.kratos.adminUrl	undefined	The admin endpoint/container address for Kratos
frontend.kratos.containerPublicUrl	undefined	The URL for you to access the Kratos Docker container from within the Docker network. This is used for backend calls to Kratos.
frontend.kratos.browserPublicUrl	undefined	The URL for you to access the Kratos Docker container from a browser outside of the Docker network. This is used for calls from a browser (what you see in the Rafiki Admin UI) to the Kratos server on the backend.

Optional values

Helm value name	Default	Description
frontend.authEnabled	true	When true, only authenticated users can be granted access to Rafiki Admin by an administrator
frontend.quoteSignatureSecret	undefined	The signature secret used to authenticate requests to the Backend Admin API.
frontend.signatureVersion	1	The signature version number used to authenticate requests to the Backend Admin API.
frontend.enableInsecureMessageCookie	true	When set to true, t, or 1, cookie will be transmitted over insecure HTTP connection. Insecure message cookies are required for flash messages to work over HTTP.
frontend.nodeEnv	production	The type of node environment: development, test, or production.
frontend.logLevel	info	Pino log level

Configure ingress

To expose Rafiki services outside the cluster, you’ll need to configure ingress. Here’s an example using NGINX Ingress Controller:

Install NGINX Ingress Controller

If you don’t already have an ingress controller installed:

# Add the ingress-nginx repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# Install the ingress controller
helm install nginx-ingress ingress-nginx/ingress-nginx \
  --set controller.publishService.enabled=true

Configure ingress in your values files

Add ingress configuration to your service values files:

# In backend-values.yaml
ingress:
  enabled: true
  className: nginx
  hosts:
    - host: rafiki-backend.your-domain.com
      paths:
        - path: /
          pathType: Prefix
          service:
            name: rafiki-backend-service
            port: 3000
  tls:
    - secretName: rafiki-backend-tls
      hosts:
        - rafiki-backend.your-domain.com

# In auth-values.yaml
ingress:
  enabled: true
  className: nginx
  hosts:
    - host: rafiki-auth.your-domain.com
      paths:
        - path: /
          pathType: Prefix
          service:
            name: rafiki-auth-service
            port: 3006
  tls:
    - secretName: rafiki-auth-tls
      hosts:
        - rafiki-auth.your-domain.com

Configure DNS

Create DNS records pointing to your ingress controller’s external IP:

• rafiki-backend.your-domain.com
• rafiki-auth.your-domain.com
• rafiki-frontend.your-domain.com

Configure dependencies

PostgreSQL

You can use an external PostgreSQL instance or deploy one using Helm:

# Add Bitnami repository
helm repo add bitnami https://charts.bitnami.com/bitnami

# Install PostgreSQL
helm install rafiki-postgresql bitnami/postgresql \
  --set auth.postgresPassword=your-secure-password \
  --set auth.database=rafiki \
  --set primary.persistence.size=20Gi

Redis

Deploy Redis for caching and session management:

# Install Redis
helm install rafiki-redis bitnami/redis \
  --set auth.enabled=false \
  --set master.persistence.size=8Gi

TigerBeetle (recommended)

For high-performance accounting, deploy TigerBeetle:

# Add TigerBeetle repository
helm repo add tigerbeetle https://tigerbeetle.github.io/helm-charts

# Install TigerBeetle
helm install rafiki-tigerbeetle tigerbeetle/tigerbeetle \
  --set replicaCount=3 \
  --set persistence.size=20Gi

Upgrade Rafiki

To upgrade your Rafiki services to newer versions:

# Update the Helm repository
helm repo update

# Upgrade each service
helm upgrade rafiki-auth interledger/rafiki-auth -f auth-values.yaml
helm upgrade rafiki-backend interledger/rafiki-backend -f backend-values.yaml
helm upgrade rafiki-frontend interledger/rafiki-frontend -f frontend-values.yaml

Uninstall Rafiki

To uninstall Rafiki services:

helm uninstall rafiki-auth
helm uninstall rafiki-backend
helm uninstall rafiki-frontend

# Optionally uninstall dependencies
helm uninstall rafiki-postgresql
helm uninstall rafiki-redis
helm uninstall rafiki-tigerbeetle

Persistent data

Uninstalling will not delete Persistent Volume Claims (PVCs) by default. To delete all data:

kubectl delete pvc -l app.kubernetes.io/instance=rafiki-postgresql
kubectl delete pvc -l app.kubernetes.io/instance=rafiki-redis
kubectl delete pvc -l app.kubernetes.io/instance=rafiki-tigerbeetle

Troubleshooting

Check service health

Verify that all services are running correctly:

# Check pod status
kubectl get pods -l app.kubernetes.io/name=rafiki-auth
kubectl get pods -l app.kubernetes.io/name=rafiki-backend
kubectl get pods -l app.kubernetes.io/name=rafiki-frontend

# Check service endpoints
kubectl get endpoints

# Check for events
kubectl get events --sort-by=.metadata.creationTimestamp

Common issues and solutions

Database connection errors

1. Verify database connectivity:

   kubectl logs -l app.kubernetes.io/name=rafiki-backend | grep -i database

2. Check PostgreSQL status:

   kubectl get pods -l app.kubernetes.io/name=postgresql
   kubectl logs -l app.kubernetes.io/name=postgresql

3. Test database connection:

   kubectl run -it --rm debug --image=postgres:14 -- psql postgresql://username:password@rafiki-postgresql:5432/rafiki

Redis connection issues

1. Check Redis status:

   kubectl get pods -l app.kubernetes.io/name=redis
   kubectl logs -l app.kubernetes.io/name=redis

2. Test Redis connectivity:

   kubectl run -it --rm debug --image=redis:7 -- redis-cli -h rafiki-redis ping

Service discovery problems

1. Verify service names and ports:

   kubectl get services
   kubectl describe service rafiki-backend-service

2. Check DNS resolution:

   kubectl run -it --rm debug --image=busybox -- nslookup rafiki-backend-service

Authentication and authorization issues

1. Check auth service logs:

   kubectl logs -l app.kubernetes.io/name=rafiki-auth

2. Verify secrets are properly mounted:

   kubectl describe pod -l app.kubernetes.io/name=rafiki-auth

3. Test auth endpoints:

   kubectl port-forward svc/rafiki-auth-service 3007:3007
   curl http://localhost:3007/healthz

Ingress and networking issues

1. Check ingress controller status:

   kubectl get pods -n ingress-nginx
   kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller

2. Verify ingress resources:

   kubectl get ingress
   kubectl describe ingress rafiki-backend-ingress

3. Check TLS certificates:

   kubectl get secrets
   kubectl describe secret rafiki-backend-tls

Resource and performance issues

1. Check resource usage:

   kubectl top pods
   kubectl top nodes

2. Monitor pod metrics:

   kubectl describe pod -l app.kubernetes.io/name=rafiki-backend

3. Scale services if needed:

   # Update replicaCount in values.yaml and upgrade
   helm upgrade rafiki-backend interledger/rafiki-backend -f backend-values.yaml

Production considerations

Security

• Use strong passwords: Replace all default passwords with cryptographically secure values
• Enable TLS: Configure HTTPS for all external communications
• Network policies: Implement Kubernetes network policies to restrict pod-to-pod communication
• RBAC: Use Kubernetes Role-Based Access Control to limit cluster access
• Secrets management: Consider using external secret management solutions like HashiCorp Vault
• Image security: Use specific image tags and scan images for vulnerabilities
• Pod security: Configure pod security standards and security contexts

High availability

• Multi-replica deployments: Run multiple replicas of critical services
• Pod disruption budgets: Configure disruption budgets to maintain availability during updates
• Resource limits: Set appropriate CPU and memory limits
• Health checks: Configure proper readiness and liveness probes
• Load balancing: Use multiple ingress controller replicas

Monitoring and observability

• Prometheus: Deploy Prometheus for metrics collection
• Grafana: Set up Grafana dashboards for visualization
• Logging: Implement centralized logging with ELK stack or similar
• Alerting: Configure alerting rules for critical issues
• Distributed tracing: Consider implementing distributed tracing for request flow analysis

Backup and disaster recovery

Database backups

PostgreSQL backup:

# Create a backup job
kubectl create job --from=cronjob/postgres-backup manual-backup-$(date +%Y%m%d-%H%M%S)

# Manual backup
kubectl exec -it rafiki-postgresql-0 -- pg_dump -U postgres rafiki > backup.sql

TigerBeetle backup:

# Create volume snapshots
kubectl create -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: tigerbeetle-snapshot-$(date +%Y%m%d-%H%M%S)
spec:
  source:
    persistentVolumeClaimName: tigerbeetle-data-0
  volumeSnapshotClassName: your-snapshot-class
EOF

Performance optimization

• Resource allocation: Right-size CPU and memory requests/limits based on actual usage
• Connection pooling: Configure appropriate database connection pool sizes
• Caching: Optimize Redis configuration for your workload
• TigerBeetle tuning: Configure TigerBeetle cluster size based on expected transaction volume

Upgrade strategies

• Rolling updates: Use rolling update strategies to maintain availability
• Blue-green deployments: Consider blue-green deployments for zero-downtime upgrades
• Database migrations: Plan for database schema migrations
• Backup before upgrade: Always backup data before major upgrades
• Testing: Test upgrades in staging environments first