Stateful application storage operations
With Portworx, you can backup and restore stateful applications and their volumes to an external object store. Specifically, you can perform the following operations:
- Backup an application
- Restore an application
- Clone an application between namespaces
- Schedule application backup operations
Prerequisites
All application storage operations require the following prerequisites:
- Stork 2.3
- Administrator-level cluster privileges if you operate between namespaces
- Portworx 2.2
Configure Application Backups
Configure application backups by creating and applying the following CRDs:
- backupLocation
- applicationBackup
Create a backupLocation CRD
Use the backupLocation CRD to specify the configuration information for your object store. Portworx supports the following object stores:
- Any S3-compliant object store
- Azure Blob Storage
- Google Cloud Storage
The example in this document uses an S3 bucket to store application backup data.
"endpoint" : "bucketEndpoint.com"
"key": "ABCDEF1234567890"
"secret": "ABCDEF1234567890ABCDEF1234567890ABCDEF1234567890"
You can specify your object store credentials directly in the BackupLocation configuration as plaintext or use a Kubernetes secret:
Plaintext credentials
Create a backupLocation YAML file, specifying the following:
- name: the backupLocation object’s name
- namespace: the namespace the backupLocation exists in
- location:
- type: the object store type
- path: the bucket Portworx will use for the backup
- s3Config:
- region: which region your s3 bucket is located in
- accessKeyID: your bucket’s accessKeyID
- secretAccessKey: your bucket’s secretAccessKey
- endpoint: the URL or IP address of your bucket
- disableSSL: whether or not to disable SSL
apiVersion: stork.libopenstorage.org/v1alpha1 kind: BackupLocation metadata: name: mysql namespace: mysql-app annotations: stork.libopenstorage.org/skipresource: "true" location: type: s3 path: "bucket-name" s3Config: region: us-east-1 accessKeyID: ABCDEF1234567890 secretAccessKey: ABCDEF1234567890ABCDEF1234567890ABCDEF1234567890 endpoint: "https://bucketEndpoint.com" disableSSL: false
Note: If you use URL as your bucket endpoint, you must include the http prefix: eitherhttps://
orhttp://
, depending on whether or not you’re using SSL.Apply the YAML:
kubectl apply -f backLo.yaml
Kubernetes secret containing your credentials
Create a Kubernetes Secret YAML file, specifying the following:
- name: the Secret object’s name
- namespace: the namespace the Secret exists in
- stringData:
- region: which region your s3 bucket is located in
- accessKeyID: your bucket’s accessKeyID
- secretAccessKey: your bucket’s secretAccessKey
- endpoint: the URL or IP address of your bucket
- disableSSL: whether or not to disable SSL
- encryptionKey: your secret’s encryption key
apiVersion: v1 kind: Secret metadata: name: s3secret namespace: mysql annotations: stork.libopenstorage.org/skipresource: "true" stringData: region: us-east-1 accessKeyID: AB123AB43678AABCD12P secretAccessKey: b7D9pA3214Vafo8432023ajksndas43242kjsnfdk endpoint: "70.0.0.141:9010" disableSSL: "false" encryptionKey: "testKey"
Apply the Secret’s YAML file:
kubectl apply -f s3secret.yaml
Create a backupLocation YAML file, specifying the following:
- name: the backupLocation object’s name
- namespace: the namespace the backupLocation exists in
- location:
- type: the object store type
- path: the bucket Portworx will use for the backup
- secretConfig: the Secret object containing your bucket’s credentials
apiVersion: stork.libopenstorage.org/v1alpha1 kind: BackupLocation metadata: name: mysql-backup namespace: mysql annotations: stork.libopenstorage.org/skipresource: "true" location: type: s3 path: "bucket-name" secretConfig: s3secret
Note: If you use URL as your bucket endpoint, you must include the http prefix: eitherhttps://
orhttp://
, depending on whether or not you’re using SSL.Apply the YAML:
kubectl apply -f backLo.yaml
Create an applicationBackup CRD
Use the applicationBackup CRD to specify what namespaces have their applications backed-up.
Create an applicationbackup YAML file, specifying the following:
- name: the applicationBackup object’s name
- namespace: the namespace the applicationBackup exists in
- spec:
- backupLocation: what backupLocation object to use to determine where to send the backup
- namespaces: the namespaces to backup
- reclaimPolicy: what happens to objects in the object store when the
ApplicationBackup
object is deleted, eitherDelete
orRetain
- selectors: define specific labels to determine which objects and volumes are backed-up
- preExecRule: what rule to run before performing backup
- postExecRule: what rule to run after performing backup
apiVersion: stork.libopenstorage.org/v1alpha1 kind: ApplicationBackup metadata: name: backup namespace: mysql-app spec: backupLocation: mysql namespaces: - mysql-app reclaimPolicy: Delete selectors: preExecRule: postExecRule:
Apply the YAML:
kubectl apply -f appBack.yaml
Verify the applicationBackup object’s status with the
storkctl get
command. You can see what stage the backup is in, as well as its status:storkctl get applicationbackup -n mysql-app
NAME STAGE STATUS VOLUMES RESOURCES CREATED ELAPSED backup Volumes Pending 0/0 0 11 Sep 19 22:10 UTC 19.023065649s
You can also describe the object to get more information about the backup. This is useful for troubleshooting:
kubectl describe applicationbackup.stork.libopenstorage.org -n mysql-app
The following error message displays when your credentials are invalid.
Warning Failed 7s stork Error starting ApplicationBackup for volumes: rpc error: code = Internal desc = Failed to create backup: Failed to validate credentials: error validating credential: AccessDenied: Access Denied. status code: 403, request id: 15C38376E0A0EA7F, host id:
Create an ApplicationBackupSchedule CRD
The ApplicationBackupSchedule CRD associates a SchedulePolicy with an application backup operation, allowing you to schedule when and how application backups are performed.
Create a SchedulePolicy YAML file, specifying the following:
- name: the SchedulePolicy object’s name
- policy:
- interval: For interval backups, how frequently Portworx will back the application up
- intervalMinutes: The interval, in minutes, after which Portworx starts the application backup
- retain: How many backups Portworx will retain.
- daily: For daily backups, Portworx will start the backup at the specified time every day
- time:
- retain: How many backups Portworx will retain.
- weekly: For weekly backups, Portworx will start the backup at the specified day and time every week
- day: The backup day, specified by string
- time: The backup time, specified in 12 hour AM/PM format
- retain: How many backups Portworx will retain.
- monthly: for monthly backups, Portworx will start the backup at the specified day and time every month
- date: The backup day, specified as an integer
- time: the backup time, specified in 12 hour AM/PM format
- retain: How many backups Portworx will retain.
- interval: For interval backups, how frequently Portworx will back the application up
apiVersion: stork.libopenstorage.org/v1alpha1 kind: SchedulePolicy metadata: name: backupSchedule policy: interval: intervalMinutes: 60 retain: 5 daily: time: "10:14PM" retain: 5 weekly: day: "Thursday" time: "10:13PM" retain: 5 monthly: date: 14 time: "8:05PM" retain: 5
NOTE: If you do not specify the value of theretain
field, Portworx will use the following default values:Policy Default value Interval 10 Daily 30 Weekly 7 Monthly 12 Create an ApplicationBackupSchedule YAML file, specifying the following:
- name: the applicationBackupSchedule object’s name
- namespace: the namespace the applicationBackupSchedule exists in
- spec:
- schedulePolicyName: the name of the schedule policy that defines when backup actions happen
- template:
- spec:
- backupLocation: the name of the backup location spec
- namespaces: namespaces which will be backed up
- reclaimPolicy: what happens to objects in the object store when the
ApplicationBackup
object is deleted
- spec:
apiVersion: stork.libopenstorage.org/v1alpha1 kind: ApplicationBackupSchedule metadata: name: backup namespace: mysql spec: schedulePolicyName: testpolicy template: spec: backupLocation: mysql namespaces: - mysql reclaimPolicy: Delete
Restore an application
You can restore an application by applying an ApplicationRestore object.
NOTE: If you’re restoring an application across namespaces on OpenShift, you must modify your destination namespace to include the same supplemental group annotation values as your source namespace:
annotations:
openshift.io/sa.scc.supplemental-groups: 1001990000/10000
openshift.io/sa.scc.uid-range: 1001990000/10000
Create an ApplicationRestore YAML file, specifying the following:
- name: the ApplicationRestore object’s name
- namespace: the ApplicationRestore object’s namespace
- spec:
- backupName: the name of the
applicationBackup
object to restore from - backupLocation: which backup location object to get application backups from
- namespaceMapping: a map of source and destination namespaces, allowing you to restore a backup to a different namespace
- replacePolicy: specifies whether you want to delete or retain any matching existing resource in the target namespace
- backupName: the name of the
apiVersion: stork.libopenstorage.org/v1alpha1 kind: ApplicationRestore metadata: name: restore namespace: mysql-app spec: backupName: backup backupLocation: mysql namespaceMapping: mysql: mysql replacePolicy: Delete
Note: You can run thestorkctl get applicationbackup -n namespace
command to help you see which backup to restore from.Apply the YAML:
kubectl apply -f appRestore.yaml
The moment you apply the YAML, the application begins to restore. Monitor the status with the storkctl get
command:
storkctl get applicationrestore -n mysql-app
NAME STAGE STATUS VOLUMES RESOURCES CREATED ELAPSED
restore Final Successful 1/1 3 11 Sep 19 23:32 UTC 35s
Verify its status with the kubectl get pods
command:
kubectl get pods -n mysql-app
NAME READY STATUS RESTARTS AGE
mysql-6d69b99774-2bv4m 0/1 Pending 0 3s
Clone an Application
You can clone an application to a different namespace or within the same namespace. You must create the ApplicationClone
object in the admin namespace, which is kube-system
by default.
NOTE:
If you’re cloning an application across namespaces on OpenShift, you must modify your destination namespace to include the same supplemental group annotation values as your source namespace:
annotations: openshift.io/sa.scc.supplemental-groups: 1001990000/10000 openshift.io/sa.scc.uid-range: 1001990000/10000
Distributed apps, such as Cassandra, may use the same node IDs on the destination namespace as their source, causing disruption when the new nodes join the source cluster.
Create a an ApplicationClone YAML file, specifying the following:
- name: the ApplicationClone object’s name
- namespace: the ApplicationClone object’s namespace
- spec:
- sourceNamespace: the namespace you want to clone applications from
- destinationNamespace: the namespace you want to clone applications to
apiVersion: stork.libopenstorage.org/v1alpha1 kind: ApplicationClone metadata: name: clone-mysql namespace: kube-system spec: sourceNamespace: mysql-app destinationNamespace: clone-mysql
Apply the YAML file:
kubectl apply -f appClone.yaml