Install on Docker Standalone


This document presents the Docker method of installing a Portworx cluster using runC containers. Please refer to the Portworx on Kubernetes page if you want to install Portworx on Kubernetes.

Why OCI

Running Portworx as a runC container eliminates any cyclical dependencies between a Docker container consuming storage from the Portworx container. It also enables you to run your Linux containers without a Docker daemon completely, while still getting all of the advantages of a Linux container and cloud native storage from Portworx.

Install

Prerequisites

  • SYSTEMD: The installation below assumes the systemd package is installed on your system (i.e. systemctl command works).
    • Note, if you are running Ubuntu 16.04, CentoOS 7 or CoreOS v94 (or newer) the “systemd” is already installed and no actions will be required.
  • SCHEDULERS: If you are installing Portworx into Kubernetes or Mesosphere DC/OS cluster, we recommend to install the scheduler-specific Portworx package, which provides tighter integration, and better overall user experience.
  • FIREWALL: Ensure ports 9001-9022 are open between the cluster nodes that will run Portworx.
  • NTP: Ensure all nodes running Portworx are time-synchronized, and NTP service is configured and running.
  • KVDB: Please have a clustered key-value database (etcd or consul) installed and ready. For etcd installation instructions refer this doc.
  • STORAGE: At least one of the Portworx nodes should have extra storage available, in a form of unformatted partition or a disk-drive. Also please note that storage devices explicitly given to Portworx (ie. px-runc ... -s /dev/sdb -s /dev/sdc3) will be automatically formatted by Portworx.

The installation and setup of Portworx OCI bundle is a 3-step process:

  1. Install Portworx OCI bits
  2. Configure Portworx OCI
  3. Enable and start the Portworx service

Step 1: Install the Portworx OCI bundle

Portworx provides a Docker based installation utility to help deploy the Portworx OCI bundle. This bundle can be installed by running the following Docker container on your host system:

# Uncomment appropriate `REL` below to select desired Portworx release
REL="/2.2"       # 2.2 portworx release
#REL="/2.3"     # 2.3 portworx release

latest_stable=$(curl -fsSL "https://install.portworx.com$REL/?type=dock&stork=false" | awk '/image: / {print $2}')

# Download OCI bits (reminder, you will still need to run `px-runc install ..` after this step)
sudo docker run --entrypoint /runc-entry-point.sh \
    --rm -i --privileged=true \
    -v /opt/pwx:/opt/pwx -v /etc/pwx:/etc/pwx \
    $latest_stable
Running the Portworx OCI bundle does not require Docker, but Docker will still be required to install the Portworx OCI bundle. If you do not have Docker installed on your target hosts, you can download this Docker package and extract it to a root tar ball and manually install the OCI bundle.

Step 2: Configure Portworx under runC

Now that you have downloaded and installed the Portworx OCI bundle, you can use the px-runc install command from the bundle to configure systemd to start Portworx.

The px-runc command is a helper-tool that does the following:

  1. Prepares the OCI directory for runc
  2. Prepares the runC configuration for Portworx
  3. Uses systemd to start the Portworx OCI bundle

Installation example:

sudo /opt/pwx/bin/px-runc install -c MY_CLUSTER_ID \
    -k etcd://myetc.company.com:2379 \
    -s /dev/xvdb -s /dev/xvdc

Command-line arguments

General options
-c <id>                   [REQUIRED] Specifies the cluster ID that this PX instance is to join
-k <kvdb://host:port>     [REQUIRED] Points to your key value database, such as an etcd cluster or a consul cluster
-b                        Use in-built kvdb. Provide the kvdb endpoints required for bootstrap with -k option.
-s <device path>          [REQUIRED unless -a/-A are used] Specify storage devices that PX should use for storing the data
-T <type>                 Specify backend storage type (<type> is dmthin, btrfs, mdraid or lvm)
-cache <device path>      Specify storage devices that PX should use for caching
-dedicated_cache          Constrain cache drive assignment from given -cache drives only
-j <device path>          Specify a journal device for PX, or "auto" (recommended) 
-metadata <device path>   Specify storage device that PX should use for storing the system meta data
-e key=value              Specify extra environment variables
-v <dir:dir[:shared,ro]>  Specify extra mounts
-d <ethX>                 Specify the data network interface
-m <ethX>                 Specify the management network interface
-z                        Instructs PX to run in zero storage mode
-f                        Instructs PX to use an unmounted drive even if it has a filesystem on it
-a                        Instructs PX to use any available, unused and unmounted drives
-A                        Instructs PX to use any available, unused and unmounted drives or partitions
-x <swarm|kubernetes>     Specify scheduler type (if PX running in scheduler environment)
-r <startport>            Start of the portrange Portworx will use for communication (dfl: 9001)
  • additional PX-OCI -specific options:
-oci <dir>                Specify OCI directory (dfl: /opt/pwx/oci)
-sysd <file>              Specify SystemD service file (dfl: /etc/systemd/system/portworx.service)
KVDB options
-userpwd <user:passwd>    Username and password for ETCD authentication
-ca <file>                Specify location of CA file for ETCD authentication
-cert <file>              Specify location of certificate for ETCD authentication
-key <file>               Specify location of certificate key for ETCD authentication
-acltoken <token>         Specify ACL token for Consul authentication
# internal-kvdb-options:
-kvdb_cluster_size <#>    Size of the internal kvdb cluster (dfl: 3)
-kvdb_recovery            Starts the nodes in kvdb recovery mode
Secrets options
-secret_type <type>        Specify the secrets type for cloudsnap and encryption features (<type> is aws-kms, dcos, docker, ibm-kp, k7s, kvdb, vault, gcloud-kms or azure-kv)
-cluster_secret_key <key>  Specify cluster-wide secret for AWS KMS or Vault and volume encryption
PX-API options
# px-api-ssl-options:
-apirootca <file>             Specify self-signed root CA certificate file
-apicert <file>               Specify node certificate file
-apikey <file>                Specify node certificate key file
-apidisclientauth             Disable api client authentication
# px-authentication-options:
-oidc_issuer   <URL>          Location of OIDC service (e.g. https://accounts.google.com)
-oidc_client_id <id>          Client id provided by the OIDC
-oidc_custom_claim_namespace  OIDC namespace for custom claims
-jwt_issuer <val>             JSON Web Token issuer (e.g. openstorage.io)
-jwt_rsa_pubkey_file <file>   JSON Web Token RSA Public file path
-jwt_ecds_pubkey_file <file>  JSON Web Token ECDS Public file path
-username_claim <claim>       Claim key from the token to use as the unique id of the user (<claim> is sub, email or name; dfl: sub)
Resource control options
--cpus <#.#>                  Specify maximum number of CPUs Portworx can use (e.g. --cpus=1.5)
--cpu-shares <#>              Specify CPU shares (relative weight)
--cpuset-cpus <val>           Specify CPUs in which to allow execution (<val> is range <#-#>, or sequence <#,#>)
--memory <bytes>              Specify maximum ammount of memory Portworx can use
--memory-reservation <bytes>  Specify memory reservation soft limit (must be smaller than '--memory')
--memory-swap <bytes>         Specify maximum ammount of RAM+SWAP memory Portworx can use
--memory-swappiness <0-100>   Specify percentage of container's anonymous pages host can swap out
Misc options
-raid <0|10>              Specify which RAID-level should PX use with local storage (dfl: 0)
-cluster_domain <name>    Cluster Domain Name for this cluster
NOTE: The -raid <0|10> option is different than the volume replication factor. For example, px-nodes using -raid 10 and hosting volumes with replication factor 3 will keep 6 copies of the data.

Environment variables
PX_HTTP_PROXY          If running behind an HTTP proxy, set the PX_HTTP_PROXY variables to your HTTP proxy.
PX_HTTPS_PROXY         If running behind an HTTPS proxy, set the PX_HTTPS_PROXY variables to your HTTPS proxy.
PX_ENABLE_CACHE_FLUSH  To enable cache flush daemon, set PX_ENABLE_CACHE_FLUSH=true.
Setting environment variables can be done using the -e option

Below is an example install command with extra “PX_ENABLE_CACHE_FLUSH” environment variable:

sudo /opt/pwx/bin/px-runc install -e PX_ENABLE_CACHE_FLUSH=yes \
    -c MY_CLUSTER_ID -k etcd://myetc.company.com:2379 -s /dev/xvdb

Examples

Installing Portworx using etcd:

px-runc install -k etcd://my.company.com:2379 -c MY_CLUSTER_ID -s /dev/sdc -s /dev/sdb2 {{ include.sched-flags }}
px-runc install -k etcd://70.0.1.65:2379 -c MY_CLUSTER_ID -s /dev/sdc -d enp0s8 -m enp0s8 {{ include.sched-flags }}

Installing Portworx using Consul:

px-runc install -k consul://my.company.com:8500 -c MY_CLUSTER_ID -s /dev/sdc -s /dev/sdb2 {{ include.sched-flags }}
px-runc install -k consul://70.0.2.65:8500 -c MY_CLUSTER_ID -s /dev/sdc -d enp0s8 -m enp0s8 {{ include.sched-flags }}

Modifying the Portworx configuration

After the initial installation, you can modify the Portworx configuration file at /etc/pwx/config.json (see details) and restart Portworx using systemctl restart portworx.

Step 3: Starting Portworx runC

Once you install the Portworx OCI bundle and systemd configuration from the steps above, you can control Portworx directly via systemd.

Below commands reload systemd configurations, enable and start the Portworx service.

sudo systemctl daemon-reload
sudo systemctl enable portworx
sudo systemctl start portworx

Upgrading the Portworx OCI bundle

To upgrade the OCI bundle, simply re-run the installation Step 1 with the --upgrade option. After the upgrade, you will need to restart the Portworx service.

Below command upgrades your installation to the latest stable Portworx version:

latest_stable=$(curl -fsSL 'https://install.portworx.com?type=dock&stork=false' | awk '/image: / {print $2}')
sudo docker run --entrypoint /runc-entry-point.sh \
    --rm -i --privileged=true \
    -v /opt/pwx:/opt/pwx -v /etc/pwx:/etc/pwx \
    $latest_stable --upgrade
sudo systemctl restart portworx

Uninstalling the Portworx OCI bundle

To uninstall the Portworx OCI bundle, please run the following:

# 1: Remove systemd service (if any)
sudo systemctl stop portworx
sudo systemctl disable portworx
sudo rm -f /etc/systemd/system/portworx*

# NOTE: if the steps below fail, please reboot the node, and repeat the steps 2..5

# 2: Unmount oci (if required)
grep -q '/opt/pwx/oci /opt/pwx/oci' /proc/self/mountinfo && sudo umount /opt/pwx/oci

# 3: Remove binary files
sudo rm -fr /opt/pwx

# 4: [OPTIONAL] Remove configuration files. Doing this means UNRECOVERABLE DATA LOSS.
sudo chattr -ie /etc/pwx/.private.json
sudo rm -fr /etc/pwx

Logging and Log files

The systemd(1) uses a very flexible logging mechanism, where logs can be viewed using the journalctl command.

For example:

# Monitor the Portworx logs
sudo journalctl -f -u portworx

# Get a slice of Portworx logs
sudo journalctl -u portworx --since 09:00 --until "1 hour ago"

However, if you prefer to capture Portworx service logs in a separate log file, you will need to modify your host system as follows:

# Create a rsyslogd(8) rule to separate out the PX logs:
sudo cat > /etc/rsyslog.d/23-px-runc.conf << _EOF
:programname, isequal, "px-runc" /var/log/portworx.log
& stop
_EOF

# Create logrotate(8) configuration to periodically rotate the logs:
sudo cat > /etc/logrotate.d/portworx << _EOF
/var/log/portworx.log {
    daily
    rotate 7
    compress
    notifempty
    missingok
    postrotate
        /usr/bin/pkill -HUP syslogd 2> /dev/null || true
    endscript
}
_EOF

# Signal syslogd to reload the configurations:
sudo pkill -HUP syslogd

Advanced usage: Interactive/Foreground mode

Alternatively, one might prefer to first start the Portworx interactively (for example, to verify the configuration parameters were OK and the startup was successful), and then install it as a service:

# Invoke PX interactively, abort with CTRL-C when confirmed it's running:
sudo /opt/pwx/bin/px-runc run -c MY_CLUSTER_ID \
    -k etcd://myetc.company.com:2379 \
    -s /dev/xvdb
[...]
> time="2017-08-18T20:34:23Z" level=info msg="Cloud backup schedules setup done"
> time="2017-08-18T20:34:23Z" level=info msg="Starting REST service on socket : /run/docker/plugins/pxd.sock"
> time="2017-08-18T20:34:23Z" level=info msg="Starting REST service on socket : /var/lib/osd/driver/pxd.sock"
> time="2017-08-18T20:34:23Z" level=info msg="PX is ready on Node: 53f5e87b... CLI accessible at /opt/pwx/bin/pxctl."
[ hit Ctrl-C ]
Migrating from Portworx containers to Portworx OCI: If you already had Portworx running as a Docker container (Portworx 1.2.10 and lower) and now want to upgrade to runC, follow the instructions at Migrate Portworx installed using Docker to OCI/runc.


Last edited: Thursday, Dec 12, 2019