Provisioning

Provision VMs with vagrant & terraform

Pigsty runs on nodes, which are Bare Metals or Virtual Machines. You can prepare them manually, or using terraform & vagrant for provisioning.


Sandbox

Pigsty has a sandbox, which is a 4-node deployment with fixed IP addresses and other identifiers. Check demo.yml for details.

The sandbox consists of 4 nodes with fixed IP addresses: 10.10.10.10, 10.10.10.11, 10.10.10.12, 10.10.10.13.

and a 3-instance PostgreSQL HA cluster: pg-test

There’s a primary singleton PostgreSQL cluster: pg-meta on the meta node, which can be used alone if you don’t care about PostgreSQL high availability.

  • meta 10.10.10.10 pg-meta pg-meta-1

There are 3 additional nodes in the sandbox, with a 3-instance PostgreSQL HA cluster pg-test.

  • node-1 10.10.10.11 pg-test.pg-test-1
  • node-2 10.10.10.12 pg-test.pg-test-2
  • node-3 10.10.10.13 pg-test.pg-test-3

Two optional L2 VIP are bind on primary instances of pg-meta and pg-test:

  • 10.10.10.2 pg-meta
  • 10.10.10.2 pg-test

There’s also a 1-instance etcd cluster, and 1-instance minio cluster on the meta node, too.

pigsty-sandbox

You can run sandbox on local VMs or cloud VMs. Pigsty offers a local sandbox based on Vagrant (pulling up local VMs using Virtualbox), and a cloud sandbox based on Terraform (creating VMs using the cloud vendor API).

  • Local sandbox can be run on your Mac/PC for free. Your Mac/PC should have at least 4C/8G to run the full 4-node sandbox.

  • Cloud sandbox can be easily created and shared. You will have to create a cloud account for that. VMs are created on-demand and can be destroyed with one command, which is also very cheap for a quick glance.


Vagrant

Vagrant can create local VMs according to specs in a declarative way. Check Vagrant Templates Intro for details

Installation

Make sure Vagrant and Virtualbox are installed and available on your OS.

If you are using macOS, You can use homebrew to install both of them with one command (reboot required).

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
brew install vagrant virtualbox ansible

Configuration

vagarnt/Vagranfile is a ruby script file describing VM nodes. Here are some default specs of Pigsty.

Templates Shortcut Spec Comment
meta.rb v1 4C8G x 1 Single Meta Node
full.rb v4 2C4G + 1C2G x 3 Full 4 Nodes Sandbox Demo
el7.rb v7 2C4G + 1C2G x 3 EL7 3-node Testing Env
el8.rb v8 2C4G + 1C2G x 3 EL8 3-node Testing Env
el9.rb v9 2C4G + 1C2G x 3 EL9 3-node Testing Env
build.rb vb 2C4G x 3 3-Node EL7,8,9 Building Environment
citus.rb vc 2C4G + 1C2G x 4 5-Node Citus/Etcd Testing Env
minio.rb vm 2C4G x 3 + Disk 3-Node MinIO/etcd Testing Env
prod.rb vp 45 nodes Prod simulation with 45 Nodes

Each spec file contains a Specs variable describe VM nodes. For example, the full.rb contains the 4-node sandbox specs.

Specs = [
  {"name" => "meta",   "ip" => "10.10.10.10", "cpu" => "2",  "mem" => "4096", "image" => "generic/rocky9" },
  {"name" => "node-1", "ip" => "10.10.10.11", "cpu" => "1",  "mem" => "2048", "image" => "generic/rocky9" },
  {"name" => "node-2", "ip" => "10.10.10.12", "cpu" => "1",  "mem" => "2048", "image" => "generic/rocky9" },
  {"name" => "node-3", "ip" => "10.10.10.13", "cpu" => "1",  "mem" => "2048", "image" => "generic/rocky9" },
]

You can switch specs with the switch script, it will render the final Vagrantfile according to the spec.

cd ~/pigsty
vagrant/switch <spec>

vagrant/switch meta     # singleton meta        | alias:  `make v1`
vagrant/switch full     # 4-node sandbox        | alias:  `make v4`
vagrant/switch el7      # 3-node el7 test       | alias:  `make v7`
vagrant/switch el8      # 3-node el8 test       | alias:  `make v8`
vagrant/switch el9      # 3-node el9 test       | alias:  `make v9`
vagrant/switch prod     # prod simulation       | alias:  `make vp`
vagrant/switch build    # building environment  | alias:  `make vd`
vagrant/switch minio    # 3-node minio env
vagrant/switch citus    # 5-node citus env

Management

After describing the VM nodes with specs and generate the vagrant/Vagrantfile. you can create the VMs with vagrant up command.

Pigsty templates will use your ~/.ssh/id_rsa[.pub] as the default ssh key for vagrant provisioning.

Make sure you have a valid ssh key pair before you start, you can generate one by: ssh-keygen -t rsa -b 2048

There are some makefile shortcuts that wrap the vagrant commands, you can use them to manage the VMs.

make         # = make start
make new     # destroy existing vm and create new ones
make ssh     # write VM ssh config to ~/.ssh/     (required)
make dns     # write VM DNS records to /etc/hosts (optional)
make start   # launch VMs and write ssh config    (up + ssh) 
make up      # launch VMs with vagrant up
make halt    # shutdown VMs (down,dw)
make clean   # destroy VMs (clean/del/destroy)
make status  # show VM status (st)
make pause   # pause VMs (suspend,pause)
make resume  # pause VMs (resume)
make nuke    # destroy all vm & volumes with virsh (if using libvirt) 

Shortcuts

You can create VMs with the following shortcuts:

make meta     # singleton meta
make full     # 4-node sandbox
make el7      # 3-node el7 test
make el8      # 3-node el8 test
make el9      # 3-node el9 test
make prod     # prod simulation
make build    # building environment
make minio    # 3-node minio env
make citus    # 5-node citus env
make meta install  # create and install pigsty on 1-node singleton meta
make full install  # create and install pigsty on 4-node sandbox
make prod install  # create and install pigsty on 42-node KVM libvirt environment
...

Terraform

Terraform is an open-source tool to practice ‘Infra as Code’. Describe the cloud resource you want and create them with one command.

Terraform can be easily installed with homebrew, too: brew install terraform. You will have to create a cloud account to obtain AccessKey and AccessSecret credentials to proceed.

The terraform/ dir have two example templates: one for AWS, and one for Aliyun, you can adjust them to fit your need, or modify them if you are using a different cloud vendor.

Take Aliyun as example:

cd terraform                         # goto the terraform dir
cp spec/alicloud.tf terraform.tf     # use aliyun template

You have to perform terraform init before terraform apply:

terraform init      # install terraform provider: aliyun (required only for the first time)
terraform apply     # generate execution plans: create VMs, virtual segments/switches/security groups

After running apply and answering yes to the prompt, Terraform will create the VMs and configure the network for you.

The admin node ip address will be printed out at the end of the execution, you can log in and start pigsty installation


Last modified 2023-08-19: update docs to v2.3.0 (f172b10)