Kind of a Gluster ... Installation woes on 3.11



Update (and TL;DR:) - make sure you are using the right version of Ansible.  For whatever reason I had believed that I should be at 2.6.  Once I updated to 2.8 everything ran correctly.



This is probably one of those times when my persistence coupled with OCD behavior was NOT on my side...
I have spent literally weeks (elapsed) and probably days (actual) of time trying to deploy my OCP 3.11 Cluster to use Gluster for the Persistent Storage (apps and infra).  And it has failed.. and failed... and failed miserably.
Started off with one hypervisor - Xeon and 96GB of memory, 2 x TB of HDD.
VMs would (eventually) lock up and require hard restarts.  Hmm.. memory?

So, I went and fired up my 3 x HP ML30s (each with 64GB and 1 TB of HDD)

Still no love.
I'm still fairly novice using Ansible and the scripts/role/playbooks/whatever they're called are actually pretty complex, not idempotent, have to be run in a specific order, etc... Huh?


So - there will be a bunch of references at the bottom which hopefully are useful.  There is no "fix" at this time, but seemingly are work-arounds.

Starting with my original Inventory which includes my Gluster nodes...
https://github.com/cloudxabide/matrix.lab/blob/master/Files/ocp-3.11-multiple_master_native_ha-2xOCS.yml

I have opted to create a "pre" Inventory file with the Gluster components commented out.
Here is the sdiff
https://github.com/cloudxabide/matrix.lab/blob/master/Files/sdiff-2xOCS-and-2xOCS-pre.txt

I have executed this successfully once now, but the procedure is:

# 

cd /usr/share/ansible/openshift-ansible
ansible all --list-hosts -i ~/ocp-3.11-multiple_master_native_ha-2xOCS-pre.yml
# The prereqs can be executed with the Gluster components present in the Inventory
ansible-playbook -i ~/ocp-3.11-multiple_master_native_ha-2xOCS.yml playbooks/prerequisites.yml -vvv | tee ocp_prerequisites-`date +%F`.logs
ansible-playbook -i ~/ocp-3.11-multiple_master_native_ha-2xOCS-pre.yml playbooks/deploy_cluster.yml -vvv | tee ocp_deploy_cluster-pre-`date +%F`.logs
ansible-playbook -i ~/ocp-3.11-multiple_master_native_ha-2xOCS.yml playbooks/openshift-glusterfs/config.yml  -vvv | tee ocp_glusterfs-config`date +%F`.logs


run prereqs with a "complete" OCS enabled inventory
run deploy_cluster with the GlusterFS components absent
run playbooks/openshift-glusterfs/config.yml referencing the complete inventory


References

Installing OpenShift Container Platform | Retrying the installation

https://docs.openshift.com/container-platform/3.11/install/running_install.html#advanced-retrying-installation

release-3.11: Running deploy_cluster.yml expects GlusterFS to be available before it is deployed #12002

https://github.com/openshift/openshift-ansible/issues/12002


Bug 1777499 - Installation is failing at TASK [openshift_storage_glusterfs : Check for GlusterFS cluster health] [NEEDINFO]
https://bugzilla.redhat.com/show_bug.cgi?id=1777499&GoAheadAndLogIn=1

Comments

Post a Comment

Popular posts from this blog

Apple OSX Catalina and VMware vCenter Server Appliance.. ugh

OSX Catalina came with some rather annoying challenges: dropped support for 32-bit apps.  I don't play Steam games that much on my Mac now that I am stuck at home all the time.  So, not a big deal. implemented a bunch of "gatekeeper" security measures.  Now, typically I would say:  if you need to circumvent security ("Disable SELinux" - for example), you're probably doing it wrong.  But, the case of all the hand-holdy, lockdown stuff, it's pretty annoying.  Several things you cannot simply "accept the risk" and move on, you get constant reminders.  And it is interesting that whenever Microsoft does this sort of thing, it's widely discussed they are vilified.  Somehow, when Apple does this.. meh, just the cost of doing business. The challenges, themselves, are not the point of this post.  The point is you need to do some stuff if you expect to use Mac OSX Catalina to deploy VMware vSphere 6.7 U3 - vCenter Server Appliance. Download the ISO (...
Read more

HomeLab - Sophos XG SSL VPN and Comcast

Ugh... Comcast.  This sucked particularly much as this was my first attempt to configure a VPN back to my HomeLab and therefore was not sure what options I should have *known* would work.  Anywho... Sophos SSL VPN uses 8443 by default.  I spent a few hours trying to get through the config on the Sophos and my Linux client.  Nothing was working. For giggles I changed the Sophos to use 443, and then updated my client on my Linux workstation.  And suddenly it all works.  Fowk.  Tried 8443 again.  No love. I don't want to utilize port 443 as I intend to actually host some web apps for testing.  So, I have finalized on an "alternate" port which had to be configured on both the Sophos and the Linux client. I'l be doing a better write-up about this later.  I still need to figure out how to configure all this via the Gnome Network-Manager GUI.
Read more

Software RAID via kickstart (not cloud, sorry)

A bunch of the following specifics are out of scope, but I am including them anyhow # System bootloader configuration bootloader --location=mbr --boot-drive=sda --append=" crashkernel=auto hpsa.hpsa_allow_any=1 hpsa.hpsa_simple_mode=1 console=tty0 console=ttyS0,9600n8" # Partition clearing information #ignoredisk --only-use=sda zerombr clearpart --all --initlabel --drives=sda,sdb,sdc # Disk partitioning information part /boot/efi --fstype="efi" --fsoptions="umask=0077,shortname=winnt" --ondisk=sda --size=256 part /boot --fstype="xfs" --ondisk=sda --size=1024 part pv.01 --fstype="lvmpv" --ondisk=sda --size=51200 --grow volgroup vg_rhel7 --pesize=4096 pv.01 logvol /              --fstype="xfs"  --size=10240 --name=root     --vgname=vg_rhel7 logvol /home          --fstype="xfs"  --size=1024  --name=home     --vgname=vg_rhel7 logvol /tmp           --fstyp...
Read more