Posts | ..:: BOFH ::..

Multiplatform Testing With Molecule and Travis

Mon, 30 Mar 2020 02:53:32 +0600

Recently I had been trying to convert all my Bourne shell scripts into possible Ansible Galaxy Roles so that I can use it every now and then in my playbooks. Normally my regular job requires concentration on FreeBSD based systems only but one of my new gigs actually has a multi platform scenario running CentOS, Ubuntu, Debian, FreeBSD, Arch Linux and what not. So thought about rewriting the shell scripts into reusable multi platform Ansible Roles.

The first place I thought about heading into was Ansible Galaxy which is a platform for sharing Ansible Roles with others. Ths site is nice and clean; has a good option for searching including name, tags and platform. For a network infrastructure the first crucial part is time synching. So I started searching for ntp which supports FreeBSD. There are lots of roles supporting that search. But some are broken, some do not work on that platform, some requires earlier version of Ansible. So rather than finding the needle in the haystack it was better to write my own version. Writing a dirty oneliner or hack for an internal project is one thing and writing and sharing something on the web which will be reused by others is another thing. It requires standardized code writing and build testing. In DevOPS philosophy what we call Continuous Integration/Continuous Deployment. Whenever we are changing a line of code the code is pushed into a testing system for a successfull build; once the build is successful it is deployed into production.

Testing Ansible Roles on multi platforms is not an easy task. For local projects we can create Docker containers to test specially on Linux systems or we can test with Vagrant using Virtualbox. For normal syntax check we can use ansible-playbook with --syntax-check or ansible-lint. But for real world scenario we need a good Testing Framework. molecule is one good framework for the purpose. It has support for multi cloud platforms to deploy our testing. We can use molecule to test locally or on remote cloud platforms or integrate it with additional CI/CD platforms like Travis or Cirrus by creating a pipline from Github or Gitlab. Both of them have commercial versions but for Open Source applications both of them are free. Ansible Galaxy has support for Travis and Github. So if my role is hosted at Github and I have a proper definition of testing in a file called .travis.yml in my root directory; whenever I have code change pushed into the git repository my Ansible role will be tested and the result will be notified in my Ansible Galaxy project. Now the big problem for me is that Travis do not have support for FreeBSD like Cirrus.

So testing for my roles in a multi platform is facing a Road Block. We have to be a bit innovative. molecule supports Vagrant and Vagrant has support for multiple Providers like Virtualbox, Docker, Vmware, libvirt. For a commandline based interface libvirt looks like a good choice. So we are going to create a Vagrant and libvirt interface within our limited access Travis platform; and push our code into that libvirt environment. Let’s jump into the code and see how we can accomplish this with molecule and Travis.

My .travis.yml looks like this.

---
dist: bionic
language: python
python:
- 3.6
env:
global:
- VAGRANT_VER: 2.2.7
matrix:
- MOLECULE_DISTRO_BOX: generic/ubuntu1604
- MOLECULE_DISTRO_BOX: generic/ubuntu1804
- MOLECULE_DISTRO_BOX: debian/stretch64
- MOLECULE_DISTRO_BOX: debian/buster64
- MOLECULE_DISTRO_BOX: centos/7
- MOLECULE_DISTRO_BOX: centos/8
- MOLECULE_DISTRO_BOX: generic/freebsd11
- MOLECULE_DISTRO_BOX: generic/freebsd12
install:
- sudo add-apt-repository universe && sudo apt update
- wget https://releases.hashicorp.com/vagrant/$VAGRANT_VER/vagrant_"$VAGRANT_VER"_x86_64.deb
- sudo dpkg -i vagrant_"$VAGRANT_VER"_x86_64.deb
- sudo apt install qemu-kvm qemu-utils libvirt-bin libvirt0 libvirt-dev
- vagrant plugin install vagrant-libvirt
- pip install molecule molecule-vagrant docker python-vagrant paramiko
- sudo chmod o+rwx /var/run/libvirt/libvirt-sock
script:
# Check ansible version
- ansible --version
# Check molecule version
- molecule --version
# Basic role syntax check
- ansible-playbook tests/test.yml -i tests/inventory --syntax-check
- molecule test
notifications:
webhooks: https://galaxy.ansible.com/api/v1/notifications/

My corresponding molecule/default/molecule.yml looks like the following:

---
dependency:
name: galaxy
driver:
name: vagrant
provider:
name: libvirt
type: libvirt
driver: kvm
options:
memory: 2048
cpus: 2
platforms:
- name: instance
box: ${MOLECULE_DISTRO_BOX:-""}
interfaces:
- network_name: public_network
dev: "virbr0"
type: "bridge"
mode: "bridge"
provisioner:
name: ansible
log: true
verifier:
name: ansible

In the future I will try to create a CookieCutter template for my roles for easier creation of directory and files layout. Please bear in mind that Travis is supporting the OSS culture with free access to resources and do not abuse this service.

Fresh Boot to My Blog

Thu, 19 Mar 2020 02:57:57 +0600

It has been a while I have stopped blogging. In the early days I used to blog in Blogger. But after the acquisition of Blogger by Google it was actually time for me to shut it down.

Since then I had been saving my documents in a local webserver running MoinMoin (incidentally my nickname is Moin too). However that was alright for a local environment and at that time finding a dedicated or shared hosting running Python WSGI was not easily available. Plus getting a dedicated hosting for hosting my platform wasn’t easy either. There were other options at that time specially Drupal, then came Wordpress, then there was Joomla!. Have gone through all of those but it seemed like all were overly complicated for a simple blogging platform. Asking about a personal blogging site my first preference is easy to write; afterall contents are the main attraction of a blog. So rather than maintaining the site it has to be the contents on which I am spending more of my time. I have worked on large scale deployments of Wordpress where the site was a daily national newspaper site with millions of view every day. Although didn’t have to go through the development side but had to take care of the deployment part.

Unfortunately none of them cattered that need. Drupal is actually considered the mother of all CMS, it’s huge. it’s robust, it’s complicated. Wordpress on the other hand incepted as a blogging platform but completely moved towards a full fledget CMS platform. But in today’s world it is completely undeniable that without the contribution of Wordpress we wouldn’t have got millions of blogs and contents running and the market share is increasing day by day. However due to this popular usage it has also been the victim of prying eyes and there are lots of bugs and exploits too. But most vulnerable seemed like Joomla!; and least vulnerable was Drupal.

However coming back to my requirements I needed something more easier to write with; not concentrating on HTML tags or missing an angle brace but something more easier to write through; where I could concentrate on contents rather than technicals. There seemed couple of options which supported an easier Markup language to concentrate on writing : Wikitext, Markdown and TEX/Latex. Generating HTML from Latex/TEX wasn’t easy at that time. Markdown was a better choice but none of those supported Markdown; which lead me to use MoinMoin which had a Markdown alike Wiki syntax. So my first requirement is markdown.

Then comes processing power. Whenever I am using another scripting language to render the HTML pages it needs to use a CGI process which was pretty slow and have been updated with FastCGI/WSGI/SGI which are much more faster than CGI. Even though what happens is for every request the CGI process parses and prepares the content in HTML format that needs to be rendered to the browser. Even though you have a .php or .pl extension in the browser addressbar the actual rendering engine is plain old HTML. So I was looking for something which actually supported simple HTML without any fuss; because I had no dynamic contents and alll I need to show are some HTML files. But when you go HTML you limit your contents to be static only. What on earth would I do with dynamic contents? I don’t have thousands of users and their dynamic contents which needs to be rendered. Unfortunately there were no CMS that actually supported plain HTML without any Server Side technologies. On top of that all the above mentioned CMS required a Database Engine where my contents are actually saved in SQL format and whenever someone is trying to access a page or a post that content is called by the CGI process and the HTML file is rendered. Lots of system calls, lots of system process, lots of CPU cycle, lots of memory requirements. Overall low performance for the viewers. So to maintain a blog like the ones from above you have to actually maintain security of the scripting language, a database system and the CMS itself. Actually these take places behind the scene for so many users and noone bothers those because a Website control panel like cPanel/DirectAdmin/Webmin manages those automagically behind the scene. Additionally we will have to use or build up customized plugins/thems/modules which are not easy to do unless you have a good grasp on the language. Plus comes the security of those modules. So my next requirement is a STATIC site with flat file support.

If someone has actually plan to build an easy to use blog and which has the possibility of getting BIG I would advice on Ghost.

Coming back to my requirements I started looking for automatically generating my site from couple of Markdown files. Jekyll seemd a very good and interesting project which does the same. Jekyll along with OCTOPRESS seemed like a good combination for me. Unfortunately it seems like OCTOPRESS development has been stalled; last update looks like in 2016. Then about Jekyll is not fit to generate thousands of pages; it’s just not that scalable and robust. On top of that I don’t know but Ruby didn’t look like something of my cup of tea. So if I required to hack something I might get stuck. In the meantime there was another project called HUGO which has been in the market for long. It was written in go so literally you will need one single binary file as most of the other modules and everything is built withing that binary. HUGO utilizes JAMSTACK technology. This application really intrigued me making my blog public again. Specially when I saw the following video.

So HUGO is going to be my next Blogging Platform. If you are keen to try out other JAMSTACK platforms you can head to StaticGen a site maintained by netlify; a Company supporting Jamstack hosting with CDN.

There are so many dead or development stalled projects listed in StaticGen. So please chose wisely.

As a techie from my point of view the deployment and maintenance didn’t look complex to me. I can maintain my contents in flat files and more over I can do Version Control by keeping them in a git repository. From there I can create a Continuous Integration/Continuous Delivery pipeline and deploy my site in seconds to the web server. Now I need a good theme. So I have chosen Academic. It really matches my requirements in point of content management and display. As I am a Conference Hopper and have papers published; it allows me to easily maintain and showcase those. Additionally I am planning to write tutorials and Courses which is supported really well. Keep an eye on this site for my upcoming tutorials on Networking/DevOPS/FreeBSD/System Administration.