In the last months my company started to move all our VMs from Ubuntu-based templates to CentOS ones.
Regardless of the reason of this choice (I do not want to start a Ubuntu vs CentOS flame since I’m a fan of both distros) one of the biggest problem was to deal with the Apache Httpd packages versions to keep both systems aligned in the middle of the migration.

Finding official up-to-date RPM packages of the latest HTTPS 2.4 branch is surprising challenging unless you are willing to trust unofficial package in your production environment. A first workaround was a source installation but it turned out being a quite fragile solution, ready to be overwrittend by the first “yum update” un on the machine for patching.

So we finally decide to walk the long path and forge our internal httpd 2.4 RPMs by ourself and then re-distribute them across all servers.
BUT this means having an rpmbuild environment for every different version of Centos installed into our datacenter, probably with conflicting dependencies. Moreover such environments would be used just once upon every HTTPS security release and then long forgotten (and leaking resources) until a new one become available.
Moreover the same procedure could change slightly at each release, causing maintenance and regression problems to the owner of these environments.

So our plan was (and the content of this guide will be):

• create a fully automated script to build Apache 2.4.x from sources tarballs
• pack all generated RPM into a portable yum repository for quick redistribution.
• Manage all the process within a docker container to manage different centos version.

Using Docker allow us to re-use the same script across different CentOS version and to get rid of the environment allocation and maintenance problem. At the end of the process the docker representing the build environment can be totally trashed, leaving behind just the desired Yum repository archive with our yearned RPMs.

Simply storing (and tagging!) our docker build and rpmbuild script on a git repository will grant us total environment and process reproducibility in few minutes with no need to waste any virtualization resource on turned off VMs.

Ok let’s start from the boring part: first of all in order to properly build apache from source… you need also to build apr and apr-utils packages which are as hard to find in their proper versions as the main packages)

Also take in account that not all available source tarball are fully working with rpmbuild (yes, bugs happens also for apache dev team 😉 ) so at the time of wiring I got v.2.4.25 available but got to get back to 2.4.20 to be able to pack RPMs successfully.

1. Install EPEL repository and all prerequisites

   yum install -y wget
   
   wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
   rpm -ivh epel-release-6-8.noarch.rpm
   
   yum install -y rpm-build gcc make libtool doxygen distcache distcache-devel autoconf zlib-devel libselinux-devel libuuid-devel apr-devel apr-util-devel pcre-devel openldap-devel lua-devel libxml2-devel distcache-devel openssl-devel postgresql-devel mysql-devel sqlite-devel freetds-devel unixODBC-devel nss-devel ccache

2. Install distcache (I know this is a third-party RPM installed manually, but at the time of writing I still have not found an official Yum Repo for this, I you find one or want to integrate with an rpmbuild let me know. However this is a build dependency and not an install dependency so the package won’t affect your prod environment)

   wget http://ftp5.gwdg.de/pub/opensuse/repositories/home:/zantekk:/Server-HTTPD24/RedHat_RHEL-6/x86_64/distcache-1.4.5-8.5.x86_64.rpm
   wget http://ftp5.gwdg.de/pub/opensuse/repositories/home:/zantekk:/Server-HTTPD24/RedHat_RHEL-6/x86_64/distcache-devel-1.4.5-8.5.x86_64.rpm
   
   rpm -i distcache-1.4.5-8.5.x86_64.rpm distcache-devel-1.4.5-8.5.x86_64.rpm

3. Download all source tarballs

   wget http://archive.apache.org/dist/httpd/httpd-2.4.20.tar.bz2
   wget http://it.apache.contactlab.it//apr/apr-1.5.2.tar.bz2
   wget http://it.apache.contactlab.it//apr/apr-util-1.5.4.tar.bz2

    

4. Start all rpmbuilds, installing output intermediate RPMs

   rpmbuild -tb apr-1.5.2.tar.bz2
   rpm -U ~/rpmbuild/RPMS/x86_64/apr-1.5.2-1.x86_64.rpm ~/rpmbuild/RPMS/x86_64/apr-devel-1.5.2-1.x86_64.rpm
   
   rpmbuild -tb apr-util-1.5.4.tar.bz2
   rpm -U ~/rpmbuild/RPMS/x86_64/apr-util-1.5.4-1.x86_64.rpm ~/rpmbuild/RPMS/x86_64/apr-util-devel-1.5.4-1.x86_64.rpm
   
   rpmbuild -tb httpd-2.4.20.tar.bz2

   A little addition I had to perform for the apr build on a vagrant VM (not on docker) was to enable IPV6 to have all tests pass. So If APR test are failing on socker simply add:

   sysctl net.ipv6.conf.all.disable_ipv6=0

    

If everything went fine you should now have all apache and apr RPMs available in the RPMs folder, ready to be installed.
Bud as anticipated we want also to distribute such packages as a Yum repository, so let’s add also a couple of RPM needed for the install and to generate the repository:

yum install -y mailcap createrepo gzip

and then then run them

mkdir -p ~/repo
mv ~/rpmbuild/RPMS/x86_64/*.rpm ~/repo
cd ~/repo
createrepo .

If you want also to provide a small repository index with packages name and description you can extract them from repo data into a more handy HTML page:

echo "<html><body><table>" > ~/repo/index.html
zcat repodata/*primary.xml.gz | grep -e "<name" -e "<summary" | sed -e "s/<name>/<tr><td>/g" -e "s/<\/name>/<\/td>/g" -e "s/<summary>/<td>/g" -e "s/<\/summary>/<\/td><\/tr>/g" >> ~/repo/index.html
echo "</table></body></html>" >> ~/repo/index.html

Now you just have to package all the folder into your portable yum repo:

cd ~
tar -zcvf FLEP-httpd24-repo.tar.gz repo

To install it on a target server you just have to unpack the archive into a temportay folder (e.g. /tmp/repo ) and create a new yum configuration into /etc/yum.repo.d/httpd24.repo

[HTTPD24_Repo]
name=Apache 2.4.x prerequisites repository
baseurl=file:///tmp/repo/
enabled=1

and install them via yum:

yum install httpd

OK, useful but boring up to now… let’s start the more interesting part, how to move all of this inside docker?
As you may notice from my other posts, I’m an Ansible fan, so we will not deal with Dockerfiles, but we will create and ansible-container project to automate the whole process.

1. the first step is obviously install  docker and ansible-container, sinche this can change depending on your system, please check thier installation page
2. Then create a new folder and init it a new ansible-container app

   ansible-container init

   it will create all the needed files to start.

3. Create a new service into container.yml

   version: "2"
   services:
   
   apache24_rpmbuilder:
    user: "{{dst_user}}"
    image: "centos:{{centosver}}"
    volumes:
    - "/tmp/apache24_rpmbuilder:{{output_path}}"
    entrypoint: ["{{vol_path}}/entrypoint.sh"]

   this will become our docker image, based on a parametrized CentOS version.
   Notice that we also add an attached volume, which will be used to persist our final repository tarball, and an entrypoint.sh script which will contain all the rpmbuild steps described above.

4. Now let’s attach an ansible role named apache24_rpmbuilder for image customization in main.yml:

   ---
   
   - hosts: apache24_rpmbuilder
    gather_facts: false
    roles:
    - apache24_rpmbuilder

   This ansible role will be responsible to customize the base docker image declared into the container.yml file, basically by uploading the entrypoint script and the yum repo config file.
   The main task will be contained in a role path: apache24_rpmbuilder/tasks/main.yml

   ---
   
   - name: "create folder {{vol_path}}"
    file: path="{{item}}" state="directory" owner="{{dst_user}}" 
    with_items:
    - "{{vol_path}}"
    - "{{ output_path }}/repo"
   
   - name: Synch script files
    template: src="{{item.src}}" dest="{{item.dst}}" mode="u=rxw,g=r,o=r"
    with_items:
    - { src: "{{roles_path}}/apache24_rpmbuilder/templates/entrypoint.sh.tpl", dst: "{{vol_path}}/entrypoint.sh" }
    - { src: "{{roles_path}}/apache24_rpmbuilder/templates/httpd24.repo.tpl", dst: "{{ output_path }}/httpd24.repo" }

   Both of them are a template version of the script containing all the rpm-build steps.

   All standard variables can be added into the /apache24_rpmbuilder/vars/main.yml 

   ---
   # Centos image tag to be used for docker
   #centosver: "6.6"
   centosver: "6.8"
   
   # Main path for the ansible-container
   base_path: "/ansible-container"
   
   # Main path for the ansible-container roles to be used
   roles_path: "{{base_path}}/ansible/roles"
   
   # Main path for the installation in the final docker image
   vol_path: "/tmp"
   
   # Path for the generated files into the final docker image
   output_path: "{{ vol_path }}/RPMS"
   
   # User running final image
   dst_user: "root"

   While all build-related vaiables will be in /apache24_rpmbuilder/defaults/main.yml

   ---
   
   flep_prereq_rpm:
    rpm-build 
    gcc 
    make 
    libtool 
    doxygen 
    distcache 
    distcache-devel 
    autoconf 
    zlib-devel 
    libselinux-devel 
    libuuid-devel 
    apr-devel 
    apr-util-devel 
    pcre-devel 
    openldap-devel 
    lua-devel 
    libxml2-devel 
    distcache-devel 
    openssl-devel 
    postgresql-devel 
    mysql-devel 
    sqlite-devel 
    freetds-devel 
    unixODBC-devel 
    nss-devel 
    ccache
   
   flep_prereq_httpd:
    mailcap
   
   src_apache_ver: "2.4.20"
   src_apache_file: "httpd-{{src_apache_ver}}.tar.bz2"
   src_apache_url: "http://archive.apache.org/dist/httpd/{{src_apache_file}}"
   
   src_apr_ver: "1.5.2"
   src_apr_file: "apr-{{src_apr_ver}}.tar.bz2"
   src_apr_url: "http://it.apache.contactlab.it/apr/{{src_apr_file}}"
   
   src_apr_util_ver: "1.5.4"
   src_apr_util_file: "apr-util-{{src_apr_util_ver}}.tar.bz2"
   src_apr_util_url: "http://it.apache.contactlab.it/apr/apr-util-1.5.4.tar.bz2"
   
   distcache_file: "distcache-1.4.5-8.5.x86_64.rpm"
   distcache_url: "http://ftp5.gwdg.de/pub/opensuse/repositories/home:/zantekk:/Server-HTTPD24/RedHat_RHEL-6/x86_64/{{distcache_file}}"
   
   distcache_devel_file: "distcache-devel-1.4.5-8.5.x86_64.rpm"
   distcache_devel_url: "http://ftp5.gwdg.de/pub/opensuse/repositories/home:/zantekk:/Server-HTTPD24/RedHat_RHEL-6/x86_64/{{distcache_devel_file}}"
   
   downloader: "yum install -y --downloadonly --downloaddir="
   downloader66: "yumdownloader --destdir="

   As you can notice the latter variables files contains all relevant properties to be maintained to perform our build as the prerequisite packages lists and the source versions and download url to be used to retrieve them.
   CentOS version for Docker base image is declared in vars file instead of defaults one since such file will be passed also to ansible-container build process in order to process the container.yml file as well

   Finally here comes the jinja2-template version of our build script/apache24_rpmbuilder/templates/entrypoint.sh.tpl

   #!/bin/bash
   
   DWN_BASE_PATH={{ output_path }}
   SRC_BASE_PATH={{ vol_path }}
   
   DWN_VERSION=$(cat /etc/redhat-release | grep -o -P "([0-9]\.?)*");
   MJR_VERSION=$(echo ${DWN_VERSION} | cut -f1 -d"." )
   
   DWN_PATH=${DWN_BASE_PATH}/repo/Httpd24/${DWN_VERSION}/{{src_apache_ver}}
   
   BUILD_PREREQUISITES="{{prereq_rpm}}"
   HTTPD_PREREQUISITES="{{prereq_httpd}}"
   
   echo "removing ${DWN_BASE_PATH}/repo/Centos/${DWN_VERSION}/latest"
   rm -rf ${DWN_BASE_PATH}/repo/Centos/${DWN_VERSION}/latest ${DWN_BASE_PATH}/httpd24-repo.*
   
   echo "creating ${DWN_PATH}"
   mkdir -p ${DWN_PATH}
   
   echo "Setting up EPEL ${MJR_VERSION}"
   command rpm -Uvh --replacepkgs https://dl.fedoraproject.org/pub/epel/epel-release-latest-${MJR_VERSION}.noarch.rpm
   
   #Install wget first for fail-fast if url not available
   yum install -y wget
   
   wget {{ src_apache_url }}
   wget {{ src_apr_url }}
   wget {{ src_apr_util_url }}
   wget {{ distcache_url }}
   wget {{ distcache_devel_url }}
   
   yum install -y {{distcache_file}} {{distcache_devel_file}}
   
   yum install -y ${BUILD_PREREQUISITES}
   
   #needed by tests
   #sysctl net.ipv6.conf.all.disable_ipv6=0
   
   rpmbuild -tb {{src_apr_file}}
   rpm -U ~/rpmbuild/RPMS/x86_64/apr-{{src_apr_ver}}-1.x86_64.rpm ~/rpmbuild/RPMS/x86_64/apr-devel-{{src_apr_ver}}-1.x86_64.rpm
   
   rpmbuild -tb {{src_apr_util_file}}
   
   rpm -U ~/rpmbuild/RPMS/x86_64/apr-util-{{src_apr_util_ver}}-1.x86_64.rpm ~/rpmbuild/RPMS/x86_64/apr-util-devel-{{src_apr_util_ver}}-1.x86_64.rpm
   
   rpmbuild -tb {{src_apache_file}}
   
   mv ~/rpmbuild/RPMS/x86_64/*.rpm ${DWN_PATH}/
   
   {% if centosver == "6.6" %}
   yum install -y yum-utils
   echo "Invoking Yum-Downloader to retrieve httpd install dependencies [${HTTPD_PREREQUISITES}]"
   {{ downloader66 }}${DWN_PATH} ${HTTPD_PREREQUISITES} > ${DWN_BASE_PATH}/yum_downloadonly.log 2>&1
   
   {% else %}
   echo "Invoking Yum Download-Only to retrieve httpd install dependencies [${HTTPD_PREREQUISITES}]"
   {{ downloader }}${DWN_PATH} ${HTTPD_PREREQUISITES} > ${DWN_BASE_PATH}/yum_downloadonly.log 2>&1
   {% endif %}
   
   echo "installing createrepo package"
   yum install -y createrepo gzip
   
   echo "indexing ${DWN_PATH}"
   cd ${DWN_PATH}
   createrepo .
   
   echo "<html><body><table>" > ${DWN_BASE_PATH}/repo/index.html
   zcat repodata/*primary.xml.gz | grep -e "<name" -e "<summary" | sed -e "s/<name>/<tr><td>/g" -e "s/<\/name>/<\/td>/g" -e "s/<summary>/<td>/g" -e "s/<\/summary>/<\/td><\/tr>/g" >>${DWN_BASE_PATH}/repo/index.html
   echo "</table></body></html>" >> ${DWN_BASE_PATH}/repo/index.html
   
   cd ${DWN_BASE_PATH}
   tar -zcvf httpd24-repo${DWN_VERSION}.tar.gz repo

   Probably less readable than the simple list of bash command but easier to maintain in an ansible-role perspective.
   Notice also that the script contains a small if-then-else clause which allow to manage a difference in yum offline download from Centos 6.6 to 6.8.

5. Now that our role is in place we just have to start ansible-container to create our base image:

   ansible-container --var-file ansible/roles/apache24_rpmbuilder/vars/main.yml  build --from-scratch

   Build will last for few minutes ( a bit more if you have to download both CentOs and Ansible-container image) and will produce our apache24_rpmbuilder image.
   Let’s check with a docker image listing:

   $ docker images
   
   REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
   apache24_rpmbuilder-apache24_rpmbuilder                           20170122105436      9f009ca5a36d        1 minute ago        203 MB
   apache24_rpmbuilder-apache24_rpmbuilder                           latest              9f009ca5a36d       1 minute ago        203 MB

    

6. Everything is in place, let’s start the real rpmbuild process:

   docker run -v /tmp/apache24_rpmbuilder:/tmp/RPMS -t -i apache24_rpmbuilder-apache24_rpmbuilder:latest

   Remember to map the output folder to your desired local folder.
   The build will last some minutes, depending on your download time for sources and dependencies plus the real rpm-build time, but in the and in your folder you will find all the RPMs both in a handful tar.gz and a unpacked format.
   As a plus a bunch of log for each rpmbuild process (just in case of trouble) and a human-readable html with all packages name and their description.

Remeber that such Httpd RPMs are slightly different from the official CentOS one, so you will still have to work a bit on their configuration after their installation, since all module will be disabled and not all defaults value could be the one you will be expecting.
I alos wrote a classic ansible role for quick install and setup sarting from the rseulting tar.gz, but this will be the topic for another post.

In the meanwhile if you are willing to contribute with testing or improvements you can find the whole source code of this post on aroundthecode github and (if you are really lazy!) the ready-to-run docker image on aroundthecode docker hub.
Such code has been tested with both CentOs 6.6 and 6.8, but I’d like to extend it also to more recent releases as soon as possible.
In such repository you will find a little bonus which is also the capability to compile mod_security RPM as well. This will be probably be another post topic since at the time of writing the source site seems to be temporary down so the process is stopping just after the httpd RPMs.