Package Apache 2.4.x RPMs via Docker and Ansible-container

In the last months my company started to move all our VMs from Ubuntu-based templates to CentOS ones.
Regardless of the reason of this choice (I do not want to start a Ubuntu vs CentOS flame since I’m a fan of both distros) one of the biggest problem was to deal with the Apache Httpd packages versions to keep both systems aligned in the middle of the migration.

Finding official up-to-date RPM packages of the latest HTTPS 2.4 branch is surprising challenging unless you are willing to trust unofficial package in your production environment. A first workaround was a source installation but it turned out being a quite fragile solution, ready to be overwrittend by the first “yum update” un on the machine for patching.

So we finally decide to walk the long path and forge our internal httpd 2.4 RPMs by ourself and then re-distribute them across all servers.
BUT this means having an rpmbuild environment for every different version of Centos installed into our datacenter, probably with conflicting dependencies. Moreover such environments would be used just once upon every HTTPS security release and then long forgotten (and leaking resources) until a new one become available.
Moreover the same procedure could change slightly at each release, causing maintenance and regression problems to the owner of these environments.

So our plan was (and the content of this guide will be):

  • create a fully automated script to build Apache 2.4.x from sources tarballs
  • pack all generated RPM into a portable yum repository for quick redistribution.
  • Manage all the process within a docker container to manage different centos version.

Using Docker allow us to re-use the same script across different CentOS version and to get rid of the environment allocation and maintenance problem. At the end of the process the docker representing the build environment can be totally trashed, leaving behind just the desired Yum repository archive with our yearned RPMs.

Simply storing (and tagging!) our docker build and rpmbuild script on a git repository will grant us total environment and process reproducibility in few minutes with no need to waste any virtualization resource on turned off VMs.

Ok let’s start from the boring part: first of all in order to properly build apache from source… you need also to build apr and apr-utils packages which are as hard to find in their proper versions as the main packages)

Also take in account that not all available source tarball are fully working with rpmbuild (yes, bugs happens also for apache dev team 😉 ) so at the time of wiring I got v.2.4.25 available but got to get back to 2.4.20 to be able to pack RPMs successfully.

  1. Install EPEL repository and all prerequisites
    yum install -y wget
    rpm -ivh epel-release-6-8.noarch.rpm
    yum install -y rpm-build gcc make libtool doxygen distcache distcache-devel autoconf zlib-devel libselinux-devel libuuid-devel apr-devel apr-util-devel pcre-devel openldap-devel lua-devel libxml2-devel distcache-devel openssl-devel postgresql-devel mysql-devel sqlite-devel freetds-devel unixODBC-devel nss-devel ccache
  2. Install distcache (I know this is a third-party RPM installed manually, but at the time of writing I still have not found an official Yum Repo for this, I you find one or want to integrate with an rpmbuild let me know. However this is a build dependency and not an install dependency so the package won’t affect your prod environment)
    rpm -i distcache-1.4.5-8.5.x86_64.rpm distcache-devel-1.4.5-8.5.x86_64.rpm
  3. Download all source tarballs


  4. Start all rpmbuilds, installing output intermediate RPMs
    rpmbuild -tb apr-1.5.2.tar.bz2
    rpm -U ~/rpmbuild/RPMS/x86_64/apr-1.5.2-1.x86_64.rpm ~/rpmbuild/RPMS/x86_64/apr-devel-1.5.2-1.x86_64.rpm
    rpmbuild -tb apr-util-1.5.4.tar.bz2
    rpm -U ~/rpmbuild/RPMS/x86_64/apr-util-1.5.4-1.x86_64.rpm ~/rpmbuild/RPMS/x86_64/apr-util-devel-1.5.4-1.x86_64.rpm
    rpmbuild -tb httpd-2.4.20.tar.bz2

    A little addition I had to perform for the apr build on a vagrant VM (not on docker) was to enable IPV6 to have all tests pass. So If APR test are failing on socker simply add:

    sysctl net.ipv6.conf.all.disable_ipv6=0


If everything went fine you should now have all apache and apr RPMs available in the RPMs folder, ready to be installed.
Bud as anticipated we want also to distribute such packages as a Yum repository, so let’s add also a couple of RPM needed for the install and to generate the repository:

yum install -y mailcap createrepo gzip

and then then run them

mkdir -p ~/repo
mv ~/rpmbuild/RPMS/x86_64/*.rpm ~/repo
cd ~/repo
createrepo .

If you want also to provide a small repository index with packages name and description you can extract them from repo data into a more handy HTML page:

echo "<html><body><table>" > ~/repo/index.html
zcat repodata/*primary.xml.gz | grep -e "<name" -e "<summary" | sed -e "s/<name>/<tr><td>/g" -e "s/<\/name>/<\/td>/g" -e "s/<summary>/<td>/g" -e "s/<\/summary>/<\/td><\/tr>/g" >> ~/repo/index.html
echo "</table></body></html>" >> ~/repo/index.html

Now you just have to package all the folder into your portable yum repo:

cd ~
tar -zcvf FLEP-httpd24-repo.tar.gz repo

To install it on a target server you just have to unpack the archive into a temportay folder (e.g. /tmp/repo ) and create a new yum configuration into /etc/yum.repo.d/httpd24.repo

name=Apache 2.4.x prerequisites repository

and install them via yum:

yum install httpd

OK, useful but boring up to now… let’s start the more interesting part, how to move all of this inside docker?
As you may notice from my other posts, I’m an Ansible fan, so we will not deal with Dockerfiles, but we will create and ansible-container project to automate the whole process.

  1. the first step is obviously install  docker and ansible-container, sinche this can change depending on your system, please check thier installation page
  2. Then create a new folder and init it a new ansible-container app
    ansible-container init

    it will create all the needed files to start.

  3. Create a new service into container.yml
    version: "2"
     user: "{{dst_user}}"
     image: "centos:{{centosver}}"
     - "/tmp/apache24_rpmbuilder:{{output_path}}"
     entrypoint: ["{{vol_path}}/"]

    this will become our docker image, based on a parametrized CentOS version.
    Notice that we also add an attached volume, which will be used to persist our final repository tarball, and an script which will contain all the rpmbuild steps described above.

  4. Now let’s attach an ansible role named apache24_rpmbuilder for image customization in main.yml:
    - hosts: apache24_rpmbuilder
     gather_facts: false
     - apache24_rpmbuilder

    This ansible role will be responsible to customize the base docker image declared into the container.yml file, basically by uploading the entrypoint script and the yum repo config file.
    The main task will be contained in a role path: apache24_rpmbuilder/tasks/main.yml

    - name: "create folder {{vol_path}}"
     file: path="{{item}}" state="directory" owner="{{dst_user}}" 
     - "{{vol_path}}"
     - "{{ output_path }}/repo"
    - name: Synch script files
     template: src="{{item.src}}" dest="{{item.dst}}" mode="u=rxw,g=r,o=r"
     - { src: "{{roles_path}}/apache24_rpmbuilder/templates/", dst: "{{vol_path}}/" }
     - { src: "{{roles_path}}/apache24_rpmbuilder/templates/httpd24.repo.tpl", dst: "{{ output_path }}/httpd24.repo" }

    Both of them are a template version of the script containing all the rpm-build steps.

    All standard variables can be added into the /apache24_rpmbuilder/vars/main.yml 

    # Centos image tag to be used for docker
    #centosver: "6.6"
    centosver: "6.8"
    # Main path for the ansible-container
    base_path: "/ansible-container"
    # Main path for the ansible-container roles to be used
    roles_path: "{{base_path}}/ansible/roles"
    # Main path for the installation in the final docker image
    vol_path: "/tmp"
    # Path for the generated files into the final docker image
    output_path: "{{ vol_path }}/RPMS"
    # User running final image
    dst_user: "root"

    While all build-related vaiables will be in /apache24_rpmbuilder/defaults/main.yml

    src_apache_ver: "2.4.20"
    src_apache_file: "httpd-{{src_apache_ver}}.tar.bz2"
    src_apache_url: "{{src_apache_file}}"
    src_apr_ver: "1.5.2"
    src_apr_file: "apr-{{src_apr_ver}}.tar.bz2"
    src_apr_url: "{{src_apr_file}}"
    src_apr_util_ver: "1.5.4"
    src_apr_util_file: "apr-util-{{src_apr_util_ver}}.tar.bz2"
    src_apr_util_url: ""
    distcache_file: "distcache-1.4.5-8.5.x86_64.rpm"
    distcache_url: "{{distcache_file}}"
    distcache_devel_file: "distcache-devel-1.4.5-8.5.x86_64.rpm"
    distcache_devel_url: "{{distcache_devel_file}}"
    downloader: "yum install -y --downloadonly --downloaddir="
    downloader66: "yumdownloader --destdir="

    As you can notice the latter variables files contains all relevant properties to be maintained to perform our build as the prerequisite packages lists and the source versions and download url to be used to retrieve them.
    CentOS version for Docker base image is declared in vars file instead of defaults one since such file will be passed also to ansible-container build process in order to process the container.yml file as well

    Finally here comes the jinja2-template version of our build script/apache24_rpmbuilder/templates/

    DWN_BASE_PATH={{ output_path }}
    SRC_BASE_PATH={{ vol_path }}
    DWN_VERSION=$(cat /etc/redhat-release | grep -o -P "([0-9]\.?)*");
    MJR_VERSION=$(echo ${DWN_VERSION} | cut -f1 -d"." )
    echo "removing ${DWN_BASE_PATH}/repo/Centos/${DWN_VERSION}/latest"
    rm -rf ${DWN_BASE_PATH}/repo/Centos/${DWN_VERSION}/latest ${DWN_BASE_PATH}/httpd24-repo.*
    echo "creating ${DWN_PATH}"
    mkdir -p ${DWN_PATH}
    echo "Setting up EPEL ${MJR_VERSION}"
    command rpm -Uvh --replacepkgs${MJR_VERSION}.noarch.rpm
    #Install wget first for fail-fast if url not available
    yum install -y wget
    wget {{ src_apache_url }}
    wget {{ src_apr_url }}
    wget {{ src_apr_util_url }}
    wget {{ distcache_url }}
    wget {{ distcache_devel_url }}
    yum install -y {{distcache_file}} {{distcache_devel_file}}
    yum install -y ${BUILD_PREREQUISITES}
    #needed by tests
    #sysctl net.ipv6.conf.all.disable_ipv6=0
    rpmbuild -tb {{src_apr_file}}
    rpm -U ~/rpmbuild/RPMS/x86_64/apr-{{src_apr_ver}}-1.x86_64.rpm ~/rpmbuild/RPMS/x86_64/apr-devel-{{src_apr_ver}}-1.x86_64.rpm
    rpmbuild -tb {{src_apr_util_file}}
    rpm -U ~/rpmbuild/RPMS/x86_64/apr-util-{{src_apr_util_ver}}-1.x86_64.rpm ~/rpmbuild/RPMS/x86_64/apr-util-devel-{{src_apr_util_ver}}-1.x86_64.rpm
    rpmbuild -tb {{src_apache_file}}
    mv ~/rpmbuild/RPMS/x86_64/*.rpm ${DWN_PATH}/
    {% if centosver == "6.6" %}
    yum install -y yum-utils
    echo "Invoking Yum-Downloader to retrieve httpd install dependencies [${HTTPD_PREREQUISITES}]"
    {{ downloader66 }}${DWN_PATH} ${HTTPD_PREREQUISITES} > ${DWN_BASE_PATH}/yum_downloadonly.log 2>&1
    {% else %}
    echo "Invoking Yum Download-Only to retrieve httpd install dependencies [${HTTPD_PREREQUISITES}]"
    {{ downloader }}${DWN_PATH} ${HTTPD_PREREQUISITES} > ${DWN_BASE_PATH}/yum_downloadonly.log 2>&1
    {% endif %}
    echo "installing createrepo package"
    yum install -y createrepo gzip
    echo "indexing ${DWN_PATH}"
    cd ${DWN_PATH}
    createrepo .
    echo "<html><body><table>" > ${DWN_BASE_PATH}/repo/index.html
    zcat repodata/*primary.xml.gz | grep -e "<name" -e "<summary" | sed -e "s/<name>/<tr><td>/g" -e "s/<\/name>/<\/td>/g" -e "s/<summary>/<td>/g" -e "s/<\/summary>/<\/td><\/tr>/g" >>${DWN_BASE_PATH}/repo/index.html
    echo "</table></body></html>" >> ${DWN_BASE_PATH}/repo/index.html
    cd ${DWN_BASE_PATH}
    tar -zcvf httpd24-repo${DWN_VERSION}.tar.gz repo

    Probably less readable than the simple list of bash command but easier to maintain in an ansible-role perspective.
    Notice also that the script contains a small if-then-else clause which allow to manage a difference in yum offline download from Centos 6.6 to 6.8.

  5. Now that our role is in place we just have to start ansible-container to create our base image:
    ansible-container --var-file ansible/roles/apache24_rpmbuilder/vars/main.yml  build --from-scratch

    Build will last for few minutes ( a bit more if you have to download both CentOs and Ansible-container image) and will produce our apache24_rpmbuilder image.
    Let’s check with a docker image listing:

    $ docker images
    REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
    apache24_rpmbuilder-apache24_rpmbuilder                           20170122105436      9f009ca5a36d        1 minute ago        203 MB
    apache24_rpmbuilder-apache24_rpmbuilder                           latest              9f009ca5a36d       1 minute ago        203 MB


  6. Everything is in place, let’s start the real rpmbuild process:
    docker run -v /tmp/apache24_rpmbuilder:/tmp/RPMS -t -i apache24_rpmbuilder-apache24_rpmbuilder:latest

    Remember to map the output folder to your desired local folder.
    The build will last some minutes, depending on your download time for sources and dependencies plus the real rpm-build time, but in the and in your folder you will find all the RPMs both in a handful tar.gz and a unpacked format.
    As a plus a bunch of log for each rpmbuild process (just in case of trouble) and a human-readable html with all packages name and their description.

Remeber that such Httpd RPMs are slightly different from the official CentOS one, so you will still have to work a bit on their configuration after their installation, since all module will be disabled and not all defaults value could be the one you will be expecting.
I alos wrote a classic ansible role for quick install and setup sarting from the rseulting tar.gz, but this will be the topic for another post.

In the meanwhile if you are willing to contribute with testing or improvements you can find the whole source code of this post on aroundthecode github and (if you are really lazy!) the ready-to-run docker image on aroundthecode docker hub.
Such code has been tested with both CentOs 6.6 and 6.8, but I’d like to extend it also to more recent releases as soon as possible.
In such repository you will find a little bonus which is also the capability to compile mod_security RPM as well. This will be probably be another post topic since at the time of writing the source site seems to be temporary down so the process is stopping just after the httpd RPMs.


Un pensiero su “Package Apache 2.4.x RPMs via Docker and Ansible-container

  1. Pingback: How to publish Ansible-container images via Travis-CI | Around the Code


Inserisci i tuoi dati qui sotto o clicca su un'icona per effettuare l'accesso:

Logo di

Stai commentando usando il tuo account Chiudi sessione /  Modifica )

Foto Twitter

Stai commentando usando il tuo account Twitter. Chiudi sessione /  Modifica )

Foto di Facebook

Stai commentando usando il tuo account Facebook. Chiudi sessione /  Modifica )

Connessione a %s...