Artifactory in the box

Today I got a very interesting problem to solve: I  had to delivery a bunch of libraries to an external developer to let him be able to fully compile a very complex build.

Tasks wasn’t easy since he could not simply connect to our local Artifactory to perform the single maven build he would need to have everything in place and packaging all libraries in a single zip and send it to him was a really error-prone procedure, even taking advantage of maven dependency plugin.

After a small brainstorming we got a really nice idea, I he cannot connect to Artifactory directly, why not providing him Artifactory itself as a Vagrant box?

Let’s do a small step behind: what is Vagrant? It can be considered a wrapper of a virtualization provider (Virtualbox, Vmware) together with some scriptable automatic provisioning tools (Ansible, Chef, Puppets) which allow you to create extensible and portable virtual machine via scripting.
A perfect example of the power of Vagrand will be illustrated in this article.

First of all I had to choose a base box to start from, since Artifactory is provided in a self-extracting RPM package, I looked for a minimal Centos box from the ones available at VagrantBox : centos65-x86_64-20140116.box

Now it was time for some Ansible’s magic.
I created a small YAML script to fully install some needed bash tools (just wget to download needed package), a Java JDK, download Artifactory RPM from the official site, install it as a startup service and finally start it!

Here it comes:

---
- hosts: all
  vars:
    pkg_list:
      - wget
  tasks:
  #########################################
  # Package install and service register  #
  #########################################
  - include: tasks/jdk.yml
  - name: create jdk link
    sudo: true
    file: src=/opt/jdk7/bin/java dest=/usr/local/bin/java state=link
  - name : install packages
    sudo: true
    yum: name={{ item }} state=present
    with_items: pkg_list
  - name: download Artifactory
    sudo: true
    shell: wget http://bit.ly/Hqvfi9 -O/tmp/artifactory.rpm
  - name: install Artifactory
    sudo: true
    shell: rpm -i /tmp/artifactory.rpm
  - name: setup Artifactory as a service
    sudo: true
    shell: chkconfig artifactory on
  - name: starting Artifactory
    sudo: true
    shell: service artifactory start

Java jdk install is delegated to a separate YML file since it’s a useful routine I reuse in a lot of other ansible script, here it comes (packages are taken from our internal artifactory to speedup setup time)

---
  #################
  # Java JDK install  #
  #################
  - name: download jdk-linux-x64-7u51.tar.gz
    local_action: get_url url=http://repo.mycompany.int/artifactory/mycompany-thirdparty/java/jdk-linux-x64/7u51/jdk-linux-x64-7u51.tar.gz dest=/tmp/jdk-linux-x64-7u51.tar.gz
    
  - name: unarchive jdk-linux-x64-7u51.tar.gz
    sudo: true
    unarchive: src=/tmp/jdk-linux-x64-7u51.tar.gz dest=/opt
    
  - name: create jdk7 link
    sudo: true
    file: src=/opt/jdk1.7.0_51 dest=/opt/jdk7 state=link

Ok, now everything is ready to be used with a Vagrant file to create our portable Artifacoty Box, let’s prepare a Vagrantfile this way:

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "centos65-x86_64-20140116.box"
  #Artifactory
  config.vm.network "forwarded_port", guest: 8081, host: 8081
  
  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "../provision/artifactory.yml"
  end
  # additional eth1 for private network access
  config.vm.network "private_network", ip: "192.168.33.100"
  config.vm.provider "virtualbox" do |v|
    v.memory = 4096
    v.cpus = 2
  end
end

As you can see I’m basically:

  • Telling vagrant I want to use the Centos box and the url where to find it (it will take care to download just at first usage)
  • Mapping my virtual box (guest) port 8081 to my own laptop (host) 8081 port so that i can access it via localhost or provide other people access via my IP address
  • Telling vagrant to use my Ansible script to provision the machine with all I want.
  • Tuning some CPU/memory settings

ok, everything on the launch pad, now its time to run it, just perform a vagrant up and see what happens.

$ vagrant up
==> default: Attempting graceful shutdown of VM...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: hostonly
==> default: Forwarding ports...
    default: 8081 => 8081 (adapter 1)
    default: 22 => 2222 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Configuring and enabling network interfaces...
==> default: Mounting shared folders...
    default: /vagrant => /Users/msacchetti/Documents/ISOS/Vagrant/work_artifactory_package
==> default: Running provisioner: ansible...
PLAY [all] ********************************************************************
GATHERING FACTS ***************************************************************
ok: [default]
TASK: [download jdk-linux-x64-7u51.tar.gz] ************************************
changed: [default]
TASK: [unarchive jdk-linux-x64-7u51.tar.gz] ***********************************
changed: [default]
TASK: [create jdk7 link] ******************************************************
changed: [default]
TASK: [create jdk link] *******************************************************
changed: [default]
TASK: [install packages] ******************************************************
changed: [default] => (item=httpd,mod_proxy_html,wget)
TASK: [download Artifactory] **************************************************
changed: [default]
TASK: [install Artifactory] ***************************************************
changed: [default]
TASK: [setup Artifactory as a service] ****************************************
changed: [default]
TASK: [starting Artifactory] ****************************************
changed: [default]

PLAY RECAP ********************************************************************

default                    : ok=9    changed=8    unreachable=0    failed=0

As you can see from the output vagrant brought up the virtual machine, configured the network interface as specified, mapped the ports and finally triggered ansible to run the script which installed and started Artifactory.

If you now open your browser and navigate to http://localhost:8081/artifactory/ you will get Artifactory homepage.

Artifactory_box_home

Ok, almost done, now we jut have to populate it with all needed library ( the screenshot refers in fact to the already populated instance since you can see the “839 artifacts” count ).

Before doing so we want to perform a dump of the machine in a clean state to be able to reuse it from scratch at will.
So we will perform a vagrant package –output artifactory.box to have vagrant produce a new .box file containing the full installed server.

now that we have your milestone box backupped we can go on: bring the machine up again with a vagrant up (packaging causes the machine to be turned of for dumping).
You will notice that this time all the Ansible provisioning will not run since it’s performed just at first boot.
You will instead get the following trace.

==> default: VM already provisioned. Run `vagrant provision` or use `–provision` to force it

Now we have to populate the newly created repository with all (and just) the needed libraries.
We also want to exclude any source or javadoc since we are dealing with a closed source project.

We need to perform the following steps:

  1. configure our official artifactory instance as a remote repository of the boxed one, with proper replication and caching configuration
  2. prepare a different user maven setting to use our boxed instance as a new mirror, so that all request goes through it instead of the official one (see “Artifactory – repository setup and configuration” post for details)
  3. configure a different local repository location in global setting to be sure no artifact is taken by our local cache instead of being requested to the remote boxed server.

First navigate to your boxed repo, log in as admin, got to Admin -> Repositories section and add a new remote repository like this:
Artifactory_localhost____Configure_Repositories

The url of the remote repository must be the one of your official one, containing all your maven libraries.
We exclude sources and javadoc to avoid source code being available into the box we will redistribute.
Finally we configure advanced setting to be sure that all requested artifact will be stored locally into our box repo.

Artifactory_localhost____Configure_Repositories

Our repo is now ready to serve incoming requests and cache them locally.

Next steps are on our own machine, where we want to perform the main artifact build.

First we have to force each request going just to the boxed repo, which will then proxy any other remote repo configured, mainly out official one we just added.
To do so create a copy of your user setting file (usually ~/.m2/settings.xml ) as ~/.m2/settings-vagrant.xml and add the mirror directive pointing to our boxed repo

<mirrors>
<mirror>
<mirrorOf>*</mirrorOf>
<name>repo</name>
<url>http://localhost:8081/artifactory/repo</url&gt;
<id>repo</id>
</mirror>
</mirrors>

Now our build is almost ready to run, we just have to be sure nothing is cached on our local maven repository, the dirty way could be simply delete the ~/.m2/repository folder, but this would means that any maven build afterward will have to re-download any library.
A more safe way is to create also a copy of the global maven settings (usually $MAVEN_HOME/conf/settings.xml ) providing a different , empty location for your local repo:

<localRepository>/tmp/m2/</localRepository>

 

save it as something like $MAVEN_HOME/conf/settings-nolocal.xml

Ok, we are ready to go, now all you have to do is to perform your maven build as usual but telling maven to use the user and global setting file we just created.

mvn clean package  –settings ~/.m2/settings-vagrant.xml –global-settings /opt/apache-maven-3.0.5/conf/settings-nolocal.xml

 

If everything has been done properly you will notice your build re-downloading any needed dependency from the boxed artifactory, while, checking repo log, you will  see that each request will be forwarded to the official one and cached locally.
Once the been has been completed your artifactory will be populated with all needed libraries to perform your build.

If you get any error on missing artifact during this phase you probably have to add some more remote repository to you boxed instance to match the ones available on your official one.

Once everything is build you just need to perform a new  vagrant package –output artifactory-full.box to dump all your work.

Now you have two boxes, a artifactory-full.box with all your artifact to be delivered to end user and which can be integrated with new libraries at each new build release,and a smaller artifactory.box with a clean installation to be reused every time you need to re-populate your full-box from scratch.

Every time you want to run the new boxes you can simply use a Vagrantfile similar to the one used for creating the box from scratch, simply changing the box name at will:

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "artifactory-full.box"
  
  #Artifactory
  config.vm.network "forwarded_port", guest: 8081, host: 8081
  
  # additional eth1 for private network access
  config.vm.network "private_network", ip: "192.168.33.100"
  config.vm.provider "virtualbox" do |v|
    v.memory = 4096
    v.cpus = 2
  end
end

Now just perform some cleanup with a vagrant destroy to deallocate running virtual instance and keep just you boxes for future reuse.

 

 

Lascia un commento