Support virtualbox and docker in the same vagrantfile.

Dockerfile

@@ -0,0 +1,17 @@
+FROM library/debian:9
+
+ENV container docker
+ENV DEBIAN_FRONTEND "noninteractive"
+ENV NOTVISIBLE "in users profile"
+
+ADD bin/dockerrun.sh /root/dockerrun.sh
+ADD bin/dockerexec.sh /root/dockerexec.sh
+ADD etc/.vimrc /root/.vimrc
+ADD etc/.vimrc /etc/skel/.vimrc
+
+RUN /root/dockerrun.sh
+
+STOPSIGNAL SIGRTMIN+3
+
+CMD ["/root/dockerexec.sh", "-D"]
+EXPOSE 22

README.md

@@ -10,8 +10,7 @@ Those developing [Library Card Platform for The Wikipedia Library](https://githu
 
 ## Requirements
 * [Vagrant](https://www.vagrantup.com/downloads.html)
-* [VirtualBox and VirtualBox Extension Pack](https://www.virtualbox.org/wiki/Downloads)
-* vagrant-vbguest plugin (eg. vagrant plugin install vagrant-vbguest)
+* [Docker](https://www.docker.com/get-started)
 * Browser configured to hit a local SOCKS proxy on a port of your choice, I use 2080
 
 ## Optional
@@ -20,30 +19,21 @@ Those developing [Library Card Platform for The Wikipedia Library](https://githu
 
 ## Notes for Linux users:
 
-For a "just works" experience, I recommend fetching Vagrant and VirtualBox packages from the vendor websites rather than using your distribution's software repositories. Those likely include fairly old versions of the required packages, and you will find yourself having to carefully managing your Vagrant, VirtualBox, and base box updates to avoid breakage, if it's not broken out of the gate.
+For a "just works" experience, I recommend fetching Vagrant and Docker packages from the vendor websites rather than using your distribution's software repositories. Those likely include fairly old versions of the required packages, and you will find yourself having to carefully managing your component updates to avoid breakage, if it's not broken out of the gate.
 
 ## Notes for Windows users:
 
 Some third-party endpoint security software, such as Dell Data Protection Encryption and several McAfee products, interfere with VirtualBox. You may need to temporarily disable these products or make different endpoint protection choices. 
 
-You'll need to add the following directory to your PATH environment variable after installing VirtualBox:
+Vagrant's (early but generally working) support for Ubuntu via the Windows Subsystem for Linux is the recommended way to run this enviroment. You should be on Windows 10 Version 1709 or later and perform a store-based Ubuntu installation. See the [Vagrant and Windows Subsystem for Linux instructions](https://www.vagrantup.com/docs/other/wsl.html). Then:
 
-```
-C:\Program Files\Oracle\VirtualBox
-```
-
-See [this example from Microsoft](https://msdn.microsoft.com/en-us/library/office/ee537574.aspx) for adding a path to the PATH environment variable.
-
-Vagrant's (early but generally working) support for Ubuntu via the Windows Subsystem for Linux is the recommended way to run this enviroment. You should be on Windows 10 Version 1709 or later and perform a store-based Ubuntu installation. See the [Vagrant and Windows Subsystem for Linux instructions](https://www.vagrantup.com/docs/other/wsl.html). You'll install VirtualBox on the Windows side, and then install exactly the same build of Vagrant in both Windows and Ubuntu. The Linux notes apply to the Ubuntu environment. Just download a fixed version of Vagrant (that matches the version you install in Windows) and install using dpkg as described in the instructions. Install any plugins in Ubuntu.
-
-On Vagrant 2.0.2 and earlier, [issue #9298](https://github.com/hashicorp/vagrant/issues/9298) means you'll need to create a symlink in the location of the deprecated lxrun installation that points to the new store-based installation. As pointed out in the reported issue, running the following powershell commands on the windows side will pull the information from the registry and create the appropriate symlink.
-
-```
-$WSLREGKEY="HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss"
-$WSLDEFID=(Get-ItemProperty "$WSLREGKEY").DefaultDistribution
-$WSLFSPATH=(Get-ItemProperty "$WSLREGKEY\$WSLDEFID").BasePath
-New-Item -ItemType Junction -Path "$env:LOCALAPPDATA\lxss" -Value "$WSLFSPATH\rootfs"
-```
+ * Install Docker on Windows and enable legacy mode (Expose daemon on tcp://localhost:2375 without TLS)
+ * Install Docker on Ubuntu. You can just use the Ubuntu-provided package, eg. `apt install docker.io`
+ * Install Vagrant on Ubuntu using dpkg as described in the Vagrant instructions. If you already have Vagrant on Windows, you'll need to keep the two at exactly the same build version.
+ * When you clone this repository in Ubuntu, make sure to do so in a location accessible to Windows, such as ``/mnt/c/Users/Username/v`` (``/mnt/c/`` corresponds to ``C:\``). This is required for the vagrant share to work properly.
+ * There are a number of environment variables that should be configured for WSL + Docker + Vagrant to work happily. As a convenience, you may just ``source bin/wsl_docker_activate.sh`` from within the project directory.
+
+The Linux notes apply to the Ubuntu environment.
 
 ## Usage
 
@@ -55,7 +45,7 @@ You might need to configure some of the settings for the [puppet module](https:/
 ```
 and configure any parameters you'd like to override, such as the git repository or revision. See the [parameters manifest in the puppet module](https://github.com/WikipediaLibrary/twlight_puppet/blob/master/manifests/params.pp).
 
-If you have a tarball that you'd like to load on provision, place it
+If you have a TWLight backup tarball that you'd like to load on provision, place it
 
 ```
 ./backup/twlight.tar.gz
@@ -64,7 +54,7 @@ If you have a tarball that you'd like to load on provision, place it
 Alternatively, scripts are included to create a superuser and generate example data. Before doing anything else, login to the platform as normal, then run
 
 ```
-sudo -u www /var/www/html/TWLight/bin/virtualenv_example_data.sh
+/var/www/html/TWLight/bin/virtualenv_example_data.sh
 ```
 
 The account you used to login will be made a superuser, giving you access to the Admin interface. The values in that file can be modified to generate more or less users, partners, and applications, but the file should only be run once.
@@ -83,5 +73,5 @@ You can now work on the running app inside Vagrant and view the changes in your
 As you are making local changes, make sure to take advantage of the included test suite. To do so, run the following command within the vagrant machine:
 
 ```
-sudo su www /var/www/html/TWLight/bin/virtualenv_test.sh
+/var/www/html/TWLight/bin/virtualenv_test.sh
 ```

Vagrantfile

@@ -1,8 +1,7 @@
 # -*- mode: ruby -*-
 # vi: set ft=ruby :
 
-twlight_puppet_version = "0.5.2"
-#twlight_puppet_version = "master"
+twlight_puppet_version = "0.5.13"
 
 # Put "--debug " in this string if you want to test the limits of your terminal
 # emulator's buffer.
@@ -27,78 +26,82 @@ Vagrant.configure("2") do |config|
 
   # Forward SSH agent from host into Vagrant machine
   config.ssh.forward_agent = true
+  config.ssh.password = "vagrant"
+
+  # Our provisioner expects /vagrant.
+  config.vm.synced_folder ".", "/vagrant"
 
-  # roughly tracking twlight VMs provisioned via wikimedia labs horizon
-  config.vm.box = "bento/debian-8"
-  #config.vm.box_version = ""
 
   # We need a little beef if we're pulling in production-scale data
   config.vm.provider :virtualbox do |v|
+    # roughly tracking twlight VMs provisioned via wikimedia labs horizon
+    v.box = "bento/debian-9"
     v.cpus = 4
     v.memory = 4096
     v.linked_clone = true
   end
 
-  # Our puppet provisioner expects /vagrant, and virtualbox is the only tested provider
-  if Vagrant.has_plugin?("vagrant-vbguest")
-    config.vm.synced_folder ".", "/vagrant", type: "virtualbox", mount_options: ['dmode=777', 'fmode=666']
-    config.vbguest.auto_update = false
+  # Build container from Dockerfile
+  config.vm.provider "docker" do |d|
+    d.build_dir = "."
+    d.has_ssh = true
+    d.create_args = ["--cap-add", "SYS_ADMIN", "-v", "/run", "-v", "/tmp", "-v", "/sys/fs/cgroup:/sys/fs/cgroup:ro"]
+  end
 
-    # Allow the SSH agent to cross the sudo barrier.
-    # Handy if you use an SSH remote and want to run the git pull script (which require root).
-    config.vm.provision "shell",
+  # Allow the SSH agent to cross the sudo barrier.
+  # Handy if you use an SSH remote and want to run the git pull script (which require root).
+  config.vm.provision "shell",
       inline: "echo 'Defaults    env_keep+=SSH_AUTH_SOCK' | sudo EDITOR='tee -a' visudo"
 
-    # Install puppet because we need it, chrony because its useful in Vagrant,
-    # and vim because the author of this Vagrantfile prefers it.
-    config.vm.provision "shell",
-      inline: "wget --quiet --timestamping --directory-prefix=/tmp \
-          https://apt.puppetlabs.com/puppetlabs-release-pc1-jessie.deb && \
-          dpkg -i /tmp/puppetlabs-release-pc1-jessie.deb && \
-          apt update && apt install -y chrony puppet-agent vim"
-
-
-    # Add github's host key to our known hosts file
-    config.vm.provision "shell",
-      inline: "ssh-keyscan -t rsa github.com >> /etc/ssh/ssh_known_hosts"
-
-    ## Handy method for fetching puppet module from github
-    #config.vm.provision "shell",
-    #  inline: "wget --quiet --timestamping --directory-prefix=/vagrant/puppet/modules \
-	#		  'https://github.com/WikipediaLibrary/twlight_puppet/archive/"+ twlight_puppet_version +".tar.gz'"
-
-    ## Install our twlight puppet module from github
-    #config.vm.provision "shell",
-    #  inline: twlight_puppet_bin_path +"/puppet module install \
-    #    "+ twlight_puppet_options +" --target-dir /vagrant/puppet/modules \
-    #    /vagrant/puppet/modules/"+ twlight_puppet_version +".tar.gz"
-
-    # Install our twlight puppet module from puppet forge
-    config.vm.provision "shell",
-      inline: twlight_puppet_bin_path +"/puppet module install \
-        "+ twlight_puppet_options +" --target-dir /vagrant/puppet/modules \
-        jsnshrmn/twlight --version "+ twlight_puppet_version +";"
-
-    # Run the puppet provisioner
-    config.vm.provision "puppet" do |puppet|
-      puppet.working_directory = "/vagrant/puppet"
-      puppet.hiera_config_path = "puppet/hiera.yaml"
-      puppet.environment = "local"
-      puppet.environment_path = "puppet/environments"
-      puppet.module_path = "puppet/modules"
-      puppet.binary_path = twlight_puppet_bin_path
-      puppet.options = twlight_puppet_options
-
-    # Run migration so any imported DB dump will work with current code.
-    config.vm.provision "shell",
-      inline: "sudo su www bash -c '/var/www/html/TWLight/bin/./virtualenv_migrate.sh >>/var/www/html/TWLight/TWLight/logs/update.log 2>&1' || :"
-
-    # Allow vagrant user to write to project .git
-    config.vm.provision "shell",
-      inline: "usermod -a -G www vagrant && chmod -R g+w /var/www/html/TWLight"
-
-    end
-
+  # Install puppet because we need it, chrony because its useful in Vagrant,
+  # and vim because the author of this Vagrantfile prefers it.
+  config.vm.provision "shell",
+    inline: "wget --quiet --timestamping --directory-prefix=/tmp \
+        https://apt.puppetlabs.com/puppetlabs-release-pc1-stretch.deb && \
+        dpkg -i /tmp/puppetlabs-release-pc1-stretch.deb && \
+        apt update && apt install -y chrony puppet-agent vim"
+
+
+  # Add github's host key to our known hosts file
+  config.vm.provision "shell",
+    inline: "ssh-keyscan -t rsa github.com >> /etc/ssh/ssh_known_hosts"
+
+  # frontload some vagrant-specific systemd config.
+  config.vm.provision "shell",
+    inline: "mkdir -p /etc/systemd/system/mariadb.service.d; \
+        printf '[Service]\nProtectHome=false\nType=simple\n' > /etc/systemd/system/mariadb.service.d/vagrant.conf; \
+        systemctl daemon-reload"
+
+  ## Handy method for fetching puppet module from github
+  #config.vm.provision "shell",
+  #  inline: "wget --quiet --timestamping --directory-prefix=/vagrant/puppet/modules \
+  #		  'https://github.com/WikipediaLibrary/twlight_puppet/archive/"+ twlight_puppet_version +".tar.gz'"
+
+  ## Install our twlight puppet module from github
+  #config.vm.provision "shell",
+  #  inline: twlight_puppet_bin_path +"/puppet module install \
+  #    "+ twlight_puppet_options +" --target-dir /vagrant/puppet/modules \
+  #    /vagrant/puppet/modules/"+ twlight_puppet_version +".tar.gz"
+
+  # Install our twlight puppet module from puppet forge
+  config.vm.provision "shell",
+    inline: twlight_puppet_bin_path +"/puppet module install \
+      "+ twlight_puppet_options +" --target-dir /vagrant/puppet/modules \
+      jsnshrmn/twlight --version "+ twlight_puppet_version +";"
+
+  # Run the puppet provisioner
+  config.vm.provision "puppet" do |puppet|
+    puppet.working_directory = "/vagrant/puppet"
+    puppet.hiera_config_path = "puppet/hiera.yaml"
+    puppet.environment = "local"
+    puppet.environment_path = "puppet/environments"
+    puppet.environment_variables = {"APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE" => "1" }
+    puppet.module_path = "puppet/modules"
+    puppet.binary_path = twlight_puppet_bin_path
+    puppet.options = twlight_puppet_options
+
+  # Run migration so any imported DB dump will work with current code.
+  config.vm.provision "shell",
+    inline: "sudo su vagrant bash -c '/var/www/html/TWLight/bin/./virtualenv_migrate.sh >>/var/www/html/TWLight/TWLight/logs/update.log 2>&1' || :"
   end
-
 end

bin/dockerexec.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+
+# Fire up systemd and detach.
+#/usr/bin/nohup /bin/systemd --system --unit=basic.target > /root/$(hostname -f).systemd.log 2>&1 &
+#service dbus start
+/bin/systemd --system --unit=basic.target &
+
+# Generate SSH host keys
+ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
+ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
+ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key
+ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key
+
+# Fire up SSH
+/bin/systemctl start sshd
+
+# Keep our container running.
+trap : TERM INT
+sleep infinity & wait

bin/dockerrun.sh

@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+
+echo "export VISIBLE=now" >> /etc/profile
+
+# Install packages.
+# Basic vagrant workflow for faking real hosts needs locales openssh-server sudo and systemd.
+# Puppet provisioner needs lsb-release. Puppet apt module needs gnupg for adding encrypted repos.
+# TWLight vagrant shell provisioner expects wget.
+apt update && apt install -y dialog gnupg lsb-release locales locales-all openssh-server sudo systemd wget
+
+# Lie if anyone anyone asks if we booted up with systemd.
+# https://www.freedesktop.org/software/systemd/man/sd_booted.html
+mkdir -p /run/systemd/system
+
+# Strip out bits that aren't going to work happily in this container. Largely cribbed from:
+# https://developers.redhat.com/blog/2014/05/05/running-systemd-within-docker-container/
+(cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done);
+rm -f /lib/systemd/system/multi-user.target.wants/*;
+rm -f /etc/systemd/system/*.wants/*;
+rm -f /lib/systemd/system/local-fs.target.wants/*;
+rm -f /lib/systemd/system/sockets.target.wants/*udev*;
+rm -f /lib/systemd/system/sockets.target.wants/*initctl*;
+rm -f /lib/systemd/system/basic.target.wants/*;
+rm -f /lib/systemd/system/anaconda.target.wants/*;
+# With these additions that @jsnshrmn found to be problematic, at least on Debian 9.
+rm -f /lib/systemd/system/user\@.service
+rm -r /lib/systemd/system/systemd-tmpfiles-setup.service
+(cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-journald.service ] || rm -f $i; done);
+
+# Prep us running SSH from a shell.
+mkdir -p /var/run/sshd
+
+# SSH login fix. Otherwise user is kicked off after login
+sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
+
+# Configure insecure root user for Vagrant
+echo 'root:vagrant' | chpasswd 
+sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
+
+# Add vagrant user
+adduser --quiet --disabled-password --shell /bin/bash --home /home/vagrant --gecos "User" vagrant
+echo 'vagrant:vagrant' | chpasswd
+
+# Setup vagrant insecure key.
+mkdir -p /home/vagrant/.ssh
+echo 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key' >>/home/vagrant/.ssh/authorized_keys
+chown -R 'vagrant:vagrant' /home/vagrant/.ssh 
+chmod 600 /home/vagrant/.ssh/authorized_keys
+chmod 700 /home/vagrant/.ssh
+
+# Setup passwordless sudo for vagrant user.
+echo 'vagrant         ALL = (ALL) NOPASSWD: ALL' >> /etc/sudoers

bin/wsl_docker_activate.sh

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+
+# Checks to see if this is a WSL environment;
+# If true, it sets environment variables in an attempt to make things universially worky.
+# Originally motivated by Windows + Hyper-V + Docker + Vagrant struggles.
+# Logic for check pulled from https://stackoverflow.com/a/43618657
+# WSL-specific Vagrant information can be found here:
+# https://www.vagrantup.com/docs/other/wsl.html
+if grep -qE "(Microsoft|WSL)" /proc/version &> /dev/null ; then
+
+  # Enable WSL vagrant
+  export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"
+
+  # Put the dotfiles in the WSL home directory so that ssh key permissions
+  # may be set.
+  here=$(basename $(pwd))
+  export VAGRANT_DOTFILE_PATH="~/.vagrant/.${here}"
+
+  # @TODO: make this check less brittle. Currenly only works with default install location.
+  # Let WSL access Windows-side Docker if it's installed.
+  if [[ -f '/mnt/c/Program Files/Docker/Docker/DockerCli.exe' || -f '/mnt/c/Program Files/Docker Toolbox/docker.exe' ]] ; then
+      # Unset minikube variables that might make vagrant angry.
+      unset DOCKER_TLS_VERIFY
+      unset DOCKER_HOST
+      unset DOCKER_CERT_PATH
+      unset DOCKER_API_VERSION
+
+      # Set variables to allow the docker service running in Windows to work in WSL.
+      export DOCKER_HOST=tcp://127.0.0.1:2375
+      export VAGRANT_DEFAULT_PROVIDER="docker"
+  fi
+else
+  echo "Not a WSL environment."
+fi

docs/host_side_file_management.md

@@ -11,7 +11,7 @@ Host default
   UserKnownHostsFile /dev/null
   StrictHostKeyChecking no
   PasswordAuthentication no
-  IdentityFile /home/$USER/Projects/vagrant/twlight_vagrant/.vagrant/machines/default/virtualbox/private_key
+  IdentityFile /home/$USER/.vagrant/.twlight_vagrant/machines/twlight/docker/private_key
   IdentitiesOnly yes
   LogLevel FATAL
   ForwardAgent yes
@@ -38,13 +38,13 @@ where $USER is your linux username and $somevalue is not consistent across insta
 To that path, add the path to the private key for the TWLight Vagrant machine that gets created upon "vagrant up." For example
 
 ```
-Projects\vagrant\twlight_vagrant\.vagrant\machines\default\virtualbox\private_key
+.vagrant\.twlight_vagrant\machines\twlight\docker\private_key
 ```
 
 In this case, the full path to your TWLight Vagrant machine's IdentityFile would be something like:
 
 ```
-%localappdata%\Packages\CanonicalGroupLimited.UbuntuonWindows_$somevalue\LocalState\rootfs\home\$USER\Projects\vagrant\twlight_vagrant\.vagrant\machines\default\virtualbox\private_key
+%localappdata%\Packages\CanonicalGroupLimited.UbuntuonWindows_$somevalue\LocalState\rootfs\home\$USER\.vagrant\.twlight_vagrant\machines\twlight\docker\private_key
 ```
 
 ## FileZilla example (Linux, MacOS, or Windows)

etc/.vimrc

@@ -0,0 +1,14 @@
+" The following are commented out as they cause vim to behave a lot
+" differently from regular Vi. They are highly recommended though.
+syntax on
+set showcmd            " Show (partial) command in status line.
+set showmatch          " Show matching brackets.
+set ignorecase         " Do case insensitive matching
+set smartcase          " Do smart case matching
+set incsearch          " Incremental search
+set autowrite          " Automatically save before commands like :next and :make
+set hidden             " Hide buffers when they are abandoned
+set mouse=r            " Enable mouse usage (all modes)
+set tabstop=4 softtabstop=0 expandtab shiftwidth=2 smarttab
+set ruler
+set background=dark