0 comments on “Github Pages with TLS and a Backend”

Github Pages with TLS and a Backend

Blogging consistently is hard, but we’re redoubling our efforts to get content out semi-regularly, even if it’s just to post something simple and (hopefully) helpful. To that end, I’d like to discuss setting up Github pages with a backend.

The Problem


Github.com allows you to host static content via its pages feature.  This is fantastic because it makes it trivial to create, deploy, and update a website.  This is how https://sunshower.io is hosted.

But what if you wanted to interact with, say, a database, to track signups?  Furthermore, what if you wanted to do it all over TLS?  This tutorial presumes that your registrar is AWS and your DNS is configured through Route53.


Set up Github DNS Alias Records For Github Pages

This one’s easy: In your Route53 Hosted Zone, create an A record that points to Github Page’s DNS servers:







Then, check in a file in your pages repository under /docs called CNAME containing your DNS name (e.g. sunshower.io)


Push that sucker to master and you should have a bouncing baby site!


Publish your backend with the correct CORS Preflights

We pretty much just have a plugin for Sunshower.io that registers unactivated users.  Create an EC2 webserver/Lambda function/whatever to handle your requests.  The thing you have to note here is that your backend will have to support preflight requests.  A preflight request is an OPTIONS request that your server understands, and responds with the set of cross-origin resource sharing (CORS​ ) that your backend understands.

This is because your page, hosted at a Github IP, will be making requests to Amazon IPs, even though both are subdomains beneath your top-level domain.  For a JAX-RS service at, say, https://mysite.com:8443/myservice, you will need two methods:


public interface SignupEndpoint {

RegistrationConfirmationElement signup(RegistrationRequestElement request);
Response getOptions();

Note that there must be an @OPTIONS method for each actual conversation method that you want to interact with (e.g. the @POST method here).  What will happen is that a preflight will be made to the same path as the request, and the server will respond with whether that request is allowed.  You can widen the scope of @OPTIONS responses, but you should have a pretty good reason for doing so.


The actual @OPTIONS method will look something like:

public Response getOptions() {
return Response.status(Response.Status.OK)
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Methods", "POST, OPTIONS")
.header("Access-Control-Max-Age", 1000)
.header("Access-Control-Allow-Headers", "origin, x-csrftoken, content-type, accept")

where the allow value and Access-Control-Allow-Methods values align with the request type of the actual backend methods.


Set up a SSL-enabled ELB

The first thing to do here is to create a certificate using AWS’s wonderful Certificate Manager Service.

Request a certificate by navigating to Certificate Manager > Request a Certificate in the AWS Management Console.  Request a public certificate, and then request it for the fully-qualified name that you want your backend to appear under (e.g. backend.sunshower.io).  If you have access to any of the follow e-mail addresses at your domain:

  • administrator@your_domain_name
  • hostmaster@your_domain_name
  • postmaster@your_domain_name
  • webmaster@your_domain_name
  • admin@your_domain_name

then select e-mail verification, otherwise select DNS verification and proceed according to the instructions provided by AWS.


Once you’ve verified your certificate, create an elastic load balancer on whatever availability zones your backend spans.  If you only have one instance in one availability zone (shame shame!), add that availability zone first.  Create a listener targeting the port that your actual backend is listening on, and add a security group with an ingress port that corresponds to the port that you want your public-facing backend to listen on (for instance, if you want your backend to respond to requests to https://backend.sunshower.io, configure your ingress to be 443.

From there, configure a target that points to your actual backend. If your backend is hosted on EC2 instances, select the instance type for the target, otherwise select ip.  Reference the IP/instances of your backend and create the ELB.


Configure a DNS A Record for your ELB

The last thing we need to do is create an A Record that points to your ELB.  If, previously, you requested a TLS certificate for backend.mysite.com, you’ll want to create an A-record whose name is backend, alias=true, with an alias target that is the ELB you created in the previous step.  Save that sucker and you are good to go!




0 comments on “Announcing Sunshower.io Preview!”

Announcing Sunshower.io Preview!

We’re thrilled to announce a preview launch of Sunshower.io. I’d like to take a little time to explain what we do!

Beautifully Simple Multicloud Management

The first thing we do is provide a simple, unified interface for managing your public clouds. This means that you provide some read-only access to your cloud, which we store securely in our vault, and then we go discover whatever infrastructure is in your cloud(s) and manage and organize it for you.

Let me step through a quick example:

Discover your Resources (aka “Systems”)

The first thing we need to do is to discover your resources. Upon logging into Sunshower.io, you’ll be presented with the System Discovery Wizard. A System (i.e. Weather System) is the set of all infrastructure associated with a set of cloud accounts. So, if you have:

  • 1 Azure Scale Set with 4 active members that is your development cluster (azure-dev)
  • 1 Azure Scale Set with 20 active members that is your production cluster (azure-prod)
  • 1 AWS Autoscaling Group with 10 active members that is your AWS dev cluster (aws-dev)
  • 1 AWS Autoscaling Group with 30 active members that is your AWS production cluster (aws-prod)

Then you will create a system with at least 2 credentials, one for AWS, the other for Azure.


You can add as many accounts from as many cloud providers as you want. Each of the cloud providers is implemented as a (relatively) simple plugin, and if you add a new plugin (e.g. Google Cloud), you’ll be able to add credentials for that cloud, too.

Once all your accounts have been added, we’ll go through and perform the actual discovery. Once that completes, you’ll be presented with a topological overview of your cloud infrastructure:


The group color is used in the topology view as the color between edges of nodes, and in the geography view to color connections between regions:


And yes, you can totally spin the globe!

Use Your Groups

Grouping is pretty fundamental to Sunshower.io:

  • Access control is based on groups
  • Management operations can be performed on entire groups (yes, you can spin down every node in a cluster by stopping its group)
  • Deployments are based on groups

For instance, you can SSH into an individual machine or an entire group. It’s pretty typical to have a ton of identical machines, so executing the same series of commands produces identical output from each machine. We de-duplicate the output and provide you with the results. Just store your private key in our vault (the actual key is stored in the excellent HashiCorp Vault). But, hey, tailing logs across a bunch of machines has never been so easy!


Visual Deployments (aka “Strata”)

One feature that we’re really excited about is visual modeling of deployments. Basically, you start off with a series of commands (e.g. shell commands), and you compose them together:


From there, you can select a deployer format (e.g. Docker or Packer or even just the userdata section of an AMI, whatever you have plugins installed for) and viola! You have a deployment that you can share with coworkers, publish to everyone, or share with a specific group. For instance, you might create a Stratus that

  • Installs Java
  • Installs NodeJS
  • Installs Gulp

And you can generate a Dockerfile for it, a Packer file, an Azure Image, or an AMI without changing a thing!

Visual Infrastructure Modeling for Systems

But let’s say you don’t have any existing infrastructure and you want to create some. Now, you can log into your various cloud providers’ consoles and spin up whatever you need, but what if you want to create some infrastructure and try it out across clouds? Enter our visual infrastructure modeling for Systems. You can quickly model your infrastructure and deploy it out to any supported cloud (below is the cluster structure for Sunshower.io’s deployment)


You can also export the model to a variety of provisioners like AWS CloudFormation, Azure Resource Manager, or Hashicorp Terraform. We’re currently figuring out what generating Kubernetes manifests looks like, but you’ll be able to do that soon too.

Finally, Anvil for Optimization

All of the data that we collect about your infrastructure we use to build a model of your tasks. If you think about purchased infrastructure as a shipping container, then it makes sense to purchase the smallest shipping container that can fit all of your packages. Anvil extends this analogy by allowing you to define new dimensions for your packages (think 5 or more), and we’ll figure out the smallest shipping container with those dimensions possible across any infrastructure for which there’s an infrastructure plugin. For instance, here’s an example of us spinning up a suboptimal configuration of resources and running Anvil on it:


It yields a much more compact (dense) configuration for your infrastructure. In fact, it typically yields an optimal configuration. You can even configure it to model packages based off of their greatest historical dimensions (like peak hours) so that you’ll never under-provision again, even while saving a substantial amount on your infrastructure.


Thanks for sticking around! I wanted to provide a list of Sunshower.io’s current features to give everyone a better idea as to how it’s used and what it can do for you. Many people will only need one or two of the features, and we’d like to get some feedback as to which might be the most valuable for your organization so that we can get them to you ASAP.

0 comments on “Installing OpenShift on ESXi and CentOS 7”

Installing OpenShift on ESXi and CentOS 7

At Sunshower, we’ve been happily using Docker Compose and Docker Swarm for development and deployment respectively. These technologies make it a snap to build and deploy code, and the effort involved in setting them up is quickly offset by their utility.

We’ll continue to use Compose for development, but for better or worse, the industry has spoken and declared Kubernetes the winner of the container orchestration wars. And by this I mean that Swarm is not offered as a service by any of the major CSPs, but each of them either have or are working towards offering turnkey Kubernetes offerings. So, no sense in swimming upstream. We decided to create a Kubernetes deployment for Sunshower. However, a lot of the Kubernetes public cloud offerings are relatively expensive, which is a barrier for a self-funded startup like Sunshower. Fortunately, we got a very generous donation of hardware that I deployed OpenShift to.

Kubernetes vs. OpenShift

OpenShift is pretty much Kubernetes with some extras that make it extra attractive if you’re going to manage your own infrastructure. We’re using it because it also supplies builds.

Getting Started

Configuring your infrastructure

Configuring your infrastructure is the most tedious and error-prone part of this exercise, but if you don’t get it right, it will bite you.

Infrastructure step 1: Create a base VM image

Download a CentOS ISO (minimal is fine) to ISO_PATH where ISO_PATH is some accessible location on your local hard drive. I was not able to get uploads to work with the ESXi web client, so you’ll need to use the older ESXi thick client. Select the following options:

  1. Hardware Compatibility (only if you’re interacting with ESXi through VMWare Workstation): Workstation 11.x
  2. Installer disc image (iso): ISO_PATH
  3. Virtual Machine Name: centos-base (or whatever)
  4. Processors: (at least 2 are required, whether that’s 2 processors, 1 core/processor or whatever)
  5. 8GB memory
  6. Network type: Bridged !important
  7. Defaults for I/O Controller types, disk type, select a disk
  8. Disk capacity: 50 GB

Once your VM is up and running, run:

yum install update 
reboot now

When your VM comes back up, install some basics:

yum install -y open-vm-tools git docker wget 
systemctl stop network #if you're ssh\'d in, you'll lose access.  Do this through the VMWare console
chkconfig network off
chkconfig NetworkManager on
systemctl NetworkManager start
nmcli dev connect ens33 # You might have a different bridged interface name--check by running nmcli

Infrastructure step 2: Create the VM inventory

This is a bit of a chore since ESXi doesn’t even let you effing clone a VM. If you’re using Workstation, I’d recommend creating all the clones locally, then upload them to the ESXi host.

If you’re not using VMWare Workstation, manually clone each VM by:

  1. Creating the base VM (previous step)
  2. Browse the datastore you want to clone a VM to, create a new folder with the VM’s name (e.g. openshift-cluster-manager)
  3. Copy every file from the base VM’s directory to the new folder except for the log files

For our installation, we’ll have 1 master and 3 workers. If you need HA, you need 3 or 5 masters and however many workers. If you need a production-ready cluster with dozens or hundreds of workers, I do consult =).

Infrastructure step 2: assign static IPs

If you’re using OpenStack/AWS/vSphere, or are running your own DNS server, this is an optional step. Since ESXi does not have any available mechanism for dynamically deploying new virtual machines, your installation will be pretty static, so pick a naming convention, pick an adequate network size, and assign each cluster-node’s interface MAC to an IP. This obviously doesn’t scale well, but eh, it’s fine for a local cluster. One day I’ll show you how to create something a little less hands-on with OpenStack.

Install OpenShift using the convenient installer script provided by Grant Shipley

Select an initiator node. This node will have the role of APIserver, so select one of your masters (or your only master). Clone (Grant Shipley’s installer script)[https://github.com/gshipley/installcentos] with

git clone https://github.com/gshipley/installcentos

Install it with:

echo "console.yourdomain.com" >> /etc/hosts

export DOMAIN=yourdomain.com
export USERNAME=<username> #maybe administrator or something
export PASSWORD=<some super strong password>
cd installcentos && ./install-openshift.sh

This takes a while. Go get some coffee. Take a walk. If your installation node does not have a static IP, it will probably change due to the network reconfiguration this step performs and hork your installation.

If it completes successfully, visit console.yourdomain.com and you should see:


Install your cluster nodes

From your initiator node, copy your ssh key to each of the nodes you want to add to the cluster:


for i in "${hosts[@]}" do
    ssh-copy-id root@$(i)

Otherwise you’ll be typing a lot of passwords.

On the initiator node, edit /etc/ansible/hosts:


openshift_deployment_type=origin  # if you bought enterprise, this would be enterprise
os_sdn_network_plugin_name=redhat/openshift-ovs-multitenant  #Important!  The installer defaults to openshift-ovs-subnet for the nodes, but the master is running multitenant.  The openshift node process will fail to start without this
os_firewall_use_firewalld=true  #iptables sucks
osm_cluster_network_cidr=  #change to whatever your network is
openshift_metrics_install_metrics=true  #optional

console.sunshower.io  #or whatever your current hostname is

console.sunshower.io  # production installations would have several of these

os1.sunshower.io        openshift_schedulable=true  # All the DNS names of your nodes.  Make sure that they're either in your /etc/hosts file or your DNS server is correctly configured
os2.sunshower.io        openshift_schedulable=true
os3.sunshower.io        openshift_schedulable=true

Then, run:

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openshift-node/scaleup.yml. These playbooks are installed by atomic-openshift-utils. After about 10 minutes, the installation process should complete and you should be able to run:

<br />oc get nodes
NAME        STATUS    ROLES            AGE       VERSION   Ready     compute,master   1h        v1.9.1+a0ce1bc657    Ready     <none>           48m       v1.9.1+a0ce1bc657    Ready     <none>           48m       v1.9.1+a0ce1bc657    Ready     <none>           47m       v1.9.1+a0ce1bc657
[root@localhost ~]# kubectl get nodes
NAME        STATUS    ROLES            AGE       VERSION   Ready     compute,master   1h        v1.9.1+a0ce1bc657    Ready     <none>           48m       v1.9.1+a0ce1bc657    Ready     <none>           48m       v1.9.1+a0ce1bc657    Ready     <none>           48m       v1.9.1+a0ce1bc657

EDIT: WordPress’s Markdown escaping is pretty broken–sorry for all the HTML escape codes–I’ll fix them when I get a moment.

0 comments on “DevOps Without the DevOps Part 4: Finally, Containers!”

DevOps Without the DevOps Part 4: Finally, Containers!

In previous segments, we discussed how to collect project dependencies, and use Maven, Gradle, and the Spring Maven Gradle plugin to organize your project dependencies in a maintainable and traceable fashion. In this post, we’re going to take that setup and create a clean, reproducible build using Docker.

Understanding Containerization

A containerized application’s lifecycle is composed of several steps:

  1. Defining the container
  2. Building the container
  3. Tagging the container
  4. Pushing the container to a registry
  5. Running the container

We’ll be using Docker as our containerization technology, but it’s not the only option. Ensure that it’s installed for your platform before you continue.

Note that these instructions are intended for Linux-based machines. Containers differ from VMs in a very important way, namely that containers share the OS kernel of their container host.

What this means is that, say that my host operating system is Debian, based on Linux Kernel x86_64. Even if my container uses a different OS base (say, Ubuntu), the container and the host should be kernel-compatible (so, no Linux Kernel 3.10.x-ARM, for instance.)

These considerations are rarely an issue for many software projects. We use Debian-based containers because they’re similar to our dev environments and tooling, but Alpine is a really good option for many projects and has some advantages in production (e.g. resource utilization is anecdotally better.)

Step 1: Defining the Container

In the same way that we identified the compile and runtime dependencies of our project in the first series, let’s think about what we need to build the project. On my development system, I typically build a project by building and publishing the project’s Bill-Of-Materials POMs (mvn clean install -f bom), then by building the project’s artifacts with Gradle and publishing them to my local Maven repository (gradle clean build publishToMavenLocal, or, with Gradle’s nifty shortcuts gradle cle b pTML). So, I obviously need Gradle and Maven installed in my container.

So, what we’re going to do here is:

  1. Install the correct version of Java
  2. Download Maven from the Maven project site and install it into the container
  3. Download Gradle from the Maven project site and install it into the container

Let’s see if we can get rid of any of these steps by selecting the correct base container. Searching Docker hub, I see that Oracle provides JDK images. openjdk:8u141-jdk

My Dockerfile becomes simply:

FROM openjdk:8u141-jdk
ENTRYPOINT /bin/bash

Breaking down these instructions:

FROM openjdk:8u141-jdk says, “ask the local Docker registry to find an image with ID openjdk:8u141-jdk and derive from that. If you can’t find that image locally, reach out to hub.docker.com and see if it’s there`

Building the container pulls the base image, then executes all of the commands in the Dockerfile, which produces a new container:

Sending build context to Docker daemon  220.7kB
Step 1/2 : FROM openjdk:8u141-jdk
8u141-jdk: Pulling from library/openjdk
3e17c6eae66c: Already exists 
74d44b20f851: Already exists 
a156217f3fa4: Already exists 
4a1ed13b6faa: Already exists 
77980e5d0a6d: Already exists 
5458607a81d3: Already exists 
e34cf8338f42: Already exists 
2f3d3da5c56e: Already exists 
2ade7a861e3f: Already exists 
Digest: sha256:4b0c879909b729d67d13e5004f5564df85a5f9c1c3820c13e41151edf1f1b1c0
Status: Downloaded newer image for openjdk:8u141-jdk
 ---> 74c95c985a85
Step 2/2 : ENTRYPOINT /bin/bash
 ---> Running in a4bd5943abcd
Removing intermediate container a4bd5943abcd
 ---> b1c22add1692
Successfully built b1c22add1692 # <-- IMPORTANT, this is your container ID, referenced as $CID.

We can check this new container out by running docker run -it --rm $CID which will drop you into a shell that looks something like:


Now, most base containers don’t have many programs installed. The JDK base containers do have Java, which is the important one.


Now, we can install Maven and Gradle really quickly.

Installing the Prerequisites

I like wget for simple downloads, but curl would work just as well. But we need to install Git anyway, so add:

RUN apt-get update
RUN apt-get install -y git-core wget

To your Dockerfile and build:

Sending build context to Docker daemon  220.7kB
Step 1/4 : FROM openjdk:8u141-jdk
 ---> 74c95c985a85
Step 2/4 : RUN apt-get update
 ---> Running in 2bfa3d396ac6
Get:1 http://security.debian.org stretch/updates InRelease [94.3 kB]
Ign:2 http://deb.debian.org/debian stretch InRelease
Get:3 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:4 http://deb.debian.org/debian stretch Release [118 kB]
Get:5 http://deb.debian.org/debian stretch Release.gpg [2434 B]
Get:6 http://deb.debian.org/debian stretch-updates/main amd64 Packages [12.1 kB]
Get:7 http://deb.debian.org/debian stretch/main amd64 Packages [9530 kB]
Get:8 http://security.debian.org stretch/updates/main amd64 Packages [468 kB]
Fetched 10.3 MB in 1s (5807 kB/s)
Reading package lists...
Removing intermediate container 2bfa3d396ac6
 ---> 4984ada9d0c8
Step 3/4 : RUN apt-get install -y git-core wget
 ---> Running in 6e6a79b1c1ab
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
The following packages will be upgraded:
1 upgraded, 1 newly installed, 0 to remove and 64 not upgraded.
Need to get 801 kB of archives.
After this operation, 8192 B of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 wget amd64 1.18-5+deb9u1 [800 kB]
Get:2 http://deb.debian.org/debian stretch/main amd64 git-core all 1:2.11.0-3+deb9u2 [1410 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 801 kB in 0s (3242 kB/s)
(Reading database ... 29522 files and directories currently installed.)
Preparing to unpack .../wget_1.18-5+deb9u1_amd64.deb ...
Unpacking wget (1.18-5+deb9u1) over (1.18-5) ...
Selecting previously unselected package git-core.
Preparing to unpack .../git-core_1%3a2.11.0-3+deb9u2_all.deb ...
Unpacking git-core (1:2.11.0-3+deb9u2) ...
Setting up wget (1.18-5+deb9u1) ...
Setting up git-core (1:2.11.0-3+deb9u2) ...
Removing intermediate container 6e6a79b1c1ab
 ---> 67169937ccdd
Step 4/4 : ENTRYPOINT ["/bin/bash"]
 ---> Running in cd4e07152d23
Removing intermediate container cd4e07152d23
 ---> 2c1ab0b17981
Successfully built 2c1ab0b17981

Now, your container will have both wget and git installed.

Setting Environment Variables

If you define an ENV variable in Docker, the value of that ENV can either be passed into the container, or you can specify a default value (or both). We want to be able to reference (and change) both the Gradle version and the Maven version so that if we want to upgrade either, we just pass in new versions when we’re building the container and viola!

# Environment Variables
ENV BASE_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Now, when we run the container, we have access to those environment variables:


Install Gradle

RUN mkdir -p /opt/build/tools/gradle #Create directory for gradle
RUN wget https://services.gradle.org/distributions/gradle-$GRADLE_VERSION-bin.zip -O /opt/build/tools/gradle.zip  # Download gradle from gradle.org 
RUN unzip -d /opt/build/tools/gradle /opt/build/tools/gradle.zip # Unzip gradle 
ENV GRADLE_HOME=/opt/build/tools/gradle/gradle-$GRADLE_VERSION/bin #Export gradle location as GRADLE_HOME

Now, one really sweet thing about Docker is that, each of these commands defines a new layer. If the textual value of the command that builds a layer doesn’t change, then that layer is retrieved from the layer cache upon subsequent builds. What this means is that the URL for the Gradle download could change, and that wouldn’t break our container.

Install Maven

RUN mkdir -p /opt/build/tools/maven
RUN wget http://www-eu.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-$MAVEN_VERSION-bin.zip \
-O /opt/build/tools/maven.zip
RUN unzip -d /opt/build/tools/maven /opt/build/tools/maven.zip
ENV MAVEN_HOME=/opt/build/tools/maven/apache-maven-$MAVEN_VERSION/bin

Once these have all executed, we have a container with Maven, Gradle, and Git installed and ready to use! In the next post, I’ll discuss how to store credentials securely and pass them into the container so that we can pull our project from source-control and build it.

Of course, if you don’t want to maintain your own, this base image is available on docker hub as sunshower/sunshower-base:latest

0 comments on “Server-Sent Events with Undertow, Spring 5, and Resteasy 4”

Server-Sent Events with Undertow, Spring 5, and Resteasy 4

Resteasy 3.5 introduced Server-Sent Events (SSE), and there weren’t any good resources for showing how to get it up-and-running, so I thought I’d put together a quick how-to guide.

Add your dependencies

Up your org.jboss.resteasy:resteasy-jaxrs version to 4.0.0.Beta2. This should automatically pull in a JAX-RS API version 2.1 unless you’ve specified a version of JAX-RS directly. If you have, upgrade to 2.1. This will provide access to the SSE component of JAX-RS 2.1 (SseEventSink, Sse, etc.).

Define an SSE method

public interface TestService {

  void subscribe(@PathParam("id") String id, @Context SseEventSink sink, @Context Sse sse);// It's ok to put JAX-RS annotations on the interface.  Recommended, in fact.

  TestEntity save(TestEntity testEntity);

  String call(@PathParam("value") String input);

Implement that biz:

  public void subscribe(String id, SseEventSink sink, Sse sse) {
        new Thread(
            () -> {
              try {
                        .data(String.class, "starting domain " + id + " ...")
                sink.send(sse.newEvent("domain-progress", "50%"));
                sink.send(sse.newEvent("domain-progress", "60%"));
                sink.send(sse.newEvent("domain-progress", "70%"));
                sink.send(sse.newEvent("domain-progress", "99%"));
                sink.send(sse.newEvent("domain-progress", "Done."))
                        (Object obj) -> {
              } catch (final InterruptedException e) {

Write a test-case:

  void ensureSseWorks() throws InterruptedException {

    ResteasyWebTarget path =
        ((ResteasyWebTarget) webTarget).path(TestService.class).path("1/events");
    SseEventSource source =
        SseEventSource.target(path).reconnectingEvery(10, TimeUnit.SECONDS).build();
    try (SseEventSource s = source) {
          e -> {

Experience failure

Failure 1

If you’ve done all that, it won’t work. The first error you’ll get is that your @Context SseEventSink is null. Unhork yourself by adding

  public SseEventSinkInterceptor sseEventSinkInterceptor() {
    return new SseEventSinkInterceptor();

to your Spring configuration.

Failure 2

The second failure you’ll experience is on the client-side: No MessageBodyReader for “text/event-stream”. This is fixed by adding a

  public SseEventProvider sseEventOutputProvider() {
    return new SseEventProvider ();

to your Spring client configuration.

Experience success

Ahh, delicious success

starting domain 1 ...


Plugging Sunshower again, you can get all this goodness by annotating your test-class with @io.sunshower.test.ws.EnableJAXRS if you have io.sunshower.test:test-ws:1.0.0-SNAPSHOT as a dependency.

0 comments on “Unit-Testing Complex External Dependencies”

Unit-Testing Complex External Dependencies

The question of how to unit-test complex external dependencies arises pretty frequently. This is near-and-dear to our hearts because we have many of them. Mocking out complex responses is tedious and error-prone, so I’ll tell you what we do instead.

1: The setup

Stratosphere has an contract for obtaining instances from a cloud service provider, viz.,

public interface ListInstancesOperation extends ProviderOperation<List<Instances>> {
      public List<Instance> perform();

      public Provider getProvider(); 
      public Secret() getSecret();


perform() will typically interact with the provider service endpoint. The body of perform is quite simple:

  public List<Instance> perform() {
    AmazonEC2 client = createClient();

    DescribeInstancesResult result = client.describeInstances();
    return toInstances(result.getReservations());

This doesn’t look like a super-testable method, so what do we do?

2: Get the actual response and write it to a file

Before you write any code that interacts with an external dependency, you have to understand how that dependency behaves. I recommend keeping a set of IAM credentials in ~/.aws/credentials for testing. Once you do that, actually perform the request and see what you get back.


Now, we serialize the response using Java’s default serialization mechanism to a file that we check into source-control.

DescribeInstancesResult result = client.describeInstances();
Objects.write(result, relativeToRoot("src/test/resources/ec2/list-instances.obj")); // our utilities for writing an object using serialization.

3: Mock it real good

Recall that our method under test had 3 statements. One to create the client, one to make the request to the client, and one to map the results. The client’s method under test is public, so we can mock that. We gave createClient default visibility so that we could mock that while not exposing it as part of our Operation API, and then we put all our actual logic into a private method, whose behavior we want to test.

We can now set up our tests to mock out the external operation with a real result:

  private Secret secret;
  private AmazonEC2Client client;
  private EC2ListInstancesOperation operation;

  void setUp() {
    secret = new Secret();
    operation = new EC2ListInstancesOperation(secret, "us-west-2", (AWS) AWS.getInstance());
    operation = spy(operation);

    client = mock(AmazonEC2Client.class);
        .willReturn(Objects.read("src/test/resources/ec2/list-instances.obj", true));

And test it as follows:

  void ensureInstanceFirewallsAreCorrect() {
    List<Instance> perform = operation.perform();
    Instance instance = perform.get(0);
    assertThat(instance.getFirewalls().size(), is(1));
    Firewall firewall = instance.getFirewalls().iterator().next();
    assertThat(firewall.getName(), is("launch-wizard-2"));
    assertThat(firewall.getSecured().contains(instance), is(true));

This approach works pretty well for external dependencies that don’t change frequently (like the public APIs of large cloud service providers), but less well for external dependencies that do. What are some approaches you use?

0 comments on “Mapping Many Properties Quickly”

Mapping Many Properties Quickly

One of the challenges in writing abstraction software like Sunshower is that we have to map a ton of vendor-specific properties into our data-model. I had considered writing a mapping language to transform, say, an EC2 Instance or Azure VM into one of our generic Sunshower compute instances, but decided against it because we just dump all the generic vendor properties like spotInstanceRequestId into properties on our internal model analog, and writing a compiler to do that is for big companies.

But how to avoid individually mapping each darn property by hand? Writing code like:

        new Property(

for each of exactly 100 hojillion properties across AWS’s and Azure’s and GCE’s and etc. data model is a breeding ground for ennui and bugs.

Attempt 1

Java 7 introduced the Beans API which makes introspecting Java beans super easy. There’s also a little-known feature of Java Matcher.replaceAll/First that allows you to reference a regular-expression capture group in the replacement string, so I whipped up:

PropertyDescriptor[] propertyDescriptors =
        Introspector.getBeanInfo(com.amazonaws.services.ec2.model.Instance.class, Object.class)

                t -> {
                  Property.Type type = resolveType(t.getPropertyType());
                  if (type == null) {
                    return Stream.empty();
                  } else {

                    Object value = ReflectionUtils.invokeMethod(t.getReadMethod(), ec2instance);

                    return Stream.of(
                        new Property(
                            t.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase(),
                                + t.getDisplayName()
                                    .replaceAll("(.)(\\p{Upper})", "$1-$2")
                            value == null ? null : String.valueOf(value)));

Which got me what I wanted:

            t -> {
                      "Name: %s, key: %s, value: %s", t.getName(), t.getKey(), t.getValue()));
Name: role, key: role, value: io.sunshower.stratosphere.core.topology.model.Instance
Name: aws.i18n.architecture, key: architecture, value: x86_64
Name: aws.i18n.client-token, key: client-token, value: sunsh-WebSe-1KKFOLRIA853V
Name: aws.i18n.ebs-optimized, key: ebs-optimized, value: false
Name: aws.i18n.ena-support, key: ena-support, value: true
Name: aws.i18n.hypervisor, key: hypervisor, value: xen
Name: aws.i18n.image-id, key: image-id, value: ami-39595240
Name: aws.i18n.instance-id, key: instance-id, value: i-05b1e0984260d51dd
Name: aws.i18n.instance-lifecycle, key: instance-lifecycle, value: null
Name: aws.i18n.instance-type, key: instance-type, value: t2.micro
Name: aws.i18n.kernel-id, key: kernel-id, value: null
Name: aws.i18n.key-name, key: key-name, value: sunshower-io
Name: aws.i18n.platform, key: platform, value: null
Name: aws.i18n.private-dns-name, key: private-dns-name, value: ip-172-31-12-114.us-west-2.compute.internal
Name: aws.i18n.private-ip-address, key: private-ip-address, value:

But it wasn’t very pretty or maintainable. So, refactoring:

Attempt 2

public class Properties {

  public static <T> void map(
      Class<T> type,
      T instance,
      Class<?> bound,
      PropertyAwareObject<?> target,
      PropertyMappingConfiguration cfg)
      throws IntrospectionException {

    PropertyDescriptor[] propertyDescriptors =
        Introspector.getBeanInfo(type, bound).getPropertyDescriptors();

            .flatMap(t -> cfg.map(t, instance))

public class ProviderPropertyMappingConfiguration implements PropertyMappingConfiguration {

  private final String prefix;

  public ProviderPropertyMappingConfiguration(String prefix) {
    this.prefix = prefix;

  public Property.Type resolveType(PropertyDescriptor descriptor) {

    Class<?> propertyType = descriptor.getPropertyType();

    if (Boolean.class.equals(propertyType)) {
      return Property.Type.Boolean;
    if (isIntegral(propertyType)) {
      return Property.Type.Integer;
    if (String.class.equals(propertyType)) {
      return Property.Type.String;
    return null;

  public boolean accept(PropertyDescriptor propertyDescriptor) {
    Class<?> propertyType = propertyDescriptor.getPropertyType();
    return Boolean.class.equals(propertyType)
        || String.class.equals(propertyType)
        || isIntegral(propertyType);

  public String mapKeyName(PropertyDescriptor descriptor) {
    return descriptor.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase();

  public String mapName(PropertyDescriptor descriptor) {
    return prefix
        + ".i18n."
        + descriptor.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase();

  public String mapValue(PropertyDescriptor propertyDescriptor, Object instance) {
    Object result = ReflectionUtils.invokeMethod(propertyDescriptor.getReadMethod(), instance);
    return result == null ? null : String.valueOf(result);

  private boolean isIntegral(Class<?> propertyType) {
    return Integer.class.equals(propertyType)
        || int.class.equals(propertyType)
        || long.class.equals(propertyType)
        || Long.class.equals(propertyType);

  public <T> Stream<Property> map(PropertyDescriptor propertyDescriptor, T instance) {
    Property.Type type = resolveType(propertyDescriptor);
    return type == null
        ? Stream.empty()
        : Stream.of(
            new Property(
                mapValue(propertyDescriptor, instance)));

Allowing us to easily map any properties:

Name: role, key: role, value: io.sunshower.stratosphere.core.topology.model.Instance
Name: aws.i18n.ami-launch-index, key: ami-launch-index, value: 0
Name: aws.i18n.architecture, key: architecture, value: x86_64
Name: aws.i18n.client-token, key: client-token, value: sunsh-WebSe-1KKFOLRIA853V
Name: aws.i18n.ebs-optimized, key: ebs-optimized, value: false
Name: aws.i18n.ena-support, key: ena-support, value: true
Name: aws.i18n.hypervisor, key: hypervisor, value: xen
0 comments on “DevOps without the DevOps part 3: Don’t Buy the Hype”

DevOps without the DevOps part 3: Don’t Buy the Hype

I’ve been asked whether this series really pertains to DevOps. The criticism, as I interpret it, is that DevOps is a conceptual framework for creating high-functioning development teams, whereas the concepts and processes outlined here are related to creating a build with specific tools.

The point at hand seems to be that DevOps as it’s practiced by its adherents and evangelists provides a universal template for whatever ails your development organization. In the DevOps utopia, managers, PMs, engineers, and QA all coexist in complete harmony to produce reliably phenomenal software on time and within the budget. Only, that’s not right, engineers and QA are coalesced in this other interpretation. And managers and PMs? Isn’t that a lot of overhead for healthy, DevOps practicing shops? In this construction, DevOps can become whatever you want it to be, or feel like it should be, because it has no grounding in actual processes.

What I’m getting at is that DevOps can’t be some abstract “template” for “building software factories.” There’s no such thing. What there is is a collective knowledge built of decades of trial and error, and everyone who tells you otherwise is trying to sell you something that you probably don’t need. Sure, containers are great. Kubernetes is great. CI/CD software is probably pretty solid. But an uncomfortable truth is that you can get every bit as good of results with Make, Cron, and Bash because the important thing in writing software is the knowledge of how it all fits together.

And that’s what this series is about: taking common tools and sharing our experience with them and what works well for us, and what doesn’t. You can derive some pretty generalizable truths from this process, like “You need a sane dependency-management process. Here’s what one looks like. There are other valid processes. The important thing is to have one.”

0 comments on “Unmarshalling generic properties with custom logic in JAXB.”

Unmarshalling generic properties with custom logic in JAXB.

I’m not entirely sure the idea behind this post will ever be useful to anyone, and certain as only a reformed sinner can be that it is not wise. Originally we had a polymorphic JAXB Property type, defined as:

public class PropertyElement<U, T extends PropertyElement<U, T>> extends AbstractElement<T> {

  @XmlAttribute String key;

  @XmlAttribute String name;

  @XmlAnyElement(lax = true)
  private U value;


Where we wanted private U value to be overridable by any subclass, provided that the actual runtime type of U exposed a public, static valueOf method that accepted a string and returned the value represented by the string. We did this because it worked for every Java primitive, plus most of the value types that I could think of. Naturally, this didn’t really work well with anything, so we just created an enumeration of possible types and forsook the notion of dynamically registering new property elements.

Abandoning this was a slightly bitter pill to swallow as it somewhat constrained the extensibility of Sunshower’s core data-model. But meh I tell you, because I spent hours fruitlessly and foolishly attempting to overcome the profound limitations of Java’s erased generics, only for my folly to haunt me for weeks longer. However, should some soul wish to soldier past, here’s what we did:

public class PropertyElement<U, T extends PropertyElement<U, T>> extends AbstractElement<T> {

  @XmlAnyElement(lax = true)
  private U value;

  private static transient Method valueOf;

I can hear some distant screaming. Possibly it’s me. That is verily a reference to a java.lang.reflect.Method It gets worse. The excellent EclipseLink MOXy library which we prefer over Jackson due to, you know, standards, allows us to define a void afterUnmarshal(Unmarshaller u, Object parent) method, which is invoked after MOXy has finished unmarshalling the object at hand. We figured that we didn’t need to write the actual value since, at runtime, its type is known to MOXy and JAX-RS, and sure enough we didn’t. Reading the object, on the other hand…(hangs head in shame):

  protected void doUnmarshal() {
    try {
      if (valueOf != null) {
        this.value = (U) valueOf.invoke(this, ((XMLRoot) value).getObject());
    } catch (Exception ex) {
      if(ex instanceof RuntimeException) { 
          throw ex;
      throw new RuntimeException(ex);

And we detected the actual type of the argument by introspecting the current type:

  public PropertyElement() {
    this.valueOf = configure(getClass());

 protected static Method configure(Class<?> type) {
    if (valueOf != null) {
      return valueOf;
    final ParameterizedType superclass = (ParameterizedType) type.getGenericSuperclass();
    final Object vt = superclass.getActualTypeArguments()[0];

    if (vt.getClass().equals(Class.class)) {
      Class<?> valueType = (Class<?>) vt;
      return Stream.of(valueType.getDeclaredMethods())
          .filter(t -> t.getName().equals("valueOf") && t.getParameterCount() == 1)
              t -> {
                final Class<?>[] ptypes = t.getParameterTypes();
                final Class<?> ptype = ptypes[0];
                return String.class.equals(ptype) || Object.class.equals(ptype);
              () ->
                  new IllegalArgumentException(
                      "Type does not supply a public, static method valueOf() accepting a string"));
    return null;

In a way, I’m sort of proud of this monster. Dr. Frankenstein would probably understand. On the other hand, our sins surely caught us, prompting a reasonable refactor to exorcise this demon.

0 comments on “Fix your indexes, homeslice”

Fix your indexes, homeslice

Database performance is the principal thing; therefore, undonk your indexes.
~Probably Confucius

Working with databases is one of the worst parts of building applications. There’s the setup, then there’s migrations, then there’s security, then there’s testing and on and on and on until you just can’t anymore. But, eventually, someone cobbles together a solution that encompasses at least a few of these things, and then you never touch it. Ever. It is forbidden to you. You will ruin everything in the application if you change even one thing. Everything will die.

But one day, something goes wrong. You’re getting transaction timeouts. Users are complaining that it’s taking 40 minutes to log into your UI (this happened once). Your application can’t handle all the data your users are feeding it.

So, someone profiles it and finds that it’s not your application, it’s the database. I mean, it’s pretty much always the database unless Keith is overusing hashmaps and locks again, but discovering which part is pretty tricky. Is it indexes? Is it disks? Is it the transaction log? Is it locks? Is it all of them? Who knows?

Fixing the code is at least a multi-month proposition that requires some poor sap go back and look at the code that interacts with your database, potentially donking the whole biz for everyone, not just this customer. The people who can actually fix it are generally not the ones who volunteer for that hell, and the people who volunteer will usually just make it worse.

So, you do the sensible thing and buy a bigger database server with faster disks. It helps for a while, then it just falls over again. You’ve got to fix this.

So, you perform an investigation, and discover that you’re using either a database sequence ID, or a UUID, or a hash function (I have seen this), and here’s the choose-your-own-misadventure part.

Database sequence ID

Your data is insertion-ordered, which is good for indexing. Heck, you can even use BRIN indexing, but you’re making a round-trip to the database for every insertion, and there’s a network round-trip and either a lock or a single thread somewhere in there. Usually there’s No escaping that.


Insertions are random. You got page-splits homeskillet. Like, everywhere. And these aren’t cheap. Your average page density is like, 0.0000000001%. Your index is like a million-page phonebook with one number per page. Oh, yeah, and you gotta maintain that B-Tree or whatever on disk because that sucker is way too big to fit into memory. Additional disk hits, baby! No BRINs for you!

Not to mention you’re probably storing the value as a string if my experience with these things is generalizable, which means you’re probably using 36 bytes/ID instead of the 16 actually required. Plus your IDs have a character set that they don’t need, and god help you if that default changes between versions or someone changes it.

Cryptographic hash (content-based addressing)

This one has all the problems of a UUID, plus some. Like, I’ve seen 128-byte (that is truly “byte” because they’re storing it as a string) MD5sum IDs used in large systems.

Ok, ok. So you stored them as a byte array, that only eliminates the size problem.

But here’s the kicker: If you’re using one of these techniques, your implementation will probably be exchanged for another of the problematic ones I just mentioned. Hax.

Use Flake! (Just do it)

At Sunshower, being ourselves the suckers who will eventually have to undonk our database, we decided to get ahead of the problem. The first post we ever did here was about Flake IDs, and now we have a well-tested, high-quality, MIT-licensed implementation that you can just use. If you’re already using UUIDs stored as byte arrays, drop this biz right in.

Step 1: Add our common library to your dependencies

  1. Make sure Nexus Snapshots are enabled by adding https://oss.sonatype.org/content/repositories/snapshots to your Maven repositories

  2. Add io.sunshower.persist:persist-api:<version> (version is currently 1.0.0-SNAPSHOT)–we’ll get a release soon (TM), but this API is totes stable.

  3. Create you a Flake Sequence. I’d recommend 1 per table for very high-scale systems since you can only generate 10,000/sequence/second. Pretty simple:

import io.sunshower.common.Identifier;
import io.sunshower.persist.Identifiers;
import io.sunshower.persist.Sequence;

Sequence<Identifier> sequence = Identifiers.newSequence(true); //'false' would have the sequence API throw an exception if you requested more IDs than you could generate in a given timespan, ~10k/sec/sequence.  Otherwise, the API blocks until the counter resets.

Then, use it however. For instance, to use it with JPA/Hibernate:

public class MyFlakeEntity {
private byte[] id;


In your database schema, just store them as a byte-array. We use bytea in Postgres which incurs 1 additional byte of overhead per row. Meh.

Swank ACLs with Flake IDs

If you really want declarative ACL goodness coupled with delicious DB goodness, and you’re using Spring Security, pull in our service-security library at io.sunshower.service:service-core:<version> (still 1.0.0-SNAPSHOT), then add these to your Spring configuration:

prePostEnabled = true, 
jsr250Enabled = true, 
securedEnabled = true

public class MySecurityConfiguration { // whatever your configuration is here

  public MutableAclService jdbcAclService(
      JdbcTemplate template, LookupStrategy lookupStrategy, AclCache aclCache) {
    return new IdentifierJdbcMutableAclService(template, lookupStrategy, aclCache, "<SCHEMA>");

  public AclCache aclCache(
      @Named("caches:spring:acl") Cache cache, // replace with your own cache.  This can just be a concurrent hashmap implementation.  We like Ignite
      PermissionGrantingStrategy permissionGrantingStrategy,
      AclAuthorizationStrategy aclAuthorizationStrategy) {
    return new SpringCacheBasedAclCache(
        cache, permissionGrantingStrategy, aclAuthorizationStrategy);

  public LookupStrategy aclLookupStrategy(
      DataSource dataSource,
      AclCache aclCache,
      AclAuthorizationStrategy aclAuthorizationStrategy,
      PermissionGrantingStrategy permissionGrantingStrategy) {
    return new IdentifierEnabledLookupStrategy(
        "<SCHEMA>", dataSource, aclCache, aclAuthorizationStrategy, permissionGrantingStrategy);

  public AclAuthorizationStrategy aclAuthorizationStrategy(GrantedAuthority role) {
    return new MultitenantedAclAuthorizationStrategy(role);

  public PermissionGrantingStrategy permissionGrantingStrategy(AuditLogger logger) {
    return new DefaultPermissionGrantingStrategy(logger);

Then, drop this schema into your migrations:

  id        BYTEA        NOT NULL PRIMARY KEY,
  principal BOOLEAN      NOT NULL,
  sid       VARCHAR(100) NOT NULL,
  CONSTRAINT unique_uk_1 UNIQUE (sid, principal)

  class VARCHAR(100) NOT NULL,
  CONSTRAINT unique_uk_2 UNIQUE (class)

CREATE TABLE <SCHEMA>.acl_object_identity (
  id                 BYTEA PRIMARY KEY,
  object_id_class    BYTEA   NOT NULL,
  object_id_identity BYTEA   NOT NULL,
  parent_object      BYTEA,
  owner_sid          BYTEA,
  entries_inheriting BOOLEAN NOT NULL,
  CONSTRAINT unique_uk_3 UNIQUE (object_id_class, object_id_identity),
  CONSTRAINT foreign_fk_1 FOREIGN KEY (parent_object) REFERENCES <SCHEMA>.acl_object_identity (id),
  CONSTRAINT foreign_fk_2 FOREIGN KEY (object_id_class) REFERENCES <SCHEMA>.acl_class (id),
  CONSTRAINT foreign_fk_3 FOREIGN KEY (owner_sid) REFERENCES <SCHEMA>.acl_sid (id)

  id                  BYTEA PRIMARY KEY,
  acl_object_identity BYTEA   NOT NULL,
  ace_order           INT     NOT NULL,
  sid                 BYTEA   NOT NULL,
  mask                INTEGER NOT NULL,
  granting            BOOLEAN NOT NULL,
  audit_success       BOOLEAN NOT NULL,
  audit_failure       BOOLEAN NOT NULL,
  CONSTRAINT unique_uk_4 UNIQUE (acl_object_identity, ace_order),
  CONSTRAINT foreign_fk_4 FOREIGN KEY (acl_object_identity)
  REFERENCES <SCHEMA>.acl_object_identity (id),

And you can totally use Spring Security annotation-driven security! For instance:

  @PreAuthorize("hasPermission(#id, 'io.sunshower.stratosphere.core.vault.model.Secret', 'DELETE')")
  public Secret delete(Identifier id) {
    Secret s = super.delete(id);
    return s;

Also, if you want to use your ACLs in JPQL (or HQL or whatever), we’ve mapped your entities for you. Pull in io.sunshower.core:core-api:1.0.0-SNAPSHOT and you’ll find the following classes:

  1. io.sunshower.model.core.auth.ObjectIdentity
  2. io.sunshower.model.core.auth.SecuredObject
  3. io.sunshower.model.core.auth.SecurityIdentity

If you need multitenancy and security groups (RBAC), that’s a topic for another post, but we have that, too.

To grant a set of permissions:

  public <T extends Persistable> void grantWithCurrentSession(
      Class<T> type, T instance, Permission... permissions) {
    final ObjectIdentity oid = new ObjectIdentityImpl(type, instance.getId());
    Sid sid = new PrincipalSid(session.getUsername());
    MutableAcl acl;
    try {
      acl = (MutableAcl) aclService.readAclById(oid);
    } catch (NotFoundException ex) {
      acl = ((MutableAclService) aclService).createAcl(oid);
    for (Permission permission : permissions) {
      acl.insertAce(acl.getEntries().size(), permission, sid, true);
    ((MutableAclService) aclService).updateAcl(acl);

To query all of the objects belonging to a user:

select e from Entity e
join e.identity oid
where oid.owner.username = :username;

Which simply requires the mapping:

  @OneToOne(fetch = FetchType.LAZY)
  @JoinColumn(name = "id", insertable = false, updatable = false)
  private ObjectIdentity identity;

  public ObjectIdentity getIdentity() {
    return identity;

In summary, you don’t have to choose between cool features and robust ACL/RBAC support and database performance with Sunshower. We’re happy to do some of that heavy lifting–and it’s all free!