0

Introducing Anvil

Go is a good language (link to my thoughts impending). I like its philosophy, which is that software is a complex enough without your tools introducing more complexity. In my 10 or so years of writing software professionally (sometimes software with demanding concurrency and performance requirements), I’ve found that you can get away with a good resizable array and hashtable implementation for 99.9% of the problems out there. If you have some nice syntactic jujubees over the array, you get your stacks and your queues for free, which is what Go opted for.

Another thing that I mostly like about Go’s hashtable (map) implementation is that it’s restrictive as to what you can use for keys. I can’t tell you how many bugs I’ve seen introduced in Java projects from someone implementing hashCode and equals methods incorrectly, or forgetting to implement one of them when they do the other. In fact, hashing is such a misunderstood topic that I have no doubt that omitting hash-based structures from the C++ standard until C++11 was in part due to that. In summary, Go made the right choice in restricting their standard container library and building it into the language–mostly.

Limitations

Online sorting

It’s a bit baffling that there is no sorted map or set. The online sorting provided by a good red-black tree implementation is frequently useful, especially when you’re creating networked middleware or (in my case), online algorithms for rewriting trees and graphs.

Iteration and Access

The idiom for iteration in go is the range operator. The only arguments that the range operator accepts are native Go types (chan, map, array), so the language does not help you in creating abstractions for something fundamental like iteration. While this is by design, it can in many cases hinder good software design. Maybe there’s a super-hot portion of an algorithm wherein iteration over the default implementation of a map just isn’t cutting it (hash tables require that you iterate over their entire backing store in the worst case, which can be substantially larger than the number of items they contain). Whatever your case may be, when you encounter a limitation in the core Go container, you must

  • Implement a new container that meets your requirement
  • Adapt the algorithm to suit your needs

I give you Anvil

Oddly, I do not notice any widely-used or maintained collections collections for Go. For Updraft, I had already had to implement a red-black tree. I then had a situation wherein an Updraft user could legitimately need different hashing characteristics. Finally, not being able to use an IR node as a key turned out to be a pain. So, I decided to implement my own collections framework with the intent that it should:

  • Be simple and similar to existing collections frameworks
  • Be Correct
  • Be Idiomatic
  • Be Consistent
  • Be Literate
  • Get the small things like serialization right

So far, the interfaces are still in flux and in a state of partial implementation, but I have implementations for a sorted map (RB tree) and an unordered map (linear probing hash table) that can be used as follows:

Map Creation

Unordered linear-probing hashmap

    hash := anvil.
        Maps().
        Hashes(anvil.Hashing.LinearProbing).
        DefaultLoadFactor().
        InitialCapacity(10).
        KeyedBy(strings.DefaultHash)

Sorted map using red-black balancing

    rbtree := anvil.Maps().Sorted(anvil.Trees.RedBlack).OrderBy(cmp)

Iteration

Stateful iterator (very fast, available on all collections)

    for iter := map.Iterator(); iter.HasNext(); {
        value := iter.Next()
        //etc
    }

Range iterator (slower than stateful iterator, but fast enough for most purposes, available on all collections)

    for value := range m.Range() {
        //etc
    }

The collections are inspired by the JCF and .Net collections classes, but have a distinct Go flavor.

Performance

One of these days when I’m not getting married and starting a new job, I’ll post full benchmarks of everything relative to builtin types. My preliminary impression is that they are competitive with the corresponding default implementations (I was actually pleasantly surprised that my linear probing hashmap is virtually indistinguishable in terms of insertion and lookup performance from the default implementation for certain keys (strings) and load-factors (.85)). As the collections mature, they’ll get even faster.

0

HP Z-Book G4 Review

I purchased an HP Z-Book G3 17 last year–Xeon processor, 64 GB of RAM, it is a true beast. Unfortunately, despite its power, it was plagued with problems like:

  • Not working with its docking station
  • Requiring a new, awful type of docking station
  • Ultra flaky drivers

Docking Station

Let’s start here because nothing quite makes me want to go Office Space on a device like repeated failures to sleep and wake up properly. I mean, to begin with, the laptop ships with Thunderbolt security enabled, which results in the computer not detecting its own docking station by default.

Sure, it’s a trip to the BIOS to fix, but really? I mean, when you’re sitting around repeatedly installing drivers and unplugging/plugging in the docking station waiting for that sweet, sweet “beep boop” chime of detection and reading through troubleshooting guides that mention this in exactly zero places, it’s hard not to feel bitter about expensive office hardware that was specifically designed to work together not working together.

But that’s not the worst of it. I guess I’m feeling my age or something, but the old docking stations worked fine. I’m, at this moment, gazing wistfully at the old HP Advanced Docking station with its 6 USB ports, SATA, etc. etc. ports, and the fact that it Just Worked asking myself “why?” To be fair, the new Thunderbolt has 4 USB ports and 2 Display Ports requiring you to only purchase 2 HDMI/DP adapters if you want to use it with 96% of the monitors available, but that presupposes that 1.) you got it to work and 2.) it continues to work. Which it mostly doesn’t.

Finally, you’d think that the Thunderbolt 3 Docking station connector would be power + garden-variety Thunderbolt cable. It’s somehow not, and it’s about 18 inches long and pretty damn inflexible, so you get wiring scenarios like this:

20170831_100227

I’ve tried most variations and have reached the conclusion that HP did this specifically to hurt me personally.

Software/firmware

My morning ritual is this:

wake-up

And yes, all the drivers and firmware are up-to-date, I’ve contacted support, etc. etc.

So what I did was

Buy a G4. The experience with the G3 was so cartoonishly bad that I thought to myself “there is no way that they will do this again. They will fix it and then I will have a functioning and powerful laptop.” Not only was this incorrect, it was horribly incorrect. How could I be so wrong?

0

Updraft

Hi folks! Been a while–just started my (paying) job a few weeks ago, and haven’t had much time to work on Sunshower. I took some time last weekend to implement expression parsing for the Pascal frontend and work on the IR (intermediate representation), so we’re still making some good progress.

You can grab the source over at https://github.com/sunshower-io/updraft

Building

It’s simple to build Updraft. It doesn’t have any external dependencies, so just clone that repo and run go build -o ucc main.go. None of the backends are fleshed out or integrated yet, but you can see some cool things about the structure of your programs.

Supported syntax structures

Right now, things are pretty basic (hahaha, nah, they’re Pascal, which is almost as bad). Pascal’s pretty easy to parse, and so it makes for a good straw-man language. It also supports most of the features that our backends and IRs will require, so using it to punch through the entire compiler stack won’t leave us in the lurch with more modern languages. Right now, the Pascal front-end supports:

Comments

{this is a comment}

{this
   is 
another
comment}

Primitive Types

{string}
str := 'string';
{integers (64 bit)}
a := 432123421341232515;

{floating-point}
c := 2.001e-6;


Compound statements

BEGIN
{ statement list }
END.

Assignments

a := 1;
b := a + a;

Arithmetic Expressions


c := 4 * 2 + 16 / 90 + (200 * (300 + 400) + 500);
a := 2 * (c - 16);

But what we’re demoing this week is some cool trace features.

Feature 1: Pluggable front-ends

While only Pascal’s currently supported, if you look into UCC (Updraft Compiler Collection)’s entry-point, you’ll see that, by using Convection, our Go dependency-injection framework, we only load the compiler toolset for a given language:

func (t *TraceConfiguration) TraceCommand(
        cmd *root.RootCommand,
        c compiler.Compiler, // yaaay.  Injected =)
        cfg *root.RootConfiguration,
) {
    //etc.
}

which keeps startup fast and memory-consumption lean. Eventually, we might support dynamically-linkable front-ends, but since Updraft is intended to be embedded, I doubt it’ll become a problem.

Feature 2: Tracing

UCC contains some pretty legit tracing facilities already, including:

Unresolved symbols:
{test.pas} 
{hello world}

BEGIN
        a := 1;
        b := 200;
        c := 1 * 20 + a;
        e := 1;
        d := c + a + e + nothere;


END.

When run with -st (-s -> summary, t -> tokens) (–only available in trace mode)(actual unresolved symbols during compilation will be reported after lexical analysis)

Results in:

ERROR{type:'syntax', token: TOKEN{text: nothere, line: 8, col: 17, type: Token('IDENTIFIER'), value: '%!s(<nil>)'}

                   3 source lines.
                   0 syntax errors.
0 seconds total parsing time

Symbol table dumps

Adding the -y (./ucc trace -e interpreter -f example.pas -l pascal -sty)
flag for sYmbols on the same file results in:

ERROR{type:'syntax', token: TOKEN{text: nothere, line: 8, col: 17, type: Token('IDENTIFIER'), value: '%!s(<nil>)'}
===== SYMBOL TABLE =====
Identifier          Line Numbers
----------          ------------
a                    004 006 008 
b                    005 
c                    006 008 
d                    008 
e                    007 008 
nothere              008 


                   3 source lines.
                   0 syntax errors.
0 seconds total parsing time

Note that nothere still got a symbol-table entry. That’s because some languages allow forward declarations, and so we’ll report it to the lexer, but subsequent phases could opt to ignore them.

IR dumps

The latest cool feature is the ability to dump the entire IR into JSON (or other formats if you choose). Optimization passes will rewrite this structure, and you can dump the IR out after any number of optimization passes you like. The reducer-interpreter backend will support tracing the IR as it is dynamically rewritten during program execution.

The IR of the above program can be shown (in conjunction with everything else) via:
./ucc trace -e interpreter -f example.pas -l pascal -styp --format json

Which results in everything above, plus:

{
 "node":{
    "type":"compound", 
    "depth":1,
    "value":"<nil>",
    "text":"a",
    "value":"%!s(<nil>)",
    "line":4,
    "position":1,
    "children":[
     {
      "node":{
         "type":"Assign", 
         "depth":6,
         "value":"<nil>",
         "text":"a",
         "value":"%!s(<nil>)",
         "line":4,
         "position":1,
         "children":[
          {
           "node":{
              "type":"var", 
              "depth":11,
              "value":"<nil>",
              "text":":=",
              "value":"%!s(<nil>)",
              "line":4,
              "position":3
           }
          },
          {
           "node":{
              "type":"int64", 
              "depth":11,
              "value":"1",
              "text":"",
              "value":"%!s(int64=1)",
              "line":4,
              "position":6
           }
          }
         ]
      }
     },
     {
      "node":{
         "type":"Assign", 
         "depth":6,
         "value":"<nil>",
         "text":"b",
         "value":"%!s(<nil>)",
         "line":5,
         "position":1,
         "children":[
          {
           "node":{
              "type":"var", 
              "depth":11,
              "value":"<nil>",
              "text":":=",
              "value":"%!s(<nil>)",
              "line":5,
              "position":3
           }
          },
          {
           "node":{
              "type":"int64", 
              "depth":11,
              "value":"200",
              "text":"",
              "value":"%!s(int64=200)",
              "line":5,
              "position":6
           }
          }
         ]
      }
     },
     {
      "node":{
         "type":"Assign", 
         "depth":6,
         "value":"<nil>",
         "text":"c",
         "value":"%!s(<nil>)",
         "line":6,
         "position":1,
         "children":[
          {
           "node":{
              "type":"var", 
              "depth":11,
              "value":"<nil>",
              "text":":=",
              "value":"%!s(<nil>)",
              "line":6,
              "position":3
           }
          },
          {
           "node":{
              "type":"+", 
              "depth":11,
              "value":"<nil>",
              "text":"+",
              "value":"%!s(<nil>)",
              "line":6,
              "position":13,
              "children":[
               {
                "node":{
                   "type":"*", 
                   "depth":16,
                   "value":"<nil>",
                   "text":"*",
                   "value":"%!s(<nil>)",
                   "line":6,
                   "position":8,
                   "children":[
                    {
                     "node":{
                        "type":"int64", 
                        "depth":21,
                        "value":"1",
                        "text":"",
                        "value":"%!s(int64=1)",
                        "line":6,
                        "position":6
                     }
                    },
                    {
                     "node":{
                        "type":"int64", 
                        "depth":21,
                        "value":"20",
                        "text":"",
                        "value":"%!s(int64=20)",
                        "line":6,
                        "position":10
                     }
                    }
                   ]
                }
               },
               {
                "node":{
                   "type":"var", 
                   "depth":16,
                   "value":"<nil>",
                   "text":"a",
                   "value":"%!s(<nil>)",
                   "line":6,
                   "position":15
                }
               }
              ]
           }
          }
         ]
      }
     },
     {
      "node":{
         "type":"Assign", 
         "depth":6,
         "value":"<nil>",
         "text":"e",
         "value":"%!s(<nil>)",
         "line":7,
         "position":1,
         "children":[
          {
           "node":{
              "type":"var", 
              "depth":11,
              "value":"<nil>",
              "text":":=",
              "value":"%!s(<nil>)",
              "line":7,
              "position":3
           }
          },
          {
           "node":{
              "type":"int64", 
              "depth":11,
              "value":"1",
              "text":"",
              "value":"%!s(int64=1)",
              "line":7,
              "position":6
           }
          }
         ]
      }
     },
     {
      "node":{
         "type":"Assign", 
         "depth":6,
         "value":"<nil>",
         "text":"d",
         "value":"%!s(<nil>)",
         "line":8,
         "position":1,
         "children":[
          {
           "node":{
              "type":"var", 
              "depth":11,
              "value":"<nil>",
              "text":":=",
              "value":"%!s(<nil>)",
              "line":8,
              "position":3
           }
          },
          {
           "node":{
              "type":"+", 
              "depth":11,
              "value":"<nil>",
              "text":"+",
              "value":"%!s(<nil>)",
              "line":8,
              "position":16,
              "children":[
               {
                "node":{
                   "type":"+", 
                   "depth":16,
                   "value":"<nil>",
                   "text":"+",
                   "value":"%!s(<nil>)",
                   "line":8,
                   "position":12,
                   "children":[
                    {
                     "node":{
                        "type":"+", 
                        "depth":21,
                        "value":"<nil>",
                        "text":"+",
                        "value":"%!s(<nil>)",
                        "line":8,
                        "position":8,
                        "children":[
                         {
                          "node":{
                             "type":"var", 
                             "depth":26,
                             "value":"<nil>",
                             "text":"c",
                             "value":"%!s(<nil>)",
                             "line":8,
                             "position":6
                          }
                         },
                         {
                          "node":{
                             "type":"var", 
                             "depth":26,
                             "value":"<nil>",
                             "text":"a",
                             "value":"%!s(<nil>)",
                             "line":8,
                             "position":10
                          }
                         }
                        ]
                     }
                    },
                    {
                     "node":{
                        "type":"var", 
                        "depth":21,
                        "value":"<nil>",
                        "text":"e",
                        "value":"%!s(<nil>)",
                        "line":8,
                        "position":14
                     }
                    }
                   ]
                }
               },
               {
                "node":{
                   "type":"var", 
                   "depth":16,
                   "value":"<nil>",
                   "text":"nothere",
                   "value":"%!s(<nil>)",
                   "line":8,
                   "position":17
                }
               }
              ]
           }
          }
         ]
      }
     }
    ]
 }
}

One of the cooler features of UCC is the ability to dynamically insert or substitute IR nodes. For instance, an early language I plan on supporting is Groovy, and it would be relatively easy to append additional nodes that convey type information about a given definition even if that type information isn’t available at compile-time. Non-compatible type-reassignments could remove that type-information, while covariant assignments could specialize it further.

0

JUnit 5 and Spring 5

I’ve been extending JUnit to provide project-specific testing infrastructure for quite some time, and have always been disappointed with the extension mechanisms prior to JUnit 5.

To begin with, JUnit 4’s primary extension mechanism was the actual test runner. If you wanted to mixin multiple extensions, you were in for a hassle. For one test-infrastructure-project, I would have the runner dynamically subclass the test case and add behaviors based on the infrastructure-specific annotations that were present. This worked really well, except that most IDEs I tried would not run the tests correctly at the method level, so you’d be left re-running the entire class.

On another project, I jumped through all manner of hoops to avoid this. Spring-test alleviated many of the problems with their TestExecutionLisiteners, but not all, and it was less lovely than subclassing in use. The result worked quite well, but it was pretty excruciating to write. Now that we’re rewriting everything from scratch, I’m implementing our testing infrastructure on JUnit 5 and Spring 5, and JUnit 5 makes this awesome!

How’s it work?

JUnit 5 provides a new extension mechanism called, well, an Extension. Extension is a marker-interface with specializations for each portion of the test lifecycle. For instance, the interface BeforeAll will equip you with a beforeAll method which is called with the container extension context, providing, among other things, the test class.

For testing purposes, we need to inject and override properties pretty routinely. Spring 5’s YAML support isn’t there when used with @TestPropertySource, so I thought I’d whip up my own (full source)

Then, to use it (given a resource-structure) suchlike:

Properties Extension Usage

@RunWith(JUnitPlatform.class)
@ExtendWith({
        SpringExtension.class,
        PropertiesExtension.class
})
@Properties({
        "classpath:classes/frap.yml"
})
@ContextConfiguration(classes = PropertiesExtensionConfiguration.class)
public class PropertiesExtensionTest {

    @Inject
    private Environment environment;

    @Value("${classes.properties.frap}")
    String frap;


    @Test
    public void ensureInjectingPropertiesWorks() {
        assertThat(frap, is("lolwat"));

    }

    @Test
    public void ensureLoadingYamlAtClassLevelWorks() {
        String prop = environment.getProperty("classes.properties.frap");
        assertThat(prop, is("lolwat"));
    }

    @Test
    @Properties("classpath:methods/frap.yml")
    public void ensureOverridingOnMethodWorksWithInjection() {
        String prop = environment.getProperty("classes.properties.frap");
        assertThat(prop, is("overridden"));
    }

    @Test
    @Properties("classpath:methods/frap.yml")
    public void ensureOverridingOnMethodWorks() {
        assertThat(frap, is("overridden"));
    }

}

0

Build you a Distributed App: Part 1

‘Morning folks,

Today’s blog will be a bit out-of-order (there’s a subsequent blog that I haven’t finished that structurally precedes this one), but I hope it won’t throw too much wire into the hay-baler.

Persistence IDs

The problem we are trying to solve is how to identify objects within a system. Most of the time, we think of object identity as having two components:

  1. Object type and
  2. Some unique name or tag for a given object instance

Selecting the correct persistence ID scheme for your application can be quite a task, and people generally don’t give it much thought at the outset. Later on, when the initial ID scheme isn’t ideal for the application, developers have to perform an ID type migration, which for live systems is frequently one of the nastier modifications to make.

About 80% of the applications that I’ve seen and worked with just use sequential, database-generated integral IDs, and that mostly works for them. There are well documented problems with them, so we’ll avoid them altogether (except under very specific circumstances, which will be discussed on a case-by-case basis). They do have one desirable property that we’ll discuss below, viz. that they’re sorted.

UUIDs

Now that database-generated sequential IDs are out, what about UUIDs? It’s a good question, and I’ve used and seen used UUIDs quite successfully in quite a few applications. But we’re not using them in Sunshower for the following reasons:

  1. They’re pretty ugly. I mean, URLs like sunshower.io/orchestrations/123e4567-e89b-12d3-a456-426655440000/deployments/123e4567-e89b-12d3-a456-426655440000 aren’t great. We previously base-58 encoded our UUIDs to produce prettier URLs along the lines of sunshower.io/orchestrations/11W7CuKyzdu7FGXEVQvK/deployments/11W7CuKz27Y9ePpV2ju9. One of the problems that we encountered was that having different string representations of IDs inside and outside our application made debugging a less straightforward than it needed to be.
  2. They’re pretty inconsistent across different databases and workloads.
    — For write-intensive workloads, UUIDs as primary keys are a poor choice if you don’t have an auxiliary clustering index (which requires that you maintain 2 indexes per table, at least). Insertions into database pages will happen at random locations, and you’ll incur jillions of unnecessary page-splits in high-volume scenarios. On the other hand, adding the additional clustering index will incur additional overhead to writes.
    — Index scans that don’t include the clustering index can perform poorly because the data are spread out all over the disk.

So, is there a better way?

How about Flake?

Twitter back in the day encountered similar issues with ID selection, so they designed the Snowflake ID scheme. There is a 128-bit extension that minimizes the need for node-coordination, which is desirable in our case (especially since we were willing to tolerate 128-bit IDs for UUIDs). The layout of the ID is as follows:

  1. The first 64 bits are a timestamp (for a single-node without modifications to the system clock, monotonically increasing).
  2. 48 random bits (usually a MAC address of a network interface, other schemes could be used)
  3. A 16-bit monotonically-increasing sequence that is reset every time the system-clock ticks forward. This is important because it places an upper limit on the number of IDs that can safely be generated in a given time-period (65,535/second). My implementation provides back-pressure, but this can cause undesirable behavior (contention) in very high-volume scenarios. To put this in perspective, Twitter accommodates an average of 6,000 Tweets/second, but even this would only consume about 10% of the bandwidth of our ID generation for a single node.

Our implementation

full source

I’m sorry. I’m pretty old-school. I like Spring and JPA (and even EJB!). Things like JPQL and transparent mapping between objects and their storage representations (e.g. tables) are important to me. I also super-like transactions, and I really really like declarative transaction management. Why? Because not having these things places a very high burden on development teams, and in my experience, reduces testability, frequently dramatically. Another requirement is that we be able to easily serialize IDs to a variety of formats, so we’ll make our ID JAXB-enabled. Here’s the important parts of the Identifier class:

//not using @Embeddable because we will create a custom Hibernate type for this--that way we can use the same annotations for everything
@XmlRootElement(name = "id")
@XmlAccessorType(XmlAccessType.NONE)
public class Identifier implements
        Comparable<Identifier>,
        Serializable {

    static final transient Encoding base58 = Base58.getInstance(
            Default
    );


    @XmlAttribute
    @XmlJavaTypeAdapter(Base58ByteArrayConverter.class)
    private byte[] value;

    protected Identifier() {

    }

    Identifier(byte[] value) {
        if(value == null || value.length != 16) {
            throw new IllegalArgumentException(
                    \"Argument cannot possibly be a valid identifier\"
            );
        }
        this.value = value;
    }

// other stuff
}

Now, ideally, we would be able to make value final. If an ID is created from thread A and somehow immediately accessed from thread B, final would guarantee that thread A and thread B would always agree on the value of value. Since neither JAXB nor JPA really work with final fields, we can’t really do that. We could partially fix value‘s publication by marking it volatile, but there are downsides to that as well. The solution that I’m opting for is protecting the creation of Identifiers by forcing the creation of IDs to occur within a sequence (note the protected and package-protected constructors of Identifier):

public interface Sequence<ID extends Serializable> {
    ID next();
}

with a Flake ID sequence (full source: [](Flake ID))

    @Override
    public Identifier next() {
        synchronized (sequenceLock) {
            increment();
            ByteBuffer sequenceBytes =
                    ByteBuffer.allocate(ID_SIZE);
            return new Identifier(
                    sequenceBytes
                            .putLong(currentTime)
                            .put(seed)
                            .putShort((short) sequence).array()
            );
        }
    }

Now, we’re guaranteed that sequences can be shared across threads, and we have several options:

  1. Each entity-type could be assigned its own sequence
  2. Entities can share sequences

We don’t really care too much about ID collisions across tables, and we can generate a ton of IDs quickly for a given sequence, so we’ll just default to sharing a sequence for entities:

@MappedSuperclass
@XmlDiscriminatorNode(\"@type\")
public class AbstractEntity extends
        SequenceIdentityAssignedEntity<Identifier> {

    static final transient Sequence<Identifier> DEFAULT_SEQUENCE;

    static {
        DEFAULT_SEQUENCE = Identifiers.newSequence(true);
    }

    @Id
    @XmlID
    @XmlJavaTypeAdapter(IdentifierAdapter.class)
    private Identifier id;


    protected AbstractEntity() {
        super(DEFAULT_SEQUENCE);
    }


    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (!(o instanceof AbstractEntity)) return false;
        if (!super.equals(o)) return false;

        AbstractEntity that = (AbstractEntity) o;

        return id != null ? id.equals(that.id) : that.id == null;
    }

    @Override
    public int hashCode() {
        int result = super.hashCode();
        result = 31 * result + (id != null ? id.hashCode() : 0);
        return result;
    }


    @Override
    public String toString() {
        return String.format(\"%s{\" +
                \"id=\" + id +
                '}', getClass());
    }
}

MOXy only allows you to use String @XmlID values, so we need to transform our IDs to strings (hence the @XmlJavaTypeAdapter)

In the next blog post, we’ll demonstrate how to make Identifiers JPA native types!

0

Announcing Sunshower.io

Hello and welcome to sunshower.io’s official blog! Josiah here excited to tell you about who we are and where we are going. Over the past 16 months, I’ve been working as the CTO of a startup. Unfortunately, my co-founders and I were not able to align our visions for the product and the project, so we decided to part ways. I had hoped that we would be able to reach an agreement to open-source the project, but I doubt that will ever fully materialize. In any case, I think it’s best to start a new project with a clean slate. This will ensure that the project is and always remains free of any legal or conceptual constraints, and I believe that with a more cohesive vision, we will be able to deliver a better product. With that in mind, let’s talk about what it is!

The goals of sunshower.io are to provide a robust, streamlined platform for Provisioning-as-a-Service (PaaS, not to be confused with Platform-as-a-Service) and Configuration-as-a-Service, so let’s talk about what we mean.

Configuration-as-a-Service

I had initially envisioned our project to be peer-to-peer, secure Git. The goal was to be able to commit arbitrary files (including very large binaries, helpful for installers), indicate that some files should be treated as templates (i.e. filtered through various data-sources available to the system, such as our key-value store, system properties, etc.), and then “push” out the repository to collections of systems. Failures could be handled according to several strategies (up to and including reverting the entirety of the push across the cluster), and you could seamlessly revert to a previous commit. Sunshower.io will not attempt to handle as many enterprise cases initially. At least, if we do, we’ll delegate down to the capabilities of the orchestration provider (e.g. Docker Swarm or Kubernetes). One of the primary goals of sunshower is to abstract the process of deployment away from the technologies used for the deployment, and so we’ll have to think carefully about how we go about it: not every orchestration provider is going to tackle the same problems the same ways.

Provisioning-as-a-Service

The other domain that we thought a lot about tackling was provisioning. Provisioning infrastructure elements such as virtual machines and security groups is conceptually the same across all the cloud providers, and so we would like to be able to take the same orchestration and deploy it to different clouds without any changes. Of course, things like our default sizing tiers won’t generally be adequate for many users needs, but there are sufficient similarities between most CSP provider instance-types that translating between them is frequently possible. In the cases where it’s not, overriding at deployment is always a possibility.

Thanks again for taking some time to learn more about us! Expect updates here frequently, and we’ll get this out the door ASAP. You can always check our progress out on Github.