← Older posts

Heartbleed security update

Heartbleed is a serious vulnerability in the SSL stack affecting a huge number of sites on the internet. In this post we address how we secured our servers in the wake of Heartbleed, and what actions customers should take to further protect themselves.

How it works

Heartbleed allows an attacker to view a window of the contents of an SSL server. For our customers, this meant that it was possible to sniff information transmitted through our API. The nature of Heartbleed makes it almost impossible to confirm an attack, so we recommend that customers take this opportunity to rotate any keys uploaded to our servers, especially those that were uploaded since the vulnerability became well-known on Monday April 7th.

What we’ve fixed

CircleCI runs on Amazon Web Services, using Elastic Load Balancer for our API, website, and traffic from 3rd party services. AWS has updated ELB to no longer be vulnerable. We have rotated our SSL key, and invalidated all customer sessions, in order to protect our customers.

GitHub has provided a way for us to cycle all GitHub Oauth tokens for our customers. We thank them for that and have rotated all GitHub Oauth tokens for our customers through this. As a result, there is no need for customers to reauthenticate CircleCI with GitHub.

We have also updated all internal servers and services to patched SSL versions. As a precaution, we are also cycling our keys for services that we rely on in production, and services our employees have access to.

What you should fix

Heartbleed affects nearly all sites on the internet, and we anticipate that many of your credentials may be affected. We suggest that customers cycle important credentials which may have been accessed or accessible by 3rd parties during this time.

If you have uploaded any of your credentials via our web site, we suggest you rotate those credentials. This includes deployment credentials such as API keys, SSH keys and passwords, CircleCI API tokens, and tokens to notify or access your chatrooms.

All credentials in CircleCI are in a project’s “Project settings” page on CircleCI, or in your personal account settings.

We strongly recommend that you update your GitHub credentials as well, as GitHub was also affected by Heartbleed. We recommend you follow their recommendations to change passwords and enable 2-factor authentication.

Contacting CircleCI

If you have concerns about how this affects CircleCI or your CircleCI settings, please get in touch – we’re always happy to help. Our in-app notification system is the best way to reach us, though you can also get in touch via sayhi@circleci.com. For more sensitive questions you can contact us using our GPG key at security@circleci.com.

We raised a $6m Series A

When Allen and I started CircleCI, just over two years ago, our vision was simple: to make developers more productive. Last Friday we took another step towards that vision by raising a $6m Series A round from DFJ.

It’s been about a year since we first raised our seed funding and since then we’ve focused exclusively on building our product. We didn’t focus on marketing or sales and we didn’t throw money into adwords. We committed every resource available to building the best product and the best experience possible for our customers, and something amazing happened: we began to grow.

about-photos

Over the last year we’ve grown dramatically, serving over 1000 customers with over a million dollars in revenue, however the most interesting thing is how we’ve grown — almost completely by word of mouth. To every developer who has helped spread the word — we can’t thank you enough!

Over the next year we are going to continue that relentless focus on product. Firstly we will build next generation tools to make Continuous Deployment more powerful and scaleable. Secondly we will be building an enterprise product which will have some very powerful features which many of you have asked for. Thirdly, we will relentlessly extend and improve CircleCI to allow developers to be more productive for years to come.

For us this money is not exciting as an achievement, its exciting as an opportunity. It allows us to grow our company in accordance with our values: a focus on productivity, a company-wide commitment to speak and listen to our customers, and to build a product for developers and by developers. These are core to what we do at CircleCI and they will always remain core as we continue to grow.

We’ve achieved our success to date by hiring people who care deeply about our customers: developers! We have a lot of work ahead of and we’re looking for the right folks to help us.

“Silence is for the weak”

In yesterday’s “Vacations are for the weak“, Seth Bannon said: “Preventing burnout is part of your job. Staying well rested is part of your job.

While we’re stating the obvious, I have another myth I’d like to put to rest: “Silence is for the weak“. Too many offices, managers, entrepreneurs and even other developers build offices full of noise and distractions, preventing developers from concentrating and doing our jobs.

I’ve spent time in offices with crazy distractions and noise. In one famous San Francisco-based startup, the table-tennis table was directly in the middle of the developers’ desks. In another, anyone could add songs to the Sonos that played across the office.

My favorite example of this sort of insanity is the office in which each time any developer deployed to production, the speaker system automatically played 6 seconds of their favorite song across the entire office. You could actually watch productivity go out the window as dozens of developers fell out of the zone.

Sometimes management prioritizes the open plan aesthetic above enabling concentration. One office had hardwood floors which allowed every conversation to bounce around the entire floor: the suggestion of carpeting was rejected because it would ruin the look of the office. In another, management felt different teams–working on completely separate projects–could collaborate if they overheard each others’ conversations. They even planned to pipe in music to raise the decibel level high enough so that it would instigate conversation between distracted teams!

Sometimes people in roles which don’t require deep concentration don’t get it: I’ve been in offices with sales calls happened in the open where the entire dev team couldn’t tune them out. I’ve seen marketing teams collaborating loudly within feet of engineers trying to concentrate. I’ve heard of sales guys playing frisbee over the heads of engineers.

Why does this happen? Because of a belief that “silence is for the weak”. You should just suck it up and get back to your desk. Put on your headphones and write me some code!

Its lunacy! Everyone tries to attract the best engineers with top-of-the-line Macbooks, expensive coffee grinders, catered lunches, and fridges full of expensive smoothies and beer. Why not attract them by giving them a great working environment, where they can spend their time deeply engrossed in their passion: writing code and building systems?

And don’t get me started on “collaboration”: people having a conversation next to me aren’t collaborating: they value their ability to chat above my ability to get work done, and they’re too rude to go find a conference room. Similarly rude and selfish is tapping someone on the shoulder to ask them a question: if you think your ability to get stuff done a few minutes faster is worth popping me out of the zone, then you show that you only care about yourself.

It’s not like the industry is not aware of the problem. Peopleware–the bible of managing engineering teams–discussed this over a decade ago. It’s a long, systematic study of productivity and it concluded, unequivocally, that noise damages engineer productivity. (It also addresses coding with music/headphones, and says music is bad for coding productivity)

Spolsky introduced a lot of us to Peopleware. He feels that concentration is so important that all his engineers get private offices. He also put “quiet working environment” in his famous “Joel test“, right up there with “do you use version control”.

I have yet to even see an office where devs have private offices. Google has 3 engineers per office in Mountain View; Microsoft’s Redmond office has two. Still, that’s better than 99% of Silicon Valley companies I’ve seen. At a recent talk I gave to developers, I asked the audience if they had private offices. About 10% of hands went up, which on the one hand seemed shockingly high compared to offices I’ve seen, and on the other hand is shockingly low: think of the productivity being thrown away at all these other companies!

The CircleCI team will soon be moving to our own office in SF. One of the highest priorities is a private office for everyone who wants one, and a quiet working environment for everyone. By the way, we’re hiring!

“Silence is for the weak” is a myth which has to die. To managers, entrepreneurs, CEOs, and even to coworkers: Preventing distractions for engineers is part of your job. Engineers being able to concentrate is part of your job. You spend a fortune hiring amazing developers, so build them an environment that lets them succeed.

Discuss on Hacker News

A Perfect Circle

How did we design our new logo? Our new designer, Danny King, explains:

From start to finish the whole thing took about two months. I knew I couldn’t make something that was uniquely us without having a firm understanding of what Circle is and how the team works, so I spent the first month or so doing as much work in our tech stack as possible, trying to gain an insight into the the product and the people who make it. Much of that time was spent on research and fighting the urge to simply use a plain circle. There was a recurring theme that I kept noticing; despite its numerous complexities, Circle is a product of ingenious simplicity. My goal with the branding was to create something equally as simple in its complexities.

simple math

I knew there were at least a few ideas I wanted to touch on going forward. There’s already a lot of meaning in the name itself, being related to a virtuous circle and the idea of producing continuously, so I knew that symbolism needed to stay intact. Beyond that I liked the idea of focusing on circuits or branches imagery. You’ll see this visual style in a lot of developer tools, like Git for example. It’s a strong visual metaphor that I wanted to draw from. If you look closely you’ll find a similar figure in the negative space of our new logo.

alignment

How complex could a circle be? Obviously the mark needed to be more than just one circle. I borrowed a lot of proportions from The Fibonacci Sequence, not just because it made for cool diagrams, but because creating a system helps give me a foundation to work from. Often times I’ll find that when starting from the beginning with a system like that, elements designed later seemingly fall into place. This gives me a sense that things are going in the right direction.

fibonacci

Similar methods were implemented for the construction of the type. I tried many variations here, mostly with typefaces already in existence. Many of them offered some of the elements I was going for, but I kept thinking it would be easier if I just took those different elements from other typefaces and merged them into one unique type. I’ve created custom typography in the past and it’s normally a pain, but the letters in “circleci” aren’t overly complex. The “c” is repeated thrice and builds most of an “e”, while “i” is repeated twice and forms most of an “l”. Only the “r” offered any resistance at all. It needed to follow the same rules as the other, very round letters, but came out looking very awkward at times.

typography

While designing the rest of the branding I never lost sight of the importance of our iconography. I believe those little symbols, used to inform users whether their build is working or not, offer a personal connection to the product. The same rules and guides I used for the construction of the mark itself helped me craft a custom set of icons to use throughout the app. I thought it was important to see a status page for someone’s project and know right away that it’s tested by Circle. The general idea is that the framework for the logomark remains intact, but its core or center is mutated into something that fits its surroundings appropriately. For example if my build is failing, an “x” seems appropriate, but if my build is queued, the arms of a clock might make more sense.

iconography

In its final form you’ll most often see our new logo appearing on a deep navy blue, but in reality it looks just as good on a light background. The sign of a great mark is its ability to look great anywhere under any conditions, and at any size. I did my best to meet that standard. I’m always more impressed with brands that were built as a whole, each piece complimenting the next. I spent much of my efforts on this project forming cohesion in both the visual identity and the process in which I craft new visual elements as well, which I think will continue to resonate as more features ship into production.

full logo

Danny joins the Circle team from The Windy City, having worked with such clients as The Tribune, Verizon, and Lego. If you want to find out more about him, you can find him on his own website or follow him on Twitter, and if you want to see more of the logo, you can do that here.

Danny has grand visions for how things will look here at Circle. He’s already working on his next projects, and you’ll start to see them roll into production more and more in the coming months.

Welcome to the team, Danny! Thanks for making us look so damned good.

Rewriting Your Test Suite in Clojure in 24 hours

This is a story about how I built a compiler to automatically translate CircleCI’s 14,000 line test suite into another testing library, in about 24 hours.

CircleCI’s test suite is probably one of the larger in the Clojure world today. Our server-side code is 100% Clojure, including the test suite, which is currently 14,000 lines, in 140 files, with 5,000 assertions. Without parallelization, it takes 40 minutes to run.

At the start of this adventure, all of those tests were written in Midje, a BDD testing library, somewhat similar to RSpec. We weren’t really happy with Midje, and decided to move to clojure.test, which is probably the most commonly used test library. clojure.test is simpler and less magical, with a larger ecosystem of tools and plugins.

Obviously, it’s not practical to manually rewrite 5,000 tests by hand. Instead, we decide to use Clojure to rewrite them automatically, using Clojure’s built-in language manipulation features.

Clojure is homoiconic, which means that all source files can be represented as data structures. Our translator parses each test file into a Clojure data structure. From there, we transform the code before writing it back to disk. Once on disk, we can load and run the tests, and even automatically check the file back in to version control if the tests pass, all without leaving the REPL.

Reading

The key to this entire operation is read. read-string is a built-in Clojure function which takes a string containing any Clojure code, and returns it as a Clojure data structure. It’s the same function the compiler uses when loading source files. A quick example: (read-string "[1 2 3]") returns [1 2 3].

We use read to turn our test code into a big nested list, which can be modified by normal Clojure code.

Transforming

Our tests were written to use midje, and we want to transform them to use clojure.test. Here’s an example test using midje:

(ns circle.foo-test
  (:require [midje.sweet :refer :all]
            [circle.foo :as foo]))
(fact "foo works"
  (foo x) => 42)

and a transformed version using clojure.test:

(ns circle.foo-test
  (:require [clojure.test :refer :all]))

(deftest foo-works
  (is (= 42 (foo x))))

The transformation involves replacing:

  • midje.sweet with clojure.test in the ns form
  • (fact "a test name"...) with (deftest a-test-name ...), because clojure.test names are vars, not strings
  • (foo x) => 42 with (is (= 42 (foo x)))
  • a number of smaller details, which we’re ignoring for now

The transformation is a simple depth-first tree walk:

(defn munge-form [form]
  (let [form (-> form
                 (replace-midje-sweet)
                 (replace-foo)
                 ...)]
    (cond
      (or (list? form)
          (vector? form)) (-> form
                              (replace-fact)
                              (replace-arrow)
                              (replace-bar)
                              ...
                              (map munge-form)))
      :else form))

The -> syntax behaves much like chaining in Ruby or JQuery, or Bash’s pipes: it passes the result of a function call as an argument to the next function call.

The first part (let [form ...]) takes a Clojure form and calls each translation function on it. The second part takes lists of forms–representing other Clojure expressions and functions–and recursively transforms them too.

The interesting work happens in the replace functions. They’re all generally of the form:

(if (this-form-is-relevant? form)
  (some-transformation form)
  form)

i.e., they check to see if the form passed in is relevant to their interests, and if so transform it appropriately. So replace-midje-sweet looks like

(defn replace-midje-sweet [form]
  (if (= 'midje.sweet form)
    'clojure.test
    form))

Arrows

Most of Midje’s testing behavior centers around “arrows”, an unidiomatic construct that Midje uses to implement a BDD style declarative test case. A simple example:

(foo 42) => 5

asserts that (foo 42) returns 5.

Depending on the arrow uses, and the types on either side of the arrow, there are a large number of possible behaviours.

(foo 42) => map?

As as example, if the right-hand side above is a function, this asserts that the result is truthy when passed to the function map?. In standard clojure this would be:

(map? (foo 42))

Here are a few examples of some midje arrows:

(foo 42) => falsey
(foo 42) => map?
(foo 42) => (throws Exception)
(foo 42) =not=> 3
(foo 42) => #"hello world" ;; regex
(foo 42) =not=> "hello"

Replacing Arrows

The actual translation is handled by about 40 core.match rules. They look like

(match [actual arrow expected]
  [actual '=> 'truthy] `(is ~actual)
  [actual '=> expected] `(is (= ~expected ~actual)
  [actual '=> (_ :guard regex?)] `(is (re-find ~contents ~actual))
  [actual '=> nil] `(is (nil? ~actual)))

(For Clojure experts, I’ve elided a lot of ~’ characters in the macros above to improve readability. Read the actual source to see what this really looks like.)

Most of these translations are straightforward. However, things get significantly more complicated with the contains form:

(foo 42) => (contains {:a 1})
(foo 42) => (contains [:a :b] :gaps-ok)
(foo 42) => (contains [:a :b] :in-any-order)
(foo 42) => (contains "hello")

The last case is particularly interesting. In the expression

(foo 42) => (contains "hello")

There are two completely different values that could make this test pass. (foo 42) could be a list that contains the item “hello”, or it could be a string that contains the substring “hello”:

"hello world" => (contains "hello")
["foo" "hello" "bar"] => (contains "hello")

In general, the contains forms are difficult to translate automatically. Some cases require runtime information (like the last example), and because there’s no existing implementation for many of the contains cases in standard Clojure, such as (contains [:a :b] :in-any-order), we decided to punt on all contains cases. The “fall through” rule looked like:

[actual arrow expected] (is (~arrow ~expected ~actual))

which turns (foo 42) => (contains bar) into (is (=> (contains bar) (foo 42))). This intentionally doesn’t compile because midje’s arrow function definitions aren’t loaded, and so we can fix these up by hand instead.

Runtime Type Information

There was one extra complication with automatic translation. If I have two expressions:

(let [bar 3]
  (foo) => bar

and

(let [bar clojure.core/map?]
  (foo) => bar

Midje’s arrow dispatch depends on the type of the right-hand expression, which can only (easily) be determined at runtime. If bar resolves to data, like a string, number, list or map, midje tests for equality. But if bar resolves to a function, midje instead calls the function, i.e. (is (= bar (foo))) vs (is (bar (foo))). Our 90% solution requires the original test namespace, and resolves functions during the translation process:

(defn form-is-fn? [ns f]
  (let [resolved (ns-resolve ns f)]
    (and resolved (or (fn? resolved)
                      (and (var? resolved)
                           (fn? @resolved)))))))

This works great in most cases, but fails when a local variable shadows a global variable name:

(let [s [1 2 3]
      count (count s)]
  (foo s) => count)

In this case, we want (is (= count (foo s))), but instead get (is (count (foo s))), which fails because in the local scope, count is a number, and (3 [1 2 3]) is an error. Thankfully there weren’t many of these situations, because solving this correctly would have required writing a full blown compiler with an understanding of the local variables in scope.

Running the tests

Once the translation code was written, we needed a way to figure out if it worked. Since we run the code in a REPL at runtime, it’s trivial to instantly run tests after translation using clojure.test‘s built-in functions.

clojure.test‘s design decisions made it easy to tie together the translation and evaluation process. All test functions are callable directly from the REPL without shelling out, and (clojure.test/run-all-tests) even returns a meaningful return value, a map containing the number of tests, passes and fails:

{:pass 61, :test 21, :error 0, :fail 0}

Being able to run the tests from the REPL made it vey convenient to modify the compiler and retest in a very tight feedback loop.

The Reader

Not everything worked quite so simply, however.

The “reader” (Clojure’s term for the part of the compiler that implement’s the read function) is designed to turn source files into data structures, primarily for consumption by the compiler. The reader strips comments, and expands macros, requiring us to review all diffs manually, and to revert lines that removed comments or contained macro definitions. Thankfully, there are only few of those in the tests. Our coding style tends to prefer docstrings over comments, and the macros tend to be isolated to a small set of utility files, so this didn’t affect us too much.

Code Indenting

We didn’t find a very good library for indenting our new code idiomatically. We used clojure.pprint‘s code mode, which although probably the best library out there, doesn’t do that great a job. We didn’t feel like writing that library during this project, so some files were written back to disk with non-idiomatic spacing and indentation. Now, when we work in a file with bad indention, we touch those up by hand. Really fixing this would have required an indenter that knows idiomatic formatting, and possibly respects file and line metadata on reader data.

There was a long delay between actually rewriting the test suite, and the publishing of this blog post. In the meantime, rewrite-clj has been released. I haven’t used it, but it looks like exactly what we were missing.

Outcomes

About 40% of our test files passed without manual intervention, which is pretty amazing given how quickly we threw this solution together. In the remaining files, about 90% of test assertions translated and passed. So 94% of the assertions in all files could be translated automatically, a great result.

Our code is up on GitHub here. Let us know if you end up using it. While we wouldn’t recommend it for unsupervised translation, especially because of the comment and macro issues, it worked great for CircleCI as part of a supervised process.

Discuss this post on Hacker News

Meet Our New Logo

CircleCI Logo

As many of you have noticed, we’ve recently been–sometimes not-so-subtly–tweaking Circle’s appearance. This has been in preparation for something we’re very excited about: our new logo.

Our designer, Danny (we’ll introduce him properly very soon), has been working on getting the new branding just right, and we rolled out some of those changes yesterday. Let us know what you think!

Why we’re supporting Typed Clojure, and you should too!

tl;dr Typed Clojure is an important step for not just Clojure, but all dynamic languages. CircleCI is supporting it, and you should too.

Typed Clojure is one of the biggest advancements to dynamic programming languages in the last few decades. It shows that you can have the amazing flexibility of a dynamic language, while providing lightweight, optional typing. Most importantly, this can make your team more productive, and it’s ready to use in production.

Even if you don’t use Clojure, you should support the Typed Clojure campaign, because its success will help developers in your language realize how great optional typing can be in everyday code. Whether you write Ruby or Python or JavaScript or whatever, what we’re learning from Typed Clojure can be applied to your language.

Why optional typing?

Dynamic languages have long been criticised for being hard to maintain at scale. When you grow to a large team or a large code base, it becomes more difficult to refactor a code base, to understand how it works, and to make sure it does what it should.

The standard solution is great testing, and obviously we at CircleCI are big fans of great automated testing. However, what we’re really about is productivity, and optional typing can make you much more productive, because you it can can provide guarantees that testing can’t. This is amazing for hard-to-test features, and to increase your coverage without spending a massive amount of time doing so.

Not your parents’ typing (unless your parents hacked in ML)

This isn’t typing as you expect from the Java world. Typed Clojure is much much more powerful than that. Most Clojure code, like other scripting languages, is about manipulating hashes and lists. Typed Clojure allows you type-check duck-typed languages.

Typed Clojure is based on important advancements in combining typed and untyped languages. Based on Typed Racket, it further proves that optional (or gradual) typing is practical to use in production code.

Technically, Typed Clojure is a different form of type-checking than Java and older languages. Those languages use nominal typing, where you type-check the names or classes of a value. Although that is still used in Typed Clojure, the major feature is about structural typing, where you can type-check the structure of an object or hash or list.

This allows you to have incredibly powerful type-checks, while staying lightweight and flexible. For example, you can guarantee that:

  • nil isn’t allowed
  • a hash must have a defined set of keys (each with their own types!)
  • a hash must have only known keys
  • a value must be one of a set of types
  • a vector has a known size
  • many other properties about set, lists, vectors, and object instances.

Not only that, but because it’s optional, adding type-checking can be done incrementally, providing only value where you need it. You can annotate a single library, a single namespace, or a small set of functions, and you can use libraries that aren’t typed.

Does it work?

Oh yes! CircleCI has been using it in production for 3 months. We started by using it in areas that are hard to test, such as code which allocates AWS machines and VMs. That’s right, we’re type-checking devops!

We now using it in 45 namespaces, about 20% of our codebase. Those are covered using only 300 type annotations. And obviously, this has made us more productive, by finding bugs before we ship them.

David's core.typed delight!

We’ll go into using more detail in a later blog post, but obviously we’re very excited.

Support Ambrose

All of this already works, and is used in production now. But Ambrose, the Typed Clojure author, isn’t stopping there. He’s launched an Indiegogo campaign to increase the scope of Typed Clojure, including:

  • good support with major IDEs/editors
  • significant speed improvements
  • support for checking Clojurescript
  • a tool to “guess” type annotations, useful for converting lots of legacy code
  • ensure support for popular libraries (eg. core.async, ring, compojure)
  • annotate the rest of clojure.core

We’ve supported him, and you should too!

UX Week Report

Last week^H^H^H^Hmonth we had our first UX week, where we spent the entire week focused strictly1 on UX, especially on the front-end UI. It went great — we fixed dozens of little things, and a number of big things, that affect you using Circle every day.

hot stuff

Let’s talk about the biggest and best changes first, shall we?

Continue reading →

Build Artifacts

Another new feature: build artifacts!

Quite a few of our users have added custom steps to the end of their builds to upload things like log files, coverage reports or deployment tarballs off the VMs for later consumption. This works okay, but it has some drawbacks — in particular, the need to insert upload credentials into the build, and that the uploaded artifacts aren’t linked from each build.

So we’ve made a feature of it! Now we can very easily save and link to any files created as part of your build. Here’s how it works: as part of your build, just write files into the $CIRCLE_ARTIFACTS directory. We’ll take it from there.

Continue reading →

← Older posts