SIGGRAPH – Anaheim, CA – The Khronos™ Group today announced the immediate release of the OpenGL® 4.4 specification,bringing the very latest graphics functionality to the most advanced and widely adopted cross-platform 2D and 3D graphics API (application programming interface). OpenGL 4.4 unlocks capabilities of today’s leading-edge graphics hardware while maintaining full backwards compatibility, enabling applications to incrementally use new features while portably accessing state-of-the-art graphics processing units (GPUs) across diverse operating systems and platforms. Also, OpenGL 4.4 defines new functionality to streamline the porting of applications and titles from other platforms and APIs. The full specification and reference materials are available for immediate download at http://www.opengl.org/registry.

In addition to the OpenGL 4.4 specification, the OpenGL ARB (Architecture Review Board) Working Group at Khronos has created the first set of formal OpenGL conformance tests since OpenGL 2.0. Khronos will offer certification of drivers from version 3.3, and full certification is mandatory for OpenGL 4.4 and onwards. This will help reduce differences between multiple vendors’ OpenGL drivers, resulting in enhanced portability for developers.

New functionality in the OpenGL 4.4 specification includes:

Buffer Placement Control (GL_ARB_buffer_storage)
Significantly enhances memory flexibility and efficiency through explicit control over the position of buffers in the graphics and system memory, together with cache behavior control – including the ability of the CPU to map a buffer for direct use by a GPU.

Efficient Asynchronous Queries
(GL_ARB_query_buffer_object)
Buffer objects can be the direct target of a query to avoid the CPU waiting for the result and stalling the graphics pipeline. This provides significantly boosted performance for applications that intend to subsequently use the results of queries on the GPU, such as dynamic quality reduction strategies based on performance metrics.

Shader Variable Layout (GL_ARB_enhanced_layouts)
Detailed control over placement of shader interface variables, including the ability to pack vectors efficiently with scalar types. Includes full control over variable layout inside uniform blocks and enables shaders to specify transform feedback variables and buffer layout.

Efficient Multiple Object Binding (GL_ARB_multi_bind)
New commands which enable an application to bind or unbind sets of objects with one API call instead of separate commands for each bind operation, amortizing the function call, name space lookup, and potential locking overhead. The core rendering loop of many graphics applications frequently bind different sets of textures, samplers, images, vertex buffers, and uniform buffers and so this can significantly reduce CPU overhead and improve performance.

Streamlined Porting of Direct3D applications

A number of core functions contribute to easier porting of applications and games written in Direct3D including GL_ARB_buffer_storage for buffer placement control, GL_ARB_vertex_type_10f_11f_11f_rev which creates a vertex data type that packs three components in a 32 bit value that provides a performance improvement for lower precision vertices and is a format used by Direct3D, and GL_ARB_texture_mirror_clamp_to_edge that provides a texture clamping mode also used by Direct3D.Extensions released alongside the OpenGL 4.4 specification include:

Bindless Texture Extension (GL_ARB_bindless_texture)
Shaders can now access an effectively unlimited number of texture and image resources directly by virtual addresses. This bindless texture approach avoids the application overhead due to explicitly binding a small window of accessible textures. Ray tracing and global illumination algorithms are faster and simpler with unfettered access to a virtual world’s entire texture set.

Sparse Texture Extension (GL_ARB_sparse_texture)
Enables handling of huge textures that are much larger than the GPUs physical memory by allowing an application to select which regions of the texture are resident for ‘mega-texture’ algorithms and very large data-set visualizations.

OpenGL BOF at SIGGRAPH, Anaheim, CA July 24th 2013
There is an OpenGL BOF “Birds of a Feather” Meeting on Wednesday July 24th at 7-8PM at the Hilton Anaheim, California Ballroom A & B, where attendees are invited to meet OpenGL implementers and developers and learn more about the new OpenGL 4.4 specification.

Do you want to use lambda expressions already today, but you are forced to use Java and a stable JRE in production? Now that’s possible with Retrolambda, which will take bytecode compiled with Java 8 and convert it to run on Java 7, 6 and 5 runtimes, letting you use lambda expressions andmethod references on those platforms. It won’t give you the improved Java 8 Collections API, but fortunately there are multiple alternative libraries which will benefit from lambda expressions.

Behind the Scenes

A couple of days ago in a café it popped into my head to find out whether somebody had made this already, but after speaking into the air, I did it myself over a weekend.

The original plan of copying the classes from OpenJDK didn’t work (LambdaMetafactory depends on some package-private classes and would have required modifications), but I figured out a better way to do it without additional runtime dependencies.

Retrolambda uses a Java agent to find out what bytecode LambdaMetafactory generates dynamically, and saves it as class files, after which it replaces the invokedynamic instructions to instantiate those classes directly. It also changes some private synthetic methods to be package-private, so that normal bytecode can access them without method handles.

After the conversion you’ll have just a bunch of normal .class files – but with less typing.

P.S. If you hear about experiences of using Retrolambda for Android development, please leave a comment.

Introduction

Qt Quick’s QML language makes it easy to do many things, especially fancy animated user interfaces. However, some things either can’t be done or are not suitable for implementing in QML, such as:

  1. Getting access to functionality outside of the QML/JavaScript environment.
  2. Implementing performance critical functions where native code is desired for efficiency.
  3. Large and/or complex non-declarative code that would be tedious to implement in JavaScript.

As we’ll see, Qt makes it quite easy to expose C++ code to QML. In this blog post I will show an example of doing this with a small but functional application.

The example is written for Qt 5 and uses the Qt Quick Components so you will need at least Qt version 5.1.0 to run it.

Overview

To expose a C++ type having properties, methods, signals, and/or slots to the QML environment, the basic steps are:

  1. Define a new class derived from QObject.
  2. Put the Q_OBJECT macro in the class declaration to support signals and slots and other services of the Qt meta-object system.
  3. Declare any properties using the Q_PROPERTY macro.
  4. Call qmlRegisterType() in your C++ main program to register the type with the Qt Quick engine.

For all the details I refer you to the Qt documentation section Exposing Attributes of C++ Types to QML and the Writing QML Extensions with C++ tutorial.

Ssh Key Generator

For our code example, we want a small application that will generate ssh public/private key pairs using a GUI. It will present the user with controls for the appropriate options and then run the program ssh-keygen to generate the key pair.

I implemented the user interface using the new Qt Quick Controls since it was intended as a desktop application with a desktop look and feel. I initially developed the UX entirely by running the qmlscene program directly on the QML source.

The UI prompts the user for the key type, the file name of the private key to generate and an optional pass phrase, which needs to be confirmed.

The C++ Class

Now that have the UI, we will want to implement the back end functionality. You can’t invoke an external program directly from QML so we have to write it in C++ (which is the whole point of this example application).

First, we define a class that encapsulates the key generation functionality. It will be exposed as a new class KeyGenerator in QML. This is done in the header file KeyGenerator.h below.

#ifndef KEYGENERATOR_H
#define KEYGENERATOR_H

#include <QObject>
#include <QString>
#include <QStringList>

// Simple QML object to generate SSH key pairs by calling ssh-keygen.

class KeyGenerator : public QObject
{
    Q_OBJECT
    Q_PROPERTY(QString type READ type WRITE setType NOTIFY typeChanged)
    Q_PROPERTY(QStringList types READ types NOTIFY typesChanged)
    Q_PROPERTY(QString filename READ filename WRITE setFilename NOTIFY filenameChanged)
    Q_PROPERTY(QString passphrase READ filename WRITE setPassphrase NOTIFY passphraseChanged)

public:
    KeyGenerator();
    ~KeyGenerator();

    QString type();
    void setType(const QString &t);

    QStringList types();

    QString filename();
    void setFilename(const QString &f);

    QString passphrase();
    void setPassphrase(const QString &p);

public slots:
    void generateKey();

signals:
    void typeChanged();
    void typesChanged();
    void filenameChanged();
    void passphraseChanged();
    void keyGenerated(bool success);

private:
    QString _type;
    QString _filename;
    QString _passphrase;
    QStringList _types;
};
#endif

Next, we need to derive our class from QObject. We declare any properties that we want and the associated methods. Notify methods become signals. In our case, we want to have properties for the selected key type, the list of all valid ssh key types, file name and pass phrase. I arbitrarily made the key type a string. It could have been an enumerated type but it would have made the example more complicated.

Incidentally, a new feature of the Q_PROPERTY macro in Qt 5.1.0 is the MEMBER argument. It allows specifying a class member variable that will be bound to a property without the need to implement the setter or getter functions. That feature was not used here.

We declare methods for the setters and getters and for signals. We also declare one slot called generateKey(). These will all be available to QML. If we wanted to export a regular method to QML, we could mark it with Q_INVOCABLE. In this case I decided to make generateKey() a slot since it might be useful in the future but it could have just as easily been an invocable method.

Finally, we declare any private member variables we will need.

C++ Implementation

Now let’s look at the implementation in KeyGenerator.cpp. Here is the source code:

#include <QFile>
#include <QProcess>
#include "KeyGenerator.h"

KeyGenerator::KeyGenerator()
    : _type("rsa"), _types{"dsa", "ecdsa", "rsa", "rsa1"}
{
}

KeyGenerator::~KeyGenerator()
{
}

QString KeyGenerator::type()
{
    return _type;
}

void KeyGenerator::setType(const QString &t)
{
    // Check for valid type.
    if (!_types.contains(t))
        return;

    if (t != _type) {
        _type = t;
        emit typeChanged();
    }
}

QStringList KeyGenerator::types()
{
    return _types;
}

QString KeyGenerator::filename()
{
    return _filename;
}

void KeyGenerator::setFilename(const QString &f)
{
    if (f != _filename) {
        _filename = f;
        emit filenameChanged();
    }
}

QString KeyGenerator::passphrase()
{
    return _passphrase;
}

void KeyGenerator::setPassphrase(const QString &p)
{
    if (p != _passphrase) {
        _passphrase = p;
        emit passphraseChanged();
    }
}

void KeyGenerator::generateKey()
{
    // Sanity check on arguments
    if (_type.isEmpty() or _filename.isEmpty() or
        (_passphrase.length() > 0 and _passphrase.length() < 5)) {
        emit keyGenerated(false);
        return;
    }

    // Remove key file if it already exists
    if (QFile::exists(_filename)) {
        QFile::remove(_filename);
    }

    // Execute ssh-keygen -t type -N passphrase -f keyfileq
    QProcess *proc = new QProcess;
    QString prog = "ssh-keygen";
    QStringList args{"-t", _type, "-N", _passphrase, "-f", _filename};
    proc->start(prog, args);
    proc->waitForFinished();
    emit keyGenerated(proc->exitCode() == 0);
    delete proc;
}

The constructor initializes some of the member variables. For fun, I used the new initializer list feature of C++11 to initialize the _types member variable which is of type QStringList. The destructor does nothing, at least for now, but is there for completeness and future expansion.

Getter functions like type() simply return the appropriate private member variable. Setters set the appropriate variables, taking care to check that the new value is different from the old one and if so, emitting the appropriate signal. As always, please note that signals are created by the Meta Object Compiler and do not need to be implemented, only emitted at the appropriate times.

The only non-trivial method is the slot generateKey(). It does some checking of arguments and then creates a QProcess to run the external ssh-keygen program. For simplicity and because it typically executes quickly, I do this synchronously and block on it to complete. When done, we emit a signal that has a boolean argument that indicates the key was generated and whether it succeeded or not.

QML Code

Now let’s look at the QML code in main.qml:

// SSH key generator UI

import QtQuick 2.1
import QtQuick.Controls 1.0
import QtQuick.Layouts 1.0
import QtQuick.Dialogs 1.0
import com.ics.demo 1.0

ApplicationWindow {
    title: qsTr("SSH Key Generator")

    statusBar: StatusBar {
    RowLayout {
        Label {
            id: status
            }
        }
    }

    width: 369
    height: 166

    ColumnLayout {
        x: 10
        y: 10

        // Key type
        RowLayout {
            Label {
                text: qsTr("Key type:")
            }
            ComboBox {
                id: combobox
                Layout.fillWidth: true
                model: keygen.types
                currentIndex: 2
            }
        }

        // Filename
        RowLayout {
            Label {
                text: qsTr("Filename:")
            }
            TextField {
                id: filename
                implicitWidth: 200
                onTextChanged: updateStatusBar()
            }
            Button {
                text: qsTr("&Browse...")
                onClicked: filedialog.visible = true
            }
        }

        // Passphrase
        RowLayout {
            Label {
                text: qsTr("Pass phrase:")
            }
            TextField {
                id: passphrase
                Layout.fillWidth: true
                echoMode: TextInput.Password
                onTextChanged: updateStatusBar()
            }

        }

        // Confirm Passphrase
        RowLayout {
            Label {
                text: qsTr("Confirm pass phrase:")
            }
            TextField {
                id: confirm
                Layout.fillWidth: true
                echoMode: TextInput.Password
                onTextChanged: updateStatusBar()
            }
        }

        // Buttons: Generate, Quit
        RowLayout {
            Button {
                id: generate
                text: qsTr("&Generate")
                onClicked: keygen.generateKey()
            }
            Button {
                text: qsTr("&Quit")
                onClicked: Qt.quit()
            }
        }

    }

    FileDialog {
        id: filedialog
        title: qsTr("Select a file")
        selectMultiple: false
        selectFolder: false
        nameFilters: 
        selectedNameFilter: "All files (*)"
        onAccepted: {
            filename.text = fileUrl.toString().replace("file://", "")
        }
    }

    KeyGenerator {
        id: keygen
        filename: filename.text
        passphrase: passphrase.text
        type: combobox.currentText
        onKeyGenerated: {
            if (success) {
                status.text = qsTr('<font color="green">Key generation succeeded.</font>')
            } else {
                status.text = qsTr('<font color="red">Key generation failed</font>')
            }
        }
    }

    function updateStatusBar() {
        if (passphrase.text != confirm.text) {
            status.text = qsTr('<font color="red">Pass phrase does not match.</font>')
            generate.enabled = false
        } else if (passphrase.text.length > 0 && passphrase.text.length < 5) {
            status.text = qsTr('<font color="red">Pass phrase too short.</font>')
            generate.enabled = false
        } else if (filename.text == "") {
            status.text = qsTr('<font color="red">Enter a filename.</font>')
            generate.enabled = false
        } else {
            status.text = ""
            generate.enabled = true
        }
    }

    Component.onCompleted: updateStatusBar()
}

The preceding code is a little long, however, much of the work is laying out the GUI components. The code should be straightforward to follow.

Note that we import com.ics.demo version 1.0. We’ll see where this module name comes from shortly. This makes a new QML type KeyGeneratoravailable and so we declare one. We have access to it’s C++ properties as QML properties, can call it’s methods and act on signals like we do withonKeyGenerated.

A more complete program should probably do a little more error checking and report meaningful error messages if key generation fails (we could easily add a new method or property for this). The UI layout could also be improved to make it properly resizable.

Our main program is essentially a wrapper like qmlscene. All we need to do to register our type with the QML engine is to call:

    qmlRegisterType<KeyGenerator>("com.ics.demo", 1, 0, "KeyGenerator");

This makes the C++ type KeyGenerator available as the QML type KeyGenerator in the module com.ics.demo version 1.0 when it is imported.

Typically, to run QML code from an executable, in the main program you would create a QGuiApplication and a QQuickView. Currently, to use the Qt Quick Components there is some additional work needed if the top level element is an ApplicationWindow or Window. You can look at the source code to see how I implemented this. I basically stripped down the code from qmlscene to the minimum of what was needed for this example.

Here is the full listing for the main program, main.cpp:

#include <QApplication>
#include <QObject>
#include <QQmlComponent>
#include <QQmlEngine>
#include <QQuickWindow>
#include <QSurfaceFormat>
#include "KeyGenerator.h"

// Main wrapper program.
// Special handling is needed when using Qt Quick Controls for the top window.
// The code here is based on what qmlscene does.

int main(int argc, char ** argv)
{
    QApplication app(argc, argv);

    // Register our component type with QML.
    qmlRegisterType<KeyGenerator>("com.ics.demo", 1, 0, "KeyGenerator");

    int rc = 0;

    QQmlEngine engine;
    QQmlComponent *component = new QQmlComponent(&engine);

    QObject::connect(&engine, SIGNAL(quit()), QCoreApplication::instance(), SLOT(quit()));

    component->loadUrl(QUrl("main.qml"));

    if (!component->isReady() ) {
        qWarning("%s", qPrintable(component->errorString()));
        return -1;
    }

    QObject *topLevel = component->create();
    QQuickWindow *window = qobject_cast<QQuickWindow *>(topLevel);

    QSurfaceFormat surfaceFormat = window->requestedFormat();
    window->setFormat(surfaceFormat);
    window->show();

    rc = app.exec();

    delete component;
    return rc;
}

In case it is not obvious, when using a module written in C++ with QML you cannot use the qmlscene program to execute your QML code because the C++ code for the module will not be linked in. If you try to do this you will get an error message that the module is not installed.

With all the talk surrounding it, crowdsourcing science might seem like a new concept and it might be true for citizen science efforts, but it is definitely an old trick to source your research to other researchers. In fact, evolutionary game theory was born (or at least popularized) by one such crowdsourcing exercise; in 1980, Robert Axelrod wanted to find out the best strategy for iterated prisoner’s dilemma and reached out to prominent researchers for strategy submissions to around-robin tournmanet. Tit-for-tat was the winning strategy, but the real victor was Axelrod. His 1981 paper with Hamilton analyzing the result went on to become a standard reference in applications of game theory to social questions (at least outside of economics), agent-based modeling, and — of course — evolutionary game theory. Of Axelrod’s sizeable 47,222 (at time of writing) citations, almost half (23,370) come from this single paper. The tradition of tournaments continues among researchers, I’ve even discussed an imitation tournament on imitation previously.

The cynical moral of the tale: if you want to be noticed then run a game theory tournament. The folks at Brilliant.org— a website offering weekly olympiad-style challange problems in math and physics — took this message to heart, coupled it to the tried-and-true marketing technique of linking to a popular movie/book franchise, and decided to run a Hunger Games themed semi-iterated Prisoner’s dillema tournament. Submit a quick explanation of your strategy and Python script to play the game, and you could be one of the 5 winners of the $1,000 grand prize. Hooray! The submission deadline is August 18th, 2013 and all you need is a Brilliant account and it seems that these are free. If you are a reader of TheEGG blog then I recommend submitting a strategy, and discussing it in the comments (either before or after the deadline); I am interested to see what you come up with.

I will present the rules in m

The Carnegie Mellon University Software Engineering Institute (SEI) has announced the slate of software engineering thought-leaders who will serve as keynote speakers for the Team Software Process (TSP) Symposium 2013. Held in Dallas, Texas, on September 16-19, the TSP Symposium 2013 keynote line-up includes Bill Curtis, senior vice president and chief scientist with Cast Software; Enrique Ibarra, senior vice president of technology of the Mexican Stock Exchange (BMV); and Robert Behler, chief operating officer of the SEI.

The symposium theme, When Software Really Matters, explores the idea that when product quality is critical, high-quality practices are the best way to achieve it.

“When a software system absolutely must work correctly, quality must be built in from the start. A disciplined approach to quality also offers the benefit of lower lifecycle costs. The TSP promotes the application of practices that lead to superior, high-quality products,” said James McHale, TSP Symposium 2013 technical chair. “Our keynote speakers and representatives from industry and government organizations from around the world will share how using TSP helps organizations build quality in from the start when there’s no room for error.”

  • Curtis will assert that the stakes for software-caused operational problems are now larger than ever, approaching a half-billion dollars per incident. Every other aspect of the business is managed by numbers, including IT operations. Software lags behind, however, because the culture of craftsmanship still prevails. Curtis’s talk will challenge that culture: Quality measurement will be challenged for under-measuring non-functional, structural quality, the cause of many operational disasters. Productivity measurement will be challenged for not penalizing baselines when rework is shifted into future releases as technical debt. Software measurement will be challenged to better express outcomes in terms that justify investments for improving quality. The word “quality” will be challenged as the wrong way to frame the argument. Curtis will propose a measurement stack or measurement pyramid to help translate software numbers to business numbers. At the foundation of this pyramid are the Personal Software Process (PSP) and TSP.
  • Ibarra will detail the Mexican Stock Exchange’s (BMV) broad plan of technological renovation that included migration to a new state-of-the-art data center and creating new operational systems with better functionalities and quality attributes. Since 2005, the BMV, which is responsible for operating the cash and derivatives market of the country and is the only exchange in Mexico, has faced the constant challenge of accommodating an exponential growth of demand for its transactional services as well as pressure from the market to offer services with better response times and functionalities. One of the most challenging software projects included in this technological renovation plan was the redesign and construction of the operational system known as the trading engine, which has strict and ambitious requirements for speed (latency), scalability, and continuous availability. The new system, which was to be designed and built internally, and the project were called MoNeT. The BMV had two goals for MoNeT: making sure a carefully considered and reviewed system architecture was in place prior to building the system and adopting a software development process that maximizes the quality of the new system and ensures that it complies with its intended quality attributes. Ibarra will describe the most relevant aspects of the MoNeT project, its performance in production, and the business impact it had on the BMV.
  • Behler, one of only 139 individuals qualified as pilots of the Lockheed SR-71 Blackbird aircraft, will describe his experience flying the fastest, most physically demanding aircraft in the world to gather vital data during the Cold War and the teamwork approach it took to develop the aircraft. The SR-71 was developed in the 1960s with myriad sophisticated sensors used to acquire highly specific intelligence data. The aircraft remains an icon of American aerospace engineering to this day and is considered to be the most effective reconnaissance aircraft in history.

In addition to the keynote speakers, substantial technical program, and organized networking events, the TSP Symposium 2013 also offers practitioners an in-depth learning opportunity with full-day tutorials on introductory and advanced TSP concepts.

“I am very excited about this year’s lineup of keynote speakers and technical presenters. The symposium should be stimulating with presentations on a broad array of topics related to quality-focused software development. It is also an excellent way for participants to network and exchange diverse ideas about how they have used the PSP/TSP approach to achieve their software quality goals,” said Mark Kasunic, Symposium co-chair.

I’ve been listening to the audiobook of Heart of Darkness this week, read by Kenneth Branagh. It’s fantastic. It also reminds me of some jobs I’ve had in the past.

There’s a great passage in which Marlow requires rivets to repair a ship, but finds that none are available. This, in spite of the fact that the camp he left further upriver is drowning in them. That felt familiar. There’s also a famous passage involving a French warship that’s blindly firing its cannons into the jungles of Africa in hopes of hitting a native camp situated within. I’ve had that job as well. Hopefully I can help you avoid getting yourself into those situations.

There are several really good lists of common traits seen in well-functioning engineering organizations. Most recently, there’s Pamela Fox’s list of What to look for in a software engineering culture. More famous, but somewhat dated at this point, is Joel Spolsky’s Joel Test. I want to talk about signs of teams that you should avoid.

This list is partially inspired by Ralph Peters’ Spotting the Losers: Seven Signs of Non-Competitive States. Of course, such a list is useless if you can’t apply it at the crucial point, when you’re interviewing. I’ve tried to include questions to ask and clues to look for that reveal dysfunction that is deeply baked into an engineering culture.

Preference for process over tools. As engineering teams grow, there are many approaches to coordinating people’s work. Most of them are some combination of process and tools. Git is a tool that enables multiple people to work on the same code base efficiently (most of the time). A team may also design a process around Git — avoiding the use of remote branches, only pushing code that’s ready to deploy to the master branch, or requiring people to use local branches for all of their development. Healthy teams generally try to address their scaling problems with tools, not additional process. Processes are hard to turn into habits, hard to teach to new team members, and often evolve too slowly to keep pace with changing circumstances. Ask your interviewers what their release cycle is like. Ask them how many standing meetings they attend. Look at the company’s job listings, are they hiring a scrum master?

Excessive deference to the leader or worse, founder. Does the group rely on one person to make all of the decisions? Are people afraid to change code the founder wrote? Has the company seen a lot of turnover among the engineering leader’s direct reports? Ask your interviewers how often the company’s coding conventions change. Ask them how much code in the code base has never been rewritten. Ask them what the process is for proposing a change to the technology stack. I have a friend who worked at a growing company where nobody was allowed to introduce coding conventions or libraries that the founding VP of Engineering didn’t understand, even though he hardly wrote any code any more.

Unwillingness to confront technical debt. Do you want to walk into a situation where the team struggles to make progress because they’re coding around all of the hacks they haven’t had time to address? Worse, does the team see you as the person who’s going to clean up all of the messes they’ve been leaving behind? You need to find out whether the team cares about building a sustainable code base. Ask the team how they manage their backlog of bugs. Ask them to tell you about something they’d love to automate if they had time. Is it something that any sensible person would have automated years ago? That’s a bad sign.

Not invented this week syndrome. We talk a lot about “not invented here” syndrome and how it affects the competitiveness of companies. I also worry about companies that lurch from one new technology to the next. Teams should make deliberate decisions about their stack, with an eye on the long term. More importantly, any such decisions should be made in a collaborative fashion, with both developer productivity and operability in mind. Finding out about this is easy. Everybody loves to talk about the latest thing they’re working with.

Disinterest in sustaining a Just Culture. What’s Just Culture? This post by my colleague John Allspaw on blameless post mortems describes it pretty well. Maybe you want to work at a company where people get fired on the spot for screwing up, or yelled at when things go wrong, but I don’t. How do you find out whether a company is like that? Ask about recent outages and gauge whether the person you ask is willing to talk about them openly. Do the people you talk to seem ashamed of their mistakes?

Monoculture. Diversity counts. Gender diversity is really important, but it’s not the only kind of diversity that matters. There’s ethnic diversity, there’s age diversity, and there’s simply the matter of people acting differently, or dressing differently. How homogenous is the group you’ve met? Do they all remind you of you? That’s almost certainly a serious danger sign. You may think it sounds like fun to work with a group of people who you’d happily have as roommates, but monocultures do a great job of masking other types of dysfunction.

Lack of a service-oriented mindset. The biggest professional mistakes I ever made were the result of failing to see that my job was ultimately to serve other people. I was obsessed with building what I thought was great software, and failed to see that what I should have been doing was paying attention to what other people needed from me in order to succeed in their jobs. You can almost never fail when you look for opportunities to be of service and avail yourself of them. Be on the lookout for companies where people get ahead by looking out for themselves. Don’t take those jobs.

There are a lot of ways that a team’s culture can be screwed up, but those are my top seven.

Prerequisites

  • A browser with WebGL – this game has been tested on Chrome and Firefox. IE still doesn’t support WebGL, unless you’re using Windows 8.1 with IE11.
  • Three.js library available for download from the Three.js website
  • The Keyboard.js helper library I used for this project, created by Arthur Schreiber at No Karma. Download it from my GitHub repository
  • A basic understanding of what Three.js does. Read this super simple, super quick tutorial by Paul Lewis. It’s basically a short-hand version of this article.

Setup

Get a base index.html running

Step one when making a web-based game is to create the host index.html file. In our case, it only needs to be a very simple set of elements, so we can bundle the CSS styling too.

Import Keyboard.js and Three.js

Three.js is a library contained in just one JavaScript file, so we can grab the minified version from the website.

For Keyboard input, we will need to referencethe aforementioned JavaScript file in our index.html as well.

Create setup() and draw() functions

The setup() function will be the start point for the game code. The draw() function will be run every frame and will handle all the rendering and game logic.

In order to loop the draw() function, we simply utilise the requestAnimationFrame() function call, and pass ‘draw’ as the parameter. Remember, not all browsers natively support the call, and you might have to use Paul Irish’s shim to gain maximum compatibility. Also, it is important to realise that requestAnimationFrame() does not guarantee a fixed frame-rate, so you need to use time-deltas to calculate realistic physics. For a basic game like Pong, we don’t really care about that.

Basic World

Set up the Three.js world and camera

Three.js includes these important elements:

  • Scene
  • Renderer
  • Camera
  • Mesh
  • Light
  • Material

Cameras, Meshes, and Lights need to be added to the scene using the scene.add() function.

Attach a WebGL Three.js Renderer to the DIV

The renderer is attached to whichever HTML DOM element you wish to render the scene to, and a render() call is made each frame to the renderer in order to draw the Three.js scene.

Add a camera to the scene

Three.js has the option to create Perspective and Orthographic cameras. For most uses, Perspective camera is the best choice. We can change position and rotation information of the camera like any other object in the scene.

Draw a sphere and light it

Meshes must be paired with Materials in order to give them a defined look and feel. Meshes can be of many types, include primitives such as Cube, Sphere, Plane and Torus. Materials can have different characteristics depending on their type. The basic Material types include Lambert, Phong, and Basic.

  • Basic renders an unlit Mesh with no shadows or dark shading. A sphere will look like a circle if rendered with Basic.
  • Lambert is a simple diffuse-like lighting that creates shading on sides facing away from a light source. It gives a basic 3D look of surfaces that are matte (non-shiny and non-reflective)
  • Phong is used for achieving a plastic-like look and feel, with the ability to gain highlights that give a much shinier appearance to the Mesh.

Show off your sphere with a Point Light. This is the most basic light, with no direction or rotation. Make sure you tweak the light’s intensity and distance to get it looking good.

Add Game Objects

Draw playing area plane

The playing area will be a Three.js Mesh object of type Plane. Make sure the plane matches the play area, giving a small buffer gap to indicate where the paddles can and can’t go.

Draw paddles

The paddles will be Mesh objects of type Cube. Position each of the paddles on opposite sides of the play area.

1234567891011121314151617181920212223242526272829303132333435363738394041
// set up the paddle vars
paddleWidth = 10;
paddleHeight = 30;
paddleDepth = 10;
paddleQuality = 1;
// set up paddle 1
paddle1 = new THREE.Mesh(
new THREE.CubeGeometry(
paddleWidth,
paddleHeight,
paddleDepth,
paddleQuality,
paddleQuality,
paddleQuality),
paddle1Material);
// add the paddle to the scene
scene.add(paddle1);
// Set up the second paddle
paddle2 = new THREE.Mesh(
new THREE.CubeGeometry(
paddleWidth,
paddleHeight,
paddleDepth,
paddleQuality,
paddleQuality,
paddleQuality),
paddle2Material);
// Add the second paddle to the scene
scene.add(paddle2);
// set paddles on each side of the table
paddle1.position.x = -fieldWidth/2 + paddleWidth;
paddle2.position.x = fieldWidth/2 – paddleWidth;
// lift paddles over playing surface
paddle1.position.z = paddleDepth;
paddle2.position.z = paddleDepth;
view rawBNG_Pong_paddlecreateThis Gist brought to you by GitHub.

If you manipulate the camera positions, as seen in the screenshot, you can give a different perspective to the player.

Basic Logic

Ball movement

The ball will have an X-direction and a Y-direction that determines the movement per frame.

// ball’s x-direction, y-direction and speed per frame
var ballDirX = 1, ballDirY = 1, ballSpeed = 2;

The ball will move at a constant speed in the X-plane every frame. To this end, we will specify a ballSpeed variable that acts as a multiplier for the direction values.

// update ball position over time
ball.position.x += ballDirX * ballSpeed;
ball.position.y += ballDirY * ballSpeed;

We want the ball to have some unpredictable characteristics (e.g. when it gets sliced quite hard) so we will allow the Y-direction to go up to a maximum of ballSpeed * 2. You can tweak the values until you’re happy with how the ball behaves.

// limit ball’s y-speed to 2x the x-speed
// this is so the ball doesn’t speed from left to right super fast
// keeps game playable for humans
if (ballDirY > ballSpeed * 2)
{
ballDirY = ballSpeed * 2;
}
else if (ballDirY < -ballSpeed * 2)
{
ballDirY = -ballSpeed * 2;
}

Ball wall bounce logic

Simple collision detection logic is required to check if the ball is touching each of the side ‘walls’. Using a series of ‘if-else’ statements, we check the ball positions against the predetermined wall positions. In the case of a collision, we simply switch the Y-direction of the ball, creating a bounce effect.

// if ball goes off the top side (side of table)
if (ball.position.y <= -fieldHeight/2)
{
ballDirY = -ballDirY;
}
// if ball goes off the bottom side (side of table)
if (ball.position.y >= fieldHeight/2)
{
ballDirY = -ballDirY;
}

Later, we will edit some of this code in order to implement scoring when the ball passes a paddle.

Keyboard input for paddles

We will utilise a very effective short-cut in order to easily get keyboard input working in this game. Using the Keyboard.js file provided, we simply have to include the reference to it in the index.html file and we are set. Only one function call is required, the Key.isDown() call. Given a parameter, the library checks if that particular key is current being pressed, and returns a boolean value.

// move left
if (Key.isDown(Key.A))
{
// code to move paddle left
}

We use the ‘A’ and ‘D’ keys to move the paddle left and right, but you can edit the Keyboard.js with additional values if you want to use your own control scheme.

var Key = {
_pressed: {},
A: 65,
W: 87,
D: 68,
S: 83,
// add your required key code (ASCII) along with the name here
// for example:
SPACE: 32,
};

While dealing with keyboard input, it is also important to ensure that the input is never blindly updated in game. We have to check that the paddle isn’t made to move off the play area, and we do that with some ‘if-else’ statements as well.

// move left
if (Key.isDown(Key.A))
{
// if paddle is not touching the side of table
// we move
if (paddle1.position.y < fieldHeight * 0.45)
{
paddle1DirY = paddleSpeed * 0.5;
}
// else we don’t move and stretch the paddle
// to indicate we can’t move
else
{
paddle1DirY = 0;
paddle1.scale.z += (10 – paddle1.scale.z) * 0.2;
}
}

Note that we use a paddle direction variable, rather than simply applying a change to the position values. This will come in handy when programming the ball to ‘slice’ when hit at an angle with a fast-moving paddle.

Opponent logic

When you code a game of this calibre, it is of utmost importance that you create a vivid, lush environment with a host of emotional, highly-relatable characters that showcase this generation’s strides forward in technology. Instead, we will code a Pong A.I. that blindly follows the ball, because that is even better.

We can update the opponent difficulty by using a variable instead of introducing magic numbers. This variable will affect the ‘reaction rate’ of the opponent by increasing the Lerp (Linear-Interpolation) time.

When using a Lerp (Linear-Interpolation) function, we must ensure the opponent plays fairly by limiting their maximum travel speed. We do that with a few more if-else statements.

// in case the Lerp function produces a value above max paddle speed, we clamp it
if (Math.abs(paddle2DirY) <= paddleSpeed)
{
paddle2.position.y += paddle2DirY;
}
// if the lerp value is too high, we have to limit speed to paddleSpeed
else
{
// if paddle is lerping in +ve direction
if (paddle2DirY > paddleSpeed)
{
paddle2.position.y += paddleSpeed;
}
// if paddle is lerping in -ve direction
else if (paddle2DirY < -paddleSpeed)
{
paddle2.position.y -= paddleSpeed;
}
}
If want to extend immersion, you could also using the paddle.scale property to stretch the paddle when it can’t be moved. This indicates an issue to the player which they can then address immediately. In order to accomplish this, we must ensure the paddle always Lerps back to the default scale size.
// We lerp the scale back to 1
// this is done because we stretch the paddle at some points
// stretching is done when paddle touches side of table and when paddle hits ball
// by doing this here, we ensure paddle always comes back to default size
paddle2.scale.y += (1 – paddle2.scale.y) * 0.2;

Adding Gameplay

Making the ball reset after missing a paddle

To get the main scoring gameplay working, we need to first remove the ball’s ability to bonce off the paddle-facing walls. To do this, we remove the bounce code from the two corresponding if-else statements.

// if ball goes off the top side (side of table)
if (ball.position.y <= -fieldHeight/2)
{
ballDirY = -ballDirY;
}
// if ball goes off the bottom side (side of table)
if (ball.position.y >= fieldHeight/2)
{
ballDirY = -ballDirY;
}
//// ——————————— ////
CHANGED CODE
//// ——————————— ////
// if ball goes off the ‘left’ side (Player’s side)
if (ball.position.x <= -fieldWidth/2)
{
// CPU scores a point
// update scoreboard
// and reset ball
}
// if ball goes off the ‘right’ side (CPU’s side)
if (ball.position.x >= fieldWidth/2)
{
// player scores a point
// update scoreboard
// and reset ball
}

We can handle scoring in many different ways. For a simple game like this, we can simply increment the corresponding score count variable.

// if ball goes off the ‘left’ side (Player’s side)
if (ball.position.x <= -fieldWidth/2)
{
// CPU scores
score2++;
// update scoreboard HTML
document.getElementById(“scores”).innerHTML = score1 + “-” + score2;
// reset ball to center
resetBall(2);
// check if match over (someone scored maxScore points)
matchScoreCheck();
}

We can then update the HUD element in the DOM by setting its innerHTML value. Finally, we have to reset the ball once someone has scored. A simple function can be written to reset the ball, with a parameter indicating which paddle just lost (so we know which paddle to send the ball to next time).

// resets the ball’s position to the centre of the play area
// also sets the ball direction speed towards the last point winner
function resetBall(loser)
{
// position the ball in the center of the table
ball.position.x = 0;
ball.position.y = 0;
// if player lost the last point, we send the ball to opponent
if (loser == 1)
{
ballDirX = -1;
}
// else if opponent lost, we send ball to player
else
{
ballDirX = 1;
}
// set the ball to move +ve in y plane (towards left from the camera)
ballDirY = 1;
}

Making the ball bounce off paddles

Alright, this is it. The big one. Literally the biggest feature of this game. It’s time to get the paddles hitting the ball. In a simple Pong game, paddle-ball physics are nothing more than a couple of if-else statements. We check the X-position and Y-position of the ball against the paddle’s rectangular bounds, and if they intersect, we bounce the ball away.

// if ball is aligned with paddle1 on x plane
// remember the position is the CENTER of the object
// we only check between the front and the middle of the paddle (one-way collision)
if (ball.position.x <= paddle1.position.x + paddleWidth
&& ball.position.x >= paddle1.position.x)
{
// and if ball is aligned with paddle1 on y plane
if (ball.position.y <= paddle1.position.y + paddleHeight/2
&& ball.position.y >= paddle1.position.y – paddleHeight/2)
{
// ball is intersecting with the front half of the paddle
}
}

It’s also important to check the direction of the ball’s travel, as we only want to check collisions in one direction (the direction towards the opponent.)

// and if ball is travelling towards player (-ve direction)
if (ballDirX < 0)
{
// stretch the paddle to indicate a hit
paddle1.scale.y = 15;
// switch direction of ball travel to create bounce
ballDirX = -ballDirX;
// we impact ball angle when hitting it
// this is not realistic physics, just spices up the gameplay
// allows you to ‘slice’ the ball to beat the opponent
ballDirY -= paddle1DirY * 0.7;
}

We will also affect the ball’s lateral movement depending on the relative speed of the paddle when hitting the ball. This is particularly useful in introducing the biggest variable in Pong: the slice. Slicing the ball is often the only way to confuse and outmaneuver the opponent, so it is vital in this game.

Remember to duplicate the code, but update the values to match the opponent’s paddle. You can use this opportunity to gimp your opponent’s ability somewhat, by reducing the hitbox size or decreasing the slice amount. It’s what we would all do.

Here is the final paddle-ball collision function:

// Handles paddle collision logic
function paddlePhysics()
{
// PLAYER PADDLE LOGIC
// if ball is aligned with paddle1 on x plane
// remember the position is the CENTER of the object
// we only check between the front and the middle of the paddle (one-way collision)
if (ball.position.x <= paddle1.position.x + paddleWidth
&& ball.position.x >= paddle1.position.x)
{
// and if ball is aligned with paddle1 on y plane
if (ball.position.y <= paddle1.position.y + paddleHeight/2
&& ball.position.y >= paddle1.position.y – paddleHeight/2)
{
// and if ball is travelling towards player (-ve direction)
if (ballDirX < 0)
{
// stretch the paddle to indicate a hit
paddle1.scale.y = 15;
// switch direction of ball travel to create bounce
ballDirX = -ballDirX;
// we impact ball angle when hitting it
// this is not realistic physics, just spices up the gameplay
// allows you to ‘slice’ the ball to beat the opponent
ballDirY -= paddle1DirY * 0.7;
}
}
}
// OPPONENT PADDLE LOGIC
// if ball is aligned with paddle2 on x plane
// remember the position is the CENTER of the object
// we only check between the front and the middle of the paddle (one-way collision)
if (ball.position.x <= paddle2.position.x + paddleWidth
&& ball.position.x >= paddle2.position.x)
{
// and if ball is aligned with paddle2 on y plane
if (ball.position.y <= paddle2.position.y + paddleHeight/2
&& ball.position.y >= paddle2.position.y – paddleHeight/2)
{
// and if ball is travelling towards opponent (+ve direction)
if (ballDirX > 0)
{
// stretch the paddle to indicate a hit
paddle2.scale.y = 15;
// switch direction of ball travel to create bounce
ballDirX = -ballDirX;
// we impact ball angle when hitting it
// this is not realistic physics, just spices up the gameplay
// allows you to ‘slice’ the ball to beat the opponent
ballDirY -= paddle2DirY * 0.7;
}
}
}
}
view rawBNG_Pong_paddlecollCompleteThis Gist brought to you by GitHub.

Scoring

In Pong, it is usually simplest to have a maximum score value, such that a game is won when either player reaches that score. To that end, we can easily create a maxScore variable and set it at the start of the match.

We then create a function to check if either player has scored equal or higher than the maximum. This function should be called only when a score has been changed (i.e. when someone scores a point.)

// checks if either player or opponent has reached 7 points
function matchScoreCheck()
{
// if player has 7 points
if (score1 >= maxScore)
{
// stop the ball
ballSpeed = 0;
// write to the banner
document.getElementById(“scores”).innerHTML = “Player wins!”;
document.getElementById(“winnerBoard”).innerHTML = “Refresh to play again”;
}
// else if opponent has 7 points
else if (score2 >= maxScore)
{
// stop the ball
ballSpeed = 0;
// write to the banner
document.getElementById(“scores”).innerHTML = “CPU wins!”;
document.getElementById(“winnerBoard”).innerHTML = “Refresh to play again”;
}
}

After a match is deemed complete, it is simplest to just return the ball to the centre and stop any movement, so that play doesnt inadvertently continue.

Prettifying the Game

HUD

It’s important to give feedback to the player so they know what’s going on. For Pong, the least we can do is keep a scoreboard ticking over. Instead of trying to draw the HUD on the same layer as the game, we can use the other DOM elements to provide the required feedback.

It’s also good to indicate the maximum score as well, so we have another element for that which we will update at match start.

// update the board to reflect the max score for match win
document.getElementById(“winnerBoard”).innerHTML = “First to ” + maxScore + ” wins!”;

Shadows

Finally, it is time to make things look a tad more polished. Three.js has the awesome ability to create shadows for primitive objects (Cube, Plane, Sphere, etc.) so we can utilise that to make the game look nicer.

Shadows can’t be created with just a Point light, so we have to add a DirectionalLight or a SpotLight. A SpotLight shines a circular beam of light onto surfaces, which DirectionalLight simply shines a light in a certain direction with no regard to positioning.

We will use a SpotLight because it clearly indicates where the light originates from and shines towards.

We can update the SpotLight to follow the ball around to give a more dynamic look and feel to the game, whilst showcasing the hard work we just put into the lighting.

// we can easily notice shadows if we dynamically move lights during the game
spotLight.position.x = ball.position.x;
spotLight.position.y = ball.position.y;

To make an object in the scene cast or receive shadows, we simply set their .receiveShadow and .castShadow variables to true. For example,

paddle1 = new THREE.Mesh(
new THREE.CubeGeometry(paddleWidth, paddleHeight, paddleDepth, paddleQuality, paddleQuality, paddleQuality),
paddle1Material);
// add the sphere to the scene
scene.add(paddle1);
paddle1.receiveShadow = true;
paddle1.castShadow = true;

Conclusion

This is but a basic introduction to the power of Three.js, which should allow you to create a basic Pong clone.

Play the latest build of this game here: LATEST BUILD*

Find the latest code at its GitHub page*

You can still do quite a number of things to polish your game, such as

  • Create animations for the paddles and ball
  • Update the HUD to look prettier
  • Import complex objects created in Modeling packages, to design a more immersive environment
  • Move HUD elements inside the game view to allow for full-screen gaming
  • Mess around with complex shaders to create reflections and other cool effects

Boost Dependency Analyzer

I have something special to announce today. A tool I’ve build over the last 2 weeks, which allows to analyze the dependencies in boost. With boost 1.53 this spring, I had the idea to build this, but not the time, as I was busy writing a series over the Papers for Bristol. Back then I realized, how easy it could be to build such a tool, as the dependencies could be read & listed by boosts bcp tool. I already had a prototype for the graphpart from 2010. But lets have a look at the tool:

The tool is very easy to handle, it is based on the out of bcp, which is a tool coming with boost. Actually bcp can help you with ripping libraries out of boost, so that you don’t have to add all of boost to your repository when you would like to use smartpointers. But bcp also has a listing mode, where it only shows the dependencies thats whats my tool build up upon. Lets have a short look at the results, the dependencies of boost 1.54:

A few words on how to read this graph. The libraries in the middle of the “starshape” are the ones with the most dependencies, each line between the nodes is a dependency. A dependency can be one or multiple files. The graphlayout is not weighted.

How to

A short introduction on what you need to get this tool to run. First boost, as this tool is build to analyze boost. I’ve tested with some versions (1.49 – 1.54) of boost. You also need a version of bcp, which is quite easy to build (b2 tools/bcp). Then you simply need to start the tool, if BOOST_ROOT is set, the tool will try to read it, other wise you will be asked to choose the location of boost when clicking on Read dependencies. Next thing is selecting the location of bcp. That is the setup, and the tool will now run for some time. On my machine its 90 seconds to 2 minutes the analysis takes, it might be lot longer on yours, depending on how much cores you got. The tool will spawn for each boost library (~112) a bcp process, and analyze this output in a thread pool. After this is done, the data is loaded into the tool, and then saved to a SQLITE database, which will be used if you start the tool a second time and select this version of boost. Loading from the database is far faster.

A screenshot to illustrate this:

tl_files/blog/bda/bda.png

To the left are all the boost libraries, the number of dependencies is shown in the braces. To the right is a Tabwidget showing all the dependencies, the graph is layouted with boost graph. When you click on show all you’ll get the full view of all dependencies in boost. The layouting is done in the background, so this will take some time to calculate, and is animated when its done. The results of the layouting are good, but not perfect, so that you might have to move some nodes. Exporting supports images, which are transparent PNGs, not all services/tools are happy with that (f.e. facebook, twitter nor G+ could handle the perfectly fine images), this can be fixed by postprocessing the images and adding a white background.

Inner workings

I’ve already written a little about the tools inside, its build with Qt5.1 and boost. Where boost is mostly used for the graph layouting. As I choose to work with Qt5, it has a few more dependencies, for windows this sums up to a 18 mb download, which you’ll find at the end. The tool depends on 3 libraries from my company Code Node: ProcessingSink, a small wrapper around QProcess, that allows to just start a bunch of processes, and lets you connect to the finished and error slot. This was necessary, as I could only spawn 62 parallel processes under windows, so this library does take care of spawning the parallel processes now. Which are currently 50 at a time. GraphLayout is the code that wraps the innerworkings of boost::graph, its a bit dirty, but lets me easily process the graphlayouting. The 3rd library is NodeGraph, which is the Graph UI, based on Qts GraphicsView Framework.
I plan to release the tool and its libraries under GPL later on github, for now I don’t have the time to polish everything.

Problems

One of the earliest questions I had when thinking about building such a tool, was where to get a list of the boost libraries? This sounds easy. But I need to have this readable by machine, not human, so HTML is a great format, but I refused to write a parser for this list yet. I talked to some people about this at C++Now, and most agreed, that the second option would be best: maintainers.txt. Thats what the tool reads currently to find the boost libraries. Unfortunately at least lexical_cast is missing in this list. So, the tool isn’t perfect yet, while lexical_cast is already patched, I’m not sure if anything else is missing. A candidate could be signals, as its not maintained anymore. Currently the tool analyzes for 1.54 112 libraries.

boost dependencies

Working for 2 weeks on this tool has given me some inside knowledge about the dependencies in boost. First, the way it is shown in the tool, is the view of bcp. Some dependencies will not affect the user, as they are internal. f.e. a lot of libraries have a dependency to boost::test, simply because they provide their tests with it. The bcp tool really gets you ALL the dependencies. Also most (or was it all?) libraries depend on boost::config. I plan to add filtering later, so that the user has the ability to filter some of the libraries in the GraphView.

The tool

Here is how to get the tool for now: there is a download for the binaries for windows and linux. I’ll try to get you a deb package as soon as I have time, but for now its only the binaries for linux, you’ll have to make sure to have Qt5.1 etc. on linux too, as I do not provide them. For Windows, its 2 archives you’ll need to download: the programm itself, and needed dlls for Qt5.1 if you don’t have the SDK installed ( in this case you also could copy them from the bin directory)

Note on linux: this is a one day old beta version. Will update this later.

As one of the developers of the Glasgow Haskell Compiler (GHC) for almost 15 years, I have seen Haskell grow from a niche research language into a rich and thriving ecosystem. I spent a lot of that time working on GHC’s support for parallelism and concurrency. One of the first things I did to GHC in 1997 was to rewrite its runtime system, and a key decision we made at that time was to build concurrency right into the core of the system rather than making it an optional extra or an add-on library. I like to think this decision was founded upon shrewd foresight, but in reality it had as much to do with the fact that we found a way to reduce the overhead of concurrency to near zero (previously it had been on the order of 2%; we’ve always been performance-obsessed). Nevertheless, having concurrency be non-optional meant that it was always a first-class part of the implementation, and I’m sure that this decision was instrumental in bringing about GHC’s solid and lightning-fast concurrency support.

Haskell has a long tradition of being associated with parallelism. To name just a few of the projects, there was the pH variant of Haskell derived from the Id language, which was designed for parallelism, the GUM system for running parallel Haskell programs on multiple machines in a cluster, and the GRiP system: a complete computer architecture designed for running parallel functional programs. All of these happened well before the current multicore revolution, and the problem was that this was the time when Moore’s law was still giving us ever-faster computers. Parallelism was difficult to achieve, and didn’t seem worth the effort when ordinary computers were getting exponentially faster.

Around 2004, we decided to build a parallel implementation of the GHC runtime system for running on shared memory multiprocessors, something that had not been done before. This was just before the multicore revolution. Multiprocessor machines were fairly common, but multicores were still around the corner. Again, I’d like to think the decision to tackle parallelism at this point was enlightened foresight, but it had more to do with the fact that building a shared-memory parallel implementation was an interesting research problem and sounded like fun. Haskell’s purity was essential—it meant that we could avoid some of the overheads of locking in the runtime system and garbage collector, which in turn meant that we could reduce the overhead of using parallelism to a low-single-digit percentage. Nevertheless, it took more research, a rewrite of the scheduler, and a new parallel garbage collector before the implementation was really usable and able to speed up a wide range of programs. The paper I presented at the International Conference on Functional Programming (ICFP) in 2009 marked the turning point from an interesting prototype into a usable tool.

All of this research and implementation was great fun, but good-quality resources for teaching programmers how to use parallelism and concurrency in Haskell were conspicuously absent. Over the last couple of years, I was fortunate to have had the opportunity to teach two summer school courses on parallel and concurrent programming in Haskell: one at the Central European Functional Programming (CEFP) 2011 summer school in Budapest, and the other the CEA/EDF/INRIA 2012 Summer School at Cadarache in the south of France. In preparing the materials for these courses, I had an excuse to write some in-depth tutorial matter for the first time, and to start collecting good illustrative examples. After the 2012 summer school I had about 100 pages of tutorial, and thanks to prodding from one or two people (see the Acknowledgments), I decided to turn it into a book. At the time, I thought I was about 50% done, but in fact it was probably closer to 25%. There’s a lot to say! I hope you enjoy the results.

Audience

You will need a working knowledge of Haskell, which is not covered in this book. For that, a good place to start is an introductory book such as Real World Haskell (O’Reilly), Programming in Haskell (Cambridge University Press), Learn You a Haskell for Great Good! (No Starch Press), or Haskell: The Craft of Functional Programming (Addison-Wesley).

How to Read This Book

The main goal of the book is to get you programming competently with Parallel and Concurrent Haskell. However, as you probably know by now, learning about programming is not something you can do by reading a book alone. This is why the book is deliberately practical: There are lots of examples that you can run, play with, and extend. Some of the chapters have suggestions for exercises you can try out to get familiar with the topics covered in that chapter, and I strongly recommend that you either try a few of these, or code up some of your own ideas.

As we explore the topics in the book, I won’t shy away from pointing out pitfalls and parts of the system that aren’t perfect. Haskell has been evolving for over 20 years but is moving faster today than at any point in the past. So we’ll encounter inconsistencies and parts that are less polished than others. Some of the topics covered by the book are very recent developments: Chapters 4, 5, 6, and pass:[14 cover frameworks that were developed in the last few years.

The book consists of two mostly independent parts: Part I and Part II. You should feel free to start with either part, or to flip between them (i.e., read them concurrently!). There is only one dependency between the two parts: Chapter 13 will make more sense if you have read Part I first, and in particular before reading “The ParIO monad”, you should have read Chapter 4.

While the two parts are mostly independent from each other, the chapters should be read sequentially within each part. This isn’t a reference book; it contains running examples and themes that are developed across multiple chapters.

CoffeeScript, as many people know, is a transpile-to-JavaScript language.1 For the most part, it does not introduce major changes in semantics. For example, this:

-> 'Hello, world'

Transpiles directly to:

function () { return 'Hello, world'; }

This is convenient syntactic sugar, and by removing what some folks call the “syntactic vinegar” of extraneous symbols, it encourages the use of constructs that would otherwise make the code noisy and obscure the important meaning. The vast majority of features introduced by CoffeeScript are of this nature: They introduce local changes that transpile directly to JavaScript.2

CoffeeScript also introduces features that don’t exist in JavaScript, such as destructuring assignment and comprehensions. In each case, the features compile directly to JavaScript without introducing changes elsewhere in the program. And since they don’t look like existing JavaScript features, little confusion is created.

equals doesn’t equal equals

One CoffeeScript feature does introduce confusion, and the more you know JavaScript the more confusion it introduces. This is the behaviour of the assignment operator, the lowly (and prevalent!) equals sign:

foo = 'bar'

Although it looks almost identical to assignment in JavaScript:

foo = 'bar';

It has different semantics. That’s confusing. Oh wait, it’s worse than that: Sometimes it has different semantics. Sometimes it doesn’t.

So what’s the deal with that?

Well, let’s review the wonderful world of JavaScript. We’ll pretend we’re in a browser application, and we write:

foo = 'bar';

What does this mean? Well, it depends: If this is in the top level of a file, and not inside of a function, then foo is a global variable. In JavaScript, global means global across all files, so you are now writing code that is coupled with every other file in your application or any vendored code you are loading.

But what if it’s inside a function?

function fiddleSticks (bar) {
  foo = bar;
  // ...
}

For another example, many people enclose file code in an Immediately Invoked Function Expression (“IIFE”) like this:

;(function () {
  foo = 'bar'
  // more code...
})();

What do foo = 'bar'; or foo = bar; mean in these cases? Well, it depends as we say. It depends on whether foo is declared somewhere else in the same scope. For example:

function fiddleSticks (bar) {
  var foo;
  foo = bar;
  // ...
}

Or:

function fiddleSticks (bar) {
  foo = bar;
  // ...
  var foo = batzIndaBelfrie;
  // ...
} 

Or even:

function fiddleSticks (bar) {
  foo = bar;
  // ...
  function foo () {
    // ...
  }
  // ...
}

Because of something called hoisting,3 these all mean the same this: foo is local to function fiddleSticks, and therefore it is NOT global and ISN’T magically coupled to every other file loaded whether written by yourself or someone else.

nested scope

JavaScript permits scope nesting. If you write this:

function foo () {
  var bar = 1;
  var bar = 2;
  return bar;
}

Then bar will be 2. Declaring bar twice makes no difference, since both declarations are in the same scope. However, if you nest functions, you can nest scopes:

function foo () {
  var bar = 1;
  function foofoo () {
    var bar = 2;
  }
  return bar;
}

Now function foo will return 1 because the second declaration of bar is inside a nested function, and therefore inside a nested scope, and therefore it’s a completely different variable that happens to share the same name. This is called shadowing: The variablebar inside foofoo shadows the variable bar inside foo.

javascript failure modes

Now over time people have discovered that global variables are generally a very bad idea, and accidental global variables doubly so. Here’s an example of why:

function row (numberOfCells) {
  var str = '';
  for (i = 0; i < numberOfCells; ++i) {
    str = str + '<td></td>';
  }
  return '<tr>' + str + '</tr>';
}

function table (numberOfRows, numberOfColumns) {
  var str = '';
  for (i = 0; i < numberOfRows; ++i) {
    str = str + row(numberOfColumns);
  }
  return '<table>' + str + '</table>';
}

Let’s try it:

table(3, 3)
  //=> "<table><tr><td></td><td></td><td></td></tr></table>"

We only get one row, because the variable i in the function row is global, and so is the variable i in the function table, so they’re the exact same global variable. Therefore, after counting out three columns, i is 3 and the for loop in table finishes. Oops!

And this is especially bad because the two functions could be anywhere in the code. If you accidentally use a global variable and call a function elsewhere that accidentally uses the same global variable, pfft, you have a bug. This is nasty because there’s this weird action-at-a-distance where a bug in one file reaches out and breaks some code in another file.

Now, this isn’t a bug in JavaScript the language, just a feature that permits the creation of very nasty bugs. So I call it a failure mode, not a language bug.

coffeescript to the rescue

CoffeeScript addresses this failure mode in two ways. First, all variables are local to functions. If you wish to do something in the global environment, you must do it explicitly. So in JavaScript:

UserModel = Backbone.Model.extend({ ... });
var user = new UserModel(...);

While in CoffeeScript:

window.UserModel = window.Backbone.Model.extend({ ... })
user = new window.UserModel(...)

Likewise, CoffeeScript bakes the IIFE enclosing every file in by default. So instead of:

;(function () {
  // ...
})();

You can just write your code.4

The net result is that it is almost impossible to replicate the JavaScript failure mode of creating or clobbering a global variable by accident. That is a benefit.

what would coffeescript do?

This sounds great, but CoffeeScript can be surprising to JavaScript programmers. Let’s revisit our table function. First, we’ll fix it:

function row (numberOfCells) {
  var i,
      str = '';
  for (i = 0; i < numberOfCells; ++i) {
    str = str + '<td></td>';
  }
  return '<tr>' + str + '</tr>';
}

function table (numberOfRows, numberOfColumns) {
  var i,
      str = '';
  for (i = 0; i < numberOfRows; ++i) {
    str = str + row(numberOfColumns);
  }
  return '<table>' + str + '</table>';
}

table(3, 3)
  //=> "<table><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr></table>"

Good! Now suppose we notice that no function calls row other than table. Although there is a slightly more “performant” way to do this, we decide that the clearest and simplest way to indicate this relationship is to nest row inside table Pascal-style:

function table (numberOfRows, numberOfColumns) {
  var i,
      str = '';
  for (i = 0; i < numberOfRows; ++i) {
    str = str + row(numberOfColumns);
  }
  return '<table>' + str + '</table>';

  function row (numberOfCells) {
    var i,
        str = '';
    for (i = 0; i < numberOfCells; ++i) {
      str = str + '<td></td>';
    }
    return '<tr>' + str + '</tr>';
  }
}

It still works like a charm, because the i in row shadows the i in table, so there’s no conflict. Okay. Now how does it work in CoffeeScript?

Here’s one possible translation to CoffeeScript:

table = (numberOfRows, numberOfColumns) ->
  row = (numberOfCells) ->
    str = ""
    i = 0
    while i < numberOfCells
      str = str + "<td></td>"
      ++i
    "<tr>" + str + "</tr>"
  str = ""
  i = 0
  while i < numberOfRows
    str = str + row(numberOfColumns)
    ++i
  return "<table>" + str + "</table>"

table(3,3)
  #=> "<table><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr></table>"

It works just fine. Here’s another:

table = (numberOfRows, numberOfColumns) ->
  str = ""
  i = 0
  row = (numberOfCells) ->
    str = ""
    i = 0
    while i < numberOfCells
      str = str + "<td></td>"
      ++i
    "<tr>" + str + "</tr>"
  str = ""
  i = 0
  while i < numberOfRows
    str = str + row(numberOfColumns)
    ++i
  return "<table>" + str + "</table>"

table(3,3)
  #=> "<table><tr><td></td><td></td><td></td></tr></table>"

Broken! And a third:

str = ""
i = 0
table = (numberOfRows, numberOfColumns) ->
  row = (numberOfCells) ->
    str = ""
    i = 0
    while i < numberOfCells
      str = str + "<td></td>"
      ++i
    "<tr>" + str + "</tr>"
  str = ""
  i = 0
  while i < numberOfRows
    str = str + row(numberOfColumns)
    ++i
  return "<table>" + str + "</table>"

table(3,3)
  #=> "<table><tr><td></td><td></td><td></td></tr></table>"

Also broken! Although the three examples look similar, the first gives us what we expect but the second and third do not. What gives?

Well, CoffeeScript doesn’t allow us to “declare” that variables are local with var. They’re always local. But local to what? In CoffeeScript, they’re local to the function that either declares the variable as a parameter or that contains the first assignment to the variable.5 So in our first example, reading from the top, the first use of str and i is inside the row function, so CoffeeScript makes them local to row.

A little later on, the code makes an assignment to i and str within the table function. This scope happens to enclose row’s scope, but it is different so it can’t share the strand i variables. CoffeeScript thus makes the i and str in table variables local totable. As a result, the i and str in row end up shadowing the i and str in table.

The second example is different. The first i encountered by CoffeeScript is in table, so CoffeeScript makes it local to table as we’d expect. The second i is local to row. But since row in enclosed by table, it’s possible to make that i refer to the i already defined, and thus CoffeeScript does not shadow the variable. The i inside row is the same variable as the i inside table.

In the third example, i (and str) are declared outside of both table and row, and thus again they all end up being the same variable with no shadowing.

Now, CoffeeScript could scan an entire function before deciding what variables belong where, but it doesn’t. That simplifies things, because you don’t have to worry about a variable being declared later that affects your code. Everything you need to understand is in the same file and above your code.

In many cases, it also allows you to manipulate whether a variable is shadowed or not by carefully controlling the order of assignments. That’s good, right?

all those against the bill, say “nay nay!”

Detractors of this behaviour say this is not good. When JavaScript is written using var, the meaning of a function is not changed by what is written elsewhere in the file before the code in question. Although you can use this feature to control shadowing by deliberately ordering your code to get the desired result, a simple refactoring can break what you’ve already written.

For example, if you write:

table = (numberOfRows, numberOfColumns) ->
  row = (numberOfCells) ->
    str = ""
    i = 0
    while i < numberOfCells
      str = str + "<td></td>"
      ++i
    "<tr>" + str + "</tr>"
  str = ""
  i = 0
  while i < numberOfRows
    str = str + row(numberOfColumns)
    ++i
  return "<table>" + str + "</table>"

table(3,3)
  #=> "<table><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr></table>"

All will be well, until you are debugging late one night, and you add:

console.log('Hello!') for i in [1..5]

table = (numberOfRows, numberOfColumns) ->
  row = (numberOfCells) ->
    str = ""
    i = 0
    while i < numberOfCells
      str = str + "<td></td>"
      ++i
    "<tr>" + str + "</tr>"
  str = ""
  i = 0
  while i < numberOfRows
    str = str + row(numberOfColumns)
    ++i
  return "<table>" + str + "</table>"

table(3,3)
  #=> "table><tr><td></td><td></td><td></td></tr></table>"

This breaks your code because the i you used at the top “captures” the other variables so they are now all the same thing. To someone used to JavaScript, this is a Very Bad Thing™. When you write this in JavaScript:

function row (numberOfCells) {
  var i,
      str = '';
  for (i = 0; i < numberOfCells; ++i) {
    str = str + '<td></td>';
  }
  return '<tr>' + str + '</tr>';
}

It will always mean the same thing no matter where it is in a file, and no matter what comes before it or after it. There is no spooky “action-at-a-distance” where code somewhere else changes what this code means. Whereas in CoffeeScript, you don’t know whether the iin row is local to row or not without scanning the code that comes before it in the same or enclosing scopes.

coffeescript’s failure mode

In this case, CoffeeScript has a failure mode: The meaning of a function seems to be changed by altering its position within a file or (in what amounts to the same thing) by altering code that appears before it in a file in the same or enclosing scopes. In contrast, JavaScript’s var declaration never exhibits this failure mode. JavaScript has a different action-at-a-distance failure mode, where neglecting var causes action at a much further distance: The meaning of code can be affected by code written in an entirely different file.

Mind you, the result of calling our row function is not affected by declaring an i in an enclosing scope. Our function always did what it was expected to do and always will. Although you and I know why the change breaks the table function is that row now uses an enclosed variable, imagine that we were writing unit tests. All of our tests for row would continue to pass, it’s the tests for table that break. So in an evidence-based programming sense, when we maintain the habit of always initializing variables we expect to use locally, changing code outside of those functions only changes the evidence that the enclosing code produces.

So one way to look at this is that row is fine, but moving i around changes the meaning of the code where you move i. And why wouldn’t you expect making changes to tableto change its meaning?

so which way to the asylum?

If you ask around, you can find people who dislike JavaScript’s behaviour, and others who dislike CoffeeScript’s behaviour. Accidentally getting global variables when you neglectvar is brutal, and action-at-a-distance affecting the meaning of a function (even if it is always within the same file) flies against everything we have learned about the importance of writing small chunks of code that completely encapsulate their behaviour.

Of course, programmers tend to internalize the languages they learn to use. If you write a lot of JavaScript, you habitually use var and may have tools that slap your wrist when you don’t. You’re bewildered by all this talk of action-at-a-distance. It will seems to you to be one of those rookie mistake problems that quickly goes away and is not a practical concern.

Likewise, if you write twenty thousand lines of CoffeeScript, you may never be bitten by its first-use-is-a-declaration behaviour. You may be in the habit of using variable names likeiRow and iColumn out of habit. You may find that your files never get so large and your functions so deeply nested that a “capture” problem takes longer than three seconds to diagnose and fix.

It’s a bit of a cop-out, but I suggest that this issue resembles the debate over strong, manifest typing vs. dynamic typing. In theory, one is vastly preferable to the other. But in practice, large stable codebases are written with both kinds of languages, and programmers seem to adjust to overcome the failure modes of their tools unconsciously while harvesting the benefits that each language provides.


  1. Yes, “transpile” is a real word, or at least, a real piece of jargon. It’s a contraction of “transcompiler,” which is a compiler that translates one language to another language at a similar level of abstraction. There’s room for debate over what constitutes a “similar level of abstraction.” https://en.wikipedia.org/wiki/Source-to-source_compiler

  2. There are other possibilities: You could write a Tail-Call Optimized language that transpiles to JavaScript, however its changes wouldn’t always be local: Some function calls would be rewritten substantially to use trampolining. Or adding continuations to a language might cause everything to be rewritten in continuation-passing style.

  3. Scanning all of the code first is called “hoisting,” in part because some declarations nested in blocks are “hoisted” up to the level of the function, and all declarations are “hoisted” to the top of the function. This is a source of confusion for some programmers, but it isn’t germane to this essay.

  4. If you don’t want the file enclosed in an IIFE, you can compile your CoffeeScript with the--bare command-line switch.