2013-08-23

Co.Labs

How An Arcane Coding Method From 1970s Banking Software Could Save The Sanity Of Web Developers Everywhere

Forty years ago, a Canadian bank pioneered a brand new computer system that allowed non-programmers to help write code. The paradigm was so disruptive that it was ignored by computer scientists for decades. But as web apps get increasingly complex, and web devs become increasingly stressed out, "flow-based programming" may be raging back to life.



Today’s web programmers grapple with problems that people in the early days never had to deal with. They’re building complex UIs, juggling a bunch of APIs, and running multiple processes at the same time. All of these tasks require mastering the flow of data between application components in real-time, something which even the most advanced developers struggle with.

Why can’t things be easier? Most modern programming techniques descend from a 60-year-old computing paradigm which stipulates, among other things, that all programs must run one step at a time—not good for handling multiple tasks at once. But there’s no reason it has to be that way. Many computer scientists have invented alternative programming techniques that attempt to solve these problems, only to be shunned by an establishment uncomfortable with thinking about programming differently. Fortunately, the dogma that exists today is now dying.

Dan Tocchini, CEO of a startup called The Grid, is part of a new generation of programmers who grew up struggling with complex multithreaded programming, asynchronous I/O, and nearly unlimited sources of data from hundreds of modern APIs. He believes that resurrecting an old paradigm from the 1970s might be the solution.

The Solution Is Not "More Programmers"

In the 1970s, a Canadian bank serving 5 million customers implemented a brand new computer system using a new and little heard-of software development paradigm called "flow-based programming" (FBP). The software was a hit, largely because it allowed the bank to build working applications using easy-to-visualize graphs of data flowing between components, which non-developers could understand. The new apps were so maintainable that some of them are still being used 40 years later, and yet FBP itself never caught on because most programmers at the time resisted adopting the new paradigm.

One of the reasons programmers didn’t warm to FBP is that it requires a new way of thinking about development. Traditional apps become easier to break as they grow larger and more complex. The usual solution to this problem is to throw more developers at the project and use careful deployment strategies like continuous integration to avoid breaking things. Lots of developers swear by the brute-force approach, but it doesn’t scale.

"What we need is not more programmers. What we need is to enable non-programmers to participate in the creation process, not just the ideation process," says Kenneth Kan, CTO of a company called Pixbi and a recent convert to flow-based programming. "Rather than making us humans think like machines, it's time to make machines to be able to think more like us."

Traditional programming paradigms force programmers to think one step at a time and combine the actual work done by an application, its program, with the order in which it runs, its logic or flow. The upshot is that applications quickly become tangled jumbles of code that rely on each other. If you want to change the order of one sequence of events, you need to rewrite everything that depends on it. And good luck getting anyone else, even another developer, to understand it.

"It just became a fucking mess. It was just unmanageable," says Tocchini about the nightmare project that prompted him to look for new solutions. His company is busy resurrecting flow-based programming with a framework called NoFlo, an implementation of FBP for NodeJS.

According to Tocchini, managing the expanding codebase of the JavaScript front-end MVC framework he was using became totally untenable as he added more features, increasing the complexity of components until it took a week just to make small changes. Switching to flow-based programming allowed him to focus on individual components, and understand bottlenecks visually.

Image by The Grid

"The workflow of sticking to the components, I love it man. It just feels so much better," he says. "You don’t have those kinds of brutal moments and if you do, it’s because you wired things wrong. Your spaghetti looks like spaghetti, right?"

Software Development Isn’t A One-Person Show

"It doesn't ‘feel’ like programming," says Kan. Through NoFlo, he’s become a convert to the FBP development process. "It's about solutions without coding much. As long as web programming remains a hacker-only party, FBP would have no room to thrive."

J. Paul Morrison, the creator of flow-based programming, has a slightly different take. He doesn’t think that we need fewer programmers, but that programming will become mainstream as developers make way for a new type of application architect.

"There’s two roles: There’s the person building componentry, who has to have experience in a particular program area, and there’s the person who puts them together," explains Morrison. "And it’s two different skills."

If NoFlo succeeds, it could herald a new paradigm of web programming. Imagine a world where anyone can understand and build web applications, and developers can focus on programming efficient components to be put to work by this new class of application architects. In a way, this is the same promise as the "learn to code" movement, which wants to teach everyone to be a programmer. Just without the programming.

How A Brilliant Paradigm Got Buried In History

FBP seems like it could be the solution to many modern development problems. It’s naturally parallelizing, which helps programmers handle multiple, concurrent tasks and complex, physics-based UIs. Because it lends itself to visualization, flow-based applications are easy for the increasing number of non-technical product stakeholders to understand. So why did it take so long to catch on?

Image by J. Paul Morrison

Morrison discovered flow-based programming a decade before he built the software for the bank, while working as an engineer at IBM. He was in Montreal experimenting with discrete simulations of cars moving through gas stations, when it occurred to him that the same techniques used in discrete modeling could be applied to software development.

"I said, ‘What if we could have some different processes that were running at the same time, with data going through them?,'" recalls Morrison. "'How hard would that be to do for programming with business data rather than cars?’" I drew some pictures and suddenly came up with this beautiful component that would make a big chunk of programming more reusable."

The idea of modeling applications as graphs of data moving between independent, reusable blocks of code was not new. In fact, Morrison admits that he lifted the idea from his experiences working with early card-based computing systems where you could literally see stacks of cards carrying data moving back and forth in the machine room. What was new about the approach was applying it to Von Neumann architecture, the basis for all modern computer design.

John Von Neumann

Named for Princeton mathematician John Von Neumann, the design replaced the program-controlled card systems that Morrison worked on with computers that could store instructions in memory. It specifies, among other things, that these instructions must pass from memory to the CPU one "word" at a time, a constraint which forces programmers to write lines of code that execute one at a time, largely in the order in which they’re written.

Morrison’s flow-based concept effectively rebuilt the earlier program-controlled approach, where data is routed in streams to and from purpose-built applications, on top of Von Neumann architecture. This allows programmers to write individual functions that execute one step at a time, but route data between these functions asynchronously and concurrently using a network definition graph. According to Morrison, flow-based programming is an improvement over other approaches because it allows programmers to build reusable components that can be integrated into any application, and because it allows them to visualize applications.

"It capitalizes on the visual capabilities of human beings. Instead of doing reams of text, you can draw pictures," explains Morrison. "We used to do these diagrams on huge sheets of paper, and the neat thing about that was you could annotate them and they would get all dog-eared and people would draw cartoons on them and comments and you could put descriptive information on them. Then you could basically turn these into flow-based programming network definitions."

The idea was so obvious that IBM declined to patent the new approach, calling it "much more like a law of nature" than a new invention, according to Morrison. Instead, the company published FBP as a technical disclosure bulletin, effectively making it public domain. And yet, except for a few rare applications like the Canadian bank, it never really caught on.

According to Morrison, FBP’s failure to take off was caused partly by his inability to properly market the approach, and partly by the programming paradigm that had already formed around Von Neumann architecture, which he compares to the scientific paradigms described by Thomas Kuhn in his groundbreaking book The Structure of Scientific Revolutions.

"You have a computer that can do anything, and one thing at a time, and that’s such a powerful paradigm, nobody sees any problem with it, any need to come up with something different," says Morrison. "All programs were loops within loops within loops. If you had a problem, you were told you just didn’t think hard enough. It was not the computer’s problem or the architecture’s problem, it was your problem."

Kuhn’s book on scientific revolutions includes a famous quote from physicist Max Planck about what really causes paradigm shifts:

A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.

Forty-five years have now passed since Morrison first discovered FBP, and the programmers who worked during the beginning of the Von Neumann era are dying off. In their place, a new generation of programmers frustrated with the shortcomings of the traditional paradigms is coming of age, and they’re willing to try something new.

How People Are Rediscovering Flow-Based Programming

Tocchini stumbled upon FBP almost by accident, when he saw some designers at Facebook using Apple’s flow-based Quartz Composer tool to create iOS app mockups instead of Photoshop. Although the flow-based approach as described by Morrison never fully caught on, some flow-based concepts did eventually make their way to niche developer communities, like game development and film compositing, where flow-like tools are used to model physical interactions. Flow-based programming is especially useful for these applications, because modeling realistic physics requires running multiple equations simultaneously. To help app designers make use of the new Core Image and Core Video advanced graphical APIs that were introduced in OS X 10.4 "Tiger", Apple included the Quartz Composer mockup tool in the software development kit in 2005.

Tocchini loved the idea of live prototyping, especially because web app UIs are becoming more complex and implementing more physics-based interactions, but wanted to take it a step further.

"One of the problems is that it has to be rebuilt," says Tocchini. "It’s purely prototyping. The only reason it’s like that is because of what you get out of Quartz Composer. There’s no reason why you can’t have a fully functioning final product with it."

So, he joined with Henri Bergius, the creator of NoFlo, and started porting his company’s applications from their old MVC framework to the new paradigm.

The NoFlo Programming Interface

NoFlo works by tying together CommonJS components with a graph, specified in JSON. It also includes a domain-specific language that allows developers to embed graphs in existing applications. It’s been available for two years and has a small, devoted user base who have helped Tocchini and Bergius build 250 components.

Like Morrison’s original FBP implementation, however, it hasn’t become widespread, especially compared with traditional Node-based frameworks like Meteor or SailsJS. Tocchini and Bergius believe that NoFlo is missing a key component: An interface.

To facilitate building an interface for their framework, Tocchini and Bergius recently launched a Kickstarter campaign to fund the development of a UI for NoFlo. The campaign raised $80,000 of the requested $100,000 in just six days, signaling that interest in FBP is growing.

One of the main benefits of flow-based programming is that it takes control logic out of the main program and instead abstracts it into a network graph that anyone, even those with no programming experience, can understand. Imagine a program as a graph of "black boxes" with inputs and outputs connected by wires that represent streams of data. Got it? Congratulations, you’ve built a program.

It will take developers giving up some of the "feel" and control over programming that Kan describes if FBP is to succeed. But in a way, the promise of FBP is every programmer’s dream. Instead of struggling to make stakeholders understand why adding a new feature that requires reading yet another API concurrently is difficult, imagine if they could simply see and understand it. Instead of dealing with connecting bits and pieces of your applications, you could leave that up to the people who use it, and just focus on making things work.

[Poster image by The Grid]






Add New Comment

53 Comments

  • John Cowan

    The underpinnings of FBP are important in themselves, quite apart from the visual side. Unix pipelines are a simple case of an FBP flow graph, and indeed are probably the most widely used FBP technology in the world. I maintained a publishing system for five years based entirely on shell scripts full of pipelines that mostly reused existing Posix-standard components, with occasional new components written in Perl. No non-programmers were harmed during the production of this system.

  • Ludovic Urbain

    The only solution to all programming problems is simply hire a better programmer.

    It's not practical because you run out of better programmers at some point.

    Flow-Based may look nice when you have 100 items on your graph, just try digesting it when there are 100.000 and you'll come back for another innovative solution to an age-old problem.

    The point is, if you can't fit the whole abstraction in your head, you can't solve the problem, and that doesn't depend on FBP or FP or anything.

    You have to be smart to solve complex problems, and no tool is going to help you with that.

  • Brett Reifers

    This is hilariously uncanny. Working with a startup (as I'm writing this) that has built an automated work"Flo" platform for online/cloud service integrations. Our [Azuqua's] vision is to open up enterprise automation and IT development in an intuitive [visual] way while avoiding any limitations in what can be built. Thereby allowing anyone to "code."

    It's one thing to find that there is an overlap in ideation... but little did I know that the majority of what we are doing now is tailored to the core principles of FBP.

    TL;DR - The 70's called... They want credit for our "Flo"

  • Alex V

    Bigger question: what's with the Amazon affiliate marketing URLs behind your Quartz Composer links? It's not a commercial application you can buy on Amazon, it's an innate part of the Mac OS / XCode.

  • Vance Feld

    I've used Quartz, vvvv and Max MSP. All three were terribly difficult to maintain. All it really did was draw your spaghetti for you.

  • Jonathan Van Matre

    I haven't used Quartz or vvvv, but Max/MSP is only difficult to maintain if you don't maintain it. If you actually modularize as you go, creating abstractions and subpatchers, you can have a very clean graph with the ability to drill down as needed.

    Whether you get spaghetti or ravioli depends on the chef, not the pan.

  • Danny Korenblum

    Kool project! Reminds me a bit of Simulink which is an add-on for Matlab that I've used in developing embedded DSP software for real-time image processing on medical devices such as digital colposcopes. It was a pain to learn, but the most useful "block" ended up being the "embedded Matlab" block which allowed me to insert customized hand-written code. So in the end, I found that the built in "blocks" were too limited in their capabilities for achieving my performance requirements and I ended up using the "custom" blocks for all of the tricky stuff. However, despite this it was sometimes helpful and enjoyable to have the graphical layout / diagram view of the software and it made code reviews almost insanely easy whereas otherwise they tend to get ignored or infrequently performed.

    I'm currently developing an open source project for HTML5 game development using CoffeeScript/JavaScript and after seeing this article, I'm wondering if I could create an interface for my code that would work with NoFlo and allow me to use my modules and objects as "blocks" in the graphical UI. Is there such a capability, and if so, an open specification or some other type of documentation on how to implement this? I'd be interested in learning more since my goal is to provide users with the easiest and most efficient tools for creating games.

    Thanks,

    Danny

  • healingshoes

    "...The most dangerous thought you can have as a creative person is to think you know what you're doing..."
    This encapsulates the magic recipe for retaining agnosticism in one's work, which translates into your creativity perpetually evolving and moving both upward and sideways at the same time.
    If you "know" what you're doing, guess what?
    You're not innovating, and never will.
    You're not on the bleeding edge, and can't reach it.
    You're not giving yourself a chance to reach for the heavens.
    Sure, this means you avoid a ton of risk of failure. Your wings won't melt if you don't fly that high.
    But I'd rather have mine melt, and fall on my ass, and then dust myself off and laugh at yet another epic fail, and then saddle up and try again, than die having known I never took the chance in the first place.
    To sum it up... Poo. Wall. Fan. Fling and duck. Eventually, a nice runny one will stick and dribble a trail to a pot of gold.

  • Alexander Golde

    One of the advantages of dataflow based programs is the development speed.

    How often needs a developer little tools to convert data from one format into another ? How often does a developer check log files or config files etc ?
    Who starts to write supporting tools in c++ or a scripting language if it's not really important ? Even little tools costs you hours of coding and debugging. With tools like ANKHOR FlowSheet you can do this in minutes.
    Of course it's possible to create really complex programs, but regardless if it's dataflow based or not, you need to be very disciplined. Otherwise you will get really fast to a point where even the developer gets lost with his solution.
    One nice side effect : because you develop the application WITH your data - the results of the last operation is immediately available on the output and can be inspected - it possible to save a lot of time of debugging or 'adjusting' the code to the desired results.
    Depending on the underlying system, you get automatically multithreaded programs. And - think of it - the underlying system could automatically transport the data to other clients on the network for distributed analytics without the pain of creating all the code which would be necessary to do this by hand.

    A dataflow program may not replace a text editor or an audio editing program due to the lack of controlling the GUI. But they can be a great help for data-driven projects.

    Alexander Golde
    www.ankhor.com

  • Chris Wilson

    How long does it take to create the components that convert data items from one format to another? For example an HTTP request into a SQL query, or a SQL response into a JSON object?

  • Alexander Golde

    Do you want me to show you a concrete implementation in an ANKHOR FlowSheet ?
    A nice example can be viewed here: http://www.youtube.com/watch?v...
    I think the video explains very well how powerful DBP's can be.
    If you use the right tools :)

  • David Gann

    Among all real-time dataflow languages there is one really hot: VVVV

    Go visit vvvv.org and see yourself that many vvvv freelancer and businesses  live from it today quite well.

    You can completely substitute any textual programming language, 3d design tool and game engine with it and lately it got an java script implementation so you do HTML5 and webGL pages with a node based, data flow approach ( vvvvjs ).

    It is so well implemented and has excellent software design patterns, as it is developed and maintained by a friendly community since over 10 years.

    I didnt find so far something that comes at least a little bit close to what vvvv offers.

    I am saying this as someone who is living from programming and patching since years.

    I very often encounter programmers (python/java/c++) that feel outperformed by my activities. It is a mixed feeling if you do things in 1/10 of the time then your collegues. But they have to face it: Meta programming is just faster and more efficient if you work project oriented. So its not longer about what language you are using but if you are using patching or not.

    What many of those programmers that are sceptical against data-flow don't realize is that most of the hundreds of nodes we use every day contain more code then they write in one or two weeks, and this code is optimized and maintained since years!

    If you encounter a problem that was not solved by the community so far, and that happens rarely, then you start coding.

    And note, I am talking about project oriented working that has close dead lines all the time. There it makes obviously sense. But we are also more and more coming to this point where we do 'real' software development incredible fast.

  • Pete Welter

    I did my Master's thesis 20+ years ago using Prograph, which is amazingly still available. Like other programming paradigms, dataflow - no matter what language you sue - is worth trying out if even just to put in one's mental toolkit.

  • jtrental

    Hi, maybe someone has already pointed this out, but the same logic applies (or can be applied) to object oriented programming, which is becoming mainstream (Java, C++, etc).

    In this context, FBP has de added value of providing an easy-to-use interface for the end user, but I don't thing that it is correct to state that FBP is a "revolution".

    Regards,

  • Jan Schrewe

    Okay, I'm not sure I understood FBP completely, but from what I understood from the text and the diagrams the developer uses predefined black boxes and models the flow of data through them. Which is a good idea actually and I would argue is what every decent developer does in one way or the other. 

    The trouble starts of course when you have to teach the computer the meaning of those nice black boxes. Black boxes that do rather sophisticated stuff still either have to be written by some developer or build from simple and stupid black boxes. The first option isn't exactly that different from what we're doing now and just shifts the communication problems between developers and business (or whoever uses the code computers understand) a little. The second option will at some point require some form of magic unicorn compiler that is able to generate fast complex black boxes from simple black boxes. Well, we're waiting for those magic unicorn compilers since the invention of compilers. And sadly still need to decide between understandable and slow and something that looks like somebody has rolled an angry armadillo over the keyboard and fast. And then try to figure out badly written guides how to interface with the mess the angry armadillo left.

    Oh, then there is the third option, which is that there really are only stupid black boxes that some poor sod has to connect with wires or tubes or whatever over and over again to get something that does perform some useful activity in the end.

  • AlexK

    Macromedia Authorware, Tibco and slightly different, Delphi and Visual Basic, perhaps even Hypercard.    

    As people said, diagrams can become horribly complex and incomprehensible.  

    This is a nice technique but it made me think of architecture rather than code.  

  • LeftWingPharisee

    You've got to be kidding me. Informatica PowerCenter. DataStage. Ab Initio. Talend. Those are just those data flow diagram programming languages that I've seen job ads for in the past week. There are others out there as well.

    Two things: one, if you use these tools, you are programming. Period. You are no longer a non-programmer. Some people find these kinds of tools easier to use. Great! I don't, but each to his/her own.

    Two, if you put real life business logic in these diagrams, they quickly turn into messes that look like wiring diagrams. If you've ever worked with hardware guys at the board level, you'll know what I mean.

    Bottom line, these are not a magic bullet. Coming up with realistic, stable requirements will go much further than some "new" tool that's not new under the Sun.