ALF — Flow-based Application Layer Framework

TL;DR. Go directly to the ALF page.

FBP and Actors

Flow-based Programming is a quite old idea (early 70th) of presenting an application as a network of independent processes (components) exchanging data via message passing. It’s like the well-known Actor model, but with a couple of significant differences. First, connections between processes (which process can send messages to which) are predefined. Second, and more important, FBP is a data-centered approach — an application is viewed as a system of data streams being transformed by processes. To put it simply, in the Actor paradigm “there are processes that exchange data”, while in FBP — “there are data flowing through processes”

There are FBP realizations in many programming languages, but in most cases, these implementations don’t provide true “independence” of components mainly because of the concurrency models of the languages.

In Elixir/Erlang ecosystem Actor model is a native one. OTP framework provides very powerful GenServer abstraction with a simple interface for sending/receiving synchronous/asynchronous messages, monitoring processes, restarting them, and so on. Moreover, in Elixir, there is the GenStage library with tools for defining “communication topology” between processes. It takes care of messages delivery between producers and consumers and also implements a back-pressure mechanism that prevents processes to be overloaded by messages. Therefore the implementation of the FBP approach in Elixir becomes an easy task.

Flowex

My first (quite limited) implementation of the FBP approach in Elixir was created almost 5 years ago. It is the Flowex library that provides the interface for creating simple linear chains of communicating processes (pipelines). I named this approach Railway-FBP. I recommend you to check Flowex Readme to get the idea of how the user’s code is put inside Elixir GenStages and what are the use-cases. Basically, the main focus of the library was on simplicity and scalability. One can configure stages to run several identical processes per stage (thus achieving higher throughput for a pipeline). Also one can execute a code written in other programming languages using Erlang ports.

We used Flowex for about 2 years in production in our “home-grown” data pipelines. The Flowex-backed application was connected to many databases inside our system and collected/transformed data before pushing them into a data lake. Pipelines looked like this:

defmodule MainAggregator.Leads.Pipeline do
  use Flowex.Pipeline

  defstruct [:ids, :leads, :call_logs]

  pipe :get_leads, count: 2
  pipe :preload_associations, count: 2
  pipe :get_call_logs, count: 3
  pipe :build_leads, count: 1
  pipe :insert_leads, count: 3
  pipe :delete_test_requests, count: 1
  ...
end

There was a simple declarative way of specifying steps of data transformation.

But the initial idea behind Flowex was not just using it in data pipelines (one can choose now the Broadway library for this purpose), but providing a general framework for a high-level business logic — application layer of a program. And consider data flowing through pipelines as not just meaningless bytes but as business events. Check, for example, this post where I created an example of simple web-app flow. The pipeline is:

defmodule GetUserPipeline do
  use Flowex.Pipeline
  defstruct [:conn, :user]  
  pipe FetchParams, 
       opts: %{auth_data: ["token"], repo_data: ["user_id"]}
  pipe AuthClient
  pipe FindRecord,
       opts: %{finder: &__MODULE__.find_user/1, assign_to: :user}
  pipe :prepare_data
  pipe RenderResponse, opts: %{renderer: UserRenderer}
  pipe SendResponse
  error_pipe :handle_error
  ...
end

The main problem with the Railway-FBP approach in Flowex is its “railway” approach. More or less complicated flow can’t be presented using just straight flows. Sometimes one need conditional branching, some kind of loops, the possibility to clone data-packet in order to execute other “aspects” of a program, etc. So in the case of Flowex-based applications, there is always additional conventional logic that adds missing functionality. That is why ALF appeared.

ALF

Since Flowex was published, many cool features were added to the Elixir language. There are DynamicSupervisor, Stream, Telemetry, GenStage library finally has reached its 1.0 version. So ALF is written from scratch with the same idea, but with a broader set of features.

From the components side, there are not just Producer, Consumer, and Stage, but many other components:

• Switch — for conditional branching;
• Clone — for creating a copy of an event;
• Plug/Unplug — for inserting components from other pipelines;
• Decomposer/Recomposer — for creating several events based on the given one, and for recomposing several events into one;
• and many other components.

The interface of sending/receiving data to/from a pipeline is also different. The primary interface now is based on streams. A pipeline receives a stream of events and produces a stream of processed events.

Telemetry is also a core part of ALF. It’s currently in the early stage of development, but the potential is huge. One can actually monitor every component in pipelines.

The important part of the project is a modeling language (a set of icons/shapes for components together with rules of combining them) that can be used during the application layer design. Below is a part of the pipeline that will be used in the online tic-tac-toe game that I’m working on currently in order to have a good example of using ALF.

And here is how it looks in the code:

defmodule Tictactoe.Pipelines.UserEnters do
  use ALF.DSL

  @components [
    stage(:validate_input),
    switch(
      :token_present?,
      branches: %{
        yes: [
          stage(:find_user_by_token),
          stage(:find_active_game),
          stage(:find_pending_game_if_no_game),
          goto(:if_there_is_game, to: :prepare_game_data_point)
        ],
        no: [
          stage(:find_users_with_the_name),
          stage(:add_postfix_if_needed),
          stage(:create_user)
        ]
      }
    ),
    stage(:find_free_game),
    switch(
      :is_there_free_game?,
      branches: %{
        yes: [stage(:assign_user_to_the_game)],
        no: [stage(:create_new_game)]
      }
    ),
    goto_point(:prepare_game_data_point),
    stage(:prepare_game_data)
  ]
  ...
end

The beauty of the FBP approach is that what you see in a diagram is actually what you’ll have in run-time. And with ALF DSL you have the same tree/graph also while coding. That is how the buzz-phrase appeared — ”design-, coding-, and run-time observability”!

I hope, you have taken an interest in the project! So, check the repo and let me know what you think!