Multi-language Flowex components

In Flow-Based Programming (FBP) paradigm each component is an independent process — “black box”. Component only transforms input to output, and this transformation is independent of internal implementation. So in general components can be written in different languages and then linked together in a single network.

Ability to run alias code is very important for such new languages as Elixir. One can reuse tonnes of libraries from more mature languages and choose the specific language for the specific problem.

In this article, I’ll show examples of multi-language Flowex components. I will use Export library to run Ruby and Python and Porcelain library to run command-line applications. There is the multi_flowex project on GitHub, for whom interesting in details.

Erlang ports

Erlang port is an abstraction that provides a byte-oriented interface to an external program over STDIN and STDOUT by sending and receiving lists of bytes, including binaries. Ports are process specific, so only process created a port can communicate with the port and if this process terminates, so does the port. The external program resides in another OS process. By default, it reads from standard input (file descriptor 0) and writes to standard output (file descriptor 1). The external program is to terminate when the port is closed.

Erlport project is not in active development but demonstrates how one can connect Erlang to other programming language. Currently, the project supports only Ruby and Python. There is a wrapper for Elixir — Export.

Porcelain allows launching and communicating with external programs.

multi_flowex

multi_flowex is a simple Flowex project with Ruby, Python and shell “pipes”. Each component says “Hello”. So passing an information packet (IP) with empty input return IP with the text: “Hello from Ruby, Hello from Python, Hello from shell”.

Let’s consider each component in details. (Read more about Flowex module pipes here)

Ruby pipe

defmodule RubyPipe do
  defstruct [:data]
  
  @ruby_dir Application.app_dir(:multi_flowex, "priv/ruby")
  @main_file “main”
  use Export.Ruby
  def init(opts \\ %{}) do
    {:ok, ruby} = Ruby.start(ruby_lib: @ruby_dir)
    Map.put(opts, :ruby, ruby)
  end
  def call(%{data: data}, %{ruby: ruby}) do
    {:ok, result} = Ruby.call(ruby, push(data, "Hello from Ruby"), from_file: @main_file)
    %{data: result}
  end
end

The init function starts Ruby process and adds its pid to options. @ruby_dir specifies the path to ruby code and @main_file the file with simple Ruby code:

def push(array, el)
  result = array.push(el)
  Tuple.new([:ok, result])
end

It just pushes element to array.

The code in the call function of the component calls the push method with initial data and “Hello from Ruby”.

Take a look at the test of the module to get the idea what each function returns.

defmodule RubyPipeSpec do
  use ESpec
  
  let :opts, do: described_module().init
  
  it "has ruby process pid in opts" do
    expect(opts().ruby).to be_pid()
  end
  it "pushes to data when .call" do
    result = described_module().call(%{data: []}, opts())
    expect(result.data).to eq(["Hello from Ruby"])
  end
end

Python pipe

The pipe with Python code looks similar:

defmodule PythonPipe do
  defstruct [:data]
  @python_dir Application.app_dir(:multi_flowex, "priv/python")
  @main_file “main”
  use Export.Python
  
  def init(opts \\ %{}) do
    {:ok, py} = Python.start(python_path: @python_dir)
    Map.put(opts, :py, py)
  end
  def call(%{data: data}, %{py: py}) do
    {"ok", result} = Python.call(py, append(data, "Hello from Python"), from_file: @main_file)
    %{data: result}
  end
end

And the Python code is:

def append(array, el):
  array.append(el)
  return (‘ok’, array)

Shell pipe

A component which uses Porcelain just runs echo program and gets its output:

defmodule ShellPipe do
  defstruct [:data]
  
  def init(opts), do: opts
  
  def call(%{data: data}, _opts) do
    output = Porcelain.shell(“echo ‘Hello from shell’”).out
    result = data ++ [String.rstrip(output)]
    %{data: result}
  end
end

The pipeline

Let’s look at the pipeline itself.

defmodule MultiPipeline do
  use Flowex.Pipeline
  defstruct data: []
  pipe RubyPipe, 3
  pipe PythonPipe, 4
  pipe ShellPipe, 2
  pipe ElixirPipe
end

The pipeline starts 3 Ruby pipes, 4 Python and 2 Shell pipes. It’s just for demonstration purposes, we definitely don’t need such parallelism is not necessary for this “Hello, world!” example. One can check that after starting the pipeline 3 Ruby and 4 Python processes appears in the system. Porcelain starts `echo` process only on demand, when IP enters the component.

The last pipe is plain Elixir pipe which just joins a list into final result string:

defmodule ElixirPipe do
  defstruct [:data]
  
  def init(opts), do: opts
  def call(%{data: data}, _opts) do
    %{data: Enum.join(data, “, “)}
  end
end

The spec below shows the pipeline in action:

defmodule MultiPipelineSpec do
  use ESpec
  
  let :pipeline, do: MultiPipeline.start()
  it “returns hello” do
    result = MultiPipeline.call(pipeline(), %MultiPipeline{})
    greetings = “Hello from Ruby, Hello from Python, Hello from shell”
    expect(result.data).to eq(greetings)
  end
end

So one can see greetings from 3 different programs!

Conclusion

Elixir/Erlang ecosystem has a flexible and safe way of running external programs thus allowing to leverage the advantages of other programming languages. At the same time, Flowex makes it easy to encapsulate alias programs inside independent processes and organize communication between them.