Declarative Interface Translation with Octopus

It is a short description of the idea behind the Octopus Elixir library I’ve recently created. It is the necessary part of the more ambitious idea of scaling the Flow-based programming approach defined in the ALF framework to the system level. For now, however, it can be used as a stand-alone project.

The problem

As an application engineer, I need a simple way of interfacing
with programs/services that provide the required functionality. These can be both internal services/tools that are developed by my colleagues and external ones, like mail service, payment gateway, cloud storage, or anything. Some of them provide HTTP JSON API, other use XML, databases uses SQL or No-SQL language, and there are gRPC, MMQT, name more! And of course, lots of cool stuff have the classic command line interface.

The conventional approach to the problem is creating client libraries. Some libraries can already exist, especially for popular services and popular programming languages. I use the Elixir programming language, not a very popular one. And even having a specific library created by the community, it sometimes seems like overengineering to add a dependency just for a couple of calls to the service I need. If there are dozens of such libraries in a project, it becomes a nightmare to change/update these dependencies.

Another annoying thing is that all these libraries are very similar (and simple) in their functionality. They do, basically, three simple things:

1. Translate data structures provided by the programming language (e.g. Elixir maps, lists, etc) to data required by another program (e.g. URL with params or CLI command or SQL query or whatever).
2. Call the program (e.g. make an HTTP request, execute a program, send a DB query, or whatever).
3. Translate the result (e.g. JSON response, text response, SQL response, etc) back to the language’s data structures.

Interface Translation is a good description of such functionality.

Explicit coding of such translations leads to a decent amount of boilerplate code, not to mention that such code is created for many programming languages existing in the world.

The idea

Can these translations be expressed in a declarative, language-agnostic, specification-like way? Then the client library can be generated based on the specification.

The solution

The specification can be provided as a JSON data structure that describes the interface to a service. The JSON is chosen as the specification language because it is widely used and easy to translate to Elixir data structures: JSON objects are translated to maps, JSON arrays are translated to lists, etc.

Consider the example of the simple interface to the Agify service:

{
  "name": "agify",
  "client": {
    "module": "OctopusClientHttpFinch",
    "start": {
      "base_url": "https://api.agify.io/"
    }
  },
  "interface": {
    "age_for_name": {
      "input": {
        "name": {"type": "string"}
      },
      "prepare": {
        "method": "GET",
        "path": "/",
        "params": {
          "name": "args['name']"
        }
      },
      "call": {
        "parse_json_body": true
      },
      "transform": {
        "age": "args['body']['age'])"
      },
      "output": {
        "age": {"type": "number"}
      }
    }
  }
}

The “client” section describes a protocol-specific client. Here it’s just an Elixir module that implements a couple of functions (start, stop, call) needed for communication via HTTP. It’s a simple reusable component that, once implemented, can be used for all the services with the HTTP API. The “start” object specifies the common configuration of the client.

The “interface” section defines the “functions” of the “service”. Here it’s only one function — “age_for_name”. Then 5 steps optional are defined:

1. “input” — describes the input data structure.
2. “prepare” — describes how the transformations needed to be done to the input data to make it ready for the call: path, method, params, headers, etc.
3. “call” — configures the actual call to the service. Here it just says that the response body should be parsed as JSON.
4. “transform” — describes how the result of the call should be transformed. In this case, it just takes the `name` field from the response body.
5. “output” — describes the output data structure.

The actual “translations” happen in the “prepare” and “transform” steps. In the example the transformations are simple. The “prepare” step says to use “GET” to path “/”, and send “name” from “args” as “params”. The “transform” step says to get “age” from the parsed JSON body and put. One can also define “helper functions” that can be used in these steps for more sophisticated data transformation.

The “input” and “output” steps are optional, but I would like to explicitly define what data are expected as the input to the function and what I’ll see at the output.

The Octopus magic

The Octopus library is the Elixir engine for defining client libraries from the specifications described above. When running: Octopus.define(definition), magic happens and the client appears in your application. After starting it with: Octopus.start("agify”), one can call the Agify service: Octopus.call(%{“name” => “Anton”}) .

One step further

Octopus provides translation from the “service language” to the Elixir programming language. However, it’s easy to build an HTTP JSON API interface to the defined “clients”.

The OctopusAgent provides the API and thus translates heterogeneous interfaces in a system into the unified JSON API.

One can deploy the “agent” to an existing system, define “services” dynamically, and, voila, different parts of the system can be called in a unified manner.

Thanks for reading! Please, give me feedback, Medium claps, and GitHub stars.