Project Abstract

MetaCall Jupyter Kernel is an open-source wrapper kernel that implements cross-language function calls through the MetaCall Core and the Polyglot REPL. MetaCall Core is an open-source library that brings the polyglot programming experience to Developers. With MetaCall, developers can embed different programming languages through an easy-to-use high-level API.

Leveraging IPython, this project aimed to create a kernel where the MetaCall Core can be loaded and launched through a Jupyter Notebook interface. With this Notebook, the users can try out writing and embedding code in different programming languages. It would also make it possible to use the MetaCall API and achieve the goal of bringing a unified interface for developers to try out and leverage MetaCall.

This year’s project achieved the following:

• Developed the initial minimal viable product centred around bringing a Jupyter kernel for MetaCall.
• Built and integrated a Polyglot REPL using the MetaCall Core to the IPython through a subprocess.
• Implemented and improved user-experience features centred around using the kernel with the Jupyter client.
• Implemented CI/CD pipelines for the project with GitHub actions and set-up container-based deployment with Binder.
• Created and delivered a full-fledged documentation website for the MetaCall Jupyter Kernel using Sphinx and ReadTheDocs.

Mentors: @giarve

Setting up the Kernel

The first step of the project was to set up the wrapper kernel using IPython and expose it on a Jupyter client. Initially decided to go with Xeus, a C++ framework used to build Jupyter kernels, we moved forward with IPython. I created a wrapper kernel, which can be built and set up using Setuptools as the build system. The initial kernel forks a MetaCall exec subprocess and loads the code on a temporary file before passing it to the exec and retrieving the standard output.

The mechanism was not sufficient to move forward with defining extra languages (we provided initial support for NodeJS), hence we moved forward with automated language inference models with guesslang. Guesslang, provided a high-level API to automatically detect programming languages through the code. This affected the processing speed and we had to do away with the same later, in favour of user magics for faster inference.

Example usage of the MetaCall Jupyter wrapper kernel

Integrating Polyglot REPL

The second step of the project was to integrate the Polyglot REPL. For this milestone, I collaborated with @viferga and worked on a NodeJS-based REPL (Read-Evaluate-Print-Loop) which loads a Python and a Node REPL through the MetaCall’s meta-object protocol. The REPL, which makes use of a high-level API, made it possible to extend its usage as per our understanding and needs and integrate it with any client interface.

The second part of this was integrating the REPL to the MetaCall Jupyter kernel. To make this possible, we made use of a subprocess that launches the REPL in the background and listens for standard inputs passed. This approach is commonly used by other wrapper kernels, hence we followed the same design principle. As each evaluation is passed to the kernel, the standard output from the REPL is fetched and displayed on the kernel in a clean logger output. This now allows a user to both, load, inter-change, inspect and evaluate the meta-object protocol from the kernel interface itself.

Example usage of the Polyglot REPL on the MetaCall Jupyter

Implementation of user-experience features

The third step of the project was to implement the user-experience features. These user-experience features defined the usability of the client interface and how the users can interact with the REPL. To achieve this, we decided to tweak the standard magics used on the Jupyter notebook interface, to match our requirements.

The user-experience centric features I added were:

1. ! : Run a Shell Command on the MetaCall Jupyter Kernel
2. $shutdown : Shutdown the MetaCall Jupyter Kernel
3. $inspect : Inspects the MetaCall to check all loaded functions
4. $loadfile: Loads a file onto the MetaCall which can be evaluated
5. $newfile: Creates a new file and appends the code mentioned below
6. %repl <tag>: Switch from different REPL (available tags: node, py)
7. >lang: Execute scripts using the MetaCall exec by saving them in a temporary file (available languages: python, javascript)
8. $loadcell <tag>: Loads a function onto the MetaCall to be evaluated
9. $help: Check all the commands and tags you can use while accessing the MetaCall Kernel
10. $available: Checks all the available REPLs on the Kernel

Implementation of CI/CD pipelines

For continuous testing, I built and managed the CI/CD pipelines for the project. The CI pipelines are built using GitHub Actions. Testing has been implemented using the pytest module, and the CI is responsible for building and validating the source and the documentation.

The primary CI file can be found on the .github/workflows/ci.yml and the codeql-analysis.yml respectively.

GitHub Actions in action with every Pull Request and push

Documentation website

For documentation, I built and published the MetaCall Kernel and the source documentation on Sphinx and ReadTheDocs. Sphinx is a documentation generator for Python and was the first choice of going ahead with documenting the project. The standard documentation ensures that the community support remains intact long after the coding period of Google Summer of Code is over. It would also ensure that future contributors can work on adding support for many languages as MetaCall Core extends the functionalities for more languages on the upstream.

Check out the documentation website on metacall-jupyter-kernel.readthedocs.io.

PyPi release

The kernel has been released on PyPi at pypi/metacall-jupyter. To get started, push on the terminal:

Pull Requests

• #4: Add Jupyter kernel setup feat
• #5: GitHub Action for CI Build CI
• #6: Documentation website feat
• #8: Linting with Flake8 and Black chore
• #10: MetaCall launcher for Jupyter kernel feat
• #12: Fixing Node execution with MetaCall fix
• #14: Adding automated language check with guesslang and examples feat
• #16: Custom log message display for unsupported languages feat
• #18: Add magic support feat
• #20: Add support for shell commands feat
• #22: Fixing user experience features fix
• #23: Adding examples + GH actions + fixing magics bug fix, CI
• #37: Embeding MetaCall REPL on the kernel feat
• #47: Adding red font for error output feat
• #50: Adding support for $newfile magics feat
• #51: Adding support for %inspect and %load magics feat
• #54: Adding support for $help and $shutdown feat
• #56: Adding support for $loadcell feat
• #60: Adding support for $available, fixing Dockerfile and adding help links feat, fix
• #62: Implement history access for the kernel feat
• #64: Adding examples for the kernel usage chore
• #68: Adding unit-tests for the kernel chore
• #73: Adding documentation docs

Tags:

Feature : feat

Bug Fix : fix

Documentation: docs

Maintenance, testing and updates: chore

Statistics

• Total PRs sent: 24
• Total issues opened: 20
• Total commits: 49
• Total lines of code added: 4372
• Total lines of code deleted: 1664
• Total lines changed: 6036

Acknowledgements

It was a great experience working with the MetaCall team for GSoC’21. Thanks to my mentor Gil Arasa Verge and Vicente Eduardo Ferrer Garcia for giving me this wonderful opportunity and mentoring me. I look forward to continuing being a part of the MetaCall community and continue contributing to Jupyter kernel as well as other projects, like Core, along with helping new contributors find their way around the community.

Harsh Bardhan Mishra (@harshcasper).
Google Summer of Code 2021 Student from MetaCall.

Java is a high-level programming language currently being used in various parts of the tech industry with more than 3 billion devices using it. Java has a wide variety of applications like mobile, desktop, web, and many more.

MetaCall is a polyglot runtime environment that lets developers call functions and methods between different programming languages. For example, we can call a Python function from a Node.js file. This is done by converting the function to a C/C++ method using an embedding API.

MetaCall supported languages like C, C++, Python, Node.js, etc. but it lacked Java as an option. So, as a part of Google Summer of Code 2021, I set out to change this by implementing the Java loader for MetaCall, which would allow MetaCall to call functions from Java.

Work Done

My journey with MetaCall started by working on a Proof Of Concept (POC) using the Java Native Interface (JNI), the reflection API, and the compilation API in Java to build a small working example that would be required to make the Java loader work in MetaCall.

After implementing the PoC successfully, the next task was to start working on the MetaCall core repo and start mapping the functions from the PoC to the loader.

In the first 4 weeks of GSoC, I worked on implementing the basic functions to initialize the loader, add the default classPaths and execution paths, the function to discover the data present in the Java file which consisted of data types, methods, and constructors. In the next 3 weeks, I worked on implementing the getters, setters, and invoke methods for static members/attributes, which are used to get and set values of static attributes and invoke the static methods present. In the last 3 weeks of GSoC, I worked on implementing the same functionality but for non-static attributes and objects.

The major problem and issues I faced during these 10 weeks was the fact that Java is not easy to embed as it is a strongly typed language compared to JavaScript or Python. All these things made the project even more complex than anticipated. For instance, supporting all the primitive data types and arrays of different data types, each had to be converted individually. Due to this, it became even more difficult to make the project dynamic. Nevertheless, all the problems were resolved and I was able to implement the whole loader.

In the end, I was also working on a whole test suite using Google Test. This was testing all the common functionality with most of the data types and edge cases.

Showcase: Calling Java from Python

Consider a simple Java file as follows:

This is a simple Java file that contains a function Add which takes two integers and returns their sum. Even though this is a simple example, MetaCall supports complex functions, data types, and constructors too.

By using the Python Port for MetaCall, we can now import Java classes directly from Python:

This script can then be called using the MetaCall command-line tool:

Design and implementation

We planned to use a mix of Java and C++ instead of using only the JNI for the proper working of the loader. This means, instead of doing more complex tasks in C++ like compiling a file, extracting the data, or converting using JNI, we used a bootstrap.java file to accomplish these tasks. Bootstrapping is the usage of the runtime/loader language and code execution for the tasks required for MetaCall’s introspection.

This approach made it easy for us to work with the JVM and the JRE and accomplish these tasks:

• Compiling a .java file in Java classes.
• Loading all the compiled classes to the JVM.
• Getting a list of classes that are loaded into JVM.
• Extracting the attributes, methods, and constructor details present in a Java class.
• Reading details such as name, signature, parameters, and return types.

We directly used JNI in C++ to accomplish all the tasks like calling the methods, getting or setting attribute values.

The loader implements the following list of methods:

• Initialization: It is the first function that MetaCall calls in the loader to initialize a runtime environment. In our case, to start the JVM and the JRE.
• Load From File: Load from file is the function that takes the absolute path of the Java files and compiles them to Java classes.
• Load From Package: Load from package is the function that is used to load files that are already compiled, like classes or JARs.
• Load From Memory: Load from Memory function takes a binary string representing a Java class and compiles and loads it to JVM.
• Discover: This function performs the most important task to “inspect” the data present in the Java class. It performs all the data extraction related to attributes, methods, and constructors and registers them to the MetaCall Core so that MetaCall can identify the type and call it in the Java loader. Discover handles the attributes, with their names, data types and Java signature, the constructors present in a class, including name, list of parameters and their types, and finally the methods, registering in all of them the visibility (public, private or protected).
• Destroy: it cleans all the resources and stops the JVM.
• Getters and Setters: These functions are used to get or set the value of an attribute, no matter if it is static or non-static. If the attribute is static, then the interface referring to class must be used, otherwise the object must be used instead.
• Invoke: These functions are used to invoke methods present in a class by passing the required parameters, if the method is static, then the class interface is used, otherwise the object is used.

For more detailed information on the development process, refer to the source code in the MetaCall Core repository and the full list of commits and PRs related to the project.

Conclusion

The addition of a Java loader to MetaCall will now allow it to support a large and expanding range of programming languages through the implementation of a single loader. While Java and consequently the loader still have a long way to go, this project provides a good base for future development.

Ketan Gupta.
Google Summer of Code 2021 Student from MetaCall.

WebAssembly is an emerging standard for a binary instruction format designed to be run on a virtual machine. It is designed to be portable, stable and fast.

It is already possible to compile to WebAssembly from many different
programming languages, including ones that are currently unsupported by
MetaCall (e.g. Go, Zig, Kotlin and Swift).

As a part of Google Summer of Code 2021, I set out to change this by
implementing a WebAssembly loader for MetaCall.
In this blog post, I will be showcasing the final product, along with
describing its design and implementation.

Showcase: Calling WebAssembly from Python

Consider the following very simple WebAssembly module written in the text format:

This module exports a single function, add, that takes two 32-bit integers as
parameters and returns a 32-bit integer as its result. Normally, one would
convert the text file above into the binary format, but the WebAssembly loader also supports loading the text format directly. By using the Python port for MetaCall, we can now import WebAssembly functions directly from Python:

This script can then be called using the MetaCall command line tool:

Design and implementation

The WebAssembly loader is implemented using Wasmtime, a small and efficient WebAssembly runtime. This runtime was chosen for two
reasons in particular:

1. Wasmtime supports the WebAssembly System Interface (WASI), an API for
   interacting with several important features of an operating system, such as the filesystem and networking. It is designed to be independent of the
   browser and is in the process of standardization.
2. Wasmtime implements the standards-track Wasm C API for embedding. Using this API will make the implementation almost completely runtime-agnostic, making it easy to change the runtime in the future if necessary.

The loader is implemented almost exclusively through the use of the Wasm C API, with any Wasmtime-specific functionality being compiled conditionally. In particular, converting from the WebAssembly text format to the binary format is not supported by the standard API and requires Wasmtime. In the future, other run-times implementing the C API may be supported.

When loading a module, imports are resolved by looking for suitable exports
within previously loaded modules in the same handle, and functions are
discovered by iterating through all functions exported by the module. It is
thus possible to hide functions from MetaCall by not exporting them.

For more detailed information on the development process, refer to the source code in the MetaCall Core repository and the full list of commits related to the project.

Conclusion

The addition of a WebAssembly loader to MetaCall allows it to support a large — and expanding — range of programming languages through the implementation of a single loader. While WebAssembly — and consequently the loader — still have a long way to go, this project provides a good base for future development, such as support for WASI when it is more mature.

Jeppe Blomgren.
Google Summer of Code 2021 Student from MetaCall.

MetaCall is a tool that allows mixing languages. You can build Polyglot Applications with it. In this article we will show you an example of a polyglot application step by step using BeautifulSoup (Python) mixed with an Express server (NodeJS) in order to build a Polyglot Scraping API.

Let’s go through the setup process together to see how easy it is to work with. First clone the repository:

git clone https://github.com/metacall/beautifulsoup-express-example

Then we need to install MetaCall. MetaCall is distributed in different ways, including Docker, Guix or Installer. It is highly portable so it should work on any Linux. Let’s install it:

curl -sL https://raw.githubusercontent.com/metacall/install/master/install.sh | sh

Now we have access to metacall command, which is a complete Command Line Interface (CLI) for developing Polyglot Applications.

After that we should install the dependencies. From Python side, we need to install BeautifulSoup dependency for scraping, and certify dependency in order to query HTTPS webs.

metacall pip3 install beautifulsoup4 certifi

From NodeJS side, we need to install express for our server and metacallas a package for allowing calls to Python.

metacall npm install metacall express

MetaCall uses its own versions of npm and pip, in order to avoid to pollute the system and maintaining portability. That’s why we prefix with metacall all package manager commands, although the syntax is exactly the same.

Great. We are done with the tedious and boring part. It is time to investigate what the application does, and how is it possible to magically mix the programming languages.

In the NodeJS side, we can see the index.js script. It implements the Express server, but also uses metacall package for loading the scraping.py script.

The first two lines (line 2 and 5) load the Python script. At this point you can easily call functions from scraping.py with metacall function. We do exactly that in the handler of our REST API in the Express server (line 14). We use metacall('links') in order to call the function links which is implemented in scraping.py script.

The links function accepts an URL, which is opened and queried in order to obtain the HTML (lines 8 to 17). Then initializes BeautifulSoup in order to parse the HTML (line 20) and then obtains all links the web has (lines 23 to 26).

Easy-peasy.

Now it is time to test it and see the magic happen. In order to run our application, we simply do:

metacall index.js

MetaCall will launch index.js and run the Express server. Logs will be streamed to $HOME/metacall.log and the server will be listening to the port 3000 so we can use curl to query it. Let’s scrape all links from NPM web:

curl localhost:3000/?url=https://www.npmjs.com/

If everything went well, we should see in the response something similar to this:

["https://docs.npmjs.com","https://npm.community","https://go.npmjs.com/npm-pkgsafe","https://docs.npmjs.com","https://npm.community","https://www.npmjs.com/advisories","http://status.npmjs.org/","https://blog.npmjs.org/"]

For the DevOps gang and in favor of containerization, we also provide a dockerized version of the application which includes automated testing. It can be built and run with the following commands:

docker build -t metacall/beautifulsoup-express-example .
docker run --rm -p 3000:3000 -it metacall/beautifulsoup-express-example

Finally we added a metacall.json file, which is a configuration used for deploying into MetaCall FaaS. In this version we only use the Python script, because MetaCall FaaS is able to export the functions and generate an API Gateway automatically, so the Express server is not needed, and we also will obtain higher performance because our FaaS server is implemented with high performance and concurrency in mind.

By this way we can deploy this as a standalone application on any platform, or through highly scalable FaaS. If you want to try the MetaCall FaaS platform, it is under Beta Testing and it is available only by early access (you can request access at https://metacall.io).

For more information about MetaCall or in order to become part of our community, you can join our Telegram Group.

The source code is licensed under Apache 2.0 License and can be found at:
https://github.com/metacall/beautifulsoup-express-example.

See you meta-programmer!

Vicente Eduardo Ferrer García.
Founder & CTO of MetaCall.

A new way to develop distributed systems that has never seen before.

A Function Mesh is a new way to inter-connect serverless functions transparently. With this new approach you can build complex distributed systems with multiple dependencies and scale only the hot parts of the system.

With MetaCall is possible to develop monolithic applications and test them locally in your laptop. The code can be later deployed to our FaaS and it will be automatically distributed in different instances. The connections between the nodes are implicit in the code and developers do not have to care about them.

You only have to focus on the business logic and the platform will do everything for you. You will not need to manage anything related to clustering and scaling. With MetaCall you can reduce time to market, auto-scale hot parts of your application and inter-connect easily your functions.

This new model also provides an efficient way to distribute the load between nodes. Nowadays functions are based on containers that run just an script with some invoker, which implies one container for each process. That is a lot of overhead just to run a small snippet of code. In our case each container can run multiple processes at the same time, and also multiple functions at the same time, using at maximum the resources available in the cluster.

A code that was designed to be monolithic will be split in a big mesh of functions that can dynamically scale up or down individual instances, or slice or merge parts depending on the workload.

The only thing you have to care about is to write a small configuration file indicating what code you want to publish and the the gateway will be automatically created for you.

Let’s take a look at the following snippets:

In this example, backend.py will be deployed as a high scalable and serverless function and frontend.js will be deployed in a complete different container, with the same characteristics. The configuration metacall.json indicates what scripts are going to be offered to the public, and the API is automatically built upon that script based on the function names.

If we connect our example repository to MetaCall it will generate the following application:

With this architecture you can eliminate vendor locking, because you can develop normal code and execute it locally. You remove all DevOps and cluster set up and all headaches related to it. As a developer, you can write complex distributed architectures easily and also reduce time to market.

Current architectures as service mesh are based on multiple containers attached to a micro-service, which makes the cluster expensive. Function Mesh communicates in a efficient way between functions and also distributes the workload between processes. So it also reduces considerably the costs of running a cluster.

If you want to reduce complexity, speed up your development and improve team performance and code quality MetaCall FaaS architecture is your best option.

MetaCall is open-sourced under Apache 2.0 license. Feel free to fork the official repository. Suggestions and contributions are welcome.

See you meta-programmer!

Vicente Eduardo Ferrer García.
Founder & CTO of MetaCall.

MetaCall is a library which allows you to execute code written in a given programming language from within different programming languages.

At this moment it supports a variety of languages such as C/C++, JavaScript, NodeJS, Ruby, Python and C#, but we are planning to add more support. For example, calling from JavaScript to Python is easy as this three lines of code.

MetaCall allows you inter-connect any programming language with a simple syntax. It was created with user experience and high performance applications in mind.

Originally designed for a Game Engine, we are now using MetaCall to power a FaaS (Function as a Service) with a new technique called Function Mesh.

MetaCall opens up a new way to develop software. With MetaCall you can connect your code across languages easily, test it locally on your laptop and then upload it to our FaaS. Your code will be converted into a high scalable distributed system with no extra effort.

Working with multiple technologies at the same time in an isolated manner will be easy. Companies can mix their own teams of developers and maintain heterogeneous software reducing costs, frictions and time to market.

MetaCall is open-sourced under Apache 2.0 license. Feel free to fork the official repository. Suggestions and contributions are welcome.

See you meta-programmer!

Vicente Eduardo Ferrer García.
Founder & CTO of MetaCall.