This is part three of How I Automated Minting My Tweets as NFTs on OpenSea article series. In this article, I discuss how I created the OpenSear-Worker UI automation system using dAppeteer.

To read part one and part two of this article series,

• How I Automated Minting My Tweets as NFTs on OpenSea
• How I Automated Minting My Tweets as NFTs on OpenSea — Part 2

In my previous articles, I discussed how I created the OpenSear-API and how to start listening to Twitter events and queue jobs according to our specific scenarios.

In this article, I discuss how to undertake the queued job, mint, and publish my tweets on the OpenSea.io platform. Since OpenSea does not let us create NFTs programmatically (using their API), I used UI automation to tackle this task of minting and publishing NFTs. As I mentioned in the first article, there are nine steps to be automated. So let’s get right to it.

1. Obtaining the screenshot of the tweet

There are so many ways to obtain a screenshot of a tweet. For this task, I use the TweetPik Twitter Screenshot API.

Twitter Screenshot API - TweetPik

The following gist shows how to use TweetPik API to obtain and store a screenshot of a given tweet.

For this, we need to provide four values.

i. The ID of the tweet

We need to set the ID of the tweet that we need to take the screenshot. Here we set tweetId body value. For example,

https://twitter.com/Niweera/status/1497455677131739137

tweetId = `1497455677131739137`

ii. The theme ID

For screenshot customization, we can set a theme by providing a TweetPik theme ID. Here we set themeId body value to TWEETPIK_THEME_ID.

iii. The timezone

We can provide a timezone for the tweet screenshot. The timezone value should be provided as in the following Wiki page.

List of tz database time zones - Wikipedia

I live in Sri Lanka, so my timezone would be Asia/Colombo. Here we set timezone body value to Asia/Colombo.

iv. The TweetPik API key

To authorize the API request, we need to set the authorization header value to the TWEETPIK_API_KEY. The TweetPik API key can be obtained for a logged-in user from the TweetPik dashboard.

Now we can obtain the screenshot for the given tweet ID. However, we need some touch-ups for the screenshot. Let’s discuss it next.

2. Add some touch-ups to the tweet screenshot

When I say, touch-ups, I meant enhancing the image etc. For that, I used Jimp, a JavaScript Image Manipulation Program.

GitHub - oliver-moran/jimp: An image processing library written entirely in JavaScript for Node, with zero external or native dependencies.

3. Log in to OpenSea.io and go to add assets page

OpenSea.io is not a usual web2 application where we can create an account using our email and password. For OpenSea.io, we need to connect our MetaMask (or any other supported wallet provider) wallet to the OpenSea platform. This is a very tedious task to be automated. However, to our help, the @chainsafe/dappeteer package comes as a knight in shining armor.

GitHub - ChainSafe/dappeteer: 🏌🏼‍E2E testing for dApps using Puppeteer + MetaMask

By using the dappeteer package, we can automate the login using MetaMask wallet. Under the hood, the dappeteer package uses the puppeteer package.

GitHub - puppeteer/puppeteer: JavaScript API for Chrome and Firefox

Let’s first spin up a puppeteer Chrome automation instance and connect our MetaMask wallet to the OpenSea platform.

What we do here is first, we create a puppeteer instance, by giving a metamaskVersion. Currently, @chainsafe/dappeteer v2.4.1 supports MetaMask v10.8.1 extension. Be sure to install them as follows.

$ npm i puppeteer @chainsafe/dappeteer@2.4.1

I have tested for the above-mentioned versions, so other combinations of versions might not work, beware of that.

After creating the puppeteer instance, we set up the MetaMask by providing the METAMASK_MNEMONIC_PHRASE which is basically the recovery key you obtained when setting up a MetaMask wallet. If you still haven’t created a MetaMask wallet or do not know how to do it, check out the following video.

The METAMASK_PASSWORD is a string that you can provide as a temporary password. Even if you omit it, it won’t be a problem since the dappeteer package will provide it for you. The weird thing about MetaMask is that it only relies on the METAMASK_MNEMONIC_PHRASE and not on the password. Password can be changed every time you connect to the MetaMask wallet.

Then I’m configuring the Polygon Mainnet in MetaMask since it is the gas-free way to mint NFTs.

• Gas free trading of NFTs on OpenSea with Polygon
• Add Polygon Network | Polygon Technology | Documentation

After that, I’m using the dappeteer to sign in to the OpenSea platform. Now we can go to the add asset page and start working on our NFT minting tasks.

4. Upload the screenshot

As mentioned in the following list, we upload the image to OpenSea.

Here we need to input two parameters, page and filepath. page is the reference to the puppeteer page and filepath is the path to the screenshot.

5. Add a unique name to the NFT

This is very essential since NFTs should have unique names.

6. Add an external link

Here, I add an external link to the original tweet which the screenshot is taken of.

7. Add a description

Here I add a small description of the tweet that is taken as a screenshot.

8. Add metadata related to the asset

Here I do three things. First I add the tries I used up for finishing the Wordle guessing game as a level (as [tries]/6). Then I count each colored box out of 30 and add them as numeric statistics for the NFT. Both of these steps are done just for fun. Finally, I check all the level metadata and statistics metadata are inserted accurately. If all the metadata is inserted accurately I move on to minting the NFT.

In the fillLevels function, I input the tries parameter where it is the number of tries out of 6. In the fillStats function, I input the statistics parameter where it is an object as follows. (As per the example below).

⬛⬛⬛⬛⬛
⬛⬛⬛🟩🟨
🟩⬛🟩🟩🟩
🟩🟩🟩🟩🟩

tries = "4"

statistics = {
               "blackBlocks" : "9",
               "greenBlocks" : "10",
               "yellowBlocks": "1"
             }

In the checkNumericTraits function, I input both tries and statistics so that I can check them against the previously entered values. If any of them mismatches, the program will notify that.

If all the metadata is entered correctly, the program will move on to the final step where it will mint the NFT. When I say mint the NFT, I mean the program will click Create button.

9. Mint the NFT

The last step is to mint the NFT by clicking the Create button. It is as simple as that.

Here I add two parameters, page is the puppeteer page and browser is the puppeteer browser. After clicking on the create button, the browser is closed and the program is exited with returning the asset URL of the NFT we minted. Now, all we have to do is create a reply tweet to the original tweet mentioning this asset URL. We will add a reply where it says, here the above tweet is for sale on OpenSea, and check it out via this asset URL.

Replying to the tweet with the asset URL

For this, we use the Twitter API v2 client for NodeJS.

Here we input the tweetID and the assetURL we obtained from the previous step. TwitterApi is constructed as mentioned in article one. After this, all of our steps are completed and the OpenSear-Worker has completed its job.

But,

There is a simple caveat where you cannot run a Chrome extension on a headless browser. So we cannot deploy this worker into a server (let’s say an Ubuntu machine). So we need to have a workaround for this. To tackle this, I used the following package to create a virtual display and trick the puppeteer browser into believing the server has a display.

GitHub - Rob--W/node-xvfb

By using xvfb I was able to overcome the above-mentioned issue and any code that runs between xvfb.startSync() and xvfb.stopSync() will believe the system has a display, so the problem is solved.

Closing remarks

So, this is the end of my story How I Automated Minting My Tweets as NFTs on OpenSea. There are so many wonderful people I have to thank. I would like to thank Andre Rabold for his wonderful article and his code for being my greatest inspiration. I would also like to acknowledge the inspiration I took from this article.

Last but not least, I would like to acknowledge the StackOverflow community, without their questions and answers (so that I can read and get inspired), this OpenSear system would still be a crazy dream that I once had.

If you want to take a look at the final product, the complete source code for the OpenSear system is hosted on GitHub (viva la open-source) below:

GitHub - Niweera/opensear: OpenSear is a system that helps me to mint and list my NFTs on opensea.io marketplace.

This is the OpenSear system in action.

1. On OpenSea.io marketplace

2. On Twitter

How I Automated Minting My Tweets as NFTs on OpenSea —Part 3 was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is part two of “How I Automated Minting My Tweets as NFTs on OpenSea article” series. In this article, I discuss how I created the OpenSear-API webhook listener using NodeJS and ExpressJS.

To read part one of this article series check out the following link:

How I Automated Minting My Tweets as NFTs on OpenSea

In my first article, (I did not want this to become an article series, but my story is too big to tell in a single piece) I discussed how I created Twitter Account Activity API webhooks. In this article, I discuss how I created the OpenSear-API which is essentially a webhook listener.

First of all, for this API, we need to create two endpoints.

GET /callback

This endpoint is used to register and authenticate the listener endpoint with the Twitter Account Activity API and to re-authenticate when needed.

POST /callback

This endpoint is used by Twitter Account Activity API to send webhook events.

The above gist is an example JSON payload that is sent to POST /callback endpoint. You can find more from here.

Account Activity data objects

So, let’s start building OpenSear-API. For this, let’s use NodeJS and ExpressJS because it’s cool to use JavaScript everywhere. However, if I go on and on about how to create an API using Node and Express, this will become extremely long not to mention boring. To learn how to create an API easily you can follow my guide on Node is Simple article series.

So long story short, let’s create this folder structure.

.
├── controllers
│   └── index.js
├── errors
│   └── index.js
├── keys
│   └── index.js
├── middleware
│   └── index.js
├── models
│   └── index.js
├── services
│   └── index.js
├── utilities
│   ├── async-wrapper.d.ts
│   └── async-wrapper.js
└── index.js

The /index.js file is the entry point to our Express API. Let’s create it as follows.

Now we need to create controllers and services which will be handling the API requests.

In this controllers file, we define the API endpoints GET /callback and POST /callback and the related services will be created in a service file. asyncWrapper is used to create a wrapper to catch errors and use our own error handling middleware to handle those errors.

To ease the development, we can use a data type declaration.

Now let’s create the validator middleware.

And let’s create the models file as well.

For model validation, we use the most powerful schema description language and data validator for JavaScript 💪. We know from the documentation, that every payload that Twitter Account Activity API sends has a for_user_id key. So we check it as the validation. (This is not necessary, but I add this just so that I can use the joi package.) Also, I will add a layer of security to these endpoints using custom-made middleware. Now let’s create the middleware functions that will handle the security and other essential Express features.

Middleware; can’t live with them, can’t live without them.

Let’s discuss the middleware used here.

1. body-parser — Body parser middleware is used to parse the request body and attach it as req.body.

2. morgan — Morgan (as in Dexter Morgan) is an HTTP request logger middleware for NodeJS.

3. cors — CORS is a package to use when you need to specify which resources can access this API. Which is also known as Cross-Origin Resource Sharing.

Cross-Origin Resource Sharing (CORS) - HTTP | MDN

4. helmet — Helmet is used to secure the Express apps by setting various HTTP headers.

Now let’s move on to the most important middleware of ’em all. The TwitterValidator.

In this TwitterValidator we check if the GET /callback endpoint is hit by the real Twitter API. To help us with that, Twitter API provides us with the x-twitter-webhooks-signature header where we can check against the received data.

Security; what about it?

1. Securing the GET /callback endpoint

As mentioned in the official guide, all we have to do is get the values for crc_token query parameter and nonce query parameter and then create the following string.

`crc_token=${token}&nonce=${nonce}`

Then we need to create the hash using the HMAC SHA-256 algorithm with the above string and TWITTER_CONSUMER_SECRET. Then we compare this hash with the value from the twitter-webhooks-signature header. (Here I do not make the hash comparison time attack safe. Theoretically, a timing attack could reveal information about the types and lengths of both hashes but not their real values.) If it matches, then we can say the GET request has been sent by Twitter. If not, someone is trying to abuse your API!

2. Securing the POST /callback endpoint

This is the trickiest part where I struggled a lot. No one ever told me (I mean the official guide) that I should use the raw buffer body to obtain the hash to check against the twitter-webhooks-signature value. I tried and tried to check this against the JSON.stringify() (stringified) body but to no avail. So later I found out that I have to use the raw body for this before bodyParser.json() middleware is being applied. So I created the verifyPostRequest custom verifier for bodyParser middleware. What it does is create the hash of the raw body and TWITTER_CONSUMER_SECRET using the HMAC SHA-256 algorithm and check the hash against the twitter-webhooks-signature value. If it matches, then we can say the POST request has been sent by Twitter. If not, someone is trying to abuse your API 😂!

Since all the other middleware is being dealt with, let’s move on to another very important middleware. Let’s build the ErrorHandling middleware.

In error-handling middleware, we check if the errors that are captured by our asyncWrapper are the ones we created in errors/index.js , or if not we handle it using a default error handler. The following is the errors/index.js file where we define our custom error classes.

Now that we have completed almost all the basic parts of the OpenSear-API Express app, let’s move on to creating the services that are used in the controllers we defined earlier.

In our services file, we handle two endpoints.

1. GET /callback endpoint — We create services.getHandler() function so that we can create and send the CRC token back to the Twitter API. For that, we create an object,

{
  response_token: `sha256=${hmac}`
}

where hmac is created using the TWITTER_CONSUMER_SECRET and the crc_token provided by the query string from Twitter API and creating a hash as we did before. The Twitter Account Activity API will hit our GET /callback endpoint like the following.

/callback?crc_token=$token&nonce=$nonce

So if the Twitter API accepts our response_token as valid, then we are in business. We can listen to Twitter events via our webhook listener.

2. POST /callback endpoint — We create services.postHandler() function to check if the Twitter event we want to focus on has actually happened. In my example, I check if the Twitter event is a tweet that I created and it contains the word wordle in the tweet text body. If the Twitter event is what I am looking for, I will queue for a job here. I have omitted the job queueing functionality here because it is up to your imagination to queue the job. For this task, I’d recommend, reading through the following.

10 Best Node.js Job Queues Libraries in 2022 | Openbase

Now we have completed implementing the OpenSear-API. We can check this by running the following commands as mentioned in my previous article.

$ node --experimental-specifier-resolution=node twitter-cli.js create-webhook

$ node --experimental-specifier-resolution=node twitter-cli.js create-susbscription

First, we created the webhook, then we added the subscription for our Twitter user. Then we can create a tweet that contains the word wordle in the text. So as for the OpenSear system, we have now completed the first half. Now we can queue a job based on a tweet using OpenSear-API.

But we need to have a worker to undertake that job and mint and publish the NFT. I’ll keep it for the part 3 of the series. So let’s continue our discussion in the next article where I will discuss how I created the OpenSear-Worker which is used to handle the job.

GitHub - Niweera/opensear: OpenSear is a system that helps me to mint and list my NFTs on opensea.io marketplace.

How I Automated Minting My Tweets as NFTs on OpenSea — Listening to Twitter Events and Queuing Jobs was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, I discuss how I turned my crazy idea into a reality through some trials and tribulations of UI automation of decentralized web apps (dApps).

It all started one day when I came across a YouTube video about Web3 apps and minting NFTs. I got curious about what are NFTs and why should I even bother about Web3 (a little late to the party, I know). I checked out the video by Jeff Delaney (Fireship.io) and got to know about NFTs and what are Web3 and dApps.

Now I have a big question; what should I mint as NFTs? Since I am not an artist, I had no clue about what to do. So here comes the thinking part.

I got to know about a site where you can mint and sell your tweets called Valuables (what they are doing is taking the screenshot of the tweet and minting an NFT token based on it). Naturally, I minted and published my first NFT on that platform. Surprisingly, I learned that I do not need the Valuables platform to mint my tweets as NFTs, and I can mint my tweets as NFTs using the OpenSea.io platform.

OpenSea is the largest NFT marketplace which is a decentralized web application running on the Ethereum blockchain. After learning how to do the deed, I created my first NFT collection in OpenSea and listed it on the marketplace.

So as I might have mentioned earlier, this NFT minting is a tedious task to do manually. If you have thousands of images, still you need to upload them by hand and provide the necessary metadata manually and mint the token. So I thought, hmm if I can do it, a computer can definitely do it.

Then I looked for a developer API for the OpenSea platform. Unfortunately, you can only view assets not mint them using the API. So I turned on to UI automation which is to follow all the manual steps of minting an NFT but by using an automated browser. The following are the steps that I followed to mint an NFT manually on OpenSea.

1. First of all, I complete a Wordle puzzle and share the results in a tweet.
2. Take a screenshot of the tweet using the TweetPik app.
3. Add some touch-ups to the screenshot (if you know what I mean).
4. Go to OpenSea.io marketplace and go to add assets page.
5. Upload the screenshot.
6. Add a unique name (this is very essential).
7. Add an external link (this link can be anything related to the asset, but here I put the tweet URL of the screenshot).
8. Add a description.
9. Add metadata related to the asset (as metadata I add levels and stats where levels are the tries I took to complete the Wordle [try/6] and as stats, I add the number of black, green, and yellow blocks from the Wordle).
10. Finally, I create the NFT by clicking the Create button.

This is the process that I’m going to automate. However, there is one last step where you can list the NFT in the marketplace by quoting a selling price.

But I am not going to automate it because that is something I should not automate. No one should let a machine quote the price for you. It is bad for everyone because the price for any asset is based on the perceived value of that asset.

And a machine cannot perceive anything because it is not human. For more info on how anything is valued, check out this great video documentary by Cleo Abram.

Enough with the storytelling; show me the code!

Well, finally I am moving on to the coding. First of all, we need to get a high-level understanding of what we are trying to build. Let me show a high-level architecture diagram of the system that I am building. Let’s call it the OpenSear System (ok, that name might’ve been taken by others but at that time I thought about a witty name, I didn’t think of that). Actually, the name doesn't matter, what it does matters right?

How did I use Twitter Account Activity API to listen to my tweets in real-time?

The first task that I have to automate is to listen whenever I post a Wordle tweet and create a new job and queue it in a job queue. For this, I used the Twitter Account Activity API which provides a webhook functionality where it hits a given API endpoint once I tweet something. All I have to do is set up a webhook API where it can be hit by Twitter Account Activity API.

First, let’s set up the Twitter Account Activity API so that it can send a payload to my webhook API.

1. Get a Twitter Developer API key if you do not have one.
2. Follow the official guide provided by Twitter on how to set up the webhooks.

The following is the article I followed to set up the webhooks.

Comprehensive Guide to Twitter Webhook

We need to create six functions to set up Twitter webhooks. For this, I used the Twitter API v2 client for NodeJS.

First, let’s create the base class for this task. Let’s call it TwitterCLIService because it will be called via a terminal.

To create the TwitterCLIService base class, we need four keys. As you might know, we need the following Twitter API keys.

1. TWITTER_CONSUMER_KEY
2. TWITTER_CONSUMER_SECRET
3. TWITTER_ACCESS_TOKEN
4. TWITTER_ACCESS_TOKEN_SECRET

These tokens can be taken from the Twitter Developer Dashboard. Watch the following video to learn how to create a Twitter App and obtain the above API tokens.

In the above gist, this.actions are the constants that we will call our functions related to the TwitterCLI we are building. First, we will create the endpoint to get existing webhooks. For a free account, Twitter allows one webhook and 15 subscribers (15 different Twitter accounts that can have the access to that one webhook).

For this function, we need to obtain a Twitter Bearer Token. It can be obtained from the Twitter Developer Dashboard just as other API tokens. Follow this article to learn how.

If we have an existing webhook configured, when we call TwitterCLIService.getWebhooks() function, the following output will be provided.

Now let’s add the create webhook function.

Before creating a webhook, you need to set up a dev environment for Account Activity API/Sandbox from Twitter Developer Dashboard. It is pretty simple and straightforward. Now, remember the name you set for your environment (For free accounts, Twitter only allows one environment). I’ll save this name as TWITTER_WEBHOOK_ENV.

Remember, Twitter only accepts HTTPS webhook URLs with no PORTs. So when you are in developing and testing you can use ngrok to set up a temporary HTTPS endpoint. Upon successfully creating a webhook (registering a webhook URL in the Twitter API), the following result will be provided.

Now let’s add the delete webhook endpoint.

In deleteWebhook() function, we are doing two things. First, we check for existing webhooks and then we delete if there are any. If the webhook deletion is successful, we don't get any results from the Twitter API, only an HTTP 204 (No Content) response.

Now let’s move on to the functions that relate to subscription. First, we created webhooks. Now the Twitter API knows which API endpoints to send their notifications to. But still, the Twitter API does not know which users to listen to. So we will create functions that will tell the Twitter API which users to listen to.

First, let’s create getSubscriptions() function.

When we call TwitterCLIService.getSubscriptions() function, if there are existing subscriptions, it will show the following results.

Now let’s add the create subscription function.

If the subscription creation is successful, we don’t get any results from the Twitter API, only an HTTP 204 (No Content) response. However, it is vital to note that in my example, we only subscribe to ourselves.

Now the Twitter Accounts Activity API listens to our tweets (the one who owns the Twitter Developer Account) and if there is activity, it will send a payload to the registered webhook.

Now if we need to unsubscribe from the Twitter Account Activity API, we need to implement the following function.

In this function, we do two things. First, we obtain the Twitter user ID. This is not the Twitter username. Twitter user ID is a unique value that every Twitter account has. You can find your own Twitter user ID from the following website.

TweeterID - Twitter ID and username converter

After obtaining the Twitter user ID, we delete the subscription for that specific user. If the subscription deletion is successful, we don’t get any results from the Twitter API, only an HTTP 204 (No Content) response.

Now we can move on to creating the CLI script. but before that take a look at the final output that we’ve been developing.

The above file can also view from the source code hosted on GitHub.

Now let’s create the CLI script to call the functions we created in TwitterCLIService. The following file can also be viewed from the source code hosted on GitHub.

Calling this CLI script is pretty simple. If you are unfamiliar with using ES6 syntax in NodeJS, do not fear, they are just the same as you would use in React or any other front-end JS framework.

But the only thing we need to remember is that out of the box, NodeJS does not support this and we need to pass a parameter to activate this powerful feature. For testing purposes, I used NodeJS v16.14.0 and NPM v8.3.1 in my development environment.

To call the CLI script we need the issue the following commands.

1. GET Webhooks

$ node --experimental-specifier-resolution=node twitter-cli.js get-webhooks 

2. CREATE Webhook

$ node --experimental-specifier-resolution=node twitter-cli.js create-webhook

3. DELETE Webhook

$ node --experimental-specifier-resolution=node twitter-cli.js delete-webhook

4. GET Subscriptions

$ node --experimental-specifier-resolution=node twitter-cli.js get-susbscriptions

5. CREATE Subscription

$ node --experimental-specifier-resolution=node twitter-cli.js create-susbscription

6. DELETE Subscription

$ node --experimental-specifier-resolution=node twitter-cli.js delete-susbscription

Now we have completed a tiny first step into building OpenSear System. But still, we cannot test these commands since we do not have our OpenSear-API up and running.

So we have to start working on our OpenSear-API which we need to set up so that it can listen to Twitter Account Activity API webhook events. But if I keep on writing, this article will be extremely large and no one would read it (I know how hard it is to read, so I have a sympathetic ear).

I will stop for now in this article and in the next article, I’ll discuss how to create the OpenSear-API and how to start listening for Twitter Account Activity API webhook events and respond to them accordingly.

This is the GitHub repo for this article and I added this just in case you are curious to see the end product:

GitHub - Niweera/opensear: OpenSear is a system that helps me to mint and list my NFTs on opensea.io marketplace.

So until we meet again with part two of this article, happy coding, and happy minting…

Want to Connect?

Shoot me on queries here.

How I Automated Minting My Tweets as NFTs on OpenSea was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

tl;dr — This is the thirteenth article of my journey into the Google Summer of Code 2021 with SCoRe Lab. Here I discuss week twelve (9nd of August to 15th of August) of my GSoC experience.

’Tis the final week of the GSoC 2021!

So, this is the final week of the Google Summer of Code program where we have to complete all of our development works and submit our final code for the final evaluations. I think it kind of saddens me to see the end of this wonderful program for this year. I know I have gained a lot of knowledge through the course of GSoC 2021. And I am proud to say that I have progressed myself in coding, software engineering best practices, and software security best practices from the experience I gained via the GSoC program.

As I have mentioned in the previous article, in this week, all I had left to do was to create the wiki for the DNSTool-Middleware-API project. So I documented all the endpoints and their usage instructions in the wiki file. The following is the link for the wiki of DNSTool-Middleware-API.

DNSTool-Middleware-API/WIKI.md at main · scorelab/DNSTool-Middleware-API

To create this wiki, I followed the Medium API Documentation and got some good insights from it.

GitHub - Medium/medium-api-docs: Documentation for Medium's OAuth2 API

So as the final notes, I would like to extend my gratitude to all my wonderful mentors, my awesome fellow developers, and our awesome organization,

Score Labs Home Page

SCoRe Lab 👏.

Last but not least, I would like to thank our wonderful benefactor Google for providing us with this wonderful opportunity and I hope Google will keep doing the good work for the foreseeable future 🙏. So until we meet again, happy coding…

GSoC 2021 with SCoRe Lab — Week 12 was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

tl;dr — This is the twelfth article of my journey into the Google Summer of Code 2021 with SCoRe Lab. Here I discuss week eleven (2nd of August to 8th of August) of my GSoC experience.

The end is nigh!

So, the end of the Google Summer of Code 2021 program is around the corner and I am so excited to complete my milestones and publish our DNSTool-Middleware-API to the world. Currently, in the API, a user can register, submit a scan, update the state of the scan and the user can download the scan results. The user can use the DNSTool-CLI to download the scan results and the DNSTool-CLI will be directly communicating with the Middleware API. For the authentication part in DNSTool-CLI, we use a custom generated service account (just like in the Google Cloud Platform), and then by using that service account, the included private key is used to sign a JWT token which provides several important (but not secret) information to the API about the scans to be downloaded. Then the user can send the JWT token to list the scan results files and by using that JWT token the user verification is done and the relevant files are returned to the user. Of course, the user does not have to do any of these tasks by themselves, that’s is why we provide DNSTool-CLI. Behind the scenes, DNSTool-CLI does all of these tasks on behalf of the user and the user only has to do is to provide the obtained service account JSON file to the CLI. For example,

$ dnstool download --service-account /path/to/service/account.json

See, how cool is that? 😁

Since there is only one week (9th August to 15th August) left to work on the DNSTool-Middleware-API, there are only a few things left to do 🙏. This week, I wrote test cases for the API using Flask Test Client and it was awesome to see all the test cases are passing with flying colors.

So in the coming week, I will complete the API documentation and Wiki for the DNSTool-Middleware-API. Until we meet again, happy coding…

GSoC 2021 with SCoRe Lab — Week 11 was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

tl;dr — This is the eleventh article of my journey into the Google Summer of Code 2021 with SCoRe Lab. Here I discuss week ten (26th of July to 1st of August) of my GSoC experience.

So what happened this week?

In this week, my task was to create the endpoint to handle scans results file downloading. In our DNSTool system, a user can submit scans and the system will do scans for the provided resources. The results of the scans are stored in Google Cloud Storage. The data is stored in buckets and it is not accessible to the general public. So my task was to provide the user the ability to download their specific scan results without directly interacting with Google Cloud Storage. For this situation, as I mentioned in my previous blog post, I created a custom service account file where I generated a private key for the specific user and for their specific scan and stored the relevant public key in our Firebase RealTime Database.

For this task, in our DNSTool-CLI, the user can provide their service account JSON file and download their respective scan files. To support this feature, I had to implement the following two endpoints.

GET /list-downloads
GET /download/<path:path>

Suppose there are 5 files in the scan results in Google Cloud Storage, now the user provides their service account JSON file and they can download all the scan results. However, there is a huge problem with this method. According to the RFCs of HTTP implementation, it can respond with one and only one file per request.

• Issue returning multiple downloads from one Flask route using MultipartEncoder
• Browser support of multipart responses

The above mentioned Stackoverflow questions provide a comprehensive idea about why we need to zip all the files before sending the response. However, there is also a problem with zipping all the files into one file and sending a single file as the response, because we can’t make assumptions about the size of the scan results. If the scan results are few Kilobytes, then we can easily compress them in the memory and return the zipped file to the user. However, if the files are few Gigabytes large, then we can’t do this in the memory (for that we need a really good server 😁). So to resolve this situation, I implemented a middle route GET /list-downloads and now the CLI can see which files are required by the user and it can provide the file names in the GET /download/<path:path> route and easily download them. However, there is another issue I faced when implementing the GET /download/<path:path> route. Since this is directly involved with downloading an unknown size file from Google Cloud Storage and I had to return it to the user. Luckily, Google Cloud Storage Python documentation provides an easy method to download chunks of data from the Cloud Storage bucket.

google-resumable-media

By using this method and Flask’s Streaming Contents API, I was able to implement the above route. Now the user can download a results file which can be small as few Kilobytes and large as few Gigabytes 😁.

So with these implementations, I submitted a PR,

Implement scan results file download API endpoint in DNSTool-Middleware-API[API-GATEWAY] by Niweera · Pull Request #15 · scorelab/DNSTool-Middleware-API

and it got merged into the main repository.

So in the coming two weeks, I will be working further on the DNSTool-Middleware-API[API-GATEWAY], and until we meet again, happy coding…

GSoC 2021 with SCoRe Lab — Week 10 was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

tl;dr — This is the tenth article of my journey into the Google Summer of Code 2021 with SCoRe Lab. Here I discuss week nine (19th of July to 25th of July) of my GSoC experience.

Life after the evaluations,

After the evaluations week, I resumed my work on the DNS-Tool-Middleware project. As I mentioned in my previous post, my mentor and I discussed the authentication flow on how to grant authorization to the users who need to download resources from Google Cloud Storage. So according to our user authentication flow, first, we need to generate a unique service account for that specific user and for that specific scan. To create this service account I decided to follow the best practices used by Google itself. The following article is a good read if you need to learn about how the Google Service Account JSON files work.

How to create your own Google service account key file

The following is a sample Google service account JSON file.

{   
  "type": "service_account",   
  "project_id": "demo-project",       
  "private_key_id": "f871b60d0617be19393bb66ea142887fc9621360",
  "private_key": "-----BEGIN RSA PRIVATE KEY-----.....",
  "client_email": "look-no-keys@demo-project.iam.gserviceaccount.com",   
  "client_id": "102234449335144000000",   
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/look-no-keys%40demo-project.iam.gserviceaccount.com" 
}

The keys in the above JSON object has the following meaning,

So, what I did was, I created my own service account file with the same above fields 😁.

The following is a sample service account JSON file of DNS-Tool-Middleware.

The private_key_id is a special unique key to identify a specific service account JSON file and it resembles the functionality of a Google service account JSON file. The private_key_id is used to check if the certain service account file is used by the user and it is also used to check if the authentication token sent by the user is valid. The private_key_id is actually a string with random 16 bytes hashed using the SHA256 hashing algorithm. So it remains considerably unique 😂. I used the following wonderful Python library to generate this random bytes string.

Welcome to PyCryptodome's documentation - PyCryptodome 3.9.9 documentation

The private_key is a 2048 bit RSA private key in PEM encoded format. It is generated using the above mentioned Python library. The client_email is the email of the user who obtained the service key, the client_id is the Firebase UID of the user who obtained the service key, the scan_id is the ID of the scan that the user wants to download resources from Google Cloud Storage, and the scans is the scans related to the specific scan_id.

By using this service account file, the user requests can be authenticated and checked against the server for validity. Special thanks to the following article for opening my mind 🙏.

Stop downloading Google Cloud service account keys!

With these implementations, I submitted a PR,

Add endpoint to obtain service account for download authorizations by Niweera · Pull Request #13 · scorelab/DNSTool-Middleware-API

and it got merged into the main repository.

So in the coming weeks, I will be working further on the authentication workflow, and until we meet again, happy coding…

GSoC 2021 with SCoRe Lab — Week 9 was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

tl;dr — This is the ninth article of my journey into the Google Summer of Code 2021 with SCoRe Lab. Here I discuss week eight (12th of July to 18th of July) of my GSoC experience.

Evaluation week!!!

After weeks of designing developing and learning about developing, finally, the evaluation week arrived. As I discussed in my previous blog, it was a little bit of a tight schedule that I had to finish up my first evaluation milestones before the evaluation week came. So finally I was able to finish up my targets before the deadline.

So to recap all of the things I developed before the evaluations, I had to develop API endpoints to support user registration, obtain domain zones list, obtain GCP zones list, check whether the provided email is a valid organization email, creating scans, and updating scans. In addition to these tasks, I created the documentation of the API endpoints using OpenAPI v3 specification Swagger UI. Since some of the API endpoints provide static outputs which can be easily cached, I integrated the Flask Caching to the Flask App in DNSTool-Middleware-API[API-GATEWAY]. In addition, to these tasks, some of the features provided by the API are strictly for human interaction only. Such as new user registration, we need to check if the requester is actually a human. For this situation, I integrated the Google reCAPTCHA v3 to check whether the request in fact came from a human user, not from a Bot 😁.

After the evaluation week, the coding starts again and we are in the endgame now. In the next phase, I have to develop the authentication mechanism for the user to download the required resources from Google Cloud Storage. My mentor and I had a discussion about the most suitable and (SECURE!) way to do the authentication for the user. At first, we were discussing creating separate user accounts for the users but there was a slight hiccup,

so much for that plan 😥. So we brainstormed our ideas on a suitable alternative method and we came to a conclusion on a superb idea. The following is the architecture of the user authentication flow.

So we decided to go with the flow 😎.

So in the coming weeks, I will be implementing this authentication workflow in the DNSTool-Middleware-API system. So until we meet again, happy coding. (Oh, one thing I forgot to mention, I passed the evaluations. YAY! 😁)

GSoC 2021 with SCoRe Lab — Week 8 was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

tl;dr — This is the eighth article of my journey into the Google Summer of Code 2021 with SCoRe Lab. Here I discuss week seven (5th of July to 11th of July) of my GSoC experience.

The first evaluation is nigh!

As the first evaluation is almost near, I was on a tight schedule since I needed to complete my first evaluation milestones. As for the first evaluation milestones, I had to complete all the supporting endpoints which are needed for DNSTool-Middleware-API[API-GATEWAY] to function properly. The supporting endpoints are as follows.

GET /zones/<query>
POST /register
POST /check-email
GET /gcp-zones/<query>
GET /scans
POST /scans
PATCH /scans/<id>
DELETE /scans/<id>

After developing all these endpoints, now in DNSTool-Middleware-API[API-Gateway] a user can register in our system, create a new scan, list all the existing scans, update the scan status of a specific scan and they can also delete a specific scan from our database if they want.

As we discussed in our online meeting, one of the tasks I was assigned was protecting the POST /register endpoint. The big reason behind protecting the POST /register endpoint is that it needed human interaction. If someone knows the endpoint, then they can utilize a BOT 😁 to create fake accounts and this will overutilize the Google Authentication and this will be an issue for real human users. So my task was to allow only human users to create accounts in our system.

Google reCAPTCHA v3 comes to the rescue 🙏

reCAPTCHA | Google Developers

Google reCAPTCHA is a free service that protects our resources from spam and abuse. It uses advanced risk analysis techniques to differentiate between humans and bots. You must have seen this reCAPTCHA prompt from time to time when visiting a website.

I know sometimes it is a real pain, but it is really important that we need to safeguard our resources from spam and abuse. Sometimes you might fail the reCAPTCHA check a few times and feel like this,

So anyway, the Google reCAPTCHA v3 removes all these hassles and it silently checks your interaction with the browser and checks whether if you are a real human user or a bot. You do not need to click on anything like those annoying fire extinguisher images or mind-bending images of traffic lights anymore 😁. The documentation of Google reCAPTCHA v3 is pretty cool and straightforward.

reCAPTCHA v3 | Google for Developers

So after adding the Google reCAPTCHA v3 to our system, I submitted a PR,

Add Google reCAPTCHA v3 verification to `/register` endpoint by Niweera · Pull Request #11 · scorelab/DNSTool-Middleware-API

and it got merged into the main repository.

So with this concluded my tasks before the evaluation one 😎. There is more to do with our DNSTool-Middleware-API system and I will be working on them in the coming weeks. Until we meet again, happy coding…

GSoC 2021 with SCoRe Lab — Week 7 was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

tl;dr — This is the seventh article of my journey into the Google Summer of Code 2021 with SCoRe Lab. Here I discuss week six (28th of June to 4th of July) of my GSoC experience.

So what happened in week six?

As my project mentors and I discussed in our online meeting, I was assigned to develop the endpoints related to making updates to the scan records stored in the database. The development of these endpoints was pretty much straightforward and I utilized the power of the all-mighty, Flask RESTful library.

Flask-RESTful - Flask-RESTful 0.3.8 documentation

In the Flask RESTful library, you can easily define a single controller for a route and define several HTTP methods. For example,

PATCH  /scans/<id>
DELETE /scans/<id>

these above two HTTP methods utilize the same endpoint /scans/<id> and if the user hits this endpoint via a PATCH method, we need to respond to it as a PATCH request and if the user hits this endpoint via a DELETE method, we need to respond to it as a DELETE request. In hindsight, we need a mechanism to capture the HTTP request method and respond to it accordingly. In plain Flask, we’ll do this as follows,

from flask import request

@app.route('/scans/<id>', methods=['PATCH', 'DELETE'])
def scan_controller(id:str):
    if request.method == 'PATCH':
        return update_scan_record(id)
    else:
        return delete_scan_records(id)

(from https://flask.palletsprojects.com/en/1.1.x/quickstart/#http-methods)

But we can utilize the Flask RESTful in the following way (in a more elegant way indeed 😁).

from flask_restful import Resource

class ScanController(Resource):
    method_decorators: Dict[str, List[Callable]] = dict(patch=[validator, authenticate])    
    model: str = "UpdateScan"

    def patch(self,id:str):
        return update_scan_records()

    def delete(self,id:str):
        return delete_scan_record()

(from https://flask-restful.readthedocs.io/en/latest/quickstart.html#full-example)

So by using this example, I was able to create endpoint handlers for PATCH and DELETE methods. Of course, I utilized other libraries such as Marshmallow and Firebase Admin to fulfill my tasks. After completing these tasks, I updated the Swagger-UI to reflect the newly added endpoints, because, documentation is your biggest ally 💪.

So after all of these, I submitted a PR,

Add Suspend and Delete Scans Endpoint DNSTool-Middleware-API[API-Gateway] by Niweera · Pull Request #9 · scorelab/DNSTool-Middleware-API

and it got merged into the main repository.

So in the coming week, I will work on enhancing the features of our DNSTool-Middleware-API[API-Gateway] component 🤞. Until we meet again, happy coding…

GSoC 2021 with SCoRe Lab — Week 6 was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.