This post outlines how I was able to setup and create the similar process as the review apps on bitbucket using bitbucket pipelines. Note that we already have appropriate app.json as our application manifest so make sure you have yours as well as appropriate for your application.

Part 1: Review Apps Generation/Updates

We start by defining bitbucket pipeline. This is done by creating bitbucket-pipelines.yml file and defining your pipeline in there.

image: techgaun/awscli-git:1.16.241

clone:
  depth: full

pipelines:
  pull-requests:
    '**':
      - step:
          script:
            - bash "${BITBUCKET_CLONE_DIR}/devops/review.sh"

BITBUCKET_CLONE_DIR is one of the several default environment variables available for builds in bitbucket. The techgaun/awscli-git:1.16.241 is the docker image I’ve created for our purpose that has aws-cli (version 1.16.241) and git installed.

In the above pipeline, for every pull-requests created or updated, we are executing a bash script that is in your <project_root>/devops/review.sh .

Next, this is what our review.sh script which is the actual meat looks like:

#!/bin/bash

cd "${BITBUCKET_CLONE_DIR}"

PROJECT_NAME="${BITBUCKET_REPO_SLUG}-stage-pr-${BITBUCKET_PR_ID}"
PROJECT_URL="https://${PROJECT_NAME}.herokuapp.com"
NOTIFICATION_TEXT="Review App for ${BITBUCKET_GIT_HTTP_ORIGIN}/pull-requests/${BITBUCKET_PR_ID} is deployed & available at ${PROJECT_URL}"

content_type="Content-Type: application/json"
accept="Accept: application/vnd.heroku+json; version=3"
auth="Authorization: Bearer ${HEROKU_API_KEY}"

echo -e "Checking if the review app for this PR already exists or not\n"
curl -ns "https://api.heroku.com/apps/${PROJECT_NAME}" -H "${accept}" -H "${auth}"
current_app_id=$(curl -ns "https://api.heroku.com/apps/${PROJECT_NAME}" -H "${accept}" -H "${auth}" | jq -r '.id')

echo -e "Current app id is: ${current_app_id}\n"

if [[ "${current_app_id}" = "null" || "${current_app_id}" = "not_found" ]]; then
  TGZ_FILE="${PROJECT_NAME}.tgz"
  S3_PATH="s3://${S3_BUCKET_ARTIFACT}/${TGZ_FILE}"

git archive --format=tar.gz -o "${TGZ_FILE}" "${BITBUCKET_COMMIT}"
  aws s3 cp "${TGZ_FILE}" "${S3_PATH}"

S3_URL=$(aws s3 presign "${S3_PATH}")

trap "aws s3 rm ${S3_PATH}" EXIT

app_id=$(curl -ns -X POST https://api.heroku.com/app-setups \
    -H "${content_type}" \
    -H "${accept}" \
    -H "${auth}" \
    -d "{\"app\": {\"name\": \"${PROJECT_NAME}\", \"personal\": false, \"organization\": \"zego\", \"locked\": false},\"source_blob\": {\"url\": \"${S3_URL}\"}, \"overrides\": {\"env\": {\"HEROKU_APP_NAME\": \"${PROJECT_NAME}\"}}}" \
    | jq -r '.id')

echo -e "Created heroku app with id: ${app_id}"
  curl -ns https://api.heroku.com/app-setups/${app_id} -H "${accept}" -H "${content_type}" -H "${auth}"

while [[ `curl -ns https://api.heroku.com/app-setups/${app_id} -H "${accept}" -H "${content_type}" -H "${auth}" | jq -r '.status'` = "pending" ]]; do
    sleep 2
    printf "."
  done

curl -ns https://api.heroku.com/app-setups/${app_id} -H "${accept}" -H "${content_type}" -H "${auth}"
  echo ""

if [[ `curl -ns https://api.heroku.com/app-setups/${app_id} -H "${accept}" -H "${content_type}" -H "${auth}" | jq -r '.status'` = "succeeded" ]]; then
    echo -e "\nCompleted setting up heroku app!!! Please visit ${PROJECT_URL}"
    curl -X POST -H "${content_type}" --data "{\"text\":\"${NOTIFICATION_TEXT}\"}" "${SLACK_WEBHOOK_URL}"
  else
    echo "Failed to setup heroku app!!!"
    false
  fi
else
  git push --force "@git.heroku.com/${PROJECT_NAME}.git">https://heroku:${HEROKU_API_KEY}@git.heroku.com/${PROJECT_NAME}.git" "${BITBUCKET_COMMIT}:refs/heads/master"
  echo -e "\nStarted deploying on heroku app!!! Please visit ${PROJECT_URL}"
  curl -X POST -H "${content_type}" --data "{\"text\":\"${NOTIFICATION_TEXT}\"}" "${SLACK_WEBHOOK_URL}"
fi

Lets dissect the above shell script:

• we construct PROJECT_NAME using repo slug and pr id and use that to create project url so the heroku app would be available at https://<your-repo-slug>-stage-pr-<pr_number>.herokuapp.com
• we setup notification text (sent to slack for notifying people watching that channel for running acceptance) and various headers necessary for heroku.
• we check if the review app already exists or not using our project name.
• if the review app does not exist, we create an tar archive of current codebase, copy that to S3 bucket, and make an API call to heroku to create a new app with the given project name and the pre-signed source url of S3 object. Once that happens, we wait until the heroku deploy completes and once it does we send a notification to slack. If the deploy fails, we exit with an error message
• if the review app already exists (pull request was created in the past and a new commit was pushed to the PR branch), then we simply perform git push operation to heroku git.

Finally, once we have the pipeline and shell setup, we need to do some configurations on bitbucket:

• Enable the bitbucket pipeline from https://bitbucket.org/<your_user_or_org>/<repo>/admin/addon/admin/pipelines/settings
• Add the appropriate repository variables in https://bitbucket.org/<your_user_or_org>/<repo>/admin/addon/admin/pipelines/repository-variables

Note that the AWS key-pair listed above must be able to write to the S3 bucket and create a pre-signed URL so that the heroku can access the code archive file.

Once everything is configured, you should now be able to see pipeline executing for your open pull requests.

You can see the logs and URL of the herokuapp printed by above bash script from earlier in pipeline build.

We’ve a working review apps generation but there’s more to it. We need to be able to clean up the review apps once the pull request is merged (or declined).

Part 2: Tearing Down The Review Apps

Once the PRs are merged (or declined), we need to clean up the generated review app. When we built out this pipeline, there was no way to do so in bitbucket pipeline. We could have sure leveraged some ways on the target branches of PRs on merge but then it would not work for declined. So, I looked for alternatives. In the meantime, I was in a need of a webhook executor for other works because most of our stuff are behind the firewall and bitbucket can not reach out many of such pieces we’ve behind the firewall. So, I built a public-facing webhook service using adnanh/webhook that allowed us to execute arbitrary shell commands for any registed webhook payload and source.

So we create a hooks.json and add matching rule for bb-review-apps:

[
  {
    "id": "bb-review-apps",
    "execute-command": "/opt/webhooks/scripts/stop-review-app.sh",
    "command-working-directory": "/opt/webhooks/scripts",
    "pass-arguments-to-command": [
      {
        "source": "payload",
        "name": "repository.name"
      },
      {
        "source": "payload",
        "name": "pullrequest.id"
      }
    ],
    "trigger-rule": {
  "match": {
    "type": "regex",
    "value": "(MERGED|DECLINED)",
    "parameter": {
      "source": "payload",
      "name": "pullrequest.state"
    }
  }
    }
  }
]

We create the stop-review-app.sh file and place it in /opt/webhooks/scripts directory:

#!/bin/bash

# ${1} -> repository.name
# ${2} -> pullrequest.id

HEROKU_API_KEY="<REPLACE_WITH_YOUR_HEROKU_API_KEY>"

PROJECT_NAME="${1}-stage-pr-${2}"

content_type="Content-Type: application/json"
accept="Accept: application/vnd.heroku+json; version=3"
auth="Authorization: Bearer ${HEROKU_API_KEY}"

echo -e "Shutting down review app for ${PROJECT_NAME}\n"

curl -s -XDELETE "https://api.heroku.com/apps/${PROJECT_NAME}" -H "${accept}" -H "${content_type}" -H "${auth}"

Now, you can run the webhook server:

./webhook -hooks hooks.json -verbose -hotreload

Finally, you can expose the webhook through nginx or other proxy. I used nginx for myself and used letsencrypt to manage the certificates.

Basically, your configuration would require an upstream definition:

upstream webhooks {
  server 127.0.0.1:9000;  
}

and then a server block with a location block that proxy_passes to the webhooks upstream.

Finally, add webhook configuration to your bitbucket repository webhooks. This would require you to choose Pull Request Merged and Declined as the triggers for the webhook to receive data. See the screenshot below:

And, you got a complete review app setup with bitbucket and heroku. I hope it becomes useful for someone out there.

Recently, we went through the process of the bitbucket to github migration and several of us were tasked with moving repositories to github for the ones we were admins.

Part 1: Moving The Repositories

So I wrote a quick bash script that takes a file with name of repositories separated by newline. The input could have been response of bitbucket API call to retrieve the list of repositories but I was not migrating all of them so it was just easier for me to compile the list in a file. It would have been a mess for me to manage 45+ repositories and configure the base branch protection rules and basic config on each of them if I had to do the manual migration.

• (bare) clones repo from bitbucket
• creates repo on github with internal visibility (we are using enterprise)
• allows write permission to engineering and qe teams
• pushes all the branches to github
• sets up branch protection rules

As long as there’re no configuration changes on branch protection and passed repo settings, this script is totally re-runnable for the migrated repository (for example, to re-sync branches).

And, usage is pretty simple:

Create a personal access token with proper access to create repo and use that as GH_TOKEN envvar. The sample command looks as below:

BB_ORG=<bb_org_name> GH_ORG=<gh_org_name> GH_TOKEN=<github_token> ./repo-mover.sh <file_with_repo_name_list>

techgaun/tg-misc-scripts

#!/usr/bin/env bash

# moves repos listed in a file from bb to gh
# usage: BB_ORG=<bb_org_name> GH_ORG=<gh_org_name> GH_TOKEN=<github_token> ./repo-mover.sh <file_with_repo_name_list>

set -euo pipefail

BB_ORG="${BB_ORG:-paylease}"
GH_ORG="${GH_ORG:-gozego}"
GH_USER="${GH_USER:-techgaun}"
GH_CRED="${GH_USER}:${GH_TOKEN}"

ROOT=$(pwd)
BRANCH_PROTECTION_BODY="$(cat <<-EOF
{
  "required_status_checks": {
    "strict": true,
    "contexts": []
  },
  "enforce_admins": true,
  "required_pull_request_reviews": {
    "dismissal_restrictions": {
      "users": [],
      "teams": [
        "qe",
        "engineering"
      ]
    },
    "dismiss_stale_reviews": true,
    "require_code_owner_reviews": true,
    "required_approving_review_count": 2
  },
  "restrictions": {
    "users": [],
    "teams": [
      "qe"
    ]
  },
  "allow_force_pushes": false,
  "allow_deletions": false
}
EOF
)"

while read repo; do
  echo
  echo "Starting repo creation for ${repo}..."

  cd "${ROOT}"
  rm -rf "${repo}.git"
  git clone --bare "git@bitbucket.org:${BB_ORG}/${repo}.git"
  cd "${repo}.git"
  curl -H 'Accept: application/vnd.github.nebula-preview+json' -u "${GH_CRED}" "https://api.github.com/orgs/${GH_ORG}/repos" -d "{\"name\": \"${repo}\", \"private\": true, \"visibility\": \"internal\", \"delete_branch_on_merge\": true}"
  curl -XPUT -u "${GH_CRED}" "https://api.github.com/orgs/${GH_ORG}/teams/engineering/repos/${GH_ORG}/${repo}" -d '{"permission": "push"}'
  curl -XPUT -u "${GH_CRED}" "https://api.github.com/orgs/${GH_ORG}/teams/qe/repos/${GH_ORG}/${repo}" -d '{"permission": "push"}'
  git push --mirror "git@github.com:${GH_ORG}/${repo}.git"
  echo "${BRANCH_PROTECTION_BODY}" | curl -H 'Accept: application/vnd.github.luke-cage-preview+json' -XPUT -u "${GH_CRED}" "https://api.github.com/repos/${GH_ORG}/${repo}/branches/master/protection" -d @-

  echo "Completed repo creation for ${repo}..."
  cd "${ROOT}"
  rm -rf "${repo}.git"
done < "${1}"

Part 2: Updating Your Local Origin

I work on several of the repositories that I had migrated. So naturally updating remotes on local filesystem can be a daunting task. So I wrote another bash script to do that.

techgaun/tg-misc-scripts

#!/usr/bin/env bash

# Usage: ${0} [OLD_REMOTE] [NEW_REMOTE] [REMOTE_NAME]

OLD_REMOTE="${1:-bitbucket.org:paylease}"
NEW_REMOTE="${2:-github.com:gozego}"
REMOTE_NAME="${3:-origin}"

find "${PWD}" -type d -name '.git' | while read dir; do
  cd "${dir}/.."
  current_remote_url=$(git remote get-url "${REMOTE_NAME}")
  if grep "${OLD_REMOTE}" <<< "${current_remote_url}"; then
    new_remote_url=$(sed "s/${OLD_REMOTE}/${NEW_REMOTE}/" <<< "${current_remote_url}")
    echo "Changing ${current_remote_url} to ${new_remote_url}"
    git remote set-url "${REMOTE_NAME}" "${new_remote_url}"
  fi
done

Basically, you can run the above script placing it on the root of your all projects and execute it by passing 3 arguments; 3rd is optional if you name your remote origin (the default).

./mass-git-remote-update.sh 'bitbucket.org:<your_bb_org>' 'github.com:<your_gh_org>'

I know these scripts may not be usable right away due to use-cases specific to you and your team but this should give good basics on how such migrations can be totally automated.

The primary motivation for me to write the Slackbot was the need for switching through various services quite frequently. For example, I would need to find address for places near me or get current time in another timezone. Likewise, I would frequently perform whois using one of the online services and I would keep on searching for programming help on Stack Overflow. Another motivation was to remind us about the long open pull requests that needs to be reviewed by someone.

We had been using Elixir for a while and I wanted to play around with Elixir for writing bot. With quick search, I found Hedwig, a bot framework with support for various adapters and I was happy that I could just offload hard work to Hedwig and just write easy bits of the bot. They even have Slack adapter so I was able to quickly bootstrap and build the bot.

I named my bot after my lovely puppy Mustang who lives in Nepal and was together with me during April 2015 Earthquake.

Hedwig’s design on high-level is obvious and intuitive to how you would think about bot design and its highly inspired by hubot. The main building block of bots is responders and with Hedwig, you can use Hedwig.Responder to build your responders. All you do to write responder is use the macros like hear and respond to define patterns to listen and then write your logic on how the bot should respond for those patterns. For more details on how to use Hedwig and adapters and how to build responders, you can refer to the official documentation. With the Beam concurrency and scalability, its pretty easy to build bot with high concurrency and low footprints. I’ve not really exploited any of these features yet but that’s the area I should look into.

The other thing I had issue with Hedwig Slack adapter was to get the current state so that I could get list of channels and then map human-readable channel names to Slack channel identifiers. However, I was able to get around that easily by using erlang sys module as seen here. Reading through documentation for Hedwig, I did not find an easy way to do this but if there’s any, feel free to suggest.

I would be happy to get new responders (or ideas in the form of issues) from everyone. Also, it can be a good way to learn Elixir if you’re just starting with it. Below are the current feature highlights:

Scheduled checks:

• Github open pull requests watcher to notify long open PRs
• Standup reminder when its time to gather for standup
• Scheduled check of list of usernames/emails against haveibeenpwned.com
• Simple uptime monitoring with support for status, content and content type checks and ability to specify method, headers, timeout, etc.

Responders:

• google maps search
• haveibeenpwned.com search
• random quotes
• slap people on slack
• time on given timezone
• unix to iso8601 time conversion
• base64 encoding/decoding
• isitup check to see if a site is down or not
• insult to get random insults for person you mention
• httpcat to get http status code cats
• howdoi to get top result for your programming question fro stackoverflow
• funny random commitmsg
• clifu to get command line fu
• get whois result of domain names

My personal favorites are howdoi and insult and I love using insult whenever I can.

Getting ExMustang up and running is pretty easy. All you need to do is create a slackbot from HERE and copy the slackbot token. Then, you can follow the setup instructions available HERE.

Some screenshots of using ExMustang follow:

Mustang — A Slackbot in Elixir was originally published in Dev Stash on Medium, where people are continuing the conversation by highlighting and responding to this story.

My credential management was pretty bad so far as I was almost always doing the following:

• generate a random password on CLI
• save the random password in a text file

I know saving to text file was pretty stupid so I decided to be better about it as I keep on saying to others about these kind of stupid mistakes. I use gnupg for some other purposes too so I knew I could perfectly use it for this use-case as well.

Meet Saancho.

Saancho is a command-line password manager that uses gnupg to store credentials with AES256 encryption. All you have to remember is a single passphrase for your credential store and you should be all set for storing your credentials without having to worry about encryption. All thanks to gnupg.

Installation is super-simple. Just run the command below:

sudo wget "https://raw.githubusercontent.com/nepalihackers/saancho/master/saancho" -O /usr/bin/saancho && sudo chmod +x /usr/bin/saancho

When you save credential on saancho for the first time, you will also enter the passphrase which you will be using for all the future credential storage. The interface is very simple menu that I could come up with.

I’ve planning for few more features and I’ve created issues for those. If you feel like hacking on this, feel free to do so and send me pull requests. I hope to get some feedback on this tool.

Github Repo: https://github.com/nepalihackers/saancho

Sharing tickets with any heroku user

An organization member can abuse help system and share the ticket to any other heroku user without the orgnization owner knowing about the sharing context. The only requirement is the other user has to be registered user on heroku.

curl 'https://support-api.heroku.com/tickets/:ticket_id/shares' -X POST -H 'Accept: application/vnd.heroku+json; version=1' -H 'Authorization: (redacted)' --data '{"user_id":"<another_user>@example.com"}'

The request like above will successfully add any other heroku user.

Information leak on comments API calls

When you load tickets on heroku, the comments API is called which loads the comments but also includes actor payload as below:

{
    "id": "0596b09e-4c56-40f9-a13a-f43acd637951",
    "created_at": "2016-10-07T18:43:58Z",
    "updated_at": null,
    "ticket_id": 409207,
    "public": true,
    "body": "<snipped>",
    "actor_id": "3ab8a77d-e1d2-446e-905f-32d0a1961dd0",
    "actor": {
        "allow_tracking": true,
        "beta": false,
        "email": "*****@heroku.com",
        "id": "3ab8a77d-e1d2-446e-905f-32d0a1961dd0",
        "last_login": "2016-09-21T16:22:10Z",
        "name": "*****",
        "sms_number": "+1 ***-***-****",
        "two_factor_authentication": true,
        "verified": true,
        "created_at": "2016-03-21T22:47:42Z",
        "updated_at": "2016-09-23T07:24:37Z",
        "suspended_at": null,
        "default_organization": null,
        "delinquent_at": null,
        "herokai": true,
        "addon_provider": false
    },
    "meta_data": null,
    "sfid": null
}

As you can see above, the actor information is quite revealing about the user.

Information leak about organization via tickets API

The ticket API leaks certain information about an organization to whoever the ticket is shared or even the ticket creator who’s probably not supposed to see those information.

curl 'https://support-api.heroku.com/tickets/:ticket_id' -H 'authorization: Bearer <auth_token>'

{
    "id": *****,
    "score": 4350,
    "complete": true,
    "created_at": "2016-10-07T16:09:23Z",
    "last_public_change_at": "2016-10-07T19:09:08Z",
    "first_responded_at": "2016-10-07T18:43:58Z",
    "last_responded_at": "2016-10-07T18:43:58Z",
    "updated_at": "2016-10-07T19:09:08Z",
    "app_name": null,
    "app_id": null,
    "permission_granted": false,
    "state": "open",
    "priority": "normal",
    "subject": "********",
    "requester_id": "d5b5b77d-6fd4-4af4-a2cc-7eb8c1bf8272",
    "requester": {
        "id": "*****@users.heroku.com",
        "email": "*****@*****.com",
        "uuid": "d5b5b77d-6fd4-4af4-a2cc-7eb8c1bf8272",
        "full_name": "Samar Acharya",
        "deleted_at": null
    },
    "assignee_id": null,
    "assignee": null,
    "actor_id": "d5b5b77d-6fd4-4af4-a2cc-7eb8c1bf8272",
    "actor": {
        "id": "****@users.heroku.com",
        "email": "******@*****.com",
        "uuid": "d5b5b77d-6fd4-4af4-a2cc-7eb8c1bf8272",
        "full_name": "Samar Acharya",
        "deleted_at": null
    },
    "category": "account",
    "addon": null,
    "requires_survey": false,
    "comment": {
        "body": "****"
    },
    "last_comment": {
        "body": "****"
    },
    "views": [{
        "id": "fc928ccc-c182-4d95-b070-a3fcb57a8f27",
        "name": "Support",
        "slug": "support",
        "external_id": "",
        "ticket_count": 48,
        "addon": false,
        "meta_data": {
            "public_name": "Support",
            "review_category": "support",
            "notes": ""
        }
    }],
    "tags": [{
        "id": "fc928ccc-c182-4d95-b070-a3fcb57a8f27",
        "name": "Support",
        "slug": "support",
        "kind": "View",
        "external_id": "",
        "ticket_count": 48,
        "assigned_count": 3,
        "addon": false,
        "created_at": "2014-10-03T11:22:06.575+00:00",
        "updated_at": "2016-02-12T16:17:17.358+00:00",
        "surveyable": true,
        "meta_data": {
            "public_name": "Support",
            "review_category": "support",
            "notes": ""
        }
    }],
    "meta_data": {
        "source": "help",
        "query": "help redirect premium support",
        "requester_browser_tz_offset": "-5",
        "requester_browser_agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0",
        "support_level": {
            "csa": true,
            "sla": true,
            "assigned_csa": "****@heroku.com",
            "account_name": "*****"
        },
        "status": {
            "status": [{
                "system": "Apps",
                "status": "green"
            }, {
                "system": "Data",
                "status": "green"
            }, {
                "system": "Tools",
                "status": "green"
            }],
            "incidents": [],
            "scheduled": []
        },
        "requester": {
            "id": "*****@users.heroku.com",
            "email": "*****@******.com",
            "uuid": "d5b5b77d-6fd4-4af4-a2cc-7eb8c1bf8272",
            "full_name": "Samar Acharya",
            "deleted_at": null
        }
    },
    "spend": 3000.0,
    "premium_support": true,
    "surveyable": "true",
    "boosted": false,
    "origin": "Support App",
    "sfid": "5003A00000******"
}

Deleting ticket by anyone with access to ticket

If you are sharing tickets with other users, you should be aware that they can just delete your ticket. Note that this is a soft-delete so you can reach out to Heroku contacts and have them undo the soft-delete but still is a hassle.

curl 'https://support-api.heroku.com/tickets/:ticket_id' -H 'authorization: Bearer <another_user_token>' -XDELETE
...json output...

None of these issues were regarded as a potential security risk by Heroku engineer and I found out recently that the information leak issues I’ve mentioned above are fixed now. I was particularly disappointed by the way the Heroku support acted on this particular issue. I also later found out that help.heroku.com is in scope in their security bounty on bugcrowd. I do understand that its up to Heroku to decide if a particular report is a security bug or not but in this case, I am amazed that they did not accept it as a security issue (although could be considered low impact).

JWT or JSON Web Token is quickly becoming the standard of choice for secure API authentication and information exchange. In this post, I will explore how to build a sample Web application using JWT for authentication of clients.

Note that there are a variety of authentication stratgies, such as ueberauth, and you’ll have to pick the right model for your project. Since we offloaded user management to Auth0, we decided to take another approach which is not necessarily superior but is simple enough for our use case.

We’re using Elixir 1.3 and Phoenix 1.2.1 for this example. Lets create a new phoenix project.

mix phoenix.new jwt_phoenix --no-brunch --no-html

Lets add joken in the list of application dependencies and fetch our dependencies first.

defp deps do
  ...
  {:joken, "~> 1.1"}
  ...
end

Then run the mix deps.get command to fetch the dependencies. We’re going to use Auth0 so lets create an appropriate config block in config.exs.

config :jwt_phoenix, :auth0,
  app_baseurl: System.get_env("AUTH0_BASEURL"),
  app_id: System.get_env("AUTH0_APP_ID"),
  app_secret: "AUTH0_APP_SECRET"
    |> System.get_env
    |> Kernel.||("")
    |> Base.url_decode64
    |> elem(1)

Once we create an application in Auth0, we have three main configuration items.

• app_baseurl — The base URL or issuer of the JWT tokens
• app_id — the application id or the audience of the JWT tokens
• app_secret — the application secret, used to sign the JWT tokens

We’re using environment variables instead of hard-coding these configuration items in config.exs but you could also use something like prod.secret.exs to inject sensitive data to your deployments. Hard-coding your app_secret is a big security no-no as malicious users can use it to create valid tokens. The other important thing to note is that we have to decode the application secret using Base64.url_decode64/1 as Auth0 base64 encodes its secret.

Now that the configuration is setup, lets build a helper module that consists of functions for verifying and creating tokens on the server-side.

defmodule JwtPhoenix.JWTHelpers do
  import Joken, except: [verify: 1]

  @doc """
  use for future verification, eg. on socket connect
  """
  def verify(jwt) do
    verify
    |> with_json_module(Poison)
    |> with_compact_token(jwt)
    |> Joken.verify
  end

  @doc """
  use for verification via plug
  issuer should be our auth0 domain
  app_metadata must be present in id_token
  """
  def verify do
    %Joken.Token{}
    |> with_json_module(Poison)
    |> with_signer(hs256(config[:app_secret]))
    |> with_validation("aud", &(&1 == config[:app_id]))
    |> with_validation("exp", &(&1 > current_time))
    |> with_validation("iat", &(&1 <= current_time))
    |> with_validation("iss", &(&1 == config[:app_baseurl]))
  end

  @doc """
  Create token from client id and secret
  Used for unit tests
  """
  def create_bearer_token(auth_scopes, config_items \\ %{:signer => :app_secret, :aud => :app_id}) do
    %Joken.Token{claims: auth_scopes}
    |> with_json_module(Poison)
    |> with_signer(hs256(config[config_items[:signer]]))
    |> with_aud(config[config_items[:aud]])
    |> with_iat
    |> with_iss(config[:app_baseurl])
    |> with_exp(current_time + 86_400)
    |> sign
    |> get_compact
  end

  @doc """
  Return error message for `on_error`
  """
  def error(conn, _msg) do
    {conn, %{:errors => %{:detail => "unauthorized"}}}
  end

  defp config, do: Application.get_env(:jwt_phoenix, :auth0)
end

In the above code, we created the verify/0 function which uses the Joken.Plug. The Joken implementation will automatically grab the authorization header and pass it along similar to how our verify/1 function works. You can see that we import all the functions but verify/1 from Joken because we’re writing our own verify/1. The verify/1 we’ve written will be used for authorization connections via phoenix channels as the normal Plug flow does not work there. You can see we’ve passed our token through a few with_validation function so that we test our token against each of the expectations such as correct issued time, issuer, expiration time, audience and client secret.

We have also written a simple create_bearer_token/2 function which we can use for creating bearer token for our unit tests. As you might have noticed, we don’t need Auth0 to create tokens; anyone with the right client id and secret can create valid tokens.

We also have an error/2 so that we can send error message for invalid tokens. We will specify both verify/0 and error/2 as part of the plug when we add Joken.Plug in the :api pipeline. Lets configure our web/router.ex

pipeline :api do
  plug :accepts, ["json"]
  plug Joken.Plug,
    verify: &JwtPhoenix.JWTHelpers.verify/0,
    on_error: &JwtPhoenix.JWTHelpers.error/2
end

We’ve added the plug in the pipeline so now all API requests will need a valid JWT in the Authorization header. To see this in action, lets create a controller that will return a simple JSON response {success: true}.

# web/controllers/status_controller.ex
defmodule JwtPhoenix.StatusController do
  use JwtPhoenix.Web, :controller

  def index(conn, _params) do
    status = %{
      success: true
    }
    render(conn, "status.json", status: status)
  end
end

# web/views/status_view.ex
defmodule JwtPhoenix.StatusView do
  use JwtPhoenix.Web, :view

  def render("status.json", %{status: status}) do
    status
  end
end

# web/router.ex ; lets add status endpoint in router.ex
scope "/api", JwtPhoenix do
  pipe_through :api

  get "/status", StatusController, :index
end

We are ready to access our /api/status endpoint. For now, we will not dive into how we can do this on the frontend side but with Auth0 lock, its fairly easy to setup. Lets see a sample curl request to see this in action.

$ curl -D - http://localhost:4000/api/status
HTTP/1.1 401 Unauthorized
server: Cowboy
date: Mon, 22 Aug 2016 17:00:00 GMT
content-length: 36
cache-control: max-age=0, private, must-revalidate
x-request-id: fco562v0k8eleuhr0oeks0i1f2boigcm
content-type: application/json; charset=utf-8

{"errors":{"detail":"unauthorized"}}

Since we didn’t specify a valid token (same for unauthorized tokens), our API now responds with status 401. For authenticated requests, it responds with the appropriate response and http status code 200.

$ curl -D - http://localhost:4000/api/status -H "Authorization: Bearer <valid_jwt_token>"
HTTP/1.1 200 OK
server: Cowboy
date: Mon, 22 Aug 2016 17:02:55 GMT
content-length: 16
content-type: application/json; charset=utf-8
cache-control: max-age=0, private, must-revalidate
x-request-id: stcjk670bdih5n2phgpm3h5nkgmc2dor

{"success":true}

Now that we’ve got the basics working, we are going to add an authorization feature. While the work we’ve done so far is good enough to check for a valid token, we don’t have a way to limit particular endpoints to particular roles. For simplicity, we are going to add two functions to the router.ex but you could easily extract it to a separate Plug module.

# web/router.ex
@doc """
Function that will serve as Plug for verifying metadata
"""
def check_admin_metadata(conn, opts) do
  claims = Map.get(conn.assigns, :joken_claims)
  case Map.get(claims, "app_metadata") do
    %{"role" => "admin"} ->
      assign(conn, :admin, true)
    _ ->
      conn
      |> forbidden
  end
end

@doc """
send 403 to client
"""
def forbidden(conn) do
  msg = %{
    errors: %{
      details: "forbidden resource"
    }
  }
  conn
  |> put_resp_content_type("application/json")
  |> send_resp(403, Poison.encode!(msg))
  |> halt
end

# create a new pipeline for admin: web/router.ex
pipeline :api_admin do
  plug :check_admin_metadata
end

# add a new scope
scope "/api/admin", JwtPhoenix do
  pipe_through [:api, :api_admin]

  get "/", StatusController, :admin
end

We just wrote a simple function to check if the user has the admin role. Auth0 provides two fields, app_metadata and user_metadata. The former one is only editable by the application so it can be used for managing roles and access. The later is user editable and can be used for things like user preferences. The above example checks the claims has the right app_metadata otherwise it will send a 403 status code. We also put admin: true in conn.assigns so that we can do certain actions later. Finally, we created a new pipeline :api_admin which we will be used under the /api/admin scope. The beauty with Phoenix is that we can pipe_through a list of pipelines giving us flexibility on transforming the %Plug.Conn{} struct. Lets add a function in our StatusController to handle the /api/admin endpoint.

def admin(conn, _params) do
  status = %{
    success: true,
    role: "admin"
  }
  render(conn, "status.json", status: status)
end

This is a simplistic example as a real world scenario would be a lot more complex but it serves our purpose of demonstrating the feature. Let’s see how curl responds to two properly signed tokens, one with the admin role and other one without.

$ curl -D - http://localhost:4000/api/admin -H "Authorization: Bearer <valid_token_with_no_role>"
HTTP/1.1 403 Forbidden
server: Cowboy
date: Mon, 22 Aug 2016 17:40:01 GMT
content-length: 43
cache-control: max-age=0, private, must-revalidate
x-request-id: 7imahtg2gnpiodeo20q9ku1udtf3jvl4
content-type: application/json; charset=utf-8

{"errors":{"details":"forbidden resource"}}

$ curl -D - http://localhost:4000/api/admin -H "Authorization: Bearer <valid_token_with_admin_role>"
HTTP/1.1 200 OK
server: Cowboy
date: Mon, 22 Aug 2016 17:40:26 GMT
content-length: 31
content-type: application/json; charset=utf-8
cache-control: max-age=0, private, must-revalidate
x-request-id: fcemumhu2kakr8e083f2u1n65ntt173h

{"success":true,"role":"admin"}

The /api/status endpoint still works for tokens without admin role but the /api/admin sends 403 status code.

This concludes our JWT and Auth0 demonstration. Hopefully now you know how to setup your web app for JWT based authentication and authorization. Some parts of this article is Auth0 specific but the flow should generally work with any provider. There’s so much more we could cover such as JWT token invalidation using short lived expiration or refresh tokens or token blacklist, but we’ll leave thatfor another day!

The sample application we built is hosted on GitHub. You can check the Ueberauth organization for authentication systems such as Guardian and Ueberauth and various strategies supported by Ueberauth. Check out joken for all your JWT needs.

Using JWT and Auth0 for an Elixir/Phoenix web app was originally published in Brightergy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

I will use Phoenix Framework which is a popular web framework written in Elixir which leverages the Erlang VM but provides beautiful syntax and other cool features to work with. However, this can be easily adapted for another Elixir or Erlang framework. Of course, runtime statistics is just one of the various metrics you would want to measure. Over the past week, we have been able to gather and measure the various important metrics of our platform such as response times, response status codes, database metrics, load average, etc.

First, lets put ExErlstats in the list of dependencies for our application by editing mix.exs:

defp deps do
  {:ex_erlstats, "~> 0.1"}
end

Run mix deps.get and we are set to use ExErlstats which is just a simple wrapper to get Erlang VM statistics in Elixir.

Now, lets create a controller to handle the status check:

# web/controllers/status_check_controller.ex
defmodule MyApp.StatusCheckController do
 use MyApp.Web, :controller
end

Lets add couple of index functions with pattern matching and configure router.ex so that we can hit the status check endpoint.

def index(conn, %{"gecko" => "true", "memory_stats" => "true"}) do
 msg = ExErlstats.memory
 |> Stream.filter(fn {_k, v} ->
   valid_stat?(v)
 end)
 |> Enum.map(fn {k, v} ->
   %{
     "title": %{
     "text": atom_to_str(k)
     },
     "description": mem_normalize(v)
   }
 end)
 response(conn, msg)
end

def index(conn, %{"gecko" => "true", "sysinfo" => "true"}) do
 msg = ExErlstats.system_info
 |> Stream.filter(fn {_k, v} ->
   valid_stat?(v)
 end)
 |> Enum.map(fn {k, v} ->
   %{
     "title": %{
     "text": atom_to_str(k)
     },
     "description": "#{v}"
   }
 end)
 response(conn, msg)
end

def index(conn, %{"gecko" => "true", "erl_stats" => "true"}) do
 msg = ExErlstats.stats
 |> Stream.filter(fn {_k, v} ->
   valid_stat?(v)
 end)
 |> Enum.map(fn {k, v} ->
   %{
     "title": %{
     "text": atom_to_str(k)
     },
     "description": "#{v}"
   }
 end)
 response(conn, msg)
end

def index(conn, _params) do
 time_start = :os.system_time(:milli_seconds)
 time_end = :os.system_time(:milli_seconds)
 time_diff = "#{time_end — time_start}ms"
 success(conn, time_diff)
end

defp success(conn, time_diff) do
 msg = %{
   success: true,
   msg: "ok",
   time: time_diff
 }
 response(conn, msg)
end

defp response(conn, msg, status \\ :ok) do
 conn
 |> put_status(status)
 |> put_resp_content_type("application/json")
 |> render(MyApp.StatusCheckView, "index.json", msg: msg)
end

defp mem_normalize(v) when is_integer(v), do: _mem_normalize(v)
defp mem_normalize(v), do: v
defp _mem_normalize(v) when v < 1024, do: "#{v} bytes"
defp _mem_normalize(v) when v < 1048576, do: "#{Float.round(v / 1024, 2)} KB"
defp _mem_normalize(v) when v < 1073741824, do: "#{Float.round(v / 1048576, 2)} MB"
defp _mem_normalize(v), do: "#{Float.round(v / 1073741824, 2)} GB"
defp valid_stat?(v), do: is_bitstring(v) || is_integer(v) || is_float(v)
defp atom_to_str(v) when is_atom(v), do: v |> Atom.to_string |> String.capitalize |> String.replace("_", " ")
defp atom_to_str(v), do: v

We need to create a simple view:

# web/views/status_check_view.ex
defmodule MyApp.StatusCheckView do
  use MyApp.Web, :view
  def render("index.json", %{msg: msg}) do
    msg
  end
end

And the router.ex configuration:

# web/router.ex
scope "/", MyApp do
  get "/status", StatusCheckController, :index
end

What we did just now is setup a /status endpoint where we can pass a couple of parameters. I used params pattern matching to generate the format Geckoboard’s List widget expects.

GET /status?gecko=true&sysinfo=true
[
 {
  "title": {
   "text": "Otp release"
  },
  "description": "18"
 },
 {
  "title": {
   "text": "Port count"
  },
  "description": "47"
 },
 {
  "title": {
   "text": "Port limit"
  },
  "description": "65536"
 },
 {
  "title": {
   "text": "Process count"
  },
  "description": "381"
 },
 {
  "title": {
   "text": "Process limit"
  },
  "description": "262144"
 },
 {
  "title": {
   "text": "Schedulers"
  },
  "description": "8"
 },
 {
  "title": {
   "text": "Schedulers online"
  },
  "description": "8"
 },
 {
  "title": {
   "text": "Version"
  },
  "description": "7.3"
 }
]

I’m not going to display the other possible json results but the endpoints to hit would be:

GET /status?gecko=true&memory=true
GET /status?gecko=true&erl_stats=true

These all return a JSON response that Geckoboard’s List widget expects. You will also notice that the code example for status controller also has a default status check, which we use for a simple check that our API is up.

GET /status
{
 "time": "0ms",
 "success": true,
 "msg": "ok"
}

And, this is what’s displayed on Geckoboard:

While we use Geckoboard, you can easily adapt the above examples to send to the monitoring dashboard of your choice.

If you are looking to use Geckoboard with Elixir, check out ExGecko. ExGecko supports Geckoboard’s Dataset and Push APIs and also provides built-in adapters for feeding Papertrail, Runscope, Heroku server, db and db backup logs to Geckoboard.

Monitoring Erlang Runtime Statistics was originally published in Brightergy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

We recently started using InfluxDB as our time-series database for our IoT platform. InfluxDB allows us to store large amount of time-series data collected from various type of devices such as power meters and solar generation monitors. We reviewed couple of time series databases such as Prometheus, OpenTSDB and Graphite and finally settled with InfluxDB as it is best suited for use-case. While InfluxDB does not have many features and there are certain shortcomings with it, we see it as a promising young time-series database. We had tried InfluxCloud but we had certain issues such as lack of control and visibility of the InfluxDB cluster servers. Thus, we decided to run our InfluxDB in house and it turned out to be a straightforward process.

In this post, we will see how to install and configure a single node InfluxDB service using AWS with an optimal configuration. For our setup, we will use two volumes, one for the WAL (write-ahead logging) and one for the InfluxDB data. The general idea is to use a higher IOPS, lower size volume for the WAL and a larger sized, but slower IOPS volume for the data.

Pre-install

From your AWS console or AWS CLI, create a new security group called influxdb-sg and allow inbound connection to TCP port 8086 from the appropriate source. Launch an EC2 instance using the Amazon Linux AMI and your desired instance size with the security group we created earlier. Since we’re going to store the data and the WAL in separate volumes, you can either choose to add these volumes as part of your launch process or later on. Or you can just choose not to use volumes based on your needs.

Once the instance is up, create a SSH session and run the following commands:

yum update
mkdir -p /opt/influx/{wal,data,ssl}

WAL and Data Volumes Configuration

Make sure your device paths are correct. If you are not using additional volumes, you can skip this step.

mkfs.ext4 /dev/sdb
mkfs.ext4 /dev/sdc
mount /dev/sdb /opt/influx/wal/
mount /dev/sdc /opt/influx/data/

Installation

At the I wrote this, the latest stable version of InfluxDB was 0.13 which we are going to use.

wget https://dl.influxdata.com/influxdb/releases/influxdb-0.13.0.x86_64.rpm
yum localinstall influxdb-0.13.0.x86_64.rpm
chkconfig influxdb on

Configuration

SSL

Its recommended to use SSL with InfluxDB whenever you can. InfluxDB requires you to concatenate your private key and your certificate bundle in a single file.

cat privkey.pem mysite.crt mysite-ca-bundle.crt > bundle.pem
scp bundle.pem root@myhost:/tmp

Now, on the influx host, you can update the InfluxDB configuration. We will also update the meta, data and WAL configurations down below.

mv /tmp/bundle.pem /opt/influx/ssl/
chown -R influxdb:influxdb /opt/influx/
cp /etc/influxdb/influxdb.conf{,-bak}
sed -i s./var/lib/influxdb/meta./opt/influx/data/meta. /etc/influxdb/influxdb.conf
sed -i s./var/lib/influxdb/data./opt/influx/data/data. /etc/influxdb/influxdb.conf
sed -i s./var/lib/influxdb/wal./opt/influx/wal. /etc/influxdb/influxdb.conf
sed -i s,/etc/ssl/influxdb.pem,/opt/influx/ssl/bundle.pem, /etc/influxdb/influxdb.conf
sed -i "s/https-enabled = false/https-enabled = true/" /etc/influxdb/influxdb.conf

Authentication

Now that we have InfluxDB ready to be run, we want to enable some form of authentication. Since we’re going to use http based data in-and-out (which is the default mechanism in InfluxDB), we need to enable authentication. We first need to create users before we can enable authentication and hence, we need to connect to the InfluxDB instance with authentication disabled (authentication is turned off by default).

influx
> create user superadmin with password 'my_password' with all privileges
> create user nonadmin with password 'na_password'
> grant all on tsdb_stage to nonadmin
> grant READ on tsdb_prod to nonadmin
> grant WRITE on tsdb_dev to nonadmin

We just created two users, an admin user and a non-admin user. Now, we can enable the authentication and we should be ready to go with our newly setup InfluxDB instance.

/etc/init.d/influxdb stop
sed -i "s/auth-enabled = false/auth-enabled = true/" /etc/influxdb/influxdb.conf
/etc/init.d/influxdb start

The single node InfluxDB is ready now. If you are using Elixir, you can use the nice Instream package to interact with InfluxDB. If you have need help porting data from one InfluxDB instance to another, you may find influx-copy helpful.

Install and Configure InfluxDB on Amazon Linux was originally published in Brightergy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Erlang supports short running programs via the escript tool. All you have to do is write your application the way Erlang scripting support expects. Also, note that you only need to have Erlang installed for escript to run even though you write an escript in Elixir as Elixir is embedded as part of the escript. We’ll dive into writing a CLI tool soon but lets first dissect the executable created by escript when you run a mix escript.build.

head -n3 influx_copy 
#! /usr/bin/env escript
%% 
%%! -escript main influx_copy_escript

As you can see above, the executable created is simple and has a shebang line just like bash or python scripts. The second line consists of an optional directive to the Emacs editor which causes Emacs to enter the major mode for editing Erlang source files. The escripts built via Elixir do not contain the directive as seen above. The third line (or second if the second directive line is not present) specifies any arguments to be passed to the emulator. In the output above, we are overriding the default behavior of escript which is to invoke main/1 function in the module with the same name as the basename of the escript. Don’t worry! This is all handled by Elixir so all you have to specify is your main_module in your mix.exs file.

After the headers, the rest of the escript can either contain the plain Erlang code or precompiled beam code which is how it works with Elixir. If you look at the output executable, it will consist of all the beam code in a single executable. And, since all the necessary dependencies are compiled and included, we don’t explicitly need Elixir to be installed for escripts written in Elixir.

Now, lets quickly dive into the process of writing escript in Elixir. You can check out the influx-copy commits which I’ve tried as much to code in small incremental steps.

mix new influx-copy — module InfluxCopy — app influx_copy
cd influx-copy
mix test

The influx-copy tool is a simple generic tool that reads from one InfluxDB source and writes to another destination with ability to select fields and transform values of particular tags. The use case was that we had to copy some data from staging to production for one of our real-time monitors but their tag values were different in different environments.

The Erlang escript expects the main module to have a main/1 function which is what is invoked when the escript is run.

Update your mix.exs file to contain the following information

def project do
  [app: :influx_copy,
   version: “0.1.0”,
   elixir: “~> 1.3”,
   escript: escript,
   build_embedded: Mix.env == :prod,
   start_permanent: Mix.env == :prod,
   deps: deps()]
end
def escript, do: [main_module: InfluxCopy]

As you can see above, whatever main_module you specify is supposed to have the main/1 function defined.

defmodule InfluxCopy do
  def main(args \\ []) do
    {opts, _, _} = OptionParser.parse(args,
      switches: [start: :integer, end: :integer, src: :string, dest: :string, update_tags: :string],
      aliases: [s: :start, e: :end, S: :src, d: :dest, u: :update_tags]
      )
    IO.puts “Not implemented yet”
  end
end

Now, your simple CLI app is ready to be compiled and used although it does not do anything.

$ mix escript.build
Generated influx_copy app
Generated escript influx_copy with MIX_ENV=dev

$ ./influx_copy
Not implemented yet

Elixir has a nice module called OptionParser that provides helpful functions to parse command line arguments which is quite useful while writing command line applications in Elixir. The influx-copy script does not yet include a helpful help message but it should be fairly simple to put.

I’d imagine adding an extra switch help and then based on truthiness of this argument, you can show help message instead.

If you are interested in the script I wrote and how I proceeded further, you can check out influx-copy.

Writing a CLI app in Elixir was originally published in Brightergy Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.