What are Metadata?

Metadata are data about data

for example lets say we have a hello.txt file which have 1 line of text “Hello World”

In this case “Hello World” are data

Metadata if this data would be:

• file size
• file format
• file created at
• filename
• file location
• created by

Other good example that i heard once:

Telephone connection

Data is the conversation itself, meaning the phone call content

Metadata:

• when was the connection
• how long was the connection
• between which numbers
• in which country or between which countries

What do you think ? Let me know in comments below to help me improve this article

Thanks for reading!

To answer what is Cloud Native I will need to first explain what does a Native Application mean

Native App is a term known from mobile applications

In short it means a application that was created for a dedicated OS/Phone where it can use some specific features of a particular device

That kind of app will not be executed in a proxy like a web browser but in OS directly

Example of Native Apps

• desktop application (native runtime would be the PC OS)
• mentioned already mobile native app (native runtime would be the phone OS)
• video game running in PS4 console (native runtime would be the gaming console OS)

Ok but how does that relates to the cloud and cloud native which is the subject of this post?

Well a cloud native app will be the app which native runtime environment is the cloud

what does that mean?

Let’s say we decided to build a product that will be designed as a Cloud Native App. Then we will need to build the app in a specific way such that we will know it will use cloud features and take advantage of doing so

What kind of features we can say are native to the cloud?

• Serverless
• Autoscaling
• Phenix infrastructure
• Limitless access to resources like memory
• Easiness of creating new microservices
• Deployments managed with CI/CD
• frequent releases
• A container-based infrastructure with containers orchestrator (for example Kuberrnetes)

I think a good example is Serverless where you can’t build a Serverless solution without using the Cloud like AWS or Azure

What is a difference between a Cloud Native App and a application that just runs in the Cloud

for example a web app like Wordpress blog that was migrated from some hosting to AWS cloud and now runs in a EC2 machine is not a Cloud Native App although it runs in AWS cloud

Why?

because it is not taking advantage of any features native to the cloud

Cloud Native applications are designed to exist in the cloud and to take advantage of the cloud’s benefits

Cloud-native apps are the ones that are originally written for cloud frameworks instead of being ported over to the Cloud as an afterthought

What do you think ? Let me know in comments below to help me improve this article

Thanks for reading!

Disclaimer: The code examples are in Scala but they will be simple enough and without any advanced features such that anyone can understand them. I am planing to run this series in multiple languages because the problems demystified here are language agnostic

Disclaimer: The code examples are in Scala but they will be simple enough and without any advanced features such that anyone can understand them. I am planing to run this series in multiple languages because the problems demystified here are language agnostic

Context (you can skip that section and go to Code example directly, then come back here):

Lets say we want to create a simple Graph that will contain unique String values, like unique names

Each unique name will represent a person

Lets assume that this graph will hold only one type of relation:

person ---(knows)---> person

Lets make it even simpler, our Graph will be one directional:

if A knows B it does not mean that B knows A

So this graph will not be generic, it will be very limited

In what scenarios in a real world we would use such a simplified graph?

We may need that graph to perform some local operations to solve a very specific problem within a function scope that may not need all the features of a sophisticated Graph

But i think this is a very good example to understand Graph and Trees by starting with such a limited and simple examples

Lets have a algo that will answer the following questions

• does person A knows B
• can person A meet B which means can we find a one directional path between A and B

A person with assigned unique name can get to other only via people that they know already

So it sounds almost like a game with made up rules: Graph represent people in the room, each person have unique name assigned, in a game round each named person knows others and they can get to each other only directly or via other friend

Sounds fun!!

Code Example:

One of the simplest way to represent it is as Map of String to list of Strings:

Map[String, List[String]]

Lets create some example graph with the data described in context section

val PeopleInTheRoom: Map[String, List[String]] = Map(
  "Jane" -> List("Bob", "Mark", "Rajesh", "Samanta", "Hussein"),
  "Mark" -> List("Tom"),
  "Bob" -> List("Lisa", "Jenny"),
  "Tom" -> Nil,
  "Lisa" -> Nil,
  "Jenny" -> Nil
)

Now lets ask questions:

• does Jane knows Mark: yes
• does Jane knows Jenny: no
• can Jane meet Jenny: yes
• does Bob knows Jane: no
• can Bob meet Jane: no

How to write code to answer those questions will be in the following episode of Graph algorithms demystified series

Side note: the same way we can model a binary tree or b-tree

val binaryTree: Map[Int, List[Int]] = Map(
  5 -> List(4, 2),
  4 -> Nil,
  2 -> List(0, 3),
  4 -> List(-1)
)

We can assume that each map entry will be a list with max size 2, 1st element of list is left node 2nd element of the list is right node

I will describe it in more detail in another following episodes of Graph algorithms demystified

What do you think ? Let me know in comments below to help me improve this article

Thanks for reading!

How to groupBy a flat dataset into a case class that will have for example a String field and list field, where that list will contain some objects

case class Example(id: String, tags: List[SomeOtherClass])

lets imagine data sets of Crypto Currency Data

Investors Portfolios and Price Reports

Basic Models

case class CryptoToken(ticker: String, count: Double)

case class CryptoInvestor(investorId: String, countryCode: String, tokens: List[CryptoToken])

case class CryptoTokenPriceReport(ticker: String, priceInUSD: Double)

Investors portfolios

val investorsPortfolios: Dataset[CryptoInvestor] = List(
  CryptoInvestor("kyc-investor1", "USA", List(
    CryptoToken("btc", 0.5),
    CryptoToken("eth", 5)
  )),
  CryptoInvestor("kyc-investor2", "UK", List(
    CryptoToken("ltc", 1.5),
    CryptoToken("eth", 3)
  ))
).toDS()

// result of investorsPortfolios.show(truncate = false)
+-------------+-----------+------------------------+
|investorId   |countryCode|tokens                  |
+-------------+-----------+------------------------+
|kyc-investor1|USA        |[[btc, 0.5], [eth, 5.0]]|
|kyc-investor2|UK         |[[ltc, 1.5], [eth, 3.0]]|
+-------------+-----------+------------------------+

current price reports

val tokensPriceReport: Dataset[CryptoTokenPriceReport] = List(
  CryptoTokenPriceReport("btc", 8000.1),
  CryptoTokenPriceReport("eth", 123.0),
  CryptoTokenPriceReport("ltc", 51.2)
).toDS()

// result of tokensPriceReport.show(truncate = false)
+------+----------+
|ticker|priceInUSD|
+------+----------+
|btc   |8000.1    |
|eth   |123.0     |
|ltc   |51.2      |
+------+----------+

Now how to perform a join of those datasets such that we will get a result in a format:

case class CryptoTokenWithCurrentValue(ticker: String, count: Double, priceInUSD: Double)

case class CryptoInvestorsReport(investorId: String, countryCode: String, tokensWithCurrentPricing: List[CryptoTokenWithCurrentValue])

To make join operation possible we will need to flatten a bit the CryptoInvestor entry such that it will be like a SQL Table row

case class CryptoInvestorTmpRow(investorId: String, countryCode: String, ticker: String, count: Double)

val investorsAndPriceReportJoin = investorsPortfolios.flatMap { investor =>
  import investor._
  tokens.map { token =>
    import token._
    CryptoInvestorTmpRow(investorId, countryCode, ticker, count)
  }
}.as[CryptoInvestorTmpRow].join(tokensPriceReport, "ticker")

After the join a new dataset will look like following

+------+-------------+-----------+-----+----------+
|ticker|   investorId|countryCode|count|priceInUSD|
+------+-------------+-----------+-----+----------+
|   btc|kyc-investor1|        USA|  0.5|    8000.1|
|   eth|kyc-investor1|        USA|  5.0|     123.0|
|   ltc|kyc-investor2|         UK|  1.5|      51.2|
|   eth|kyc-investor2|         UK|  3.0|     123.0|
+------+-------------+-----------+-----+----------+

Now the main subject of this Post: how to perform the groupBy ?

we will use here collect_list and struct functions from org.apache.spark.sql.functions package

import org.apache.spark.sql.functions.{collect_list, struct}

investorsAndPriceReportJoin.groupBy("investorId", "countryCode").agg(
  collect_list(struct("ticker", "count", "priceInUSD")) alias "tokensWithCurrentPricing"
).as[CryptoInvestorsReport]

the result after groupBy will look like the following

+-------------+-----------+---------------------------------------+
|investorId   |countryCode|tokensWithCurrentPricing               |
+-------------+-----------+---------------------------------------+
|kyc-investor1|USA        |[[btc, 0.5, 8000.1], [eth, 5.0, 123.0]]|
|kyc-investor2|UK         |[[ltc, 1.5, 51.2], [eth, 3.0, 123.0]]  |
+-------------+-----------+---------------------------------------+

Checkout the full code on GitHub:

mkuzdowicz/spark-examples-for-blogs

What do you think ? Let me know in comments below to help me improve this article

Thanks for reading!

Do you need to be good at Math to be a successful Software Engineer ?

Before i will try to answer that question i will ask another question

Why we are even asking that question at the first place ?

The main problem is not if you are good at Math or not, but the problem is with our common believes that came from Education System where we are sometimes divided into 2 groups (A: good at Math, B: not good at Math) and then making some assumptions if you are in group A you can do that things and if you are in group B you can do other things

I think we should question that and think for our selfs!

To think for yourself you must question authority and learn how to put yourself in a state of vulnerable open-mindedness, chaotic, confused vulnerability to inform yourself

TIMOTHY LEARY, How to Operate Your Brain

Firstly just remember that you will need to learn new skills your whole life Especially if you are in Tech you have to learn all the time where new approaches on how to build Software emerge continually. I think that a large percentage of what a Software Engineer actually do is learning, at lest this in my case

Ok going back to the subject

Answer:

No, yo don’t need to be good at Math to be a successful Software Engineer!

Everything depends on how motivated you are to became a Software Engineer and that is all !!! Not some believes that stayed with you because you heard them about yourself in the past

The fact that you haven’t been proficient in such field as Math or whatever else during your School time doesn’t matter today because you can work on improving it. Why not !?

There are some domains of Software Engineering were knowing more advanced Math will be required like Game Development or Machine Learning

For majority of Software Jobs basic Math is good enough

But, if you will like to do Software that will require being good at Math you can just learn that! We live in a online world now where there is plenty of knowledge and tutorials available in the internet probably in a much better quality then classic school materials

I decided to write about because that was one of the questions that i asked myself couple years ago, because i rebranded myself form Musician to Software Engineer and i haven’t though about myself as someone that is good at Math. That assumption was not objective it was based on my experience from School

And there you go! I managed to become a successful Software Engineer. I even had the opportunity to work on Machine Learning projects and if i had some knowledge gaps i was working to fill them by using materials available over the internet

Post scriptum

if you would like to fill some gaps in your Math knowledge, that is a awsome youtube channel that i was using:

Eddie Woo

What do you think ? Let me know in comments below to help me improve this article

Thanks for reading!

How to create a AWS CDK template for API gateway that will have a Resource Policy

I needed to implement something like that recently and i was struggling to find a simple example on internet, so i decided to share my solution

API Gateway Resource Policy can be very useful if you want to improve security of Api Gateway that endpoint type is Edge

More about AWS Api Gateway Endpoint types:

API endpoint types for REST APIs in API Gateway

AWS API gateway with endpoint type is Edge is visible from public internet by default. What i wanted to achieve is to be able to whitelist ip ranges for my Api Gateway such that only chosen ones can hit my API

You can configure that by adding Api Gateway Resource Policy

Anyway, here is the example AWS CDK code in TypeScript:

aws-cdk version used here: 1.27.0

What do you think ? Let me know in comments below to help me improve this article

Thanks for reading!

Yes it means Software as a Service. You may know some products that are like SaaS For example Twilio, PayPall, Zuora, Stripe and many others

The platforms that i mentioned above are services that we can integrate our Applications

Before we explain what is a Webhook it is important to know the above

Webhook or Callout is a user defined HTTP callback. What does that mean ?

Short answer:

If some Event take place in a SaaS platform that your App is using, then that Event may trigger a http call to your App, which will be a call to endpoint that you created on your App

long answer (with example):

Imagine that you have an online ebooks store and customers can buy your ebooks by PayPall

lets imagine a Synchronous flow without using Webhook

This solution have some problems:

• How will your App know how long to wait for a payment confirmation? Can we guarantee a fixed max time ?
• What if the user refresh the page during that flow or lose the connection ?
• What if the payment confirmation takes a long time and your App will timeout during that flow? That may result with: you’re client not getting the ebook but being charged anyway. Isn’t that Scary !?

This is were Webhooks and Callouts comes into play!

Thy will help you to implement the flow in a more Asynchronous manner

lets imagine a Asynchronous flow:

Here the flow ends on your app and will continue on the SaaS platform where the Payment Confirmed Event will trigger a Webhook

TaDam!

That flow is much more reliable and solves one of the scary scenarios: when client paid for the product but due to latency the synchronous flow may decide that the payment was unsuccessful

Payment is one of the examples, there are many more examples where our App is using some external SaaS platform and we would like to consume some Events form those platforms in our Apps

What do you think ? Let me know in comments below to help me improve this article

Thanks for reading!

Is writing code like Art?

hmm…

Can we say that being a Programmer, Software Developer or Software Engineer may be like being an Artist (Musician or Painter) ?

Does Code creation is like an act of Art creation ?

I don’t think so

It is a very close subject to me personally as i have been a Musician before i became a Software Developer/ Software Engineer

Well OK there are some similarities, but i don’t think we can say that writing code is like creating Art or code that works behind the scenes can bee seen as work of art

I don’t think that reading code can give you the same experience as listening to music or watching paintings

Why?

• A piece of art like painting or music may become more valuable over time. Software usually gets worst over time, it usually comes to a point that it will need to be replaced with a new code
• A work of art need to be finished at some point of time and usually the artists is not releasing it until he feels like it is nearly perfect
• I think we can say that about a lot of master works of Beethoven or Picasso that they are perfect and changing anything with them will be just change but definitely not improvement
• Whit software is different, there will be always something that can be improved and over time a new version of language or framework will be released and it is very likely that a change will be a improvement in that System
• Software development is done iteratively not until a developer feels that it is nearly perfect and he will not change it afterwards

So i don’t think that writing Code can be same as Composing Musing because the end result and lifecycle of the “product” is very much different

But

On the other hand there is one similarity. When a musician creates music then she\he is in specific flow like a creational flow. I think a lot of programmers know what is a flow or how it feels like to be in it

Passionate Artist and a passionate Developer can feel the same type of inspirational flow while creating something

Both Artists and Developers CREATES

Checkout my music channels:

• Martin Kuzdowicz
• Freeway Quintet

What do you think ? Let me know in comments below to help me improve this article

Thanks for reading!