Cookie*- Maintain the active state of the browser by storing a bit of data between the **server and the client** with every request.*

Session*- Record the **server and client session state**.*

Token- A random string generated by the server*.*

This article is tailored for those who are looking for a basic understanding of what sessions, tokens, and cookies are, how they work, and why they exist.

When you log into a website, you declare who you are (identification). Your identity is verified (authentication), and you are granted the necessary permissions (authorization). Many solutions have been proposed in the past, and the list keeps growing.

Authentication

Authentication is the act of verifying user credentials in terms of either correctness or time.

• Correctness: The user credentials are verified based on existing details. At the sign-in request, an authentication token is assigned to the user. It will be used to authorize the user and authenticate subsequent interactions with the application.

• Time: The authentication token assigned to the user is only valid for a specific period of time. If the token becomes invalid, the user needs to be re-authenticated before access can be granted.

The term “authentication token” is used here to describe any form of authentication credential that might be implemented by the application, not explicitly an access token.

Why use authentication

There are two reasons why authentication is necessary to allow full access to an application:

• Authentication establishes the identity of the user and verifies that the user is who (or what) it says it is. Authentication protects your resources by denying access to unauthenticated users.

• Authentication gives each user a distinct identity, protecting your data and theirs. Applications that require users to create an account give each user a unique profile, which is what determines the data shown to the user.

For example, PayPal or any payment portal requires users to sign in before displaying their account balance and transactions.

Generally, the process works like this:

• You sign in.

• The server verifies your sign-in details and assigns you an authentication token.

• The authentication token is used to make a request to your homepage that displays your unique dashboard.

Both session cookies and access tokens allow users to make requests to the server without needing to re-authenticate at each request.

Sessions

When thinking of building out a website, a session is all interactions made by a user within a timeframe. This time could be set to be a certain period before the session expires, or even until the user closes out the webpage. A session could include a variety of activities, such as searches, scrolling, clicking through links, or even filling out web forms. It can be anything if it represents the time a visitor spends on the website. Any activity on that website is recorded through the session ID.

A session ID is used to store the data server-side. However, session IDs are also used for users’ privacy, as the ID can be passed instead of login or other sensitive information. By gathering information, data on how the website is used can be viewed. This information aids in creating better user experiences, which may even be personalised and tailored to the users. That session ID is still tied to a timeframe, however, so at some specified point, it will expire.

When we think about a session expiring, this means that the user has exceeded the duration that the developer-defined in their code. Sessions can be brief periods, such as how a bank will timeout if you are inactive for more than a few minutes, or they could be set as a longer period, such as an entire day. Inside the session, information from the first arrival until the session ends is recorded. Because a session refers to a certain duration of time, a single user can have multiple sessions within a short period. For example, if we think of the banking example, after a period of inactivity the page times out and logs you out. But if it does log you out, you could log back in almost right away, which would create a new session. Another example is that a computer can open multiple tabs. This means you could be looking at a series of different websites, all of which are considered sessions for the different websites. But what if you decide to open a page on the same website in a new tab or new window? Would that generate a new session ID? Not exactly. That’s where cookies come in.

Cookies

If information about your session is going to be stored, something must temporarily retain that data. A cookie is a small file that stores pieces of data gathered in a session. For example, when segments, addresses, or passwords are automatically filled in, it is because there is a cookie holding that information. It is also what can be gathered from a session to let the website see where you clicked or what you did during your session. Cookies allow you to stay logged in even if you exit the page.

As you’ve likely seen a thousand times when visiting a website, you can choose to “allow cookies” for a website or to ignore them when prompted. This means you can choose to not allow those cookies. But that does not mean something will not collect data. If you check your browser, it collects cookies as well as makes a cache of your visited websites. You can choose to clear those, but you may find yourself having to fill out your usernames that are otherwise usually saved. The cookies found on websites are first-party cookies, which are used to track just your session on the website to tailor/personalise your visit.

If your Facebook ads are looking suspiciously close to your Google searches, you may have found third-party ads. Third-party ads are typically stored inside cookies, and sometimes they can be enabled without even clicking on the ad. These are the types of cookies that draw privacy concerns. They can track not only the activity on the website you got them from, but they can track your browsing history in general.

One even more difficult third-party cookie is a zombie cookie. They are more difficult because they are often invisible, and even after being deleted, they can reappear.

Stateful cookies

Stateful cookies contain a pointer to a database record that stores the session information.

Pros:

• Have no limitations on the amount of session information stored.

• Can easily clear a user’s session -just remove the record from the database.

Cons:

• Requires a database to store the session data (but most web applications already have this).

• Increases latency because you have to make database calls to read the session (and sometimes write it) for each HTTP request a user makes.

Stateless cookies

Stateless cookies are self-contained; they include all session information that you need (for authenticated users, the user ID) and reside on the client. To prevent external tampering, stateless cookies should be encrypted (or at least signed).

Pros:

• Reduces latency because you don’t have to call a database.

• They require little storage

• They reduce manipulation by client-side JavaScript

• Cookies use the same session across subdomains

Cons:

• Makes it difficult to revoke a session, because there is no record in a database you can delete, you’ll need to find other methods to forcefully clear a session.

• If using multiple web servers, must make sure all servers have the key to encrypt/decrypt or sign the cookie.

• Not good for API authentication

• Cross-site request forgery attacks (XSRF or CSRF)

Cookies aren’t the only way to store session IDs; other options include URLs and form fields. Cookies are more secure than those two, but how secure are cookies?

• Cookies are only secure in an HTTPS connection. Enforcing the Secure flag ensures that cookies are only sent via an encrypted HTTPS connection. The use of HTTPS prevents disclosure of session ID in person-in-the-middle (MITM) attacks.

While cookies can be made secure by setting the appropriate attributes and following best practices, they can also be made insecure by neglecting these steps.

Because of these other types of cookies and the chance for security breaches, many people opt out of all cookies. Privacy can be more important than the ease of login. But what if you’re designing a website and all this cookie talk has you nervous? What’s the solution to have “sessions” without the need for “cookies”? That’s where tokens come in.

Tokens

Tokens are a self-contained and compact way to securely transfer sensitive information between two parties. The composition of a token includes the header, which defines the type and algorithm used, the payload, which carries the metadata from the user much like a cookie would gather, and a signature, which is used to verify the identity of the sender. In this transmission type, the algorithm is used to secure the information by encoding it. It also does not require the use of cookies, because a token can be stored in Web Storage instead. This is either local storage or session storage. If a token is tampered with, it can be either revoked or blacklisted, which furthers its security.

In a session, a server must store the state, for example, if it is still active and ongoing. With tokens, the state is not stored. It is self-contained and therefore contains all the required information for authentication.

An example of a popular token system is JSON Web Tokens (JWT). A JSON Web Token is encoded using Base64, but this does not mean they are encrypted. The message is available for viewing, but only if they have the secret. There is a secret to decode the message, and without the secret or with the incorrect secret the message would be useless.

Process:

1. The server generates a token value by using a hashing algorithm and then returns the token value to the client when the user logs in successfully for the first time.

2. After the client gets the token value, it stores it locally

3. When the client requests again, it attaches the token value to the request and sends it to the server

4. After the server receives the client’s request, it will extract the token value and compare it with the token value stored locally on the server

5. Token value from client = Token value from the server-The user is logged in successfully

6. Token value =/ token value from the server — The original login information has expired, and the user is required to log in again

7. No such token value — the login is not successful

Tokens Are Signed, Not Encrypted

A JSON Web Token is comprised of three parts: the header, payload, and signature. The format of a JWT is header.payload.signature.

• JWTs are cryptographically signed and base64-encoded. They’re only secure when they aren’t exposed, so they should be treated like passwords.

• A JWT can be viewed but not manipulated on the client side. You can take your token to jwt.io, choose the algorithm you used to sign, and see the data. You just can’t tamper with it because it’s issued on the server.

• The lifespan of a JWT should be kept short to limit the risk caused by a leaked token.

{
  "sub": "67554887",
  "name": "Alpha",
  "admin": true
}

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwi
bmFtZSI6IkFkbyBLdWtpYyIsImFkbWluIjp0cnVlLCJpYXQiOjE0NjQyOTc4ODV9.
Y47kJvnHzU9qeJIN48_bVna6O0EDFiMiQ9LpNVDFymM

What is SSO?

Single sign-on (SSO) is a centralized session and user authentication service in which one set of login credentials can be used to access multiple applications. Its beauty is in its simplicity; the service authenticates you on one designated platform, enabling you to then use a variety of services without having to log in and out each time.

In the most common arrangement, the identity provider and service provider establishes a trust relationship by exchanging digital certificates and metadata and communicating with one another via open standards such as Security Assertion Markup Language (SAML), OAuth, or OpenID.

Single sign-on example

Imagine you’re the user in an environment with a single sign-on and you’re trying to get access to some resource on a server. The sequence of events for how SSO works go like this:

1. You attempt to access the service provider- again, this generally is an application or website you want to access.

2. As part of a request to authenticate the user, the service provider sends a token that contains some information about you, like your email address, to the identity provider, a role played by your SSO system.

3. The identity provider first checks to see whether you’ve already been authenticated, in which case it will grant you access to the service provider application and skip to step 5.

4. If you haven’t logged in, you’ll be prompted to do so by providing whatever credentials the identity provider requests.

5. Once these credentials have been validated, the identity provider will send a token back to the service provider, confirming that it’s authenticated with you.

6. This token is passed through your browser to the service provider.

7. Once received, the token is validated according to the trust relationship that was set up between the service provider and the identity provider during the initial configuration.

8. The user is granted access to the service provider.

Some common SSO protocols include:

• Security access markup language (SAML): SAML is an XML-based open standard for transmitting user information.

• Open Authorization (OAuth): OAuth is an open standard to transmit user information without revealing the password.

• Kerberos: Kerberos issues a ticket-granting ticket (TGT) for every valid user authentication. Both the user and server identify each other with tickets issued by the TGT. This makes it the ideal standard for untrusted networks.

• OpenID Connect: OpenID Connect is built on top of OAuth 2.0. It is fast gaining traction for being easier to implement than SAML. Since it works with restful application programming interfaces (APIs), it is an ideal choice for mobile applications.

• Smart card authentication: This involves smart cards loaded with cryptographic keys. When plugged into a computer, pre-loaded software interacts with it and authenticates the user. It is expensive and is usually used in addition to other modes of authentication.

OAuth

OAuth is an open-standard authorization protocol or framework that describes how unrelated servers and services can safely allow authenticated access to their assets without actually sharing the initial, related, single logon credential. In authentication parlance, this is known as secure, third-party, user-agent, delegated authorization.

It is also helpful to remember that OAuth is about authorization in particular and not directly about authentication. Authentication is the process of a user/subject proving its ownership of a presented identity, by providing a password or some other uniquely owned or presented factor. Authorization is the process of letting a subject access resources after successful authentication, oftentimes somewhere else. Many people think that OAuth stands for open authentication, but it’s more helpful to understand it by thinking about it as open Authorization.

Additionally, OAuth 2.0 is a framework, not a protocol (like version 1.0).

How OAuth works

Let’s assume a user has already signed into one website or service (OAuth only works using HTTPS). The user then initiates a feature/transaction that needs to access another unrelated site or service. The following happens (greatly simplified):

1. The first website connects to the second website on behalf of the user, using OAuth, providing the user’s verified identity.

2. The second site generates a one-time token and a one-time secret unique to the transaction and parties involved.

3. The first site gives this token and secret to the initiating user’s client software.

4. The client’s software presents the request token and secret to their authorization provider (which may or may not be the second site).

5. If not already authenticated to the authorization provider, the client may be asked to authenticate. After authentication, the client is asked to approve the authorization transaction to the second website.

6. The user approves (or their software silently approves) a particular transaction type at the first website.

7. The user is given an approved access token (notice it’s no longer a request token).

8. The user gives the approved access token to the first website.

9. The first website gives the access token to the second website as proof of authentication on behalf of the user.

10. The second website lets the first website access its site on behalf of the user.

11. The user sees a completed transaction occurring.

Conclusion

As we learned, sessions are durations of time that a user spends on a website and includes everything that they do in that timeframe such as clicking on different pages or filling out a form. Sessions give you the ability to track what the user is doing in their time on the website. You collect that information to personalize the user’s experience, or even to learn more about what the typical user is doing on your website and what may or may not be getting clicks. However, a session only tracks the user instead of storing the data. For this, you need cookies. Cookies are small files that hold all the information gathered about the user’s experiences during a session. They are what store the data to allow it to be used later. Cookies can be enabled automatically, but many websites choose to prompt the user to allow for cookies. However, browser and third-party cookies also exist which are not in control of the website and can be more harmful in terms of user privacy.

Instead of a session, tokens can be generated as a secure way to track user information and activities. Tokens can also use cookies, but they can avoid cookies with web storage. One way to store the data is with JSON Web Tokens (JWTs). These tokens are stored in a JSON file and are encoded using Base64, which means a secret is required to decode the message. Without that secret, the message is meaningless. It is an alternative to the session/cookie route, and tokens can even be signed by third parties.

ACID is an acronym of the following words:

1. Atomicity

2. Consistency

3. Isolation

4. Durability.

If we have a certain operation in a database that satisfies these properties, we can call this operation “transaction”.

Ok, that’s nice, but these words are just words without an explanation. Let’s dig deeper into each of them!

Transactions

A quick note about transactions. A transaction is a collection of instructions. All transactions must follow ACID properties in order to guarantee database validity. I’ll use banking transactions between customers below to illustrate the 4 properties.

Atomicity

Atomicity ensures that each transaction is treated as a single unit that follows the “all or nothing” rule; a transaction either happens completely or not at all. If any part of the transaction fails, the entire transaction will fail.

For example, customer A wants to withdraw $100 from their account and then deposit it into customer B’s account. If any of the instructions fail (insufficient funds or server crashes, etc.), the transaction will fail and any changes will be rolled back.

Consistency

Consistency guarantees that a transaction must be valid before it gets written to the database. A transaction is valid when it follows all of the defined rules, including constraints, cascades, triggers, and any combination of these. Inconsistent transactions will result in the database being rolled back to a previous state that complies with the given database rules.

If customer A wants to withdraw $100 from their account but only has $50 in their account, consistency prevents them from withdrawing the funds and the transaction will be aborted.

Isolation

As transactions are typically executed at the same time (i.e. multiple transactions that read and write to a table at the same time), isolation sees that each transaction acts as an individual transaction and would receive the same state if the transactions were executed sequentially. Isolation is important for concurrency control.

If customer C has an account balance of $1000 that both customers A and B can make withdrawals from (say $50 and $100, respectively), one of the customers will have to wait until the other customer transaction is finished, in order to avoid inconsistent data.

Durability

Durability ensures that once a transaction has been completed and committed, it will remain committed even in the event of a system failure (crash or power failure). These completed transactions are stored in non-volatile memory.

If customer A successfully deposited $500 in their account, this transaction should not disappear if any system failure occurs.

As I mentioned above, a majority of SQL/relational databases are ACID compliant. NoSQL databases tend to sacrifice ACID compliance for faster performance. There is strong data reliability and a guarantee that the transactions are safe.

There may be clear-cut scenarios where an ACID-compliant database is the best solution, such as e-commerce, healthcare, financial services apps (when data integrity is of the utmost concern). Yet, there are other times where you may want to take other factors into consideration in deciding between a SQL or NoSQL database.

All the things that can go wrong with your transactions

Data Scientists worry about long analytical queries and warehousing, but for developers, databases are all about transactions. A database transaction is a series of logically grouped database operations: insert a row here, update a record there, and more stuff like that. Your application code is constantly making transactions every time you sign up a new user or that user updates their account information.

The thing about transactions, though, is that they can go very wrong. Any number of things can happen when you’re trying to write to your database: you can lose connection to a remote instance, you can encounter value errors or anything else under the sun. You’ve seen it, you’ve dealt with it, and it can mean disaster for your underlying data. Let’s take a look at a quick example that a company like Amazon might run into:

User updates order quantity and clicks “order now” →
- Update order quantity in the pending orders table
- Add row to orders table
- Apply the purchase to user’s balance / charge credit card

If something goes wrong in the middle of this group of operations but the system continues executing them, the user will get charged the wrong amount. And if the charge doesn’t work, they’ll get their order for free. It turns out that these kinds of data errors have names, and there are a bunch of them. A few examples:

Dirty Reads

If a transaction is in the middle of updating some data and hasn’t committed yet, and another transaction is allowed to read that uncommitted data, that’s dirty, and could lead to your app showing incorrect data that got rolled back.

An example of a dirty read could be a transaction that invalidates login tokens when a user changes their password. If as the first transaction loads the token, a second one reads that token before the first invalidates it, you’d have yourself a dirty read.

In terms of actual SQL, here’s what a dirty read might look like:

### Transaction 1 ###
SELECT user_login_token_id
FROM tokens
UPDATE tokens
SET token_status = "INVALID"
WHERE token_id = user_login_token_id
### Transaction 2 ###
SELECT user_login_token_id
FROM tokens

Non-Repeatable Reads If you’ve got two consecutive reads in one transaction with a concurrent update in between, those reads are going to show different results even though they’re part of the same transaction.

An example might be two writers working on a blog. Our first user starts a transaction that reads a post’s title, writes to the post, and then reads that post’s title again. If a second user changes that post’s title in the middle of the first user’s transaction, the first user is going to see different values for the title across the two reads; or in other words, a non-repeatable read.

Here’s what a non-repeatable read might look like in SQL:

### Transaction 1 ###
SELECT post_title
FROM posts
SELECT
    post_title,
    post_content
FROM posts
### Transaction 2 ###
UPDATE posts
SET post_title = "something_new"
WHERE post_title = post_title

Phantom Reads

If a transaction reads data and then a concurrent transaction inserts data that would have been read in the original transaction, that’s a phantom read.

Let’s use the same example as a non-repeatable read: if our second user adds content in between our first user’s two reads, the first read will be missing data that appears in the second read (this is actually really similar to a non-repeatable read, which is why the same example works).

In SQL, a phantom read might look like this:

### Transaction 1 ###
SELECT post_title
FROM posts
SELECT
    post_title,
    post_content
FROM posts
### Transaction 2 ###
INSERT INTO posts
VALUES "something_new", ...

These are the 3 transactional errors as defined by the SQL standard — the big three, you might say. A lot of these errors sound like one another and tend to overlap in practice, so don’t sweat the details.

ACID concepts in NoSQL and distributed systems

ACID was a building block of the reliable relational DBs we’re familiar with today, but NoSQL has changed the game: many NoSQL DBs are built as distributed systems, and they can’t always ensure complete transactional consistency. There’s actually a theory that governs this — called CAP theorem — that in distributed systems, you can never have full consistency and full availability; you need to choose one. So what does ACID look like for something like MongoDB or Cassandra?

A new quasi-standard has emerged for NoSQL databases called BASE (a rare overlap between SQL and chemistry jokes), and it’s a weak or soft consistency model that relaxes some of the assumptions of ACID in order to achieve scalability:

-Basic Availability: the database basically works most of the time, even though it’s not perfect

-Soft-State: nodes of the database aren’t necessarily consistent with each other all the time

-Eventual Consistency: data will be consistent across nodes eventually, like by reading time

In a lot of ways, this is the exact opposite of ACID, and prioritizes availability over perfect consistency; but that’s kind of the point of NoSQL in the first place. As NoSQL becomes a more established part of app development (more than 25% of developers are using MongoDB already), expect more advancements on this front.

Today, we’re dealing with more data than we ever have, and we’re building database systems that can scale and handle that load. ACID might not be the exact transactional standard for the future, but its building blocks will certainly make their way in there.

• What is Distributed Systems
• How does Distributed Systems work
• Why Distributed Systems
• Characteristic of Distributed Systems
• Challenges of Distributed Systems.
• Risks using Distributed Systems
• Distributed System Architecture & Types
• Distributed Data Stores(Cassandra)
• CAP Theorem
• Distributed Messaging
• Data Copying
• Decentralized vs Distributed Systems
• Cloud vs Distributed Systems

What is a distributed system?

A distributed system in its most simplest definition is a group of computers working together as to appear as a single computer to the end-user.

These machines have a shared state, operate concurrently and can fail independently without affecting the whole system’s uptime.

A distributed system is any network structure that consists of autonomous computers that are connected using a distribution middleware. Distributed systems facilitate sharing different resources and capabilities, to provide users with a single and integrated coherent network. The opposite of a distributed system is a centralized system. If all of the components of a computing system reside in one machine.

Unlike traditional databases, which are stored on a single machine, in a distributed system, a user must be able to communicate with any machine without knowing it is only one machine. Most applications today use some form of a distributed database and must account for their homogenous or heterogenous nature.

In a homogenous distributed database, each system shares a data model and database management system and data model. Generally, these are easier to manage by adding nodes. On the other hand, heterogeneous databases make it possible to have multiple data models or varied database management systems using gateways to translate data between nodes.

Distributed Information Systems: distribute information across different servers via multiple communication models.

Distributed Pervasive Systems: use embedded computer devices (i.e. ECG monitors, sensors, mobile devices).

Distributed Computing Systems: computers in a network communicate via message passing.

Distributed systems must have a shared network to connect its components, which could be connected using an IP address or even physical cables.

How does a distributed system work?

Distributed systems have evolved over time, but today’s most common implementations are largely designed to operate via the internet and, more specifically, the cloud. A distributed system begins with a task, such as rendering a video to create a finished product ready for release. The web application, or distributed applications, managing this task — like a video editor on a client computer — splits the job into pieces. In this simple example, the algorithm that gives one frame of the video to each of a dozen different computers (or nodes) to complete the rendering. Once the frame is complete, the managing application gives the node a new frame to work on. This process continues until the video is finished and all the pieces are put back together. A system like this doesn’t have to stop at just 12 nodes — the job may be distributed among hundreds or even thousands of nodes, turning a task that might have taken days for a single computer to complete into one that is finished in a matter of minutes.

There are many models and architectures of distributed systems in use today. Client-server systems, the most traditional and simple type of distributed system, involve a multitude of networked computers that interact with a central server for data storage, processing or other common goal. Cell phone networks are an advanced type of distributed system that share workloads among handsets, switching systems and internet-based devices. Peer-to-peer networks, in which workloads are distributed among hundreds or thousands of computers all running the same software, are another example of a distributed system architecture. The most common forms of distributed systems in the enterprise today are those that operate over the web, handing off workloads to dozens of cloud-based virtual server instances that are created as needed, then terminated when the task is complete.

Why distribute a system?

Systems are always distributed by necessity. The truth of the matter is — managing distributed systems is a complex topic chock-full of pitfalls and landmines. It is a headache to deploy, maintain and debug distributed systems, so why go there at all? What a distributed system enables you to do is scale horizontally. Going back to our previous example of the single database server, the only way to handle more traffic would be to upgrade the hardware the database is running on. This is called scaling vertically.

Scaling vertically is all well and good while you can, but after a certain point you will see that even the best hardware is not sufficient for enough traffic, not to mention impractical to host.

Scaling horizontally simply means adding more computers rather than upgrading the hardware of a single one.

It is significantly cheaper than vertical scaling after a certain threshold but that is not its main case for preference.

Vertical scaling can only bump your performance up to the latest hardware’s capabilities. These capabilities prove to be insufficient for technological companies with moderate to big workloads. The best thing about horizontal scaling is that you have no cap on how much you can scale whenever performance degrades you simply add another machine, up to infinity potentially. Easy scaling is not the only benefit you get from distributed systems. Fault tolerance and low latency are also equally as important.

Fault Tolerance - a cluster of ten machines across two data centers is inherently more fault-tolerant than a single machine. Even if one data center catches on fire, your application would still work.

An important part of distributed systems is the CAP theorem, which states that a distributed data store cannot simultaneously be consistent, available, and partition tolerant.

Low Latency — The time for a network packet to travel the world is physically bounded by the speed of light. For example, the shortest possible time for a request‘s round-trip time (that is, go back and forth) in a fiber-optic cable between New York to Sydney is 160ms. Distributed systems allow you to have a node in both cities, allowing traffic to hit the node that is closest to it.

For a distributed system to work, though, you need the software running on those machines to be specifically designed for running on multiple computers at the same time and handling the problems that come along with it. This turns out to be no easy feat.

Key Characteristics of Distributed Systems

Scalability

Scalability is the capability of a system, process, or network to grow and manage increased demand. Any distributed system that can continuously evolve to support the growing amount of work is considered scalable.

A system may have to scale because of many reasons like increased data volume or increased work, e.g., number of transactions. A scalable system would like to achieve this scaling without performance loss.

Generally, although designed (or claimed) to be scalable, the performance of a system declines with the system size due to the management or environmental cost. For instance, network speed may become slower because machines tend to be far apart from one another. More generally, some tasks may not be distributed, either because of their inherent atomic nature or some flaw in the system design. At some point, such tasks would limit the speed-up obtained by distribution. A scalable architecture avoids this situation and attempts to balance the load on all the participating nodes evenly.

Horizontal vs. Vertical Scaling Horizontal scaling means that you scale by adding more servers into your pool of resources, whereas Vertical scaling means that you scale by adding more power (CPU, RAM, Storage, etc.) to an existing server.

With horizontal scaling, it is often easier to scale dynamically by adding more machines into the existing pool; Vertical-scaling is usually limited to the capacity of a single server. Scaling beyond that capacity often involves downtime and comes with an upper limit. Good examples of horizontal scaling are Cassandra and MongoDB, as they both provide an easy way to scale horizontally by adding more machines to meet growing needs. Similarly, a good example of vertical scaling is MySQL, as it allows for an easy way to scale vertically by switching from small to bigger machines. However, this process often involves downtime.

Reliability

By definition, reliability is the probability a system will fail in a given period. In simple terms, a distributed system is considered reliable if it keeps delivering its services even when one or several of its software or hardware components fail. Thus, reliability represents one of the main characteristics of any distributed system. Any failing machine can always be replaced by another healthy one in such systems, ensuring the completion of the requested task.

Take the example of a large electronic commerce store (like Amazon), where one of the primary requirements is that any user transaction should never be canceled due to a failure of the machine running that transaction. For instance, if a user has added an item to their shopping cart, the system is expected not to lose it. A reliable distributed system achieves this through the redundancy of both the software components and data. If the server carrying the user’s shopping cart fails, another server with the replica of the shopping cart should replace it. Redundancy has a cost, and a reliable system has to pay to achieve such resilience for services by eliminating every single point of failure.

Availability

By definition, availability is when a system remains operational to perform its required function in a specific period. It is a simple measure of the percentage of time that a system, service, or machine remains operational under normal conditions. For example, an aircraft that can be flown for many hours a month without much downtime has high availability. Availability takes into account maintainability, repair time, spares availability, and other logistics considerations. If an aircraft is down for maintenance, it is considered not available during that time.

Reliability is availability over time, considering the full range of possible real-world conditions that can occur. For example, an aircraft that can make it through any possible weather safely is more reliable than one that has vulnerabilities to possible conditions.

Reliability Vs. Availability

If a system is reliable, it is available. However, if it is available, it is not necessarily reliable. In other words, high reliability contributes to high availability. Still, it is possible to achieve a high availability even with an unreliable product by minimizing repair time and ensuring that spares are always available when they are needed.

Let’s take the example of an online retail store with 99.99% availability for the first two years after its launch. However, the system was launched without any information security testing. The customers were happy with the system, but they don’t realize that it isn’t very reliable as it is vulnerable to likely risks. In the third year, the system experiences a series of information security incidents that suddenly result in extremely low availability for extended periods. This results in reputational and financial damage to the customers.

Efficiency

To understand how to measure the efficiency of a distributed system, let’s assume an operation that runs in a distributed manner and delivers a set of items as a result.

Two standard measures of its efficiency are the response time (or latency) that denotes the delay to obtain the first item, and the throughput (or bandwidth), which denotes the number of items delivered in a given time unit (e.g., a second). The two measures correspond to the following unit costs:

• The number of messages globally sent by the nodes of the system, regardless of the message size.

• Size of messages representing the volume of data exchanges.

The complexity of operations supported by distributed data structures (e.g., searching for a specific key in a distributed index) can be characterized as a function of one of these cost units. Generally speaking, the analysis of a distributed structure in terms of the ‘number of messages’ is over-simplistic. It ignores the impact of many aspects, including the network topology, the network load, variation, the possible heterogeneity of the software and hardware components involved in data processing and routing, etc. However, it is quite difficult to develop a precise cost model that would accurately consider all these performance factors. Therefore, we’ve to live with rough but robust estimates of the system behavior.

Serviceability or Manageability

Another important consideration while designing a distributed system is how easy it is to operate and maintain. Serviceability or manageability is the simplicity and speed with which a system can be repaired or maintained

If the time to fix a failed system increases, then availability will decrease. Things to consider for manageability are the ease of diagnosing and understanding problems when they occur, ease of making updates or modifications, and how simple the system is to operate (i.e., does it routinely operate without failure or exceptions?).

Early detection of faults can decrease or avoid system downtime. For example, some enterprise systems can automatically call a service center (without human intervention) when the system experiences a system fault.

Challenges of distributed systems

Distributed systems are considerably more complex than monolithic computing environments, and raise a number of challenges around design, operations and maintenance. These include:

Increased opportunities for failure: The more systems added to a computing environment, the more opportunity there is for failure. If a system is not carefully designed and a single node crashes, the entire system can go down. While distributed systems are designed to be fault tolerant, that fault tolerance isn’t automatic or foolproof.

Synchronization process challenges: Distributed systems work without a global clock, requiring careful programming to ensure that processes are properly synchronized to avoid transmission delays that result in errors and data corruption. In a complex system — such as a multiplayer video game — synchronization can be challenging, especially on a public network that carries data traffic.

Imperfect scalability: Doubling the number of nodes in a distributed system doesn’t necessarily double performance. Architecting an effective distributed system that maximizes scalability is a complex undertaking that needs to take into account load balancing, bandwidth management and other issues.

More complex security: Managing a large number of nodes in a heterogeneous or globally distributed environment creates numerous security challenges. A single weak link in a file system or larger distributed system network can expose the entire system to attack.

Increased complexity: Distributed systems are more complex to design, manage and understand than traditional computing environments.

Risks of distributed systems

The challenges of distributed systems as outlined above create a number of correlating risks. These include:

Security: Distributed systems are as vulnerable to attack as any other system, but their distributed nature creates a much larger attack surface that exposes organizations to threats.

Risk of network failure: Distributed systems are beholden to public networks in order to transmit and receive data. If one segment of the internet becomes unavailable or overloaded, distributed system performance may decline.

Governance and control issues: Distributed systems lack the governability of monolithic, single-server-based systems, creating auditing and adherence issues around global privacy laws such as GDPR. Globally distributed environments can impose barriers to providing certain levels of assurance and impair visibility into where data resides.

Cost control: Unlike centralized systems, the scalability of distributed systems allows administrators to easily add additional capacity as needed, which can also increase costs. Pricing for cloud-based distributed computing systems are based on usage (such as the number of memory resources and CPU power consumed over time). If demand suddenly spikes, organizations.

Distributed System Architecture

Distributed systems must have a network that connects all components (machines, hardware, or software) together so they can transfer messages to communicate with each other.

• That network could be connected with an IP address or use cables or even on a circuit board.

• The messages passed between machines contain forms of data that the systems want to share like databases, objects, and files.

• The way the messages are communicated reliably whether it’s sent, received, acknowledged or how a node retries on failure is an important feature of a distributed system.

• Distributed systems were created out of necessity as services and applications needed to scale and new machines needed to be added and managed. In the design of distributed systems, the major trade-off to consider is complexity vs performance.

Types of Distributed System Architectures:

Distributed applications and processes typically use one of four architecture types below:

Client-server: In the early days, distributed systems architecture consisted of a server as a shared resource like a printer, database, or a web server. It had multiple clients (for example, users behind computers) that decide when to use the shared resource, how to use and display it, change data, and send it back to the server. Code repositories like git is a good example where the intelligence is placed on the developers committing the changes to the code.

Today, distributed systems architecture has evolved with web applications into:

Three-tier: In this architecture, the clients no longer need to be intelligent and can rely on a middle tier to do the processing and decision making. Most of the first web applications fall under this category. The middle tier could be called an agent that receives requests from clients, that could be stateless, processes the data and then forwards it on to the servers.

Multi-tier: Enterprise web services first created n-tier or multi-tier systems architectures. This popularized the application servers that contain the business logic and interacts both with the data tiers and presentation tiers.

Peer-to-peer: There are no centralized or special machine that does the heavy lifting and intelligent work in this architecture. All the decision making and responsibilities are split up amongst the machines involved and each could take on client or server roles. Blockchain is a good example of this.

Distributed Data Stores

Distributed Data Stores are most widely used and recognized as Distributed Databases. Most distributed databases are NoSQL non-relational databases, limited to key-value semantics. They provide incredible performance and scalability at the cost of consistency or availability.

We cannot go into discussions of distributed data stores without first introducing the CAP Theorem.

CAP Theorem The CAP theorem states that a distributed data store cannot simultaneously be consistent, available and partition tolerant.

Some quick definitions:

• Consistency — What you read and write sequentially is what is expected

• Availability — the whole system does not die — every non-failing node always returns a response.

• Partition Tolerant — The system continues to function and uphold its consistency/availability guarantees in spite of network partitions

In reality, partition tolerance must be a given for any distributed data store. As mentioned in many places, one of which this great article, you cannot have consistency and availability without partition tolerance.

Think about it: if you have two nodes which accept information and their connection dies — how are they both going to be available and simultaneously provide you with consistency? They have no way of knowing what the other node is doing and as such have can either become offline (unavailable) or work with stale information (inconsistent).

In the end you’re left to choose if you want your system to be strongly consistent or highly available under a network partition.

Practice shows that most applications value availability more. You do not necessarily always need strong consistency. Even then, that trade-off is not necessarily made because you need the 100% availability guarantee, but rather because network latency can be an issue when having to synchronize machines to achieve strong consistency. These and more factors make applications typically opt for solutions which offer high availability.

Such databases settle with the weakest consistency model — eventual consistency (strong vs eventual consistency explanation). This model guarantees that if no new updates are made to a given item, eventually all accesses to that item will return the latest updated value.

Those systems provide BASE properties (as opposed to traditional databases’ ACID)

• Basically Available — The system always returns a response

• Soft state — The system could change over time, even during times of no input (due to eventual consistency)

• Eventual consistency — In the absence of input, the data will spread to every node sooner or later — thus becoming consistent

Examples of such available distributed databases — Cassandra, Riak, Voldemort

Of course, there are other data stores which prefer stronger consistency — HBase, Couchbase, Redis, Zookeeper

The CAP theorem is worthy of multiple articles on its own — some regarding how you can tweak a system’s CAP properties depending on how the client behaves and others on how it is not understood properly.

Cassandra

Cassandra, as mentioned above, is a distributed No-SQL database which prefers the AP properties out of the CAP, settling with eventual consistency. I must admit this may be a bit misleading, as Cassandra is highly configurable — you can make it provide strong consistency at the expense of availability as well, but that is not its common use case.

Cassandra uses consistent hashing to determine which nodes out of your cluster must manage the data you are passing in. You set a replication factor, which basically states to how many nodes you want to replicate your data.

When reading, you will read from those nodes only.

Cassandra is massively scalable, providing absurdly high write throughput.

Even though this diagram might be biased and it looks like it compares Cassandra to databases set to provide strong consistency (otherwise I can’t see why MongoDB would drop performance when upgraded from 4 to 8 nodes), this should still show what a properly set up Cassandra cluster is capable of.

Regardless, in the distributed systems trade-off which enables horizontal scaling and incredibly high throughput, Cassandra does not provide some fundamental features of ACID databases — namely, transactions.

Distributed Messaging

Messaging systems provide a central place for storage and propagation of messages/events inside your overall system. They allow you to decouple your application logic from directly talking with your other systems.

Known Scale — LinkedIn’s Kafka cluster processed 1 trillion messages a day with peaks of 4.5 millions messages a second.

Simply put, a messaging platform works in the following way: A message is broadcast from the application which potentially create it (called a producer), goes into the platform and is read by potentially multiple applications which are interested in it (called consumers).

If you need to save a certain event to a few places (e.g user creation to database, warehouse, email sending service and whatever else you can come up with) a messaging platform is the cleanest way to spread that message.

Consumers can either pull information out of the brokers (pull model) or have the brokers push information directly into the consumers (push model).

There are a couple of popular top-notch messaging platforms: RabbitMQ — Message broker which allows you finer-grained control of message trajectories via routing rules and other easily configurable settings. Can be called a smart broker, as it has a lot of logic in it and tightly keeps track of messages that pass through it. Provides settings for both AP and CP from CAP. Uses a push model for notifying the consumers.

Kafka — Message broker (and all out platform) which is a bit lower level, as in it does not keep track of which messages have been read and does not allow for complex routing logic. This helps it achieve amazing performance. In my opinion, this is the biggest prospect in this space with active development from the open-source community and support from the Confluent team. Kafka arguably has the most widespread use from top tech companies. I wrote a thorough introduction to this, where I go into detail about all of its goodness.

Apache ActiveMQ — The oldest of the bunch, dating from 2004. Uses the JMS API, meaning it is geared towards Java EE applications. It got rewritten as ActiveMQ Artemis, which provides outstanding performance on par with Kafka.

Amazon SQS — A messaging service provided by AWS. Lets you quickly integrate it with existing applications and eliminates the need to handle your own infrastructure, which might be a big benefit, as systems like Kafka are notoriously tricky to set up. Amazon also offers two similar services — SNS and MQ, the latter of which is basically ActiveMQ but managed by Amazon.

Data Copying

Copying is another foundational concept in computing. In software design we use the concept of taking data copies to achieve speed and resilience in several different ways.

Replicas A data replica is an exact copy of a database. Replicas are constantly, iteratively synced with one another so their contents are kept as identical as possible. They generally come in two flavours, active and passive. Active replicas support read and write access and play an (unsurprisingly) active role in serving clients. Active replicas can help with scale, resilience and location-base performance. Passive replicas are generally maintained for failover purposes; they don’t support read or write and are only synced in one direction (from the master data). The job of a passive replica is to be ready to take over from the main replica if it fails.

Caches Unlike a replica, a cache is a not-necessarily-identical, read-only, copy of your data, usually maintained as a way to serve read requests faster and more cheaply than querying your main database. Caches may take advantage of

• Location (keeping information physically close to the reading client).

• Medium (keeping information in faster-to-access physical media like memory rather than disk).

• Design (maintaining information in data structures optimised for fast read like key/value stores or graphs).

Backups A backup is a copy of your data for emergency use in the case of catastrophic data loss. Usually a backup is written to a permanent storage medium like disk in multiple physical locations. Backups are slow to write to and restore from and usually they are not kept constantly in sync with the master data. Instead, they are periodically updated. They are useful but have severe limitations and can easily lull folk into a false sense of security. This false security becomes especially dangerous as the scale and complexity of a system increases.

Denormalization for Decoupling Data duplication goes in and out of fashion as an architectural technique. It is often frowned upon because it’s a common cause of bugs. On the other hand, it is very useful for performance. Basically, like everything in life it has pros and cons and we need to make a judgment on whether to use it based on the current circumstances.

A decade ago we spent ages “normalizing” our databases, or removing data duplication. However in a distributed system we really need to have data available in more than one location. Every service cannot be calling back to the same database all the time or we lose all the benefits of distribution. Microservices usually maintain local equivalents of certain pieces of data that they can write and read without worrying about anyone else.

BUT the potential bugs have not mysteriously gone away. We have to somehow keep all those data copies in alignment (that doesn’t necessarily mean identical). By using data duplication we have to accept that we’ve just included an architectural technique that can be very bug prone in our design. It’s a tradeoff. It just means that we have to use development techniques that help resolve those problems such as Domain Driven Design, or operational techniques like managed stateful services that handle some of these issues for us.

Decentralized vs Distributed Before we go any further I’d like to make a distinction between the two terms.

Even though the words sound similar and can be concluded to mean the same logically, their difference makes a significant technological and political impact.

Decentralized is still distributed in the technical sense, but the whole decentralized systems is not owned by one actor. No one company can own a decentralized system, otherwise it wouldn’t be decentralized anymore.

This means that most systems we will go over today can be thought of as distributed centralized systems — and that is what they’re made to be.

Decentralized is essentially distributed on a technical level, but usually a decentralized system is not owned by a single source.

If you think about it — it is harder to create a decentralized system because then you need to handle the case where some of the participants are malicious. This is not the case with normal distributed systems, as you know you own all the nodes.

This has been debated a lot and can be confused with others (peer-to-peer, federated). In early literature, it’s been defined differently as well. Regardless, what I gave you as a definition is what I feel is the most widely used now that blockchain and cryptocurrencies popularized the term.

Cloud vs distributed systems Cloud computing and distributed systems are different, but they use similar concepts. Distributed computing uses distributed systems by spreading tasks across many machines. Cloud computing, on the other hand, uses network hosted servers for storage, process, data management.

Distributed computing aims to create collaborative resource sharing and provide size and geographical scalability. Cloud computing is about delivering an on demand environment using transparency, monitoring, and security.

Compared to distributed systems, cloud computing offers the following advantages:

• Cost effective

• Access to a global market

• Encapsulated change management

• Access storage, servers, and databases on the internet

However, cloud computing is arguably less flexible than distributed computing, as you rely on other services and technologies to build a system. This gives you less control overall.

Priorities like load-balancing, replication, auto-scaling, and automated back-ups can be made easy with cloud computing. Cloud building tools like Docker, Amazon Web Services (AWS), Google Cloud Services, or Azure make it possible to create such systems quickly, and many teams opt to build distributed systems alongside these technologies.

C# was developed by Microsoft in 2000 in response to JAVA. They have some similarities, such as strongly typed,compiled, Object-Oriented ,working on a virtual machine and many more. C# provides support for Modern days functionalities for all kind of software development such as front-end ,back-end, mobile ,etc.

Lets talk about some pros and cons PROS Jobs- The question a person asks when choosing a programming language is “Can I use this to find a job?” No matter what career site you search , C# and .Net developers are most assault after programmers on the platform.There are a lot of jobs out there .

Tools- Visual studio is one of the best and most popular IDE in the market.

Frameworks- .NET Core is cross platform variant of .NET framework & Xamarin used for cross platform Mobile ddevelopment

Cross Platform- It means it allows our C# programs to run on multiple platform instead of only windows machine.

Speed- C# is fast compared to same Modern day interpreted programming languages such as Python and Javascript.

CONS Learning Curve- Compared to some modern programming languages such as Python or Javscript .C# is quite had to learn.

Is C# Dying..?Should you learn C# in 2021?

My answer would be NO, C# is not dying and YES you should learn it in 2021 because of such new and upcoming features for every programming need.

If you are building a web application, you will consider where it should be hosted, and .Net allows you to host it on a host of your choice. In the past, you had to host to Windows & it would be more expensive compared to a linux host. For an enterprise this can be very important, because you can now host on AWS, Heroku, Windows, Azure, and Linux (in short, you can choose to host).

Let’s talk about some key points in C# Web Application Development- There is a framework called ASP.Net, under this framework it brings MVC and Web assembly world using Blazor. Many Enterprise solutions are made using C# because of built in scaffolding ,OEM using entity framework ,security called Identity which allows you to build very fast, secure and scalable application.

Another approach to web development is Blazor, because Microsoft puts Blazor at the top of the MVC and Razor pages, because it is the future of web development. So, if u need to learn one thing it’ll be ASP.net but if you are already coding for a living and already doing things then take a look at Blazor because it has the best implementation so far in web Assembly till date & Another approach to web development is Blazor, because Microsoft puts Blazor at the top of the MVC and Razor pages, because it is the future of web development.

Mobile Development- Writing a single code base that supports Android and IOS devices gives us cross-platform mobile development. There are basically three ways to do this: React Native, Google’s Dart / Flutter, and Microsoft’s Xamarin.

Xamarin allows us to create a user interface that supports IOS and Android natively, and it also allows us to share business layer logic code between multiple platforms. Applications developed by Xamarin can run on Android devices, IOS, and Windows / Mac OS desktops. Since it is now open source, the community can jump in and build things that can extend Xamarin to infinity

With C #, you can use ASP.net and the Web API to create microservices or service-based sites, allowing you to run these functions sometimes without using infrastructure in your traditional web hosting or service functions. You can create these microservice patterns and host them on Azure, AWS, or a host of your choice.

Game Development- Build a game inside of C# in Unity that will allow you be break into Game Development.

There’s Embeded, Machine Learning as well as Internet of things(IOT) which are being supported by C# & Microsoft.

CLOUD- Microsoft provides you with one of the most powerful cloud platforms known on Azure. It provides you with cognitive services, enables you to perform speech recognition and text-to-speech services, and uses ML.Net for machine learning.

The great thing about C # and .Net is that it is now open source and the tools are free earlier it was not