Step into the realm of AI innovations once again! Previously, Lexis Solutions introduced you to some of the top AI-powered tools for videos, content, and images. In this second chapter, get ready to hit all the right notes with a collection of tools for producing the ultimate music, audio, and design masterpieces.

Music

AIVA

AIVA is the AI-powered composer that crafts unique soundtracks for your every need. Whether for a gaming adventure, a marketing masterpiece, or a special event, this musical prodigy will generate the perfect score. Select a predefined style or upload your sound to guide AIVA’s creative flow.

Supertone

Get ready to be amazed by the audio alchemy of Supertone, the innovative Korean studio that blends art and technology to create music, voices, and more! They’re the masters of speech enhancement, song revival, and dubbing — and if you want to hear ABBA singing in Korean, they’ve got you covered.

Magenta

Magenta Studio comprises a set of music plugins that are constructed upon Magenta’s open-source tools and models. Leveraging advanced machine learning methods for musical composition, Magenta is accessible both as an independent application and as an integrated plugin for Ableton Live.

Orb Producer Suite

Assisting producers in crafting melodies, basslines, and wavetable synthesizer sounds, Orb Producer employs cutting-edge technology to generate limitless musical patterns and loops. These serve as inspirational foundations for musicians to build their creative ideas upon.

MuseNet

Under the ownership and operation of OpenAI, MuseNet can produce songs featuring a diverse range of up to 10 instruments spanning across 15 distinct styles. While MuseNet offers a rich array of AI-generated music for enjoyment, it must provide the functionality to create original music independently.

Audio

Cleanvoice

Introducing Cleanvoice, the AI-powered solution tailored to enhance your podcasts or audio recordings by eliminating filler sounds, stutters, and unwanted mouth noises. You can easily upload your audio, allow the AI to perform its cleaning magic, and obtain the refined outcome effortlessly. Experience the benefits firsthand with a complimentary 30-minute trial, allowing you to explore its functionality and witness its effectiveness.

FakeYou

FakeYou is a text-to-speech generator offering diverse options featuring hundreds of voices. You can effortlessly select any voice and have it articulate the text you input. Whether it’s the voice of your favorite actor, singer, or even your cherished animated character, the choices are abundant. It’s a creative and entertaining means to prank a friend playfully, wouldn’t you agree? And the best part? It’s free.

LALAL.AI

LALAL.AI emerges as an advanced AI audio tool, harnessing the capabilities of machine learning algorithms to swiftly and accurately accomplish this task. This exceptional AI system empowers you to effectively extract voices or instruments from audio files while preserving the original quality. By offering a complimentary 10-minute trial period, you can experience its prowess firsthand. Furthermore, another 5 minutes can be unlocked by simply following them on Reddit.

Uberduck

Uberduck’s text-to-speech technology boasts an impressive selection of over 5,000 expressive voices at your disposal, ready to enhance your voiceovers. This platform goes further by enabling you to replicate your voice. Whether for entertainment — like creating singing or rapping voices — or for professional endeavors, Uberduck caters to both. It’s a dynamic tool offering versatile applications, including the creative and the commercial. With subscription plans commencing at just $10 per month, it’s an accessible gateway to a world of vocal possibilities.

Design

Design Beast

Design Beast is a comprehensive AI-powered design platform that unites many design functionalities into a single hub. Encompassing the Mockup Engine, Logo Factory, Image Editor, Object Remover, Background Remover, and Pixel Perfect tools, this platform emerges as a valuable solution, particularly for those who may not have a design background. Its vast library of pre-designed templates caters to various applications, offering a user-friendly experience. With pricing starting at an affordable one-time payment of $67, Design Beast provides a cost-effective avenue into the world of versatile and professional designs.

Beautiful.ai

Beautiful.ai boasts an array of customizable templates, a vast selection of stock photos and videos, audio track uploads, and collaborative functionalities to enhance your workflow. Subscription plans commence at $12/month, while the option to purchase individual projects is also available at $45 per project.

Uizard

Uizard can convert screenshots into editable designs and can even automatically transform your sketches into polished designs. Whether your focus is crafting landing pages, designing apps, or creating wireframes, Uizard offers an intuitive and efficient solution with its time-saving features. Uizard’s innovative functionality comprehensively addresses your design needs.

Tome

Tome crafts incredible presentations as a seamless experience. Enter your prompts into the command bar, and watch as Tome creates mesmerizing slideshows. Whether you’re turning strategy papers, creative briefs, websites, or extensive content into engaging presentations, Tome has you covered.

As an engineer at Lexis, I would like to share some helpful bits of my experience solving complex business intelligence problems. Recently, I was assigned a rather daunting task of extracting financial information from over half a million documents of financial reports from various companies. This task, though challenging, allowed me to delve deeper into the applications of the more recent AI advances, mainly through OpenAI embeddings and vector databases, for business intelligence problems. How, you ask? Let’s take a journey through the process in my comprehensive case study.

The problem

We were tasked to extract key-value pairs from inconsistent structure and text from business documents. Therefore, we couldn’t quickly develop an algorithm to solve the problem and had to implement a solution to leverage the new advances in LLM. We used a cloud solution to train an AI model to extract the key-value pairs. However, it was still pretty challenging to deal with the sheer volume of documents, as not all documents held relevant information, and if we had to process each page, not only would it take more time, but it would be much more expensive. Manually sifting through these documents would be even more time-consuming and economically unviable. The solution? A classification program that combines OpenAI embeddings and ChromaDB vector databases, using Pytesseract for Optical Character Recognition (OCR).

Before we delve into how these technologies helped me overcome the challenge, let’s understand what vector embeddings and vector databases are.

Vector embeddings

In a high-dimensional space, vector embeddings are mathematical representations of objects, such as words or documents, capturing semantic meanings based on their context. These vectors provide a way for algorithms to understand the content and context of documents.

Vector databases

Vector databases, on the other hand, are databases designed to store and query these vector embeddings efficiently. They enable us to perform similarity search at scale, which is critical in tasks such as semantic search, relatedness search, or, in our example — classification.

Implementation

This is where OpenAI and ChromaDB came into the picture. OpenAI provides a powerful tool to generate embeddings for our documents, while ChromaDB allows us to store and query these embeddings efficiently. By leveraging the power of vector embeddings and vector databases, we can classify documents based on how close their vector representations are, which will help us identify relevant pages in a document.

First, I split the PDF documents into individual pages. For each page, I created a MySQL record, applied OCR using Pytesseract, and generated an embedding using OpenAI. Using ChromaDB, I queried the page embedding with the embeddings of already classified pages (that I organized personally) stored in the vector database. The result of the query would give me the most related page to the one I am querying, and I would classify it with the same type, storing the type back in the database.

This process enabled me to filter out irrelevant pages, leaving only the ones that needed further processing. This streamlined the process and reduced the cost by about 80% of the original estimate.

A basic example

I will give you a simple example to showcase an implementation of a Python program that does a similar job. To implement this solution, you’ll need to create a MySQL database that contains the information for each image and a ChromaDB vector database, which you will use to query the vector embeddings.

Here’s a simple schema for the MySQL database:

CREATE TABLE documents (
    id BIGINT UNSIGNED PRIMARY KEY,
    name VARCHAR(255),
    file_path VARCHAR(255), # path of the document in the filesystem
);

CREATE TABLE document_pages (
    id BIGINT UNSIGNED PRIMARY KEY,
    document_id BIGINT UNSIGNED,
    type ENUM(‘invoice’, 'balance_sheet’, ‘income_sheet’, ‘none), # here we store the type of the image once we classify it
    file_path VARCHAR(255),
    FOREIGN KEY (document_id) REFERENCES documents(id)
);

We need to manually classify at least a couple hundred images and save them in a folder. The assigned type of each image should be contained in the name of the image, separated by a dash, for example: '[name]-invoice.jpg' or '[name]-none.jpg'. Then, we will embed these images into a ChromaDB vector database using the following example:

import chromadb
import glob
import os
import pytesseract as pt
from openai import OpenAI
import uuid

openai = OpenAI('your-api-key')

client = chromadb.PersistentClient(path="chromadb")

# create the classification chromadb collection
classification_collection = client.get_or_create_collection(name="classification_collection",  metadata={"hnsw:space": "cosine"})

classified_images = glob.glob("classified_images/*.jpg") # the location of the folder containing the manually classified images

for classified_image in classified_images:
    # get base name of image
    image_type = os.path.basename(classified_image).split(".")[0].split('-')[1]

    # get the ocr text
    ocr_text = pt.image_to_string(classified_image)

    embedding = openai.embeddings.create(
        input=ocr_text.lower(),
        model="text-embedding-ada-002"
    ).data[0].embedding

    # embed the image in the chromadb database
    classification_collection.add(
        embeddings=embedding,
        metadatas={'type': classified_image.type},
        ids = str(uuid.uuid4())
    )

After creating the classification database, you can use the following Python code to perform OCR, generate embeddings, and classify the pages of a given document page:

from models import DocumentPage
import chromadb
import pytesseract as pt
from openai import OpenAI

openai = OpenAI('your-api-key')

client = chromadb.PersistentClient(path="chromadb")

# create the classification chromadb collection
classification_collection = client.get__collection(name="classification_collection")

for DocumentPage in DocumentPage.all():
    # get the ocr text
    ocr_text = pt.image_to_string(DocumentPage.file_path)

    embedding = openai.embeddings.create(
        input=ocr_text.lower(),
        model="text-embedding-ada-002"
    ).data[0].embedding

    # embed the image in the chromadb database
    query = classification_collection.query(
        query_embeddings=embedding,
        n_results=1
    )

    DocumentPage.type = query['metadatas'][0][0]['type']

    DocumentPage.save()

Conclusion

This simple example should illustrate the process of classifying images using the latest trends in LLMs and vector databases. The synergy of OpenAI embeddings and ChromaDB vector databases has revolutionized our approach to document classification, making it more efficient and cost-effective.

The article has given you a glimpse into the power of AI tools in business intelligence. As we continue to explore and experiment, we are excited about the endless possibilities that AI holds for us, and we will keep sharing our insights about how we can use them in real-world scenarios.

Nikola Popov — Laravel Dev & Project manager at Lexis Solutions

Why choose Cypress

Testing capabilities

Cypress has a friendly user interface that provides simple, human-friendly syntax, which makes it great for both beginners and experts. You can simulate user interactions like clicking buttons and filling out forms, making it perfect for testing your web applications thoroughly.

Consistent cross-browser testing

Another of Cypress’ benefits is that it supports multiple browsers, ensuring your application works well for all your users, regardless of which browser they prefer. In testing, this is helpful to identify and fix browser-specific issues early on in the development process.

Community and documentation

Cypress has a relatively big community and excellent documentation. The community actively contributes, meaning you can find answers to your questions and difficulties. At the same time, there are also plenty of tips and even pre-written code snippets that are easily findable. Additionally, many people and channels contribute to the community by posting tutorials about practically everything you search for. There is also a well-maintained documentation website that provides clear explanations for every code snippet and good general guidance, making it easy to begin using Cypress and stay aware of the situation.

Setting up your testing environment

Installation and setup

The first step in setting up your testing environment with Cypress is installing it on your device. Before attempting that, make sure to check your device is compliant with the requirements from the documentation. After that, you can start making a designated folder and name it correctly, as this will be your project folder. Afterward, open the code editor you prefer (I recommend VS Code, so let’s use it for reference) and open the folder in the editor. The next step is to open your terminal in VS Code and to install Cypress in the project folder itself with the npm install Cypress --save-dev command, after which, as soon as the installation finishes, you will have several files generated on the left side. By default, your tests will be written in the spec.cy.js file, but you can always change that and set it per your preferences. So far, this is all you need, and you can begin putting the configurations for your project.

provided with freely accessible code from the article in a public Github repo for your convenience

Project configuration

In this step, you need to set the best testing environment specifications for your project, which you can do from here. That means setting a specific browser to test with particular parameters like height and width, and you can also set a base URL to be tested, which means that you can avoid it as a step in your tests. Of course, some of these steps are non-obligatory, so excluding the ones you cannot test without, it is entirely up to you to set whatever pre-conditions you prefer. Remember that the available documentation will also give you plenty of other recommendations. You should check it out here to get some ideas.

Integrating with your application

By this, I mean that before testing whatever website you’re about to test, ensure it’s accessible on the web and exceptionally reachable by Cypress. If it has any access limitations or restrictions, this would potentially interrupt the testing process or prevent it from starting. Writing your first test

Creating test files

So, you set your project specifications, and now you must decide where to write your tests. Naturally, the first step is creating a test file, usually in the e2e folder. Also, if your test file has a longer name — you can connect it with dashes — for example, 'edit-my-profile-page.cy.js.' __Always remember that the extension for your testing suite files is not just .js it's .cy.js.

Clear and precise name given to test files becomes a crucial part of keeping project cleanliness in time, and proper naming pays off as the number of tests increases. This is particularly evident for above, let’s say, 100 tests.

Writing the tests

Here is the step where the essence of our job takes place: we begin writing tests. We will proceed here by considering that we already have test specifications prepared and reported test cases; now, we must transform the written scenarios into difficulties.

Here is a simple overview of how it looks:

describe('Most basic Google tests', () => {
    it('Successfully opens Google url', () => {
        cy.visit('https://www.google.com')
        cy.url().should(‘eq’, ‘https://www.google.com’)
    })
})

You begin by using describe() to define your whole test suite. It's good to clarify that the test suite contains as many tests as you want. Still, it's strongly recommended that all of them are related to one application module and that they are located in different folders - for example, the folder Authentication should only contain the file authentication.cy.js, the folder Navigation bar should only contain navigation-bar.cy.js and likewise for home page, etc. If you're working on a more extensive website that can potentially have login options from a third party like Facebook or Google - you can have several files in the authentication.cy.js - one file for login with credentials, let's call it standard-login.cy.js, Google login file called google-login.cy.js and a Facebook login file called facebook-login.cy.js, etc. Let's analyze the test, which is supposed to check that the Google URL opens successfully.

describe(Most basic Google tests, () => { ... })

We give our suit a proper name (which often overlaps with the test file name)

it('Successfully opens google.com', () => { ... })

Here is where we give our test a proper name specifically related to what it should do. This means that if the idea is for the test to visit a website successfully, then we should specify the “successful” part especially. That way, we can differentiate between positive and negative test scenarios еasier and not mess up with test names. After all, every test is supposed to test something different from all others, so the uniqueness in the name is a must.

cy.visit('https://www.google.com')

The cy.visit() command instructs Cypress to visit a specific URL. This step cannot be skipped in any way, whatever you test, because Cypress is designed to test websites. Before writing your first command, it is an excellent reminder to refer to the documentation. Constant checks will save time, prevent code inconsistencies, and boost effectiveness.

cy.url().should('eq', 'https://www.google.com')

In all honesty, this is the last but crucial part of the test, which a beginner QA is likely to forget once or twice. The assertion part is where we instruct our Cypress to compare the URLs it visited specifically and to check if they are the same. If they are not, Cypress didn’t successfully visit the URL we told it to visit, or we were redirected, and the assertion will fail. And so will the whole test, respectively.

Running and debugging

After writing our test, we can return to the terminal and note that the npx cypress is open to run Cypress. After that, click E2E Testing from the window:

After you do that, this window will open:

Click Start E2E testing on the browser we choose; I recommend Chrome. That will open a new window containing the specs page. From here, you should select Create new spec:

And then you will see this:

Directly click on Create spec and here:

Okay, run the spec. This will generate your first test from Cypress, and you can observe how the program runs the tests.

The test will pass because everything assigned by default is correctly written. However, when we write a test, and it fails, Cypress offers a little bit of info to identify and fix the problem, such as what exactly did not happen as expected and on which line of code in your file. So you can inspect the application’s state and understand why a test didn’t behave as expected.

You can now go back to your code editor at spec.cy.js, the file that contains one automatically generated test, which is situated in the e2e folder in Cypress (cypress/e2e/spec.cy.js).

From here on, we can edit our file name according to our test coverage and edit the test correctly, like, for example, changing the name of the test suite, the name of the test itself, and also the website in the trial from example.cypress.io to www.google.com, the commands, etc., so that we can simply perform our first exploratory testing and see how Cypress works.

Cypress best practices

Pinpoint the objects you interact with

Make the best possible use of the page elements (header, footer, login modal, search results field, etc.) when building your selectors. It will help you make a logically correct test that’s easy to read and maintain. Let’s say you’re writing a test where you have to open a login modal via a button in the site’s header. The right approach, in this case, is to incorporate the footer in the selector. Its ID can do this, or if it doesn’t have one — by its tag name and class. Chances are that you won’t be the single QA in your company. Someone will look at your test and read: you open this site, in this site, you go to the footer, and within the footer, you click on a button that opens a login modal — plain and simple. It is understandable to read.

I would also suggest that if you have no prior knowledge — you go through a crash course in HTML and CSS; it will expand your general idea of how they are built because we QAs interact with them all the time.

Write compact tests

Think of test writing as if you’re packing a suitcase — you want to include only the essentials, not have the whole house stuff. A proper test doesn’t need to include steps that don’t particularly add any use. You don’t need to instruct Cypress to open a modal that doesn’t need to be opened, just like you won’t take knives and forks when you go to a restaurant. A compact test means no unnecessary code writing, saving time as unnecessary steps will not be executed. It’s also a good idea to have your code prettier because it’s natural to become messy when writing tests. A prettifier comes in handy to tidy up the code once you finish writing. Here’s the one I use for Visual Studio Code.

Write versatile tests

Writing versatile tests, from my perspective, means several things — in this case, we’re talking about Cypress, which is JS-based. Therefore, almost all functions and capabilities of JS can be applied to test writing (such as loops, functions, variables, etc.) despite Cypress itself having its syntax. As for how to maximize the effectiveness of the tool to the best degree — we can make use of custom commands, for example. Suppose you have ten tests containing identical first two or three steps. You can combine the respective steps in one test. For example:

Cypress.Commands.add('login', () => {
 cy.get('#username-input-field').type('My username')
 cy.get('#password-input-field').type('My password')
 cy.get('#login-button').click()
})

Afterward, this command, which we named ‘login,’ we can apply like this:

describe('Authentication', () => {
  it('Successful login into the system', () => {
      cy.visit('https://www.mywebsite.com')
      cy.login()
      cy.url().should(‘contain’, ‘/profile’)
  })
})

This saves us the effort of writing the same steps for many tests and keeps tests cleaner, and when you make changes to the code, you just change only one place. All commands should be stored in the commands.js file, after which you can import them into the test file with

import '../../support/commands

Also, include constants in your tests to avoid magic numbers. If they’re to be used in one test suite only, you can write them in the test files you’re working with. Here is an example of how a constant can look if it’s stored and used in one test file:

const inputField = '#input-field’

However, if your const is to be used in three different test suites (which means three different files, respectively) — then you should consider creating a separate file called constants.js and store them there with the export keyword:

export const PASSWORD_INPUT_FIELD = '#password-input-field’'

After that, to apply it in your desired test suite file, you should import it at the beginning of the file this way:

import { PASSWORD_INPUT_FIELD } from '../../support/constants'

Conclusion

For anyone considering a testing tool for their automation journey, I can confidently say that Cypress is an excellent choice. I’ve enjoyed using Cypress for over a year, and I can attest to its remarkable user-friendliness and various capabilities. If you’re about to embark on your automation testing adventure, I strongly encourage you to try Cypress!

Radoslav Kosev — QA Engineer at Lexis Solutions

In this article, we’ll explore all the steps necessary to containerize your application and deploy it to a Kubernetes cluster on Amazon Web Services using Elastic Kubernetes Service.

A quick overview of Docker and Kubernetes

Before deploying our app to EKS, let’s first quickly go over what Docker and Kubernetes are and how they help us deploy our applications.

Docker

Containerization is a technology that allows you to package your application and its dependencies into a single, portable unit called a container. Docker is the most popular containerization platform; it will enable you to encapsulate an application and its dependencies into a self-sufficient container, ensuring that it runs consistently across different environments, from a developer’s PC to a production server. Docker containers are lightweight, fast to start, and easy to share, making them the ideal building blocks for modern applications. Containers ensure that what you develop and test locally behaves the same way in any environment where Docker is installed, eliminating the “but it was working on my machine” problem. All this helps speed up the development process, simplifies the deployment, and improves the overall reliability of applications.

Kubernetes

On the other hand, Kubernetes is an open-source container orchestration platform that automates container deployment, scaling, and management. With Kubernetes, you can define how your applications should run, ensuring high availability, fault tolerance, and efficient resource utilization.

A Kubernetes cluster comprises two fundamental components: the control plane (master node) and multiple worker nodes. The control plane oversees essential cluster-wide tasks like scheduling, scaling, and maintaining the desired state. On the other hand, the worker nodes serve as the execution engines, hosting pods — the smallest deployable units within Kubernetes. Each pod may contain one or more containers, sharing network and storage resources for efficient communication.

Some of the main features of Kubernetes include:

• Orchestration: Kubernetes automates the deployment, scaling, and management of containerized applications, reducing manual work.
• Load Balancing: It offers built-in load balancing for distributing network traffic to containers.
• Auto-Scaling: Kubernetes can automatically scale the number of containers (pods) based on CPU or other metrics to handle changing workloads.
• Self-Healing: It constantly monitors the state of your applications and automatically replaces containers if they fail.
• Rolling Updates: Kubernetes supports rolling updates, allowing you to update your application without downtime or service interruption.
• Resource Management: You can define and control resource limits and requests for CPU and memory, ensuring efficient resource utilization.

Prerequisites

Before continuing with this article, make sure you have the following:

• An AWS account
• Access to the EKS (Elastic Kubernetes Service) and ECR (Elastic Container Registry) services on AWS
• The AWS CLI tool, we’ll be using it to interact with the AWS services
• kubectl, the Kubernetes CLI tool
• eksctl, a CLI tool that will simplify our work with Amazon EKS, abstracting away the complexity of the AWS CLI
• Docker installed on your machine. We’ll use it to build our app image and push it to the container registry

Deploying the app

Initial project setup

For this example, we will deploy a simple express hello world app. This is the initial project structure that we’ll be working with

hello-world-app
├── package.json
├── package-lock.json
└── src
    └── index.js

If we run the app, it should start on port 3000.

$ node src/index.js
App running on port 3000

And a GET http://localhost:3000 request should return "Hello World!"

$ curl http://localhost:3000                          
Hello World

Containerizing the application

Before deploying our application to Amazon EKS, we need to build a Docker image and push it to Amazon’s container registry. This registry serves as the centralized repository from which EKS retrieves the image, enabling the execution of our application in the cluster.

Building the Docker image

Let’s create two new files in the root directory of our project: “Dockerfile” and “.dockerignore”.

hello-world-app
├── package.json
├── package-lock.json
└── src
    └── index.js
├── Dockerfile
└── .dockerignore

Dockerfile is a script used to create a Docker image. It contains a series of instructions that describe how the image should be built. Here’s the content of our Dockerfile:

FROM node:16

WORKDIR /app

COPY package*.json ./

RUN npm install

COPY . .

EXPOSE 3000

CMD ["node", "src/index.js"]

Let’s break it down line by line:

• FROM node:16: This instruction specifies the base image for building this Docker image. In this case, it starts with a Node.js version 16 base image, which already includes Node.js and npm, making it suitable for Node.js applications.
• WORKDIR /app: This instruction sets the working directory within the image to /app. This is where the rest of the commands will be executed.
• COPY package*.json ./: Here, the COPY instruction copies the package.json and package-lock.json files from the host (the directory where the Dockerfile is located) to the /app directory within the image.
• RUN npm install: You can use RUN to run any command during the build process. In this case, we use npm to install the dependencies specified in the package.json file.
• COPY . .: This instruction copies all the files and directories from the host into the /app directory in the image.
• EXPOSE 3000: The EXPOSE instruction informs Docker that the container will listen on port 3000 when it runs.
• CMD ["node", "src/index.js"]: The CMD instruction specifies the default command to run when a container based on this image is started. In this case, it runs the Node.js application by executing node src/index.js within the container, starting our app.

The .dockerignore file tells Docker what files should be ignored during the build. In this case, we can add node_modules so it doesn’t get copied into the docker image. While not required, this helps us keep the image size small and improve the build time.

Now that we’ve created the Dockerfile, we can build the image using the docker build command:

$ docker build -t hello-world-app .

Using the -t flag, we've specified what we want to name our image. The dot at the end tells Docker to look for the Dockerfile in the current directory. If it's somewhere else, or you named your script file differently, you can use the -f flag to pass Docker the path to the file. Once the command has been executed, you can run docker images to get a list of available images. The output should look something like this:

REPOSITORY  	 TAG  		   IMAGE ID 	   CREATED  SIZE TAG
hello-world-app  ec5cdedc65fc  26 seconds ago  861MB    latest

Pushing the image to ECR

Now that we’ve built the image, it’s time to create a repository in ECR and push it there.

To create the repository, run the following command:

$ aws ecr create-repository --repository-name hello-world-app

You should see output similar to this:

{
    "repository": {
        "repositoryArn": "arn:aws:ecr:eu-central-1:721145219880:repository/hello-world-app",
        "registryId": "721145219880",
        "repositoryName": "hello-world-app",
        "repositoryUri": "721145219880.dkr.ecr.eu-central-1.amazonaws.com/hello-world-app",
        "createdAt": "2023-10-05T13:26:15+03:00",
        "imageTagMutability": "MUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": false
        },
        "encryptionConfiguration": {
            "encryptionType": "AES256"
        }
    }
}

Take note of the repositoryUri value, which we'll use later to push our image to the repository and deploy it to the Kubernetes cluster. Still, before that, since this is a private repository, we need to authenticate Docker first. To do this, you can run the following command with your region and account ID:

$ aws ecr get-login-password --region [region] | docker login --username AWS --password-stdin [aws_account_id].dkr.ecr.[region].amazonaws.com

You will get a “Login Succeeded” message in the console if it succeeds.

Now that we’ve authenticated Docker, we can push our image to the repository. We must tag it with a specific value using the repositoryUri: [repositoryUri]:[tag]. The tag value here can be anything: an image version, "latest", or anything else that makes sense in your case. Then, you can run docker push with the same value you tagged the image with:

$ docker tag hello-world-app 721145219880.dkr.ecr.eu-central-1.amazonaws.com/hello-world-app:latest

$ docker push 721145219880.dkr.ecr.eu-central-1.amazonaws.com/hello-world-app:latest

Once the command executes, you can confirm that the image has been pushed to ECR by running:

$ aws ecr list-images --repository-name hello-world-app

{
    "imageIds": [
        {
          "imageDigest": "sha256:511a3e2fef5790fca4ac7b03fec6dba0b5339c4acea8a74b7f1ea3fc16f5904f",
            "imageTag": "latest"
        }
    ]
}

With this, we’ve successfully built our image and pushed it to ECR. Now, it’s time to deploy it to EKS.

Deploying the app to EKS

First, we’ll need to create a new cluster in EKS:

$ eksctl create cluster --name hello-world-app --region eu-central-1

Creating a new cluster might take a while. Once the command finishes executing, you should see in the terminal that it has created a new config file. In my case, it was at ~/.kube/config. This file contains the credentials necessary for kubectl to access our cluster.

We will create two new files to deploy the app to the Kubernetes cluster: k8s/deployment.yaml and k8s/service.yaml.

hello-world-app
├── k8s
	├── deployment.yaml
    └── service.yaml
├── package.json
├── package-lock.json
└── src
    └── index.js
├── Dockerfile
└── .dockerignore

The deployment.yaml file describes how the container should be run within the cluster. Here, we can specify the number of replicas, docker image, env variables, etc. On the other hand, the service.yaml file allows us to connect our app to a network. Let’s examine the files’ contents and explain them in more detail.

First the deployment.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world-container
        image: 721145219880.dkr.ecr.eu-central-1.amazonaws.com/hello-world-app:latest
        ports:
        - containerPort: 3000

• The apiVersion and kind fields are used to specify the kind of Kubernetes resource that's being defined. In this case it's deployment, which is used for deploying containerized applications. Still, many other resource types exist, like ConfigMap for storage configurations or PersistanceVolume for storage.
• The metadata section contains metadata about the deployment.
• The spec section contains information about the deployment itself:
• replicas specifies the number of running instances of your application.
• selector defines what selector we can use later on to identify pods managed by this deployment
• template contains the template of the pods. Here, we've configured the container, for example, what image should be used and what port is exposed. Here, you can also set other things, such as env variables, CPU and memory limits, data volumes for the container, and so on.
• And here’s the service.yaml file:

apiVersion: v1
kind: Service
metadata:
  name: hello-world-service
spec:
  selector:
    app: hello-world
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
  type: LoadBalancer

The service.yaml file will allow our deployment to connect to a network. In this case, the service will listen to port 80 and route the traffic to the container on port 3000.

One of the more important fields in this resource type is the type field, which decides who can access the app. There are four possible values:

• ClusterIP — the deployment won’t be publicly accessible. It can only be accessed by other apps within the Kubernetes cluster.
• NodePort — this service type will open a port that we specify on all nodes in the cluster, making the deployment publicly available
• LoadBalancer — This is another way to expose the deployment to public access. Still, it only works if you use Kubernetes with a cloud provider that supports it (AWS, in our case). It will create a load balancer service to route the traffic to our pods.
• ExternalName — this will map our service to a DNS name. We can specify the name with the spec.externalName field.

And now, we’re ready to deploy the app to the cluster by calling kubectl apply to apply the changes that we defined earlier to our cluster:

$ kubectl apply -f k8s/deployment.yaml 
deployment.apps/hello-world-deployment created

$ kubectl apply -f k8s/service.yaml   
service/hello-world-service created

Once this is done, we can check whether everything has been appropriately deployed by running:

$ kubectl get all

NAME  										READY 	STATUS  RESTARTS 	AGE
pod/hello-world-deployment-574fbf949b-5xcrf 1/1 	Running 0  			4m
pod/hello-world-deployment-574fbf949b-bfghm 1/1 	Running 0  			4m

NAME  						TYPE 			CLUSTER-IP  	EXTERNAL-IP PORT(S)  												 					AGE
service/hello-world-service LoadBalancer 	10.100.159.38 	a2c68e692ec944c48a22d7a8b10aff98-2831416.eu-central-1.elb.amazonaws.com 80:30955/TCP 	4m
service/kubernetes  		ClusterIP  		10.100.0.1  	<none>  443/TCP  																		17m

NAME 									READY 	UP-TO-DATE AVAILABLE 	AGE
deployment.apps/hello-world-deployment 	2/2 	2  			2 			4m

NAME  												DESIRED CURRENT READY	AGE
replicaset.apps/hello-world-deployment-574fbf949b 	2 		2 		2 		4m

Here, we can see all resources deployed to the cluster, including our two instances and the service. For service/hello-world-service, we can see it has been assigned an external IP address. Using that, we can test our app:

$ curl a2c68e692ec944c48a22d7a8b10aff98-2831416.eu-central-1.elb.amazonaws.com
Hello World!

As we can see, everything is working correctly. With this, we’ve finished our deployment.

You can find the complete project from this article here: https://github.com/lexis-solutions/kubernetes-demo.

Conclusion

This article covers the basics of Docker and Kubernetes, essential components in modern application deployment. However, it’s important to note that Docker and Kubernetes are vast and intricate topics with numerous advanced features and capabilities waiting to be explored. While we’ve provided a solid foundation to get you started, there’s much more to discover and master in these powerful technologies.

Stefan Mitov — Full-Stack Developer at Lexis Solutions

Introduction

In the early stages of my software development journey, I was tasked with a project that utilized “Power Apps,” Microsoft’s low/no-code platform. The experience with the low/no-code tech could have been more impressive and filled with shortcomings. As a result, I chose to stay away from low/no code development for the past few years. However, a recent opportunity within the company led me to the low/no-code ecosystem again, prompting me to explore a platform called Bubble.io.

What is Bubble.io?

Bubble.io is one of the more popular no-code development frameworks. The platform follows the “drag & drop” design approach using its visual editor, which allows anyone with or without a coding background to start immediately.

With the simple UI, it is easy to navigate, and with the search functionality, you can find what you are looking for without feeling overwhelmed.

Nevertheless, its primary appeal lies with individuals lacking technical expertise who wish to create & design web applications, all without needing familiarity with the conventional tools of the industry.

Visual Editor & UI Elements

By utilizing the visual editor, designing and building responsive apps can be done quickly and previewed immediately.

Inside the editor, you have access to various built-in UI elements. You can install new elements, or if you have knowledge of HTML & CSS, you can create your custom elements. Also, you can preview & deploy your app.

Workflow Editor & Actions

The workflow editor is where you handle the app state; it lets you create actions and add logic to every element built inside the UI Builder.

Inside the editor, you can assign dynamic or static variables during the different states of the page or when the input value is changed.

Built-in Database

Bubble.io, the no-code development platform, creates a customizable & scalable database to meet all your needs and comes with privacy settings where you can set rules for what content is available for which user and so on.

Styles and Re-usable components

In tradition, web development must have clean and reusable UI elements to maintain and scale applications. I am impressed that Bubble.io gives you the ability to have the same level of maintainability and scalability as traditional Javascript libraries such as React.js.

Inside the style editor, you can apply and define the application layout theme and colours and design the templates for buttons, inputs, and so on. When the time comes to update/change any UI element, Bubble.io allows you to change the templates, which, as a result, will change all instances of that element.

Plugins and Third-Party Integrations

Bubble.io has many pre-built and ready-to-be-used plugins, making third-party integrations such as Stripe and Google easy to set up and use. Of course, the no-code development platform provides the tools to create custom plugins, so any limitations on that front are covered.

What Bubble.io does well from a developer’s point of view

With little prior experience with low/no-code development platforms, it was easy to start building a web application. When you create an account, you’re led straight to the main dashboard; I read through the documentation to see how to work with plugins & workflows and was ready to go. The Bubble.io UI is intuitive and easy to navigate. I liked how the workflows work, and with the containers in the UI builder, it is easy to set element hierarchy and groups and have a responsive design. I could have a working MVP in a matter of hours, and I was impressed by how effortless it was.

What Bubble.io doesn’t do so well?

After a few days of working with Bubble.io, and as my project grew in pages/elements/events, I started to experience different issues, mundane tasks to solve in traditional web programming but annoying and sometimes hard to solve in Bubble.io. After having hundreds of visual elements, it was challenging to make changes. When workflows grew in size and complexity, maintaining them and making changes became much more complicated.

Conclusion

Overall, Bubble.io is one of the better low/no-code development platforms. It has reached a point worth considering as an alternative to traditional web development. Just like React.js and Angular.js, both libraries have their use cases. Strong and weak points, so for a simple application that may display information or is small in scale and with a simplistic UI, Bubble.io is an excellent option. For larger, more complex applications, I wouldn’t use Bubble.io. However, this may change in the future as the platform keeps on improving.

Ivan Todorov — Software Developer at Lexis Solutions

Introduction

Guzzle is a popular PHP HTTP client that simplifies making HTTP requests. It has been around for over ten years and is utilized by the Laravel framework under the hood. Guzzle abstracts the complexities of creating HTTP requests and handling responses by wrapping them in intuitive PHP objects, which you can easily manipulate. It is super flexible and allows you to specify timeouts, auth and proxy details, SSL certificates, etc. Guzzle is used for various HTTP-related tasks, such as making API requests, scraping web pages, and interacting with web services.

Features

Guzzle has all the features you’d expect from an HTTP client. Here are some of its key functionalities in detail: HTTP Request Methods & Params Guzzle supports all primary HTTP request methods, including GET, POST, PUT, DELETE, HEAD. You can easily specify the desired manner when creating requests.

Get query without parameters

$client = new GuzzleHttp\Client();
    $request = $client->get('https://example.com');

Delete query

$client = new GuzzleHttp\Client();
   $request = $client->delete('https://example.com');

Guzzle also makes it easy to add parameters to your requests. This is useful for passing data in the URL for GET requests or specifying the body for the JSON API you interact with.

Get query with parameters

$client = new GuzzleHttp\Client();
 $response = $client->request('GET', 'https://example.com', [
   'query' => [
     'param1' => 'value1',
     'param2' => 'value2',
    ],
]);

Post query with parameter

$client = new GuzzleHttp\Client();
 // Define the parameters as an associative array 
   $parameters = [
     'param1' => 'value1',
     'param2' => 'value2'
  ];
  // Convert the parameters to a JSON string if there are two or more parameters, 
  and skip this step if you pass only one parameter
 $json_ parameters = json_encode($parameters); 
 // Send the POST request
 $response = $client->request (POST, 'https://example.com', [
    'body' => $json_ parameters,
    'headers' => [
'Content-Type' => 'application/json',
    ],
 ]); 
 // *__Get the response body if needed
 echo__* $response->getBody()->getContents();

Custom Request Headers

You can set custom HTTP headers for your requests, essential for authentication, user agent identification, and specifying content types.

Guzzle HTTP Client Authentication Example

You can use Guzzle to send authenticated requests using various authentication methods such as Basic Authentication, Bearer Token, or other custom authentication mechanisms. Here’s an example of basic authentication:

$client = new GuzzleHttp\Client();
    $response = $client->request('GET', 'https://example.com', [
       'auth' => [
         'your_username',
         'Your_password'
         ],
       ]);

Guzzle HTTP Client Specifying Content Types Example

You can specify the Content-Type and Accept headers to indicate the content type of the request and the expected response content type:

$client = new GuzzleHttp\Client();
    $response = $client->request('POST', 'https://example.com', [
       'headers' => [
          'Content-Type' => 'application/json',
          'Accept' => 'application/json',
        ],
        'json' => [
           'key1' => 'value1',
           'key2' => 'value2',
          ],
        ]);

Guzzle HTTP Client User Agent Identification Example

You can set a custom User-Agent header to identify your client to the server:

$client = new GuzzleHttp\Client();
  $response = $client->request('GET', 'https://example.com', [
     'headers' => [
        'User-Agent' => MyCustomUserAgent/1.0,
      ],
    ]);

Request Options

Guzzle allows you to configure various request options, such as timeouts, SSL verification settings, and more.

// Timeout if a server does not respond in 5 seconds
  $client->request('GET', 'https://example.com', ['timeout' => 5]);

 // Automatically retry the request if a client or server error occurs - 
  it is possible by installing an additional Guzzle Retry
  Middleware package via 'composer require caseyamcl/guzzle_retry_middleware'. 
  Then, add an option in the request
    $response = $client->get('https://example.com', [
       'max_retry_attempts' => 5, // maximum number of retries. It is ten by default.
      ]);

// *If you're using Laravel, 
you can set retries with this code where 3 is the number of request attempts, 
and 50 is the number of milliseconds Laravel should wait between shots
Use Illuminate\Support\Facades\Http;
$response = Http::retry(3, 50)->post('https://example.com'); 

// Use a custom SSL certificate on disk
$client->request('GET', '/', ['verify' => '/path/to/directory/certificate.pem']);

Middleware

Middleware is a powerful feature that enables you to modify requests and responses flexibly and reusable. You can attach middleware to the Guzzle client to perform tasks like logging, authentication, or response processing.

// An example of how to log Guzzle HTTP requests and responses
  for debugging purposes using the Monolog package, which sends logs to a file,
  and the MessageFormatter instance that controls what gets logged
  $logger = new Logger('guzzle');
  $logger->pushHandler(new StreamHandler('guzzle.log'));
  $stack = \GuzzleHttp\HandlerStack::create();
  $stack->push( \GuzzleHttp\Middleware::log(
       $logger,
       new MessageFormatter('{method} {uri} - {code} - {res_body}')
     )
    );
    $client = new GuzzleHttp\Client(['handler' => $stack]);
    $response = $client->request('GET', 'https://example.com');

Cookies

Guzzle includes a CookieJar for managing cookies in your HTTP requests. This allows your application to handle session and authentication cookies seamlessly.

// An example of managing cookies 
   across multiple requests using CookieJar
   $cookieJar = new CookieJar();
   $client = new GuzzleHttp\Client(['cookies' => $cookieJar]);
   $response = $client->request('GET', 'https://example.com');
   $cookies = $cookieJar->toArray(); // Access cookies from the response
   print_r($cookies);

Async Requests

Guzzle supports asynchronous requests, which means you can send multiple requests in parallel and process their responses as they become available. This can significantly improve the performance of your applications when dealing with numerous remote resources.

// An example of how to send asynchronous requests for improved performance
  $client = new GuzzleHttp\Client(); 
  $api_urls = [
    'https://api.example.com/resource1',
    'https://api.example.com/resource2',
    'https://api.example.com/resource3',
];
$promises = []; 
foreach ($api_urls as $url) {
    $promises[$url] = $client->getAsync($url);
}
$responses = Promise\settle($promises)->wait(); // 
A settled Promise represents the outcome of 
an asynchronous operation which can be 
successfully (fulfilled) or with an error (rejected)
foreach ($responses as $url => $response) {
if ($response['state'] === 'fulfilled') {
    $response_data = $response['value']->getBody()->getContents();
    echo "Response from $url: $response_data \n";
  } else {
     echo "Request to $url failed.\n";
   }
 }

Streaming Responses

Guzzle can stream large response bodies, which helps download large files or process streaming data without loading it all into memory.

// You can stream large responses to save memory
$client = new GuzzleHttp\Client(); 
$api_url = 'https://api.example.com/large-resource';
$response = $client->request('GET', $api_url, ['stream' => true]); 
// Stream the response
$stream = $response->getBody();
while (!$stream->eof()) {
    echo $stream->read(1024); // Process data in chunks
}

Proxy support

Proxy support in Guzzle allows you to route your HTTP request through an intermediary server (a proxy server) before reaching the target server. This can be useful when accessing resources behind a firewall or for privacy reasons. Guzzle makes configuring and using a proxy server for your requests is straightforward.

// An example of how to set proxy servers. Define the proxy server details
    $proxy_servers = [
    first_server => 'https://first-proxy-server:port',
    second_server => 'https://second-proxy-server:port',
];
// Create a Guzzle HTTP Client with proxy configuration
$client = new GuzzleHttp\Client([
    RequestOptions::PROXY => $proxy_servers,
]);
// Sent a GET request through the proxy
$response = $client->get('https://example.com');
// Handle the response
echo $response->getBody()->getContents()

Sink

Guzzle’s “sink” feature allows you to efficiently download large files or responses directly to a file instead of loading the entire response into memory. This is particularly useful when dealing with large files, as it helps prevent memory exhaustion.

// An example of downloading a file with the sink function
$client = new GuzzleHttp\Client();
// Define the file path where you want to save the downloaded file
$file_path = 'path/to/directory/file.txt';
// Send a GET request and save the response directly to the file
$client->get('https://example.com/large-file.zip', [
    RequestOptions::SINK => $file_path,
]);

Error Handling

The library provides error-handling mechanisms and exceptions for various HTTP-related errors, making it easier to handle issues like connection failures, timeouts, and invalid responses gracefully.

// Handle errors gracefully with Guzzle
$client = new Client();
$api_url = 'https://example.com/resource';
try {
    $response = $client->request('GET', $api_url);
    // Handle the response here
} catch (RequestException $e) {
    if ($e->hasResponse()) {
        // Handle HTTP errors
        $response = $e->getResponse();
        $status_code = $response->getStatusCode();
        echo "HTTP Error: $status_code \n";
    } else {
        // Handle other request errors
        echo 'Request Error: ' . $e->getMessage() . "\n";
    }
}

Request and Response Logging

Guzzle supports request and response logging, which is valuable for debugging and monitoring the communication between your application and external services.

// You can log both requests and responses for debugging. 
Create a PSR-3 compatible logger (e.g., Monolog)
$logger = new \Monolog\Logger('guzzle');
$logger->pushHandler(new \Monolog\Handler\StreamHandler('guzzle.log'));
$stack = HandlerStack::create();
$stack->push(Middleware::log($logger, new MiddlewareFormatter()));
$client = new GuzzleHttp\Client(['handler' => $stack]);
$api_url = 'https://api.example.com/resource';
try {
    $response = $client->request('GET', $api_url);
    // Handle the response here
} catch (Exception $e) {
    echo 'Error: ' . $e->getMessage();
}

Note: In this example, you must implement a custom MiddlewareFormatter to format the log entries per your requirements. Scenarios

Certainly, Guzzle HTTP Client is commonly used in various scenarios in PHP development. Let’s dive into the most common use cases for Guzzle, along with detailed examples for each:

Web Scraping and Content Retrieval

Scraping web content or fetching HTML data from websites.

$client = new GuzzleHttp\Client();
        $response = $client->get('https://example.com');
        $html_content = $response->getBody()->getContents()

Asynchronous Requests

Sending multiple HTTP requests in parallel for improved performance, e.g., concurrently fetching data from various APIs.

$client = new GuzzleHttp\Client();
$promises = [
    $client->getAsync('https://api.example.com/resource1'),
    $client->getAsync('https://api.example.com/resource2'), 
    // Add more async requests as needed
];
// Wait for all promises to settle and get their results
$responses = GuzzleHttp\Promise\unwrap($promises);

Authentication and Token Retrieval

Authenticating with an OAuth 2.0 server to obtain an access token for API access.

$client = new GuzzleHttp\Client();
$response = $client->post('https://auth.example.com/token', [
    'form_params' => [
        'grant_type' => 'client_credentials',
        'client_id' => 'YOUR_CLIENT_ID',
        'client_secret' => 'YOUR_CLIENT_SECRET',
    ],
]);
$token_data = json_decode($response->getBody()->getContents(), true);

Web Service Testing

Writing automated tests to check the functionality of a web service’s API endpoints. You can use PHPUnit, a popular testing framework for PHP, to create and run these tests.

$client = new GuzzleHttp\Client();
// Send a GET request to https://example.com/your-endpoint
$response = $client->get('https://example.com/users/123');

// Check if the response status code is 200, and then parse the JSON response content and make assertions about its structure and values

$this->assertEquals(200, $response->getStatusCode());
$data = json_decode($response->getBody(), true);
$this->assertArrayHasKey('expected_key', $data);
$this->assertEquals('expected_value', $data['expected_key']);

Step-by-step with Guzzle HTTP Client

Picture yourself in the process of constructing a web application that needs to interact with various online services. Let’s say you intend to retrieve video information from a video database and present it within your application. You will surely need an HTTP client, and Guzzle is the perfect choice. Let’s start with authentication and then proceed with different download modes (synchronous and streaming). Follow these steps:

1. Install Guzzle: If you haven’t already, install Guzzle by running:

composer require guzzlehttp/guzzle

2. Create a PHP file: Start by creating a PHP file where you’ll write the code for downloading the video.

3. Import necessary classes: In your PHP file, you’ll need to import the Guzzle classes you’ll be using and include the “autoload” file in the script part of the code to load all the classes and methods.

<?php
require 'vendor/autoload.php';
Use GuzzleHttp\Client;
Use GuzzleHttp\RequestOptions;

4. Specify the URL of the video and authentication credentials: Set the URL of the video and any authentication credentials if required:

$username = 'your_username';
  $password = 'your_password';
  $video_url = 'https://example.com/path/to/video.mp4';
  $file_path = 'path/to/directory/video.mp4';

5. Initialize Guzzle Client: Create an instance of the Guzzle Client. This client will handle the HTTP requests:

$client = new GuzzleHttp\Client([
        'auth' => [$username, $password]
]);

6. Option 1: Synchronous Download: This method downloads the entire file simultaneously. Use this approach if you’re working with reasonably sized files:

try {
    $client->get($video_url, [
        RequestOptions::SINK => $file_path,
    ]);
    echo 'Video downloaded successfully!';
} catch (GuzzleHttp\Exception\RequestException $e) {
    echo 'Error: ' . $e->getMessage();
}

7. Option 2: Streamed Download: This method downloads the file in chunks, which can be helpful for large files. It consumes less memory compared to synchronous downloads:

try {
    $response = $client->request('GET', $video_url, [
        'auth' => [$username, $password], // Add authentication
        'stream' => true // Enable streaming
    ]); 
    if ($response->getStatusCode() === 200) {
        $stream = $response->getBody();
        $file_handle = fopen('video.mp4', 'w');
        while (!$stream->eof()) {
            fwrite($file_handle, $stream->read(1024));
        }
        fclose($file_handle);
        echo 'Video downloaded successfully!';
    } else {
        echo 'Error: Status code - ' . $response->getStatusCode();
    }
} catch (Exception $e) {
    echo 'Error: ' . $e->getMessage();
}

Replace ‘https://example.com/path/to/video.mp4', ‘your_username’, and ‘your_password’ with your actual URL and authentication credentials.

Note: Ensure you have the proper rights or permissions to download and use the video, considering any legal or copyright considerations.

Conclusion

Guzzle is an incredible HTTP Client for PHP because of its simplicity, flexibility, extensive feature set, security, and strong community support. Whether you’re building web applications, APIs, or scripts that interact with remote services, Guzzle can streamline your HTTP-related tasks and enhance the overall quality of your projects.

Antoniya Ivanova — PHP Developer at Lexis Solutions

In the digital age, where information is at our fingertips, the efficiency of retrieving data from the web has become paramount. To address the challenges posed by data retrieval, web scraping, and crawling have emerged as essential techniques. Let’s delve into web scraping and crawling, exploring their benefits, intricacies, and potential pitfalls.

Navigating the Data Labyrinth

Web scraping involves extracting specific information from websites by parsing the HTML and other relevant data. It allows applications to access up-to-date information from the web without relying on manual input.

Benefits of Web Scraping

Real-time Data

Web scraping ensures that applications always have access to the latest data without the delay associated with manual updates.

Automation

By automating data collection, web scraping saves valuable time and resources that can be allocated to more critical tasks.

Customization

Scraping allows you to extract only the necessary data, eliminating the need to sift through irrelevant information.

Competitive Insights

Businesses can gain a competitive edge by monitoring competitors’ websites for pricing, product details, and other market insights.

Comprehensive Indexing

Crawlers explore the entire web, indexing a wide range of content for users to search.

Timely Updates

Crawlers revisit websites periodically, ensuring the current indexed content.

Structured Data

Crawlers organize information in a structured manner, making it easier for search engines to retrieve relevant results.

Challenges and Considerations

Server Load and Rate Limiting

Frequent crawling can strain website servers, potentially affecting their performance and leading to access restrictions. Websites protect themselves from being overwhelmed by requests by imposing rate limits or blocking IP addresses exhibiting suspicious behavior.

Legal and Ethical Concerns

Some websites prohibit scraping through their terms of use. It’s crucial to respect the website’s policies and not overload its servers with excessive requests.

Duplicate Content

Crawlers might inadvertently index the same content, leading to inaccurate search results.

Data Integrity

One challenge lies in ensuring the accuracy and integrity of the harvested data. Websites may update their structure or content, leading to data extraction errors. Additionally, ensuring that the scraped data is legally and ethically sourced is of utmost importance.

Website Structure

Websites often change their structure, which can break scraping scripts. Regular maintenance is necessary to adapt to such changes. Websites often use dynamic content-loading mechanisms like JavaScript, which can complicate the scraping process. Extracting data from such sources requires more advanced techniques to correctly interpret and capture the information.

Conclusion

Scraping and crawling web pages are essential methods for optimizing web searches. They allow us to access up-to-date information from the Internet without manual input. However, it is crucial to be aware of the problems and considerations associated with these methods, such as website structure, legal and ethical issues, and data quality. By following best practices, web scraping and crawling can be used to efficiently and effectively extract data from the Internet.

Oleksandr Suprun — Junior Software Developer at Lexis Solutions

In this article, I will discuss the benefits of using DevOps-as-a-Service (DaaS) over traditional DevOps practices, as seen from our experience at Lexis Solutions. We have had positive and negative experiences when choosing the right cloud infrastructure for our clients, and I will elaborate on that aspect.

(source: AWS)

To quote AWS’s Model definition directly, it is the combination of practices and tools that increase an organization’s ability to deliver applications and services at high velocity: evolving and improving products faster than companies using traditional software development and infrastructure management. This speed enables organizations to serve customers better and compete more effectively in the market.

DevOps vs. DaaS

In that regard, the traditional DevOps Model applies to mostly in-house infrastructure management. This means hiring and managing a dedicated team, purchasing and setting up servers, and configuring everything from the ground up to meet one’s business needs.

On the other side, DaaS is mainly cloud-based, with many possible layers of functionality, abstraction, and “ease of use” focus.

It integrates selected elements of DevOps tools into one comprehensive system to enhance collaboration, monitoring, management, and reporting. This service model contrasts with the in-house toolchain approach, where the DevOps team employs a disjointed set of individual tools.

DaaS is an attractive solution for businesses lacking in-house DevOps knowledge or the financial resources to train employees in these skills. It streamlines the intricate process of managing data and information flows throughout the chain. With this method, various members and teams participating in the process can use user-friendly interfaces to access necessary tools without the need for a complete understanding of the whole toolchain. For instance, using the same DevOps as a Service system, a developer can utilize source code management tools, a tester can access application performance management tools, and the IT operations team can implement changes using more top-level configuration management settings. This setup enables team-wide monitoring and reporting on activities.

Unlike traditional DevOps, DaaS focuses more on the end result and the complete process, from code compilation to production deployment.

Popular DevOps service providers

Amazon Web Services (AWS)

Amazon Web Services (AWS) has carved out a substantial space in this domain, building a robust global network for virtually hosting some of the world’s most complex IT environments. Key to their suite are AWS CodeBuild, AWS CodePipeline, and AWS CodeDeploy — a trinity of services designed to manage continuous cloud delivery. These services offer accessible solutions that enable a secure, scalable, continuous delivery model in the cloud, making migration to the cloud a worthwhile consideration for many organizations.

Even more, many platforms competing for a slice of this market choose to integrate some AWS tools to offer their own packaged DevOps solutions.

Google Cloud Platform (GCP)

Google Cloud Platform (GCP) also boasts a global network and an ever-growing list of capabilities. The StackDriver suite of development tools, GCP Deployment Manager, and GCP Cloud Console constitute a robust set of offerings to manage the cloud-based continuous delivery cycle. These tools and the platform’s ability to support complete cloud development solutions for various platforms make GCP a strong contender in the DevOps sphere.

Microsoft Azure

Microsoft’s Azure enters the scene with various interoperable tools to manage the cloud environment. As a cloud management platform, Azure offers a complete set of tools such as Azure App Service, Azure DevTest Labs, and Azure Stack. With a long-standing presence in the IT industry, Azure may offer the most seamless transition to hybrid or full cloud environments for organizations already using Microsoft products and services.

Digital Ocean

Digital Ocean is another platform that we at Lexis Solutions use quite often. They offer a more “lean” approach with configurable and optimized out-of-the-box tools for quick time-to-market needs. Their App Platform is convenient for deploying web services from a GitHub repository while offering scalability and load-balancing solutions. Their ecosystem has grown continuously in the last few years while offering competitive pricing, a combination ideal for start-up caliber projects.

Since July this year, we have also officially partnered with Digital Ocean, which grants our clients $200 credit for usage.

Different varieties of DaaS

One thing to note is that DaaS may refer to a broader range of possible services. On the one hand, there are platforms like the ones described in the previous section. On the other — IT teams that offer DevOps services and know-how, including in-house integrations. Teams may even partner with a platform for more complete and flexible solutions.

As mentioned earlier, we have an established partnership with Digital Ocean at Lexis Solutions and have successfully set up environments for multiple clients there. Although we are not letting ourselves be limited to a single provider.

For example, there have been client specifications that required us to integrate with specific third-party APIs. The App Platform that Digital Ocean offers makes it relatively easy to deploy production-ready software on quick notice. The main drawback there, however, is a lack of a static IP address of the environment due to how the continuous delivery is implemented on their side. This limitation meant that the third party’s API couldn’t whitelist our setup for authorization, so we had to consider other options.

Examples such as these illustrate limitations that some packaged solutions may have, but also how turning to a team might save extra effort and costs by tailoring for the specific case.

DaaS considerations

The general speed from software implementation to production deployment is the main benefit and potential cost-saving from not relying on an internal DevOps team. This is, of course, of significant importance to emerging businesses and startups. However, one may also encounter challenges later down the road.

First, there might be security risks if the DaaS provider fails to adhere to security protocols. Next to consider is the limited control over one’s setup — platforms may encapsulate their services, allowing for partial control only.

In conclusion, a mixed approach between an automated cloud setup and hiring an IT team for consulting is the best of both worlds. Either part-time or for an extended period, an external team could guide you and help with unforeseen challenges later down the road of your business growth.

Milush Karadimov — Co-founder at Lexis Solutions

Over the past year, here at Lexis Solutions, we have been growing our offerings in the custom data solutions space. Having worked on numerous initiatives, we’ve expanded into data integrations, data acquisition, data aggregation, and sentiment analysis to achieve our clients’ diverse needs. In this article, we’ll look at one of the core data services we provide — data enrichment.

What is Data Enrichment?

Data enrichment refers to enhancing existing data by adding valuable and relevant information. The goal is to enrich the data, making it more comprehensive, accurate, and actionable.

Data enrichment involves various techniques and methodologies, such as appending missing information, standardizing data formats, validating data accuracy, and integrating multiple data sources. This process results in a more complete and valuable dataset that can deliver significant insights.

Data enrichment involves augmenting existing data with additional information from external sources. This includes demographic details, geographic information, firmographics, social media data, purchasing behavior patterns, etc. By adding these extra dimensions to the data, businesses gain a more comprehensive understanding of their customers, allowing them to tailor their strategies, improve customer experiences, and drive growth.

For example, a retail company may enrich customer data by integrating social media. By analyzing customers’ social media activity, the company can gain insights into their interests, preferences, and influencers they follow. This information can then be used to create personalized marketing campaigns, recommend products that align with customers’ interests, and build stronger relationships.

Data enrichment is not limited to customer data. A manufacturing company may enrich its product data by appending detailed specifications, images, and customer reviews. This enriched product data can be used to improve inventory management, optimize pricing strategies, and enhance the customer experience.

The Importance of Data Enrichment in Today’s Business Landscape

In the digital age, businesses are generating vast amounts of data. However, more than raw data is needed to extract meaningful insights. With proper enrichment, companies can make decisions based on complete or updated information, hindering their growth potential.

Data enrichment plays a crucial role in ensuring data quality. By cleansing and enriching data, businesses can minimize inaccuracies and inconsistencies arising from human errors, data entry mistakes, or outdated information. Clean and reliable data is the foundation for practical analysis and decision-making.

Furthermore, data enrichment enables businesses to unlock the true power of business intelligence. Companies can gain deeper insights into market trends, customer preferences, and emerging opportunities by integrating external data sources and enriching internal datasets. This valuable intelligence can drive marketing strategies, optimize operations, and foster innovation.

Moreover, data enrichment facilitates better decision-making. Enriched data provides a holistic view of customers, allowing businesses to personalize their offerings, anticipate customer needs, and provide tailored experiences. This level of understanding empowers organizations to make data-driven decisions that boost customer satisfaction, enhance their competitive edge, and drive revenue growth.

Data enrichment is an ongoing process. As new data becomes available and business needs evolve, organizations must enrich their datasets to stay ahead of the competition continuously. By investing in data enrichment, businesses can unlock the full potential of their data and gain a competitive advantage in today’s data-driven business landscape.

Data Enrichment Services Use Cases

Enhancing Data Quality

Data enrichment services are instrumental in enhancing data quality. Data cleaning processes rectify duplicate entries, inaccurate information, and inconsistencies. By standardizing data formats and validating data accuracy, businesses can ensure their data is reliable and consistent, enabling more accurate analysis and decision-making.

Retail companies collect customer data through various channels, including online purchases, in-store transactions, and loyalty programs. With data enrichment services, this company may end up with multiple entries for the customer, making it easier to analyze behavior. The company can identify duplicate entries by utilizing data enrichment services and creating a comprehensive profile for each customer. This improves data quality and provides a holistic view of customer interactions and preferences.

In addition to eliminating duplicate entries, data enrichment services can rectify inaccurate information. For instance, if a customer’s address is misspelled or incomplete, data enrichment services can validate and correct the address based on reliable sources, such as postal databases. This ensures the company has accurate and up-to-date customer information, crucial for effective communication and personalized marketing efforts.

Boosting Business Intelligence

Data enrichment services play a crucial role in boosting business intelligence capabilities. Businesses can enrich their internal data by integrating external data sources, such as market research reports, social media insights, and public databases, creating a comprehensive view of their industry, target market, and customers.

Imagine a technology company that wants to launch a new product in a specific market segment. The company can better analyze market research reports, industry trends, and competitor data by leveraging data enrichment services to understand the target market’s needs and preferences. This enriched intelligence allows the company to tailor its product features, pricing, and marketing strategies to align with the market’s demands, increasing the chances of success.

In addition to external data sources, data enrichment services can leverage social media insights to provide valuable information about customer sentiment, preferences, and behavior. By analyzing social media conversations and trends, businesses can identify emerging opportunities, monitor brand reputation, and engage with customers in a more targeted and personalized manner.

Facilitating Better Decision Making

Data enrichment services facilitate better decision-making by giving businesses a deeper understanding of their customers and target market. With enriched data, companies can segment customers more effectively, identify their unique needs, and develop personalized marketing strategies.

Let’s consider an e-commerce company that sells a wide range of products. The company can analyze customer purchase history, browsing behavior, and demographic information by utilizing data enrichment services to create customer segments based on preferences and interests. This segmentation allows the company to tailor its product recommendations, promotional offers, and marketing campaigns to each customer segment, resulting in higher customer satisfaction and conversion rates.

Furthermore, data enrichment services enable businesses to track and analyze customer interactions across multiple touchpoints, such as website visits, email communications, and social media engagements. This comprehensive view of customer interactions helps businesses identify patterns, preferences, and pain points, allowing them to make data-driven decisions that align with customer expectations and drive business growth.

Key Components of Data Enrichment Services

Data Cleaning

Data cleaning is a fundamental feature of data enrichment services. It involves identifying and rectifying errors, inconsistencies, and inaccuracies within datasets. Businesses can improve data quality and reliability by removing duplicate entries, standardizing data formats, and validating data accuracy.

Data Integration

Data integration is another essential aspect of data enrichment services. It involves consolidating data from multiple internal and external sources to create a comprehensive, unified dataset. By integrating data silos, businesses can gain a holistic view of their operations, customers, and market, facilitating more accurate analysis and decision-making.

Data Validation

Data validation is an integral part of data enrichment services. It involves verifying the accuracy, completeness, and consistency of data. By running validation checks, businesses can ensure that their data is reliable and error-free, enabling confident decision-making based on accurate information.

Choosing the Right Data Enrichment Service Provider

When selecting a data enrichment service provider, several factors should be considered.

Firstly, consider the provider’s experience and expertise in data enrichment. Look for a provider that offers a wide range of enrichment techniques and has a proven track record of delivering high-quality results. Some good places to check for reputation are Clutch and LinkedIn.

Secondly, consider the provider’s data sources and data coverage. Ensure that the service provider has access to diverse and reliable data sources covering various aspects relevant to your business. It’s best if the provider has partnerships with data acquisition platforms to give you guidance and discounts on your usage, such as our partnership with Apify.

Thirdly, consider the scalability and flexibility of the provider’s solutions. As your business grows, your data needs may evolve. Choose a provider that can accommodate changing data requirements and scale their services accordingly. Usually, small companies can offer greater flexibility and better pricing.

In conclusion, data enrichment services hold immense potential for businesses looking to unlock the power of their data. By understanding the concept of data enrichment, exploring its importance in today’s business landscape, and recognizing its key features and benefits, businesses can make informed decisions to leverage data enrichment services effectively. By choosing the right data enrichment service provider and learning from success stories through case studies, companies can harness the power of data enrichment to drive growth, enhance customer experiences, and stay ahead in today’s competitive business environment.

Bilyal Mestanov — Co-founder at Lexis Solutions

Imagine having a brilliant business idea bubbling with potential, just waiting to become a reality. But there’s a catch: you need a developer to make it happen. The problem is the world is brimming with freelancers and development companies, all claiming to be the key to your dreams.

When it comes to outsourcing, the path ahead can be murky. Hiring a freelancer is a different ballgame than enlisting a software development agency. An agency boasts a powerhouse team of specialized experts with unique skill sets. On the other hand, a freelancer may offer cost savings and flexibility. So, what path should you take when faced with the dilemma of freelancer vs. software development agency? Your decision today holds the power to shape the future success of your project.

Sorting through these choices can feel overwhelming. But fear not! Let’s embark on another of Lexis Solutions’ journeys to demystify the selection process.

The agency vs. freelancer quandary may appear deceptively simple, yet its implications must be balanced. Making the correct choice between an agency and a freelancer can transform your final product, for better or worse. So, let’s dive headfirst and explore the ideal fit for your project.

Embrace the freedom

When it comes to freelance developers, they operate in a world of boundless possibilities, offering not just one but three unique modes of engagement. Picture this: they can sign temporary contracts, take freelance gigs, or work remotely full-time. However, like any endeavor, there are risks to be mindful of when collaborating with freelancers, particularly as they juggle multiple projects simultaneously.

Cost efficiency

The allure of freelance development lies in its ability to provide exceptional value for your investment. Freelancers, armed with nothing more than a trusty computer and an endless supply of caffeine, keep their expenses low and, as a result, offer rates significantly lower than those of software agencies. But remember, cost efficiency doesn’t necessarily mean your entire project will come at a reduced price when you hire a freelancer.

Revel in unparalleled flexibility

Many freelancers embrace working beyond traditional hours and remain responsive to communication. Liberated from the constraints of others’ schedules, they can often work with unmatched adaptability, leading to quicker project delivery.

Harness specialization

Freelancers are masters of their craft, honing their expertise in specific domains. Before bringing a freelancer on board, it’s essential to carefully examine their portfolio to understand their specialized skill set. While some freelancers may tout themselves as Jacks-of-all-trades, we all know that true mastery lies in specialization. You can unlock remarkable results by aligning your unique talents with your project’s requirements.

The bad points

While the realm of freelance developers offers enticing advantages, it’s crucial to be aware of the potential drawbacks that lurk beneath the surface. Let’s shed light on the cons associated with working with freelancers:

Questionable work quality

Not all freelancers prioritize the quality of their deliverables. Generic platforms like Upwork need more rigorous vetting processes for their listed freelancers. This means you may encounter individuals solely driven by quick cash, rushing through projects without paying attention to the level of craftsmanship. For such freelancers, reputational risks hold little importance, as they can easily create new accounts or work anonymously. Heightened risks

Freelancers present inherent risks due to their solitary nature. Even if your chosen freelance developer is highly reliable, unforeseen circumstances can disrupt your project. Without a contingency plan, your project may come to a screeching halt. It’s essential to consider the potential consequences and have measures to mitigate those risks.

Organizational challenges

Working with freelancers can introduce complexities as they juggle multiple clients simultaneously. This increased workload can lead to errors and delays in your product’s development. It’s crucial to clearly understand your freelancer’s capabilities and establish realistic deadlines to minimize the chances of misunderstandings or poor outcomes.

Vanishing acts

All terms and conditions have been meticulously defined, payments have been made, and the project is ready to take off. Suddenly, your chosen developer disappears into thin air. It may sound like a nightmare, but it’s a scenario that can occur if you engage with unreliable freelance platforms or individual contractors lacking the proper means to ensure timely and consistent delivery.

The triumphs of an agency

A successful outsourcing business typically has a diverse team of experts, including web/mobile app developers, testers, product managers, content creators, designers, analysts, and more. When you hire a development agency, you gain access to a professional support system that can handle all your work efficiently. Their reliability and accountability ensure adherence to work processes and procedures while prioritizing security. Seamless teamwork

An agency is a pre-established team that has spent years refining its development strategies. While freelancers may not guarantee cooperation, agency employees work together seamlessly daily, fostering unparalleled teamwork and collaboration. Their familiarity with each other’s strengths, weaknesses, and work habits enables them to navigate significant challenges effortlessly. No matter how many freelancers you hire, they won’t match the level of cooperation an agency guarantees.

Extensive expertise

The primary reason agencies cover more ground is their larger workforce. Even the most talented individual can only possess limited knowledge, making an agency more valuable when your project requires diverse areas of expertise. Each employee can specialize in a specific development phase, eliminating the need to search for additional resources. Furthermore, agencies can provide long-term benefits like maintenance, marketing, feature enhancements, and other valuable services by overseeing your product from start to finish.

Superior quality

Freelancers generally produce high-quality work, but agencies often adhere to even more rigorous quality standards. While freelancers may complete a project without consideration, agencies employ strict review and QA processes. Additionally, freelancers usually need more resources available to agencies for thorough testing. How many independent freelancers have multiple phones for testing purposes?

Ongoing maintenance & support

Reputable agencies prioritize building long-term customer relationships and upselling additional services. As a result, they are always ready to offer maintenance services, bug fixes, functionality modifications, and other refinements to the products they create or maintain.

Accurate development estimation

Since most vendors charge hourly, knowing the time required to implement specific functionality by hiring a top software development company can eliminate concerns about unpredictable expenses is crucial. They usually have a well-defined project estimation process, accurately determining the development time required for each feature. And hiring a software development company is an excellent choice for startups and large corporations seeking unique skills, specific timelines, and a reliable contractor committed to project completion.

Unveiling potential pitfalls

Beware of hidden costs

When outsourcing software development, be mindful of potential hidden costs. Read the contract the software development company offers. Pay attention to any fine print that may reveal unexpected expenses.

Furthermore, remember that cheaper sometimes doesn’t mean better, especially for complex projects like software development. Opting for offshore developers with lower hourly rates may seem enticing initially, but it could result in higher costs in the long run. Conversely, although some developers may appear expensive initially, they may complete the project more efficiently, ultimately reducing the total cost.

Increased security and business risks

While reputable software development companies pose minimal risks, hiring the wrong talent can complicate matters. Since software often collects users’ data, mishandling or data breaches can have severe consequences. Consider the case of Meta, which was fined $276 million for a data leak impacting millions of Facebook users. Such incidents can be catastrophic for startups and small to medium businesses operating on limited budgets. Additionally, developers are at risk of mishandling sensitive information or intellectual property related to your project.

Quality control challenges

Failure to properly vet software development companies can lead to partnering with organizations driven primarily by profits rather than a commitment to quality. Conduct thorough due diligence to mitigate this risk by reviewing companies’ portfolios and customer feedback.

Potential for non-delivery

There is always a possibility that the software development company you hire may fail to deliver as promised. They may disappear after collecting upfront costs, struggle to meet your requirements or expectations, provide a flawed product, or become unresponsive. Signing a contract and carefully reviewing its terms beforehand is crucial to protect yourself. Choosing a software development company in a country with a reliable justice system can provide legal recourse if you encounter exploitation.

Language barriers and cultural differences

Hiring software developers fluent in English and capable of seamless verbal and written communication is advisable. This minimizes the risk of miscommunication and misunderstandings that could negatively impact your working relationship and project outcomes. Additionally, your developers’ location in a different time zone may affect decision-making speed and issue resolution. Consider nearshore or onshore developers if this is a concern. Furthermore, be mindful of cultural and political disparities that could hinder effective communication and software development progress, such as cultural differences in values, opinions, and holidays.

The end of the debate

When hiring a freelancer or a software development agency for your project, opting for a software development agency is a compelling choice. The numerous advantages agencies offer, such as seamless teamwork, extensive expertise, superior quality, long-term maintenance and support, and accurate development estimation, make them a preferred option for many businesses.

But remember, ultimately, the ideal choice depends on your project’s requirements, budget, timeline, and risk tolerance. Careful evaluation of the pros and cons, thorough vetting of potential candidates, and clear communication of expectations are essential for making an informed decision that aligns with your project’s goals.

Deyan Denchev — CEO at Lexis Solutions