Probably the only tips one needs for their career

Back in college, I was captivated by programming and machine learning. Coding late into the night and poring over tech blog tutorials became my world. Social events and networking felt like distractions from mastering my craft. But as my career progressed, I realized that technical skills alone wouldn’t get me where I wanted to go. Networking — growing and nurturing relationships — proved to be just as vital for career success.

It’s not just “who you know” but “who knows you” — and the depth of that connection — that truly matters.

Don’t get me wrong, “what you know” is also important. This is the foundation everyone needs to have to start networking. You don’t want to be known by the CEO of the company, but as someone who can’t do his/her job.

After you have enough skillset to excel at your job, who you know will start to feel more and more important. You need to know people and listen to their insights to expand your understanding. You need to know who can grant you access to certain data/information within an organization, who has the authority and needs to be convinced of your solution. Otherwise, it will be difficult for you to utilize what you know to make an impact.

Then, to move up in your career, who knows you will play a serious role.

In my previous company (as is the case in many others), who gets promoted is decided by a group of senior leaders in an annual meeting and they will vote on who gets the promotion. If senior leaders don’t know you well enough, particularly about your contributions, your chances of getting a promotion and a raise are pretty slim, as no one can advocate for you in these meetings. I was fortunate to have worked with several directors, including the managing director, when I was in consulting. This made my promotion to Senior and Principal Data Scientist much easier.

Getting a new job is also somewhat similar. When I moved from consulting to Indonesia’s top private telecommunication company, I had already worked on a couple of projects with them so the HR and VP could easily see the value I could bring to the table.

But how does one grow and nurture their network to gain better opportunities?

I have read several leadership books, watched career videos on YouTube, and asked several professionals with more experience than me on this matter. Up until this post, I can summarize them into two key things on how one can grow and develop their professional network.

To Grow Your Network — Just Go Out and Be Nice To People

If you can even help them, great!

That is it.

Just go out and be kind

Get to know people. Be curious about their stories. Everyone has got some stories to tell.

And you’ve started to be part of a new network.

One key trick here that many people don’t tell explicitly (at least when I asked them for networking advice) is — DON’T EXPECT ANYTHING IMMEDIATELY !!!

Over the years, several colleagues and friends shared their struggle to grow their network with me. Some were looking for a new job. Some wanted to grow their business. A trait many of them shared — they weren’t comfortable networking just for the sake of getting something out of it.

Networking is a long-term game.

Expecting immediate returns often leads to disappointment, especially in today’s competitive job market. Building connection and trust takes time; focus on forming genuine connections rather than asking for favors right away. Imagine you’re at a party, and you’ve just met someone for the first time. You barely know their name, and suddenly you ask them to lend you their car for a week. It’s awkward, right? Just like you wouldn’t ask a stranger for such a big favor without building some rapport first, it’s equally strange to ask someone you’ve just met for a job or referral.

Building a relationship takes time and trust, much like getting to know someone before asking to borrow their car.

You should just go out, meet new people, get to know each other, exchange stories, and that is it. Don’t ask for anything other than contact/business number/work e-mail/LinkedIn (even these should still be gotten with consent).

If for instance, you’re in a networking event. And you recently read about a company facing some challenges in the news. Someone high-level from that company is also attending there. If you are in a relevant role/company that is capable of helping, then you can use that in your discussion and also hint at a potential partnership. This is one of the key ideas of attending networking events.

To Nurture Your Network — Stay Connected, and Exchange Stories or Insights of Value

To nurture your network, it depends on where you are with them.

Nurturing your network is easier when you’re in the same company. Collaborate well on projects or, at the very least, greet colleagues warmly when you cross paths — regardless of their role.

If one of you has moved on to another opportunity, you can keep in contact by asking them how they are doing, saying congrats on their latest professional achievements/milestones, or wishing them a Happy New Year. When I share my career advancements, some network contacts congratulate me, and others even suggest a coffee catch-up. Other times, I also share some news/updates in the industry (tech and artificial intelligence) and ask them for their opinions.

The point here is to keep the relationship alive by maintaining communication. You never know what sort of collaboration you can have with your network in the future, and how you can help each other grow.

A few years back, I collaborated with my senior from my first company in a national machine learning hackathon via Kaggle. We were able to be in the top rank. This experience became a good story for me to tell during an interview for a project in the US (yes, I got the project). Recently I’ve reached out to an old friend from work in Indonesia as well, who is now working in the US. I told him I was in the city and asked if we could meet up. We did not do a project together from this encounter, but it was a fun and insightful reunion. We exchanged our experiences and learned from each other.

Stay Connected

Growing and nurturing your network is an ongoing process that requires genuine effort and patience. It also requires practice. By staying connected, exchanging valuable insights, and building relationships based on trust and mutual respect, you can create a strong and supportive professional network.

Remember, networking is not about immediate gains but about fostering long-term connections that can lead to meaningful opportunities. So, go out, be kind, and take the time to get to know people. Your future self will thank you for the relationships you build today.

The Best Two Tips on Professional Networking was originally published in Practice in Public on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you are a starting software developer/engineer, or just someone who is interested in learning about software architectures, then you must have at least heard of microservices at least once. This article aims to shed some light on the subject.

What are Microservices?

Microservices is an architectural style for developing software applications as a collection of small, independent, and loosely coupled functions, or services. Each service in a microservices fulfills one specific function and works together with other services to fulfill the overall software’s requirements.

What existed before Microservices?

Before microservices, IT organizations typically develop and utilized what is called as monolithic architecture. In a monolithic architecture, the entire software application is developed as a single, tightly integrated unit. All the components of the application, such as user interface, business logics, data storage, etc. are packaged together into a single codebase and deployed as one unit.

This posed several problems:

• Tight integration — Components and functions are tightly coupled within the codebase. Changes to one part of the application may require changing several other parts and redeploying the entire application. Failure in one part of the application can impact neighboring parts.
• Scalability Challenges — since the entire application is tightly coupled, scaling the application, or introducing new features and changes can be very inefficient and resource intensive.
• Development and Deployment Challenges — Monolithic applications are harder to manage. More so as they grow in size and complexity. Developers will likely face difficulties coordinating changes, and deploying updates may require more effort than it appears.

These problems cost companies so much that according to a survey conducted by Statista in 2023, 81.5% of companies in 2021 already use microservices, and 17.5% of businesses plan to switch to this architecture type.

What are the characteristics of Microservices?

Several key characteristics of Microservices are:

• Loose Coupling — Microservices are loosely coupled, meaning they interact with each other through well-defined interfaces without relying on the internal implementation details of other services. This reduces dependencies and promotes flexibility in making changes or updates.
• Scalability — Microservices architecture enables horizontal scaling, where individual services can be scaled independently based on demand. This scalability is essential for handling varying workloads and ensuring optimal performance.
• Resilience — Failure in one microservice does not necessarily impact the entire application. Services are designed to be resilient, with built-in fault tolerance and mechanisms for handling errors gracefully. Hence it reduces the risk of single point of failures.
• Technology Diversity — Different services within a microservices architecture can be implemented using different technologies, programming languages, and frameworks. This allows teams to choose the most appropriate tools for each specific service based on its requirements.

Overall, microservices has become increasingly popular due to its ability to address the challenges of developing large, monolithic applications in rapidly changing environment.

Examples of Microservices

Let’s talk about the example of a microservice.

In a simple (let’s say) web application, a single functionality like login would generally be handled by a single microservice. Take the user’s username/email/mobile number and password, check them against the database, and either deny or grant access.

However, with increasing complexities such as demand for security and third-party integrations (using Google and Facebook logins), the login functionality becomes bigger and harder to manage. With the microservice architecture, we can breakdown the now huge login functionality into several more specific ones. In other words, instead of having one big microservice, it’s a good idea to break it into multiple smaller, easier-to-manage microservices.

Here are some example microservices that handles specific functions within the login’s big functionality:

1. Authentication Service — One microservice might be responsible for authenticating users based on their credentials (e.g., username/password or OAuth tokens). This service verifies the user’s identity and returns an authentication token or session identifier.
2. Authorization Service — Another microservice might handle authorization, determining what actions or resources a user is allowed to access after they have been authenticated. This service checks the permissions associated with the user’s account and the requested resources.
3. User Management Service — Separate microservice(s) might handle user management tasks, such as user registration, profile updates, or password resets. While not directly related to the authentication process, this service provides functionality that interacts with user accounts.
4. Session Management Service — In systems requiring session management, there might be a microservice dedicated to managing user sessions. This service handles tasks like session creation, expiration, and invalidation.
5. Single Sign-On (SSO) Service — In environments where multiple applications or services need to share authentication credentials, a dedicated microservice might handle single sign-on functionality. This service validates authentication requests and issues tokens that can be used across multiple applications.
6. Identity Provider Integration — If the application integrates with external identity providers (e.g., social media logins, third-party authentication services), there might be microservices responsible for interfacing with these providers to handle authentication requests.

How Microservices are Implemented

Microservices can be implemented in various ways depending on the organization’s technology landscape and business needs:

1. Containerization — Microservices can be packaged and deployed as lightweight, portable containers using containerization technologies such as Docker.
2. Orchestration — Orchestration tools like Kubernetes or Docker Swarm can manage the deployment, scaling, and lifecycle of microservices deployed in containers.
3. Serverless Computing — Microservices can be implemented using event-driven architectures like serverless functions, where code is executed in response to events triggered by external sources.
4. RESTful APIs — Microservices communicate with each other via RESTful APIs (Representational State Transfer), using HTTP requests and responses to exchange data.
5. gRPC (Remote Procedure Call) — Services across multiple servers define their interface using Protocol Buffers, and communication is performed via remote procedure calls.
6. Message Brokers — Microservices can communicate asynchronously through message brokers like Apache Kafka, RabbitMQ, or Amazon SQS.
7. Service Mesh — Service mesh architectures, such as Istio or Linkerd, manage communication between microservices by injecting a sidecar proxy alongside each service instance.
8. GraphQL — Microservices can expose GraphQL APIs, allowing clients to query and mutate data using a single endpoint and flexible queries.
9. Domain-Driven Design (DDD) — Microservices architecture can be aligned with domain-driven design principles, where services are organized around specific business domains or subdomains.
10. Polyglot Persistence — Microservices can use different databases or storage solutions optimized for specific data models or access patterns, following the polyglot persistence approach.
11. Continuous Integration/Continuous Deployment (CI/CD) — Microservices can be integrated and deployed using CI/CD pipelines, where automated tests are run, and new versions are deployed to production rapidly and frequently.

Several companies have published how they have incorporated microservices in their business, unlocking scalability and growth. In the case of Netflix, you can read about them here.

Challenges

Microservices architecture is a paradigm shift in software development, offering a modular, scalable, and resilient approach to building complex applications. By breaking down applications into smaller independent components, it can ensure iterative delivery and easier scalability. However, there is no such thing as a free lunch. Microservices has their own challenges, such as maintaining distributed systems, ensuring communication between services, and implementing effective monitoring and observability.

Closing

If you found this post insightful and want to continue the conversation, feel free to connect with me on LinkedIn. I’d love to hear your thoughts and discuss further. :)

Here’s the story

So you’ve built and deployed your model. Be it using simple logistic regression, SVM, random forest, or the infamous deep learning.

The business users are also excited to see its impact. Whether it keeps customers around with the new personalized and targeted campaigns, increased transaction volumes and sales through that up-sell/cross-sell, or whatever KPIs you promised to be achieved may be.

Congrats.

The first couple of months, everything was going great.

Then suddenly you check the company dashboard/reports that suddenly KPIs are going back to its pre-model state, or perhaps it is worse. Stakeholders are bombarding your push notifications demanding answers. They are questioning your model’s performance.

What happened?

The most common explanations are:

1. Your model overfitted. Perhaps it did not take into consideration some key factors, like seasonality. Perhaps you did not properly sample the data.
2. Late data issue. Perhaps the load balancers had a malfunction that it made the system not update the data for a whole day. So either the latest reports are not accurate or the model made its inference on incomplete data. Simply do a count(*) on the tables in question and escalate it to IT/data engineering team. Or, it could be…
3. The data itself has shifted.

What does it mean for the data to be shifted?

It means there is a fundamental change in the data that the model you’ve built can no longer represent the current situation of the business. Be it from internal or external factor. In other words, the data to which the model is trained upon is no longer relevant, therefore the model is outdated.

This is very much likely to happen to business everywhere. Take a look at covid. Remember how much change it brought? Or the current hot news — Inflation. Both of which causes customers’ and businesses to change the way they behave and work in a significant way.

Or let’s take a more simple, common, and less apocalyptic, example. Say you are working in a big telecommunications company. One of your competitors are offering a huge discount in their prepaid packages with generous benefits. Something which no other company has ever done before. Turns out, your market loves this so much that they decided to abandon you for your competitor. It’s not you, it’s them.

All of these are external factors. What about internal ones?

Well, this is definitely your companies’ doing. A change in policy/management. The business is growing/loosing more money than it is making overall, so there are new products/cutbacks. Perhaps they soft-launched a new product variation that is very unique compared to existing ones. Or your star retail employees for 10 years has left/retired/finally use their holiday allowances at once, and customers are not loving the replacements’ service.

Can this be avoided? Yes. If your businesses captures all of these events into your data storage. Which, you can understand, if you’ve worked for any company, to be extremely difficult and costly to do. So, you’ve just got to do with what you have.

This is why it is very important to implement a model monitoring practice in place BEFORE the results are sent out. Every time the model gives inference on the latest data, you need to look out for these data shifts before the results are given to business users. If all is good, then the results can be blasted. If not, and depending on the severity, you can either easily fix it quick or raise awareness that something different has happened AS WELL as the measurable proof.

How can you send measurable proof that the data has changed?

There are three easy ways to do it:

• Descriptive Statistics

A simple time series report could easily tell if there is a shift in your data. For instance, a dip in a simple monthly revenue MoM trend is a clear indicator your business is not doing so well overall. If the business keeps doing worse every month, it’s a matter of time until the model won’t recognize the sales data as it used to.

• Population Stability Index (PSI)

This index basically measures the population of the model’s result and how much of it has shifted class/groups. You need to use the latest data and compare it to when the model performed good, e.g. its training data, or the month after that (which also performed good).

Let’s say the model produces N classes/groups/categories. Or it could also be binary classification like churn, in which case, you can for example take the probability of the customers churning, bin them into N equal or non equal groups. For instance, 0–10% as group 1, 11–20% as group 2, etc. It can also be like 0–50% group 1, 51–60% group 2, so on. The important thing is consistency throughout the entire process. Determining this bin could require some business acumen as well, as different bins from the same data and model would impact the model monitoring metrics significantly.

Simply count how many cases/customers fall within those groups for both the training data and latest one. Take the percentage of their respective total data. Then multiply the difference between the training (DT) and latest data (DL) with the natural log of DT/DL .

This is an example of the calculations, you can try to recreate the formula in Excel.

The rule of PSI is:

• At least one bin >20% — Data shift definitely has occurred. Retrain the model. If not, then
• At least one bin is 10–20% — Slight change is required. This would likely give a bit of a drop in the model’s performance. If not, then
• Less than 10% — no significant data shift. Carry on

We can see the above example already has two groups whose PSI is >20%. Therefore we need to investigate what happened to our customer’s behavior and retrain the model.

Bear in mind that these thresholds are not fixed. It depends on how much you and the business are willing to tolerate the change. For instance, in bin 3 you can clearly see theres is a huge difference in absolute numbers but the PSI is small.

• Characteristic Stability Index (CSI)

If PSI determines if there is a data shift on the population, then CSI is to determine which features impacted it. The calculation is basically the exact same one as the PSI. Only difference is, we drill down into the problematic bins (in the example above, group 5, 8, and 9) and group them further based on the features.

Let’s say you have 10 features. Age. Monthly expenses. Outstanding bills. Whatever the case may be. Group them into accordingly and do the same calculations as you did in PSI.

The below example is 2 features’ CSI for bin 8.

From the above example, we can speculate that for bin 8, the number of young customers has risen and elder ones have declined. So much so that it has caused a shift in our data for these bins. Your monthly expenses for this group has also reduced significantly.

Key Takeaways

Descriptive statistics, PSI, and CSI are very simple and quite effective metrics in monitoring your model’s performance. But one thing that is better than these metrics in determining if the data has shifted, is business and market understanding that is regularly updated. Always stay updated to your business strategies, market, and customers needs.

Monitoring Model Performance was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Setup configurations for Docker is different for Windows than it is for Mac and Linux

If you are starting to learn about setting up a system environment using Docker, then you definitely want to make sure it uses the right amount of memory and processing power.

After installing Docker in Windows, you might think that changing these settings would be as easy as changing configurations in VirtualBox (via UI).

Well, not quite.

Unlike Linux and Mac, we cannot directly change the configuration via Docker’s UI.

The latest release of Docker for Windows requires that we install the Windows Subsystem for Linux version 2 (WSL2) that allows you to configure global options that will be used by all WSL2 Linux distributions installed in Windows 10.

And to change the setting for Docker means that we need to create a .wslconfig file. Note that there are two types of config files that we can create. The .wslconfig file would let us create a global configuration. The other one, wsl.conf allows us to configure settings per -distribution for Linux distros in our machine. You can learn more about it here.

1. Make sure you shut down Docker and any other instances of wsl2. You can do this by going to a command prompt and type the command wsl — shutdown.
2. To create a .wslconfig file, simply open your File Explorer, and type and enter %UserProfile% to go to your profile directory in Windows.
3. Docker or WSL2 by default does not create these config files so we should do it ourselves. Create a new file called .wslconfig (make sure there are no .txt at the end)

4. Add the (either all or some of the) following commands in that file to configure docker

# Settings apply across all Linux distros running on WSL 2
[wsl2]

# Limits VM memory to use no more than 4 GB, this can be set as whole numbers using GB or MB
memory=4GB

# Sets the VM to use two virtual processors
processors=2

# Specify a custom Linux kernel to use with your installed distros. The default kernel used can be found at https://github.com/microsoft/WSL2-Linux-Kernel
kernel=C:\\temp\\myCustomKernel

# Sets additional kernel parameters, in this case enabling older Linux base images such as Centos 6
kernelCommandLine = vsyscall=emulate

# Sets amount of swap storage space to 8GB, default is 25% of available RAM
swap=8GB

# Sets swapfile path location, default is %USERPROFILE%\AppData\Local\Temp\swap.vhdx
swapfile=C:\\temp\\wsl-swap.vhdx

# Disable page reporting so WSL retains all allocated memory claimed from Windows and releases none back when free
pageReporting=false

# Turn off default connection to bind WSL 2 localhost to Windows localhost
localhostforwarding=true

# Disables nested virtualization
nestedVirtualization=false

# Turns on output console showing contents of dmesg when opening a WSL 2 distro for debugging
debugConsole=true

You can configure it however you want depending on your system’s need and available resources.

Once those are done, you can start your Docker and continue development.

Hope this is helpful in your journey to learn about Docker. Let me know if you have questions by replying to this post

Sources

1. https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig
2. https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container

I previously wrote about a simple way to plot maps using Geopandas and Matplotlib.

A Beginners Guide to Create a Cloropleth Map in Python using GeoPandas and Matplotlib

Now another very common geospatial analytics use case is to tag a specific location to a predetermined location/area/region/etc. For example, let’s say you are an analyst for a brick-and-mortar store that wants to map the amount of competition in their key areas. These key areas can either be a business predefined area or based on predetermined country regions (like a province, district, etc.).

Your team already collected enough list of competitor addresses and their lat-long coordinates. Either purchased from a third-party company, or an arduous Google Search exercise, or both. So now you are tasked with mapping these addresses’ latitude and longitude with let’s say, a custom city name. Your company already has the shapefile containing the strategically drawn borders. All you need to do is to tag the City name to competitors’ locations.

You don’t need any fancy algorithm, GeoPandas package already got your back!

For this exercise, we will use the Indonesian province shapefile which you can get here, and some random Indonesian malls’ addresses that I gathered. You can get it along with the complete source code at my Github repo.

The first step would be to load our required packages and then our reference shapefile and list of addresses that we would like to tag.

Now, Geopandas is actually a very powerful geospatial package that makes this task so easy. It already provides us with the within command so we only a few short lines of codes!

The within code checks if our addresses’ latitude and longitude points are within our reference shapefile. Note that we need to check it with one province reference at a time. Otherwise, we would not be able to properly tag it (since two addresses can be in two different provinces, we would have difficulty correctly tagging it if we did it on all references at once).

And that’s it! Hopes this tutorial can be of use to your geospatial analysis. For the full source code and data, you can check them out on my Github repo.

Tagging a Location to a Shapefile Area using Geopandas was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Mini can get the job done, even better than the big snake

When I first started out my career as a data scientist, one of the tools I kept getting recommended is Anaconda. It is even said so in the original docs.

For a while, I also thought Anaconda was cool. I mean look at the first page of Anaconda Navigator that you see once you open the tool.

When I first saw that landing page, during my first tenure as a data scientist, I felt like how a kid imagined a real scientist works, in front of his complex and awesome tools.

All I needed then to complete the set was a pair of glasses.

I also agreed at the time why Anaconda was so recommended for data scientists. All the tools, including extra packages, code editors, and viz tools, were already provided.

One or Two Years later…

After working on dozens of analytics use cases, each with its unique challenges and requirements, I have grown wearier and wearier of Anaconda.

In summary, it became one bloated useless app on my computer.

1. The thousands of packages are total storage and memory eater. And even worse, useless. All those preinstalled packages really weighed my computer’s performance down. Storage taken was a few GB, which can make a pretty big difference, especially on a Macbook Pro 128GB. Even with conda clean --all it is still taking up quite a space. I did not use many of the packages that came with Anaconda, like pomegranate, proj4, pyopengl, and so many more. I’m not sure what they all are for.
2. Managing the Python packages became a very slow process. Even updating a single package withconda update [some package] felt like it dragged on for way too long. I have tried using from only one environment (not really recommended, by the way) to different virtual environments with specific packages for different use cases (one for data exploration, one for numerical analysis and modeling, and another one for image processing). Both were still slow.
3. The Navigator became obsolete. I can just open my preferred code editor app either from the Start menu/Application folder or via command prompt/Terminal. In fact, opening the app through Navigator is even way slower. Updating and managing packages is also more convenient via command prompt. Making new virtual environments? Open command prompt/terminal and type in the conda command. Why bother with the Environment page in Anaconda Navigator. Even more so, why bother with it at all?
4. Makes my job a lot harder. Once the deployment schedule arrives for some of my clients/users, I have to create a virtual environment with the same required packages and config to ensure the model(s) can run smoothly in their production server(s). For the exact same reasons that I have just listed above, no way I was going with Anaconda (or likely ever).

So what’s the alternative…

Miniconda!

Basically, it is just the conda package management system + Python + its base packages.

That’s it.

• No extra (useless!) tools and installations.
• Need some packages for a specific use case/project? You can still create virtual environments and install just the packages you need there.
• Need some code editors installed? Just download directly from the official web. Directly managed by the developers, and just as good (or even better) than the one installed via Anaconda Navigator.

With Miniconda, I actually felt like making lean data science and analytics projects.

Conclusions

I am not saying Anaconda is an outdated tool that no data scientist/analyst should ever use. It still has its potential. There is even an Enterprise version of it. I think first-time data scientists/analysts could benefit from starting their career/training using Anaconda. Like, learn what is an optimal virtual environment for your data projects. They might even have some uses for those 1,500 extra packages. Who knows.

I am saying that if you have a firm grasp of what you want to build or use with Python (e.g. you want to be a time series expert, deep learning engineer, or a data-driven marketing specialist), Miniconda is an efficient and recommended tool to use.

Switch from Anaconda to Miniconda for your Data project environment was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

A standard, but crucial, functionality in image processing tasks

There are several use cases in image processing that can be solved if we know the most common color(s) of an image or object is. For example in the field of agriculture, we might want to determine the maturity of a fruit, an orange or strawberry for instance. We can simply check if the color of the fruit falls in a predetermined range and see if it is mature, rotten, or too young.

As usual, we can solve this case using Python plus simple yet powerful libraries like Numpy, Matplotlib, and OpenCV. I will demonstrate several ways on how to find the most frequent color in an image using these packages.

Step 1 — Load Packages

We’ll load the basic packages here. We’ll load some more packages as we go along. Also, since we are programming in Jupyter, let’s not forget to include %matplotlib inline command.

Step 2 — Load and show sample images

In this tutorial, we will be showing two images side by side a lot. So, let’s make a helper function to do so.

Next, we’ll load some sample images that we’ll be using in this tutorial and show them using the function above.

Now we are ready. Time to find out the most common color(s) in these images.

Method 1 — Average

The first method is the easiest (but ineffective one) — simply find the average pixel values.

Using numpy's average function, we can easily get the average pixel value across row and width — axis=(0,1)

We can see that the average method can give misleading or inaccurate results, as the most common colors it gave are a bit off. This is because the average took into consideration all pixel values. This will be really problematic when we have images with high contrast (both “light” and “dark” in one image). This is much more clearer in the second image.

It gave us a somewhat new color that is not visibly clear/noticeable in the image.

Method 2 — Highest Pixel Frequency

The second method will be a bit more accurate than the first one. We’ll simply count the number of occurrences in each pixel value.

Fortunately for us, numpy again gives us a function that gives us this exact result. But first, we must reshape the image data structure to only give us a list of 3 values (one for each R, G, and B channel intensity).

We can simply use numpy ‘s reshape function to get the list of pixel values.

Now that we have the data in the right structure, we can start counting the frequency of the pixel values. We can just use numpy's unique function, with the parameter return_counts=True .

Done, let’s run it to our images.

This makes more sense than the first one right? The most common colors are in the black area. But we can go further. What if we take not just one most common color, but more than that? Using the same concept, we can take the top N most common colors. Except, if you look at the first image, many colors with the highest frequencies would most likely be neighboring colors, probably with a difference of a tiny few pixels.

In other words, we want to take the most common, different color clusters.

Method 3 — Using K-Means clustering

Scikit-learn package comes to the rescue. We can use the infamous K-Means clustering to cluster groups of colors together.

Easy, right? Now, all we need is a function to display the clusters of colors above and display it right away.

We simply create an image with a height of 50, and a width of 300 pixels to display the color groups/palette. And for each color cluster, we assign it to our palette.

Beautiful isn’t it? K-Means clustering gives great results in terms of the most common colors in the images. In the second image, we can see that there are too many shades of brown in the palette. This is most likely because we picked too many clusters. Let’s see if we can fix it by choosing a smaller value of k.

Yep, that solved it. Since we use K-Means clustering, we still have to determine the appropriate number of clusters ourselves. Three clusters seem to be a good choice.

But we can still improve upon these results plus still solve the number of cluster issues.

How about we also show the proportion of the clusters towards the whole image?

Method 3.1 — K-Means + Proportion display

All we need to do is to modify our palette function. Instead of using fixed steps, we change the width of each cluster to be proportionate to how many pixels are in that cluster.

Much better.

Not only it gives us the most common colors in the images. It also gives us the proportion of occurrences of each of the pixels.

It also helps answer how many clusters should we use. In the case of the top image, two to four clusters seem reasonable. In the case of the second image, looks like we need at least two clusters. The reason we don’t use one cluster (k=4) is that we’ll run into the same problem as the average method.

Conclusion

We have covered several techniques to get the most common colors in images using Python and several well-known libraries for it. Plus we’ve also seen the advantages and disadvantages of those techniques. So far, finding the most common colors using K-Means with k > 1 is one of the best solutions to finding the most frequent colors in images (at least compared to the other methods we’ve gone through).

Let me know if you have problems with the script in the comments, or in my Github.

Finding Most Common Colors in Python was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the responsible things to do when a year is ending is to reflect on it. What accomplishments you have made, what challenges did you face, what did you learn, and how you can make the remainder of the year count.

One experience that I can definitely share, and hopefully it would be beneficial to readers, is being awarded the 2019 Bertelsmann Tech Scholarship and receive the Deep Learning Nanodegree from Udacity, completely free of charge. And this year, Bertelsmann Tech is opening another scholarship application, which you should definitely try if you have a passion for data and cloud tech.

Many people have asked online what it was like to apply for the Bertelsmann Tech scholarship, win it, and complete the Nanodegree from Udacity. Also, what benefit can one receive from achieving all that? So hopefully this article can help answer some of those questions that you might have.

Let’s begin!

1. What exactly is the Bertelsmann Tech scholarship

Well, Bertelsmann is a media, services, and education company that operates in about 50 countries around the world. Their CEO, Thomas Rabbe, spoke that it was their mission to empower many individuals across the world with the increasing demand for digital skills. They work together with Udacity to provide free nanodegree courses that would otherwise be expensive if paid in full personally.

In 2019, they offered nanodegree in Data Analysis, AI/Deep Learning, and Data Science. This year, they are offering three new programs. I’ve included the link below if you want to find out more.

2. Applying for the Scholarship

Last year’s application required applicants to write short essays about (1) why should you receive the scholarship, and (2) what do you plan to do with the skills that you will have acquired in the scholarship or how you would have benefitted from this scholarship. We were also asked about our confidence in our Python skills.

This year though, the application is a lot more simple and easy. You simply need to answer a few questions about yourself and your current skills in Python and SQL.

3. How the selection is done

There are two main stages that you have to pass in order to receive the full scholarship.

The first one is the application stage. There are thousands and thousands of people around the world applying, and only about 10k-15k are accepted into the three programs, that means about 3–5k for each ND program are selected to pass the application process.

The second stage is called Phase 1. Phase 1 is where you are given the basics or fundamental courses of the ND program that you applied for. To see which candidates can actually finish the entire course. The duration is about 3 months, the same as last year. Of the 3–5k applicants, only about 300–500 (10%) are awarded the full scholarship, i.e. Phase 2, where the awardees begin the full ND program and finish the remaining courses.

That’s it. after that, it is up to the scholars to finish the ND in 6 months.

Below is the timeline for this year’s scholarship

Do not be fooled, these time window may seem long, but participating in Phase 1 and Phase 2 is not as easy as it seems. Especially with Covid-19 pandemic. It took me longer than I expected to finish the course thanks to Covid-19 and the “new normal” everyone had to adapt into.

4. My Experience

Now, we get to the fun part. I will share with you my story from applying all the way to graduating.

4.1 Application process

In 2019, I chose the Deep Learning program because I believed the resources (learning materials, forums, tips and tricks, best practices, etc.) for Deep Learning are still harder to attain than the other two if I chose to learn by myself in other online courses or websites.

I wrote the essays and submitted my application. One or two months later, I received this.

I was so happy and ecstatic that my application stood out and I got accepted into Phase 1.

The key was that I had to really know what I want out of the program and why I (and also Bertelsmann) would benefit from this receiving (and giving) of scholarship.

So, if there is another scholarship with the similar application process, the key here is that You really, really have to sell it.

But this year is a bit simpler, and therefore in my opinion, harder. Why? Well, with no essays in the application and only a few checkboxes, it can be different to set yourself apart from the competition.

But, that shouldn’t stop anyone from trying, am I right? :)

4.2 Phase 1

This is where the fun of the Bertelsmann Tech scholarship begins.

So, in each ND program, the students were gathered into three global Slack groups, each according to the program that they applied for. And each student had been given access to the basics or foundational courses for their ND programs.

The instructions for this phase, and to get to Phase 2, were simple and straightforward.

We had to pass the fundamental courses in time AND actively participate in the Slack groups.

Sounds simple right? Complete the course and participate in the group Slack for 3–4 months, easy right?

Not so fast,

Because it was “easy” and “simple”, everyone could easily do it, i.e. the competition is also fierce and it was easy to slip up. The most common slip ups people made were:

1. They finished the course so fast, got bored, and no longer participated in the group
2. They finished the course in time and participated in the group, but not enough

Mistake #1 was likely due to boredom, urgent matters in the office, etc. The point is they did not consistently participate in the Slack group. Mistake #2 is the most dangerous. Because many people did finish their course in time, they did participate in the Slack groups, but they got drowned by those who were way more active. Some of them even complained to Udacity, saying that they disagreed/protested the selection outcomes for their friends. These highly active people usually participate in study group sessions (we called them Study Jams), posted in the Slack groups on a daily basis or a few days in a week, either useful thinks, answers to other student’s questions, motivational quotes, or possibly even a funny meme to relax.

The point is, the ones who passed onto Phase 2, were those who“madly” wanted these scholarships AND they showed it. In ALL 3–4 MONTHS of Phase 1.

What did I do in Phase 1? I gathered with my fellow Indonesian scholars after office hours, worked together in the course, organized and participated in regular meetings, and kept working on it. I even published one of the projects I made from the skills I got in Phase 1.

Chinese Zodiac Sign Classification Challenge with Pytorch

And after a lot of hard work, those 3–4 months felt way longer. But it was worth it, especially with this e-mail.

4.3 Phase 2

Now the real hard work begins.

There were 5 main courses, 5 projects that we had to submit before 17th of Sept. Plus one extracurricular if one feels extremely motivated. Working on these 5 projects were not the biggest challenge. It was Covid-19.

The world was thrown a huge wrench into its machine and everyone around the globe had to adapt to a “new normal”, including in Indonesia. Work from office from 9 AM to 6 PM turned to Work from Home with longer, random work hours. Sometimes it was still the same, other times you had to work from the moment you opened your eyes until 10 PM. It was really unpredictable. I couldn’t plan my time properly. I had to prioritize my work first, of course. Since many people were laid off because of the pandemic.

I had no choice but to work on my nanodegree late at nights and weekends full time. It was exhausting.

Nearly six months had passed. In the end, with the help of praying, coffee, Google, and Udacity’s mentors and student hub, I managed to finish it before the big deadline and graduated. Plus I managed to do it without sacrificing my health.

Certificate of graduation from a Udacity Nanodegree program

4.4 What did I get

That was the process and journey in 2019’s Bertelsmann Tech Scholarship and Udacity’s Deep Learning ND. I was also asked, what benefit did I get from doing all this. Here are some of them

1. I met and befriended lots of awesome people with the same drive and passion during the scholarship and nanodegree. We exchanged ideas and awesome topics and motivated each other to progress. I still have contact with some of them. Who knows how we will benefit from this network in the future. And who knows whom you might meet in 2020’s Bertelsmann Tech Scholarship, too
2. No doubt the skill and knowledge was as good as I thought. The lecturers, mentors, and student hub were so amazing. The lectures were relatively easy to follow and understand. The practice projects were also very intuitive, they also provided the solutions as well in case we get stuck with one or two lines of codes or the concept behind them.
3. The projects were very unique. The experience I got from working the Nanodegree projects were unbelievably so good. The problems were unique. The cases were real. Udacity also provided students with 99 hours of GPU enabled workspace to run our codes and model faster than using our own private laptops. They also gave us a set of clear objectives for us to achieve in the project (min accuracy, max loss, clear and concise explanation of our logic/thought process). No other online course could have provided the same experience for me.

Overall, it was an amazing experience. Udacity did not exaggerate the term nanoDEGREE for their Deep Learning ND. It almost felt like a mini master’s program. One that I recommend to others to achieve.

Last words

Hopefully, this post is helpful to those who want to apply and learn in the Bertelsmann Tech scholarship and Udacity ND, or at least motivate you to keep learning.

“Stay hungry, stay foolish” — Steve Jobs

“Stay safe and healthy” — Anyone with common sense

My Experience as a Bertelsmann Tech and Deep Learning Nanodegree Graduate was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you have worked for a company or startup for many years, it is very likely that you have dreaded Mondays. Be it because of a looming deadline(s), annoying co-worker(s), demanding boss(es)/client(s), or because of the nasty traffic caused by construction work for who knows how long. We have all felt something uncomfortable about going back to work after a nice relaxing weekend that makes us roll back into the comfort of our beds and not have a care in the world.

To make it worse, the mere thought that tomorrow is a regular, non-public holiday Monday, can cause the anxieties to be way worse. There is even a name for that, the Sunday Scaries. It can prove to be an even bigger annoyance. You waste your Sundays thinking about Mondays and stress yourself even more.

Dude, I never felt anything like that. I love my work so much. It is my passion. It is my calling on this earth.

Well, either you are a fresh graduate and just starting your first job, or you are about to feel the dread. No matter how much you love something, you will eventually get bored/find something wrong with it that makes you love it less.

Why is this such a big deal?

Worrying or hating so much about something will give you stress, which obviously can be bad for our health, either physical or mental. Especially when we are doing it almost every week. Our immune systems get weaker the more stressed we are.

We can also develop more negative thoughts about our work — anxiousness, paranoia, etc. It can also make your work productivity suffer. Or it can put a strain in your relationship with coworkers or (worse) your boss.

These are pretty good reasons for why we have to manage our Monday dreads. It may not seem much of a big deal now. But if one carries this out for a long period of time, it will affect your day to day lives. Work productivity. Health. Personal relationships. You name it.

So what do we do about it?

Back in the early days of high school when I started feeling these Monday dreads, I go tell my school counselors or close friends or my mom, hoping to get some advice on how to deal with it. But even in the career world, the advice we get to deal with these Monday dreads is usually in at least one of the following archetypes.

The motivational speaker/saint — “Try looking at it in a new way. Today is a gift, you should be grateful you are given the chance for another adventure”

The pragmatic/tough guy — “You are not the only one, deal with it and get the job done”, “Don’t whine, you’re a man, so shut up and take it like one”

They are all good, reasonable points. But if you are the kind of person who prefers doable realistic actions along with the shared wisdom/advice, then we are pretty much alike. I mean, what good is advice if you can’t act on it, right? Yes, a smile can help, but sometimes our negativity is way heavier than putting a frown on your face (cause, you know, it takes more muscles to frown than to smile). I needed something more than that.

Fortunately, with some years of experience, and a little bit of research (especially after I started focusing on my career), I learned to better manage my Monday dreads.

Here are some of the things that helps me, and hopefully you as well.

1. Acknowledge It

Feeling nervous, panicky, or anxious that Monday is near is completely fine. It is completely natural. As I’ve said before, everyone has felt it.

What’s not fine, is if you respond to it in an unproductive, negative, or another bad way. And this is where all the difference is.

What good does incessantly looking at the clock all day on a Sunday do? Hoping you still have more hours left in the day? Nothing. In fact, it can be worse than nothing. You end up worrying all day, with nothing to show for it. You could have spent more time relaxing, going to the mall shopping for new clothes, eat at a fancy restaurant, playing with your kids, spending quality time with your family, the list goes on.

If you acknowledge you are stressed and hating Mondays, you have a better chance of actually dealing with it. This is how so many professionals are able to return on Mondays and do kick ass at work (among many other qualities). They know there is an issue that needs to be addressed and they do just that.

Acknowledging that there is a problem is the first step towards fixing said problem.

2. Plan for your next week

“If you fail to plan, you plan to fail” — Benjamin Franklin

There are things in this world which we can control. And those which we cannot. That is an undeniable fact.

It can be somewhat depressing when we enlist all things that can go wrong and outside our control. So it makes perfect sense to do everything we can so that those within our control goes in our favor, not against us. Nothing short of proper planning from the beginning followed by good execution will do the job for this. Even if your week still sucks afterward, you can still find comfort in knowing you did what you could.

A piece of good advice I got was to use my Friday to plan my next week. The reason being is that Friday I was done with work for the week. My brain was still humming with all the tasks I had to do next, all the notes and feedback from my supervisors are still fresh in my head. I can still think about work and the momentum from the day helped to keep me going.

3. Keep away from anything that reminds you of work

We are feeling the Monday dread, likely because we are in dire need of spending quality time outside of work. Either with friends, family, or just ourselves. Go to the mall, to the movies to see the new Marvel movies (oh, how I miss this in this hard time of Covid-19), that new restaurant you heard your friends talking about. Or take a short trip to someplace nice. Or just cover yourself under a warm blanket and watch Netflix or read one of your favorite books. Anything you enjoy.

Another pro tip I got from one of my friends, is to shut down (at least set to silent) my work e-mail and Whatsapp work group. I was very surprised at just how effective this method was in reducing work stress. I wasn’t able to fully relax, to enjoy my weekend because a percentage of my brain is still allocated to work. In the end, I ended up having no real weekends, and no work done either. One or two days without thinking or be reminded about work turns out to be a necessity.

4. Little to no work on weekends

I know. This contradicts the previous point. No one likes to work on weekends. It is the time to rest and relax, starting after coming home from work Friday.

But, there are some times where some coworkers/clients/boss sent a late email, sending you the required materials for you to do your work, and they ask you to present it first thing next week. No one likes that. I’m sure not even they like having to send a late email on Friday nights or, hopefully on more less often, rarer occasions, weekends in the first place (is what I tell myself to be positive and not complain).

Which is why I cram everything on Friday night. Prepare a fresh cup of coffee, order pizza or a burger to aid me. Shut off any distraction (Youtube, Netflix, Instagram, etc.) and just try to reach that goal of finishing that last annoying report and send it to whoever is supposed to receive it. I do not want ANY work to do on weekends. If it is not possible to finish by Friday night, what can I do? But I won’t go crazy with it. I’ll just allocate three hours on Sunday night to do it.

5. Communicate

If you cannot change the situation yourself and still feel like the office is a bleak house, it’s time to ask for help. It’s not wrong if you talk to your supervisor/manager about your work condition and ask for help. Ask to appoint another team member to help carry the massive load. Or to be moved to other less demanding projects that you can help. What’s wrong is if you clearly need help and you keep the problem to yourself.

Often times it is not just a one-time quick meeting. Your boss may need some convincing that you need help. More than once. Show him your workload and the unreasonable demand behind it. That it is having a negative impact on yourself. Missing deadlines, getting sick more frequently, or whatever the case may be.

6. Find a new place

Sometimes the problems are embedded so deep, it becomes too difficult to fix. Whether it is poor budget allocations for new staff, difficult boss(es), or any other forms of toxic work culture. If we cannot change it despite our best efforts, maybe your job isn’t right for you. Maybe you need to ask yourself if its time to move on.

Work is not always great or easy, but it should never make you feel uncomfortable, anxious, or depressed.

Final words

Hopefully, this post can help you. It may take time to grasp this and apply it to our lives and keep practicing to make it more natural. But once you understand and practice this consistently, you will soon realize the Monday dread is getting smaller, by a lot. And this is obviously no small win, at the very least, for your career.

Whenever I want to write Python code, whether it’s for testing or deployment purposes, I will just open up Jupyter Notebook and start writing. And why not, right? It’s free, open-source, and everyone seems to love it (for example).

And if your company wants to deploy your code on their production environment, you can just use Anaconda Enterprise to do it.

But, if you need to write Python modules in your project, it can be less than ideal to write and test the module in Jupyter, save it as Python file, and then including it in your main Jupyter notebook and all that.

Plus, the downside of deploying with Anaconda Enterprise? — well….

Which is why I looked for an alternative tool. One that can save the hassle of writing production, modular codes, but in which I can still run codes line by line, and also display the data and variables inline.

In comes Hydrogen package for Atom!!!

What is Hydrogen?

Well, like the tagline in the official documentation says

… All the power of Jupyter kernels, inside your favorite text editor.

And so far, it is true and I’m loving it on my MacOS 10.15. It turns your Atom text editor into a Jupyter-like notebook! Plus Atom is a lightweight text editor with plenty of packages to make it an awesome code editor.

This post will walk you through the steps and workarounds to the issues I’ve faced (and some will likely face as well) during installation on MacOS 10.15.

1. Install Atom

This is, of course, a very easy step to do. Just head over to https://atom.io and download the latest version of Atom. Once downloaded, simply copy and paste the Application file to your, well, Applications folder.

“Then just double click to open it, right?” — Well, not really.

Many Mac users, including myself, found that we cannot open the editor, and were greeted with a message like this.

“Atom 2” can’t be opened because Apple cannot check it for malicious software.
This software needs to be updated. Contact the developer for more information.

But don’t worry.

Just right-click the Application file, and then click “Open”.

2. Install Hydrogen Package

Click on Atom > Preferences > Install > type “hydrogen” and click Install.

Once it is done, you will need to install the kernels for Python, otherwise, Hydrogen will not work. Open up your Terminal, and type in the following code.

python -m pip install ipykernel
python -m ipykernel install --user

This will install the IPykernel, which is a kernel for Python so that Hydrogen can make Atom execute Python code like how Jupyter does.

Once it is done, restart Atom, and you will be able to run your code line by line and even display your pandas data frames and other visuals you normally make in Jupyter. To see the command shortcuts, you can check it on Packages > Hydrogen and there you will find the list of shortcuts you can use. You’ll find they are quite similar to those in Jupyter.

To run only a selection of lines, just select which line(s) of code you want to execute and press the Run shortcut.

Awesome, right?

To see the full set of Hydrogen’s capabilities, you can check them on the official page.

Plus — Autocomplete

Lastly, it wouldn’t be much of an editor if there are no autocomplete.

Hydrogen already provides an autocomplete function, but when I tried it, several variables or built-in Python functions were not shown. Therefore I recommend you add the autocomplete-python package.

Click on Atom > Preferences > Install > type “autocomplete-python”. and click Install.

If you find some troubles in autocompleting basic Python keywords and variables, it may be that the grammar file is not copied correctly. To fix it, you can try the following code in your Terminal.

cd ~/.atom/packages/autocomplete-python/lib/jedi/parser
cp grammar3.6.txt grammar3.7.txt

Final words

Hope this post is useful and that everyone has a better time writing complex, modular machine learning, data preprocessing, or data analysis codes. Especially during these hard times due to the Covid-19 pandemic.

Atom’s Hydrogen for writing data science projects using Python was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.