Avil Page

Moving from RayCast to TinyCast

Anand Reddy Pandikunta — Sat, 19 Sep 2026 09:33:06 GMT

I have been using RayCast from ~4 years. I built 2 extensions, ~20 custom scripts, contributed to existing extensions.

I raised support tickets related to automating RayCast exports & settings. Unfortunately, there was no good resolution.

FOSS Alternatives

When looking at open source alternatives I stumbled on Vicinae, RustCast long back but they were not drop in replacements.

Recently TinyCast came out and I am using it as a primary launcher on one of the Mac. It is working well as a drop-in replacement so far without any major issues.

It is just ~6MB app and consumes only a fraction of CPU/Memory compared to other tools.

Software as Moat

With AI, it is getting easy to build & customise software. There is a growing trend where users are shifting towards open source alternatives(since I am in FOSS space, I might be biased).

If you think software only is a moat, it is time to re-think again!

How I run LLMs locally on Mac

Anand Reddy Pandikunta — Thu, 27 Aug 2026 17:03:44 GMT

Harness

OpenCode & oh-my-pi come with batteries included and works out of the box.

Pi is barebones & lightweight. There are quality plugins available to extend the functionality & customise the behaviour. It is extremely extensible.

Inference Server

Ollama MLX doesn't generate tokens efficiently.

Rapix-MLX, omlx are excellent at utilizing resources and generating tokens efficiently. With M5 Pro 48GB RAM, they are able to generate ~50 tokens/second with Qwen3.6-35B-A3B-4bit model.

Llama.cpp is cross-platform and it is extremely fast as well.

Local LLMs

Qwen3.6-35B-A3B model made locall LLMs usable on decent hardware.

Qwen3.6-35B-A3B - has 35B parameters & only 3B are active at a time. I use 4bit quantized model as it provides a better tradeoff between speed and accuracy. It fits in 48GB RAM with some headroom for other applications.

Bonsai has highest intelligent densisty and it can run on ~5 GB RAM.

It is surprising how fast & powerful the local LLMs have become.

Handy - Best cross platform Dictation app (Free)

Anand Reddy Pandikunta — Tue, 28 Jul 2026 23:49:39 GMT

Context

Around 2016, I explored various speech-to-text (STT) tools to reduce strain on writs due to RSI(Repetetive Strain Injury) and gave up on them due to their limitations.

With AI advancements, these tools have come a long way.

Open Source STT Tools

There are couple of paid tools but they didn't work well in noisy environment.

After trying out all these tools with local models like Parakeet, Voxstral, and Cohere Transcribe, I found that Handy + Cohere Transcribe is the best combination for my needs.

Handy has post processing as an experimental feature where we can send output of STT to other LLM models and get a fine grained output.

For example, when I dictate "git push dash dash force", the output is "git push --force" after post processing.

Conclusion

I let AI to write handy_stats.py script to track my dictation stats. I am dictating more than 10K characters per day which reduces strain on my fingers/writs.

Moving from TailScale to NetBird

Anand Reddy Pandikunta — Sat, 27 Jun 2026 05:43:54 GMT

Why?

I have been using TailScale from 4 years. Recently, I wanted to self-host HeadScale(open source TailScale server) on my own server.

During self hosting, I realised that HeadScale is not a drop-in replacement for TailScale.

App connectors are main reason I was using TailScale and HeadScale doesn't support app-connectors.

NetBird

NetBird is completely open source and has a self-hosted server. Setting up split DNS tunneling and routing is easier with NetBird.

It has a clean web UI to manage network/policies which is way better than TailScale's web UI.

It also has a cool control center to visualize the network and connected devices.

It still doesn't have support for light theme yet. Since I am not accustomed to dark theme, I am using Stylus extension and changing the theme to light theme.

html { filter: invert(1) hue-rotate(180deg); background: #fff; }
img, svg, video, canvas, [style*="background-image"] { filter: invert(1) hue-rotate(180deg); }

Conclusion

NetBird can be self-hosted in few minutes and migration is much smoother than I expected.

Everyday AI Tools

Anand Reddy Pandikunta — Sun, 31 May 2026 17:47:04 GMT

CLI Tools: GitHub Copilot CLI, Claude Code & Open Code

IDE: GitHub Copilot Plugin

Caveman

Reduces token usage by eliminating clutter. It doesn't affect thinking/reasoning.

Alternatives: Ponytail

These tools will help in saving more than 30% of our time to read model output and money(tokens).

Claude-mem

Claude-mem preserves context across sessions by automatically capturing tool usage observations, generating semantic summaries, and making them available to future sessions. This enables Claude to maintain continuity of knowledge about projects even after sessions end or reconnect.

This will improve developer productivity as well.

Alternatives: AgentMemory (backed by Linux Foundation)

CodeGraph

We work with multiple code bases on a daily basis and claude-code spawns Explore agents that scan files with grep, glob and read. They consume tokens on every tool call.

CodeGraph provides a pre-indexed knowledge graph to those agents, which reduces 50% tokens.

Alternatives: Graphify

rtk

CLI proxy that reduces LLM token consumption by ~75% on common dev commands like ls, cat, git, etc.

Similar tool: Headroom

Misc

andrej-karpathy-skills

ccstatusline

Custom instructions for agents in plain markdown files

First 30 Days in a Senior Technical Role

Anand Reddy Pandikunta — Thu, 30 Apr 2026 06:17:29 GMT

A little trust goes a long way

As a senior technical person, you will have certain professional authority. But to be effective, you need to build personal trust with your team and peers. That can happen only if you are approachable & transparent.

Set up 1:1 meetings with your team members and cross-functional team leads. Go on a listening tour and take your notes.

Be specific on how people can reach you and how you will communicate with them. Setup routine 1:1 meetings if required.

Follow the money

It is important to have high level overview of the business. This will help you make informed decisions and prioritize your work effectively.

Sankey chart of Apple's revenue

If the company is public, you can get quarterly & annual reports. If not, try to get a good sense of where the money is coming from and where the money is going.

Follow the data

Get a high level overview of product(s), infrastructure, tools, data flow. This is crucial to make informed technical decisions or architectural changes.

Start Small

For the first 30 days, you should be in "listening mode" or "absorbing mode". Try to understand the company, the team, the product, the customers, the culture, and the processes.

It is better to avoid any big changes in the first 30 days. There will be plenty of time for that later.

30-60-90 Day Plan

Note down action items for next 60 and 90 days.

It could be anything from setting up a new monitoring system, to improving the deployment process, to hiring new team members, providing simple tools for self service, etc.

Commnicate the plan with all stakeholders and avoid any surprises.

Reduce Cloud & SaaS Costs

If these are a significant part of the company's expenses, then review the costs and see if there is any room for reducing these costs.

On cloud, moving compute instances to reserved or spot can save 70-90% of costs.

That can have a big impact on the company's bottom line.

Buy or build good tools

On the other hand, if there are any tools that can help the team to be more productive, then it is worth investing in those tools.

To do that, you should already have a good toolset under your belt.

Sometimes building a simple tool which can make team self sufficient can have a big impact on the team's productivity and morale.

Conclusion

The first 30 days in a senior technical role are crucial for building trust, understanding the business, and setting the foundation for the next step.

Without much work to do, you can have a big impact on the company by following the tips mentioned above.

Build/Deploy Full Stack Web Apps with only Config

Anand Reddy Pandikunta — Sat, 28 Mar 2026 06:17:29 GMT

The best code is the code you don't have to write

Config First Tools

Any tool that provides first class support for configuration files can be considered a Config First Tool. These tools allow developers to setup/build/deploy applications by writing configuration files instead of writing code.

In DevOps world, tools like Terraform, Ansible, and Kubernetes are examples of Config First Tools. These kinds of tools are not that popular in the web development world.

In this article, lets see how we can build a full stack web application using only configuration files.

Todo - Web App

A simple backend API that allows users to manage todo lists. A simple frontend that uses the backend API. A simple script to deploy the backend and frontend applications.

Backend API

PocketBase is a backend API that provides a simple way to create and manage databases, authentication, and file storage. It is a great tool for building simple backend APIs without writing any code.

$ brew install pocketbase
$ pocketbase serve

We can create a collection called Todo with required fields and pocketbase will automatically generate the API for us.

Frontend

Lowdefy is a config first tool that allows us to build web applications using yaml config.

$ npm install -g lowdefy
$ lowdefy init
$ lowdefy dev

For basic crud apps, instead of us writing the boilerplate config, we can ask an AI agent to write the config for us.

Deployment

For deployment, we can use Ansible to deploy both the backend and frontend applications. Here also we can just AI agent to write the playbook for us.

$ pip install ansible
$ ansible-playbook deploy.yml

OliveTin

Even though we can use ansible to deploy the applications, it is not that convenient to run ansible playbooks on mobile. OliveTin is a config first tool that allows us to create web UIs for our scripts.

  - title: "Deploy ToDo App"
    popupOnStart: execution-dialog
    shell: ansible-playbook deploy.yml
    timeout: 6000
    icon: "🤖"

We can create a simple web UI for our deployment script with just 5 lines.

Conclusion

For quick prototyping and simple applications, config first tools can be a great choice as they allow us to build applications without writing any code. They also allow us to focus on the application logic rather than the implementation details.

With the rise of AI, it is much easy to maintain 500 lines of config than 5000 lines of code.

Talk: Classify Billion base pairs per Second

Anand Reddy Pandikunta — Sat, 28 Feb 2026 01:27:43 GMT

During 25th Feb 2026, a joint meetup was hosted by BangPypers & SciPy India.

There was a slot for lightning talk & I gave a talk on Kraken2 performance optimisations titled as "Classify billion base pairs per second".

Since it was a lightning talk, I didn't had time to polish the slides. During the talk, I briefly covered:

What is metagenomics?
Overview of open source classification tools
Brief intro to Kraken2
Performance optimizations in Kraken2

Few years back, I wrote a series of posts on Mastering Kraken2 where I covered all the optimizations in detail. If you want to dig deeper into the topic, you can check out the series.

In Praise of OliveTin

Anand Reddy Pandikunta — Fri, 30 Jan 2026 20:13:11 GMT

In a world of complex internal tools, OliveTin bridges the gap between raw CLI and usable web UI with just yaml config.

Server Admin Panel

I used admin panels like CockPit, Ajenti, etc to provide simple web UI for non-developers to provide access to services. Developing custom widgets is time consuming and requires programming skills as well.

With OliveTin, I can provide web UI directly with just yaml config. This saves a lot of time and effort. It also provides a clean UI as it starts from scratch.

Backend Driven UI

For existing python scripts, I often use tools like Streamlit, NiceGUI, etc to create web UIs. Organizing the UI components for multiple scripts is time consuming.

OliveTin can provide web UI directly python scripts and they can be organized in groups with just yaml config.

Mobile Friendly

To do a simple deployment on mobile, I need to open an app that supports ssh, ssh into the server, navigate to a directory, run a script to deploy.

OliveTin provides a "Deploy" button on mobile browser which is way convenient.

Authentication

OliveTin provides authentication(local users, OAuth2, JWT, etc), authorization(ACLs) & accounting(logs) out of the box. With just yaml config, we can secure the web UI.

Conclusion

There are hundreds of other features provided by OliveTin like scheduling, file uploads, webhooks, etc.

If you ever want to provide a simple web UI for scripts with low code or no code, give OliveTin a try!

Remote Access to k3d/k3s Kubernetes cluster

Anand Reddy Pandikunta — Wed, 17 Dec 2025 03:10:39 GMT

Introduction

We learnt how to deploy kubernetes cluster anywhere with a single k3d command. By default, k3d cluster is accessible only from the host machine.

Remote Access

Create new cluster with

$ k3d cluster create cloud-k8s \
  --api-port 6443:6443@loadbalancer \
  --k3s-arg "--tls-san=<remote-ip>@server:*"

--api-port 6443:6443@loadbalancer maps port 6443 on the host machine to the cluster's API server load balancer. This can be changed (e.g., 8080:6443) if 6443 is taken on the host.
--k3s-arg "--tls-san=..." adds the host's public IP to the certificate, preventing SSL errors later.
Ensure remote server firewall allows incoming traffic on the chosen port (e.g., 6443 or 8080). To verify, run telnet <remote-ip> 6443 from local machine.

On the remote server, get kubeconfig:

$ k3d kubeconfig get cloud-k8s
---
apiVersion: v1
clusters:
- cluster:
    server: https://0.0.0.0:6443
  name: k3d-rk
   ... # truncated for brevity

Copy this kubeconfig, replace 0.0.0.0 with the remote server's public IP.

Open Free Lens, paste this config and connect to the remote k3d cluster.

Conclusion

You can use your favorite Kubernetes tools (kubectl, FreeLens, k9s, etc.) to manage the remote k8s cluster.

Setup Tailscale App Connector in 3 easy steps

Anand Reddy Pandikunta — Sat, 22 Nov 2025 12:48:18 GMT

Problem Statement

In the earlier post, we have seen how to route specific websites to an exit node using a SOCKS proxy & browser extension(or a PAC file).

With this approach, when we need to work on multiple computers, we need to setup the SOCKS proxy & browser extension individually.

Instead we can use Tailscale's App Connectors to route specific websites through selected nodes.

Tailscale docs are too complicated for this App connectors. Here is a simplified version of the steps.

Tailscale - App Connectors

Note: There is no need for publicly reachable IP address for the Tailscale node which is used as App Connector. (Thanks Chris Smith for pointing it out. Tailscale docs are misleading in this regard.)

In the Tailscale admin console, go to the "Access control" section & add tagOwners.

    "tagOwners": {
        "tag:home-connector": [
            "autogroup:member",
        ],
    },

Use any Linux machine (it doesn't need static IP) to setup the App Connector. Install Tailscale on it. Advertise it as an app connector.

$ sudo tailscale up --advertise-tags=tag:home-connector --accept-routes

Enable IP forwarding on the machine. If your Linux system has a /etc/sysctl.d directory, run

echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
sudo sysctl -p /etc/sysctl.d/99-tailscale.conf

Otherwise, run

echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf

From the Tailscale admin console, create a new app connector.

In the "Target" field, we can select any existing app or we can use custom target so that we can route our desired websites. For now, I am adding ip.me domain so that we can verify if the traffic is routed through the connector.

From the "Machines" page, select the device and approve the requested subnets.

From browser, lets visit 2ip.io to see our device ip address. Next, lets visit ip.me to ensure we are seeing the connector's IP as this domain will be served by app connector.

Conclusion

We can use App Connectors to route specific websites through a Tailscale node without needing to setup SOCKS proxy or browser extensions on each device.

TailScale - Use Exit Node for Specific Websites only

Anand Reddy Pandikunta — Wed, 22 Oct 2025 22:20:58 GMT

Problem

Tailscale¹ provides zero-config mesh VPN built on top of Wiregaurd.

It provides an option to use an exit node for routing all traffic through it. But it can't route only specific domains through the exit node.

Solution

Setup a SOCKS proxy using ssh command line tool.

ssh -D 1080 -f -C -q -N user@exit-node-ip

-D 1080 : Specifies the local port to listen on for SOCKS connections.

-f : Requests ssh to go to background just before command execution.

-C : Enables compression.

-q : Quiet mode.

-N : Do not execute remote commands.

Once it is started, we can use ZeroOmega² browser extension to route specific domains through the SOCKS proxy.

In the auto switch profile, we can setup specific domains which needs to be directed to the proxy. For all other domains, it will use the direct connection.

This works only for the browser where the extension is installed.

To apply this at the system level, we can use pac file and update the system proxy settings to use the pac file.

Conclusion

Tailscale provides app connector feature to route specific apps through the exit node, but it is needs publicly reachable IPs & Linux based OS for now.

Setting up SOCKS proxy & using a browser extension or PAC file is much easy & straight forward solution to route specific domains through Tailscale exit node.

MacBook - Natural Scrolling for Mouse

Anand Reddy Pandikunta — Tue, 30 Sep 2025 18:20:58 GMT

Problem

If we connect a mouse to macOS, the scrolling direction changes to "unnatural" for the mouse. This is because macOS has a single setting for natural scrolling that applies to both trackpad and mouse.

Solution

To have natural scrolling for both trackpad and mouse, we can use third-party tools like UnnaturalScrollWheels.

If we already use Karabiner-Elements, we can use it to set natural scrolling for mouse separately.

Open Karabiner-Elements and go to the "Devices" tab, select your mouse, and click on "Open mouse settings".

In the mouse settings, check the "Flip mouse vertical wheel" option.

Now, both trackpad and mouse will have natural scrolling.

Mutual Funds - Behind the scenes

Anand Reddy Pandikunta — Tue, 19 Aug 2025 02:32:52 GMT

As investors, we buy mutual funds from fund houses, platforms, or from distributors. But have you ever wondered what happens behind the scenes? How do these funds operate, and what processes are involved in managing them? In this post, we'll explore the behind-the-scenes operations of mutual funds.

Structure of a Mutual Fund

In India, mutual funds have 3-tier structure consisting of the Sponsor, the Trust & the Asset Management Company (AMC).

The sponsor establishes the mutual fund and is responsible for its operations.
The trust & trustees are responsible for safeguarding investor interests, supervising the AMC's operations, and ensuring compliance with regulations. Custodians are appointed to hold the securities of the mutual fund and ensure their safekeeping.
AMC is the core entity responsible for the day-to-day management of the mutual fund.

The legal seperation of Trust & AMC ensures that the interests of investors are protected, and the fund's assets are managed independently.

All AMCs are registered with the Securities and Exchange Board of India (SEBI) and the list is available on the SEBI website.

We can look at SBI MF details from AMFI to understand the structure better.

Mutual Fund Transactions

RIA: Investors can use RIA (Registered Investment Advisor) platforms like Groww, Zerodha Coin, to invest in direct mutual funds.

MFD Alternatively, they can also buy regular mutual funds through distributors like NJWealth, PhonePhe etc.

TEP: To process these transactions, Trading & Execution Platforms like BSE Star MF, NSE MFSS, etc are available. These platforms facilitate the buying and selling of mutual fund units, ensuring that transactions are executed efficiently and accurately.

RTA: RTA(Registrar and Transfer Agent) plays a crucial role in maintaining investor records, processing transactions, and ensuring compliance with regulatory requirements.

Investors can also buy mutual funds directly from the fund house's website or through their mobile app.

These mutual funds can be stored in Demat account or in physical form as account statements. It depends on the platform or fund house you choose to invest with.

Conclusion

This post provided a brief overview of the behind-the-scenes operations of mutual funds in India.

Setup Raspberry Pi without Monitor & Keyboard

Anand Reddy Pandikunta — Thu, 31 Jul 2025 03:50:56 GMT

Requirements

Raspberry Pi (any model)
MicroSD card (8GB or larger)
Power supply

Setup

Download & install Raspberry Pi Imager from here.

After installation, connect SD Card to Computer, and start installation process.

In the config step, setup WiFi credentials and enable ssh server as well.

Once the installation is complete, insert the SD card into the Raspberry Pi and power it on.

Find the IP Address

You can find the IP address of your Raspberry Pi by checking your router's connected devices list or using a network scanning tool like nmap or arp.

arp -an

# or using nmap
nmap -sn

Once you have the IP address, you can SSH into the Raspberry Pi.

ssh pi@<IP_ADDRESS>

Conclusion

When we don't have a monitor and/or keyboard, this process allows us to set up a Raspberry Pi headlessly. This is particularly useful for projects where the Pi will be used as a server or in a remote location.

Raycast replaced 10 macOS apps

Anand Reddy Pandikunta — Mon, 30 Jun 2025 15:51:21 GMT

Since I started using Raycast, I have replaced many macOS apps with it. Here is a list of those apps and their Raycast equivalents.

Espanso

Instead of using Espanso for text expansion, I now use Raycast's built-in text snippets feature.

Hotkey

I used to use Hotkey to assign keyboard shortcuts to various actions. Now, I use Raycast's built-in hotkeys feature to achieve the same functionality.

Fly Cut

I used to use Fly Cut for clipboard management, but now I rely on Raycast's clipboard history feature to access my clipboard items quickly.

FinderGO

I used to use FinderGO to quickly navigate to specific folders in Finder. Now, I use Raycast's built-in file navigation feature to achieve the same functionality.

Rectangle

To move and resize windows, I used to use Rectangle. Now, I use Raycast's built-in window management feature to achieve the same functionality.

Textbar

Instead of using Textbar to copy TOTP for two-factor authentication, I now use Raycast's extension "One Time Password" to generate and copy TOTP codes directly.

Caffenate

To prevent my Mac from sleeping, instead of using Caffenate, I now use Raycast's built-in "Caffeinate" command to keep my Mac awake.

Calculator

Instead of using Python prompt for basic arithmetic calculations, I now use Raycast's built-in calculator feature to perform quick calculations. This is super convenient and saves me from opening a separate calculator app.

Hammer Spoon (partially)

To assign shortcuts to browser urls, shell scripts, and other actions, I used to use Hammer Spoon. Now, I use Raycast's built-in scripting capabilities to create custom commands and assign shortcuts to them.

Terminal (partially)

For things like killing processes, installing packages, and running shell commands, I used to rely on Terminal. Now, I use Raycast's built-in terminal integration to run shell commands directly from the Raycast interface.

Conclusion

With Raycast, I have been able to streamline my workflow by replacing multiple macOS apps with its powerful features.

Migrating from Nginx to Caddy

Anand Reddy Pandikunta — Fri, 30 May 2025 15:51:21 GMT

When deploying web apps in production, a web server is essential for serving static files, handling reverse proxying, and managing SSL/TLS certificates.

I have been using Nginx for a long time, but I have recently switched to Caddy as it provides automatic HTTPS by default. In addition, Caddy has a simpler configuration syntax and is easier to set up for basic use cases.

Nginx vs Caddy

To illustrate the differences, here are some examples of how to configure a simple web server in both Nginx and Caddy.

Nginx Configuration

server {
    listen 80;
    server_name avilpage.com;

    location / {
        root /var/www/html;
        index index.html index.htm;
    }

    location /api {
        proxy_pass http://localhost:3000;
    }
}

With Nginx, you need to manually handle SSL/TLS certificates, which can be cumbersome. You typically use tools like Certbot to obtain and renew certificates.

Caddy Configuration

avilpage.com {
    root * /var/www/html
    file_server

    reverse_proxy /api localhost:3000
}

Caddy automatically manages SSL/TLS certificates for you, so you don't need to worry about obtaining or renewing them. It also provides default redirection from HTTP to HTTPS, making it easier to secure your site.

Conclusion

Caddy's automatic HTTPS and simpler configuration syntax make it a compelling choice for web servers, especially for those who want to get started quickly without dealing with the complexities of SSL/TLS management.

Python UV in 100 seconds

Anand Reddy Pandikunta — Wed, 30 Apr 2025 17:46:24 GMT

UV is a fast Python package manager that replaces tools like pip, pyenv, virtualenv, etc.

Install uv

# On macOS and Linux.
curl -LsSf https://astral.sh/uv/install.sh | sh
# On Windows.
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Installing Python

uv python list

uv python install 3.13

This makes it easy to install and manage multiple versions of Python on your system without needing of pyenv.

Using UV with existing requirements.txt

uv venv --seed -p 3.13

source .venv/bin/activate

uv pip install -r requirements.txt

This is way faster than using any other tool to create virtual environments.

Managing virtual environments

uv init -p 3.13 --name demo

uv add pandas

This creates a new virtual environment named demo with Python 3.13 and adds the pandas package to it.

Runs scripts with inline dependencies

# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "requests",
# ]
# ///
import requests

print(requests.get("https://avilpage.com"))

uv run script.py

uvx

Similar to pipx, it can install and run Python applications in isolated environments.

uv tool install glances

uv tool run glances

# shortcut
uvx glances

Conclusion

UV speeds up Python development by providing a fast package manager, virtual environment management, and inline dependency management. It is a great alternative to traditional tools like pip, pyenv, and virtualenv.

Autoscale k8s pods with queue size (KEDA)

Anand Reddy Pandikunta — Thu, 27 Mar 2025 06:07:52 GMT

Introduction

Kubernetes(k8s) is a popular container orchestration tool, and it provides Horizontal Pod Autoscaler(HPA) to scale pods based on CPU and Memory usage.

To scale pods based on the queue size we can use KEDA.

Setup

For this demo, we will use PostgreSQL table to store tasks and KEDA to scale the pods based on the queue size.

As written earlier, we can use k3d and setup k8s cluster with a single command anywhere.

Once the cluster is up, we can set up a simple shell script to produce and consume tasks from the PostgreSQL table.

I am adding only relevant snippets here. You can find the complete code in the GitHub repo.

PostgreSQL deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-deployment
spec:
  template:
    spec:
      containers:
        - name: postgres
          image: postgres:13

Producer shell script:

#!/bin/bash

psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" <<-EOSQL
  CREATE TABLE IF NOT EXISTS tasks (
      id SERIAL PRIMARY KEY,
      description TEXT NOT NULL,
      status TEXT NOT NULL
  );
EOSQL

# Continuously insert tasks
while true; do
    psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" <<-EOSQL
    INSERT INTO tasks (description, status)
    VALUES ('$DESCRIPTION', '$STATUS');
done

Producer dockerfile:

FROM postgres:13

RUN apt-get update && apt-get install -y bash curl

COPY producer.sh /producer.sh

RUN chmod +x /producer.sh

CMD ["/producer.sh"]

Producer deployment:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: task-producer
          image: task-producer
          imagePullPolicy: Never

Consumer shell script:

#!/bin/bash

TASK=$(psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" -t <<-EOSQL
  SELECT id, description, status FROM tasks
  WHERE status != 'Completed'
  ORDER BY id
  LIMIT 1;
EOSQL

TASK_ID=$(echo "$TASK" | awk '{print $1}')

psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" <<-EOSQL
  UPDATE tasks
  SET status = 'Completed'
  WHERE id = $TASK_ID;
EOSQL

Consumer dockerfile:

FROM postgres:13

RUN apt-get update && apt-get install -y bash

COPY consumer.sh /consumer.sh

RUN chmod +x /consumer.sh

CMD ["/consumer.sh"]

Consumer deployment:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: task-consumer
          image: task-consumer
          imagePullPolicy: Never

Instead of pushing these images to any container registry, we can directly load the images into cluster using k3d image.

k3d image import task-producer --cluster demo-cluster
k3d image import task-consumer --cluster demo-cluster

After that, we can deploy postgres, consumer and producer using the above deployment files.

Once the deployment is done, we can monitor the tasks from pod logs.

Lets install KEDA in the cluster and setup a scaled-object to scale the consumer pods based on number of pending tasks.

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda

ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: task-consumer-scaler
  namespace: default
spec:
  scaleTargetRef:
    name: task-consumer  
  pollingInterval: 1    
  cooldownPeriod: 1     
  minReplicaCount: 0    
  maxReplicaCount: 50   
  triggers:
    - type: postgresql
      metadata:
        host: postgres-service  
        query: "SELECT COUNT(*) FROM tasks WHERE status != 'Completed';"
        targetQueryValue: "10"

With this, KEDA will scale the consumer pods based on the number of tasks in the PostgreSQL table. The targetQueryValue is set to 10, which means if there are more than 10 tasks in the table, KEDA will scale up the consumer pods.

In the cluster, after a while, a bunch of tasks will be created in the PostgreSQL table. Now, we can set the status of all tasks to empty so that we can see auto-scaling in action.

kubectl exec -it task-producer-<pod-id> -- psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" <<-EOSQL
  UPDATE tasks
  SET status = '';
EOSQL

On k8s dashboard, we can see the consumer pods scaling up and down based on the number of tasks in the PostgreSQL table.

Conclusion

KEDA is a powerful tool to scale k8s pods based on custom metrics. In this post, we have used PostgreSQL but we can use KEDA with variety of other data stores as well. Full code for this post is available here.

Free DockerHub Alternative - ECR Public Gallery

Anand Reddy Pandikunta — Sun, 09 Feb 2025 16:08:34 GMT

DockerHub started rate limiting¹ anonymous docker pulls. When testing out a new CI/CD setup, I hit the rate limit and had to wait for an hour to pull the image. This was a good time to look for alternatives.

AWS ECR Public Gallery² is a good alternative to DockerHub as of today(2025 Feb). It is free and does not have rate limits even for anonymous users.

Once we find the required image from the gallery, we can simply change the image name in the docker pull command to pull the image from ECR Gallery.

docker pull public.ecr.aws/ubuntu/ubuntu

In Dockerfile, we can use the image from ECR Gallery as follows:

FROM public.ecr.aws/ubuntu/ubuntu

That is a quick way to avoid DockerHub rate limits.

Postman - Auto Login & Renew OAuth2 Token

Anand Reddy Pandikunta — Fri, 31 Jan 2025 17:20:17 GMT

Introduction

When using Postman to interact with APIs behind an OAuth2 authentication, we need to login and renew the token manually. This can be automated using the following steps.

Set credentials in environment variables
Create a pre-request script to login and renew the token
Use the token in the request headers

Automating Login & Renewal

var e = pm.environment;
var isSessionExpired = true;

var loginTimestamp = e.get("loginTimestamp");
var expiresInSeconds = pm.environment.get("expiresInSeconds") || 86400;

if (loginTimestamp) {
  var loginDuration = Date.now() - loginTimestamp;
  isSessionExpired = loginDuration >= expiresInSeconds;
}

if (isSessionExpired) {
  pm.sendRequest({
    url: e.get('host') + "/auth/connect/token",
    method: 'POST',
    header: {
      'Content-Type': 'application/x-www-form-urlencoded',
      'Accept': 'application/json'
    },
    body: {
        mode: 'urlencoded',
        urlencoded: [
          { key: "username", value: e.get('username') },
          { key: "password", value: e.get('password') },
          { key: "grant_type", value: "password" },
          { key: "client_id", value: e.get("client_id") }
        ]
    }
  }, function (err, res) {
    jsonData = res.json();

    e.set("access_token", jsonData.access_token);

    if(res.json().expires_in){
        expiresInSeconds = res.json().expires_in * 1000;
    }
    e.set("expiresInSeconds", expiresInSeconds);
    e.set("loginTimestamp", Date.now())
  });
}

We can copy this script to the pre-request script of the collection.

Most of the script is self-explanatory. The script checks if the session is expired and sends a request to the token endpoint to get a new token. The token is stored in environment variables and used in the request headers.

Conclusion

This is a one time setup for Postman collection and it saves a lot of time in the long run. The script can be modified to handle different grant types and token renewal strategies.

Install Cockpit on Remote Linux VM

Anand Reddy Pandikunta — Mon, 30 Dec 2024 22:54:07 GMT

Introduction

Cockpit is an easy to use web-based interface(like a cPanel) for managing Linux servers. When we want to provide access to non-developers or people who are new to linux, it is a good idea to get them started with Cockpit. It provides a user-friendly interface to manage services, containers, storage, logs, and more.

Setup

Let's create a new Ubuntu VM and install Cockpit on it.

sudo apt update
. /etc/os-release
sudo apt install -t ${VERSION_CODENAME}-backports cockpit

Once the installation is complete, we can get the public ip of the VM and access the Cockpit web interface running on port 9090.

It will be difficult to remember the public ip of the VM. So, let's create a DNS record for the VM. Let's add an A record in DNS settings to point cockpit.avilpage.com to the public ip of the VM.

Reverse Proxy

Let's set up a reverse proxy to access the Cockpit web interface using a subdomain.

sudo apt install caddy

Add the below configuration to /etc/caddy/Caddyfile.

cockpit.avilpage.com {
    reverse_proxy localhost:9090
}

We need Origins to Cockpit configuration at /etc/cockpit/cockpit.conf to allow requests from the subdomain.

[WebService]
Origins = https://cockpit.avilpage.com

Restart both services and open https://cockpit.avilpage.com in browser.

sudo systemctl restart cockpit
sudo systemctl restart caddy

Conclusion

Cockpit web UI is a great tool to manage Linux servers even for non-developers. Users can browse/manage logs, services, etc. It also provides a terminal to run commands on the server

Mastering "Partial Covered Calls" - Part 1

Anand Reddy Pandikunta — Sat, 30 Nov 2024 06:52:10 GMT

Covered Calls

In a covered call strategy, we buy one lot of stocks (or 1 Future) and sell at the money call option. The payoff diagram looks like this:

There are 2 drawbacks of this strategy:

It requires a lot of capital to buy the shares. We can't fully use the margin from pledging the stocks.
We need to sell the stocks at expiry if stock price closes above the strike price.

To overcome this limitation, we can use a strategy called "partial covered calls."

Partial Covered Calls

Instead of buying one lot of shares, we can buy "partial" or "fractional" lot of shares and then sell a far away call option instead of at the money call option.

For example, we can buy 0.15 lot of shares and sell a call option which is 10% away from the current price. The payoff diagram looks like this:

Here we can pledge the stocks we have bought and use the margin to sell call option using that margin.

Since we are selling a call option which is far away from the current price, the probability of the call option getting exercised is very low. So we can keep the premium we received from selling the call option.

In addition to that, we get the long term capital appreciation of the stocks we have bought and there won't be short term capital gains tax on them.

Conclusion

If you want to hold the stock for long but still want to generate regular income from it, then partial covered calls is a good strategy to consider.

Cube & Cubicle

Anand Reddy Pandikunta — Thu, 31 Oct 2024 03:35:37 GMT

Rubiks Cube

When I was in college, I was traveling to a friend's place and missed bus at midnight. The next bus was at 4 AM. While I was bored waiting for the bus, I found Rubik's Cube in a shop.

I scrambled the cube and spent the next 4 hours trying to solve the cube. I managed to solve one color. When I tried to solve the next color, the pieces in the previous layer started missing.

Even after spending a lot of time in the next 3 weeks, I couldn't solve it and gave up.

After a couple of years, when I "learnt" about the internet, I searched and found simple algorithms to solve the cube. Within a few days, I was able to solve the cube in a minute.

Office Cubicles

In the final year of college, there were placements. When I was preparing resume, I included "I can solve Rubik's Cube in a minute" in it.

During the interview, interviewer asked me if I can really solve the cube in a minute. He asked me to get my cube and show him during the lunch break. I did. Luckily, I got hired.

Even though, I was hired for Wipro I didn't join. I went to Bangalore and started applying for start-up jobs.

I went for an interview at a web development company in Malleswaram, Bangalore. The CEO looked at my résumé, took out a cube from his desk. He handed the cube to me, showed an empty cubicle behind me and said, "If you solve the cube in a minute, that cubicle is yours."

Just by learning the cube, I was able to land a job an at an MNC(Multi National Company) and a startup as well.

tailscale: Resolving CGNAT (100.x.y.z) Conflicts

Anand Reddy Pandikunta — Sat, 07 Sep 2024 07:20:05 GMT

Introduction

In an earlier blog post, I wrote about using tailscale to remotely access any device¹. Tailscale uses 100.64.0.0/10 subnet² to assign unique IP addresses to each device.

When a tailscale node joins another campus network³ (schools, universities, offices) that uses the same subnet, it will face conflicts. Let's see how to resolve this.

Private Network

In the above scenario, node C1 will be able to connect C2 & C3 as they are in the same network.

Once we start tailscale on node C1, it will get a 100.x.y.z IP address from tailscale subnet. Now, node C1 will not be able to connect to node C2 & C3.

To avoid conflicts with the existing network, we can configure tailscale to use a "smaller" subnet using "ipPool".

{
    "acls": [
        "..."
    ],
    "nodeAttrs": [
        {
            "target": [
                "autogroup:admin"
            ],
            "ipPool": [
                "100.100.96.0/20"
            ]
        }
    ]
}

Once it is configured, taiscale will start assigning IP addresses from the new subnet. Even though ip address allocation is limited, we can't still access nodes in other subnets due to a bug⁵ in tailscale.

As a workaround, we can manually update the iptables to route traffic to the correct subnet.

Lets look at the iptables rules added by tailscale by stopping it and then starting it.

The highlighted rule drops any incoming packet that doesn't originate from tailscale0 interface, and source IP is 100.64.0.0/10 (100.64.0.0 to 100.127.255.255).

Let's delete this rule and add a new rule to restrict the source IP to 100.100.96.0/20 (100.100.96.1 to 100.100.111.254).

$ sudo iptables --delete ts-input --source 100.64.0.0/10 ! -i tailscale0 -j DROP
$ sudo iptables --insert ts-input 3 --source 100.100.96.0/20 ! -i tailscale0 -j DROP

Conclusion

By configuring tailscale to use a smaller subnet, we can avoid conflicts with existing networks. Even though there is a bug in tailscale, we can manually update iptables to route traffic to the correct subnet.

Mastering Kraken2 - Part 4 - Build FDA-ARGOS Index

Anand Reddy Pandikunta — Sat, 24 Aug 2024 09:58:00 GMT

Mastering Kraken2

Part 1 - Initial Runs

Part 2 - Classification Performance Optimisation

Part 3 - Build custom database indices

Part 4 - Build FDA-ARGOS index (this post)

Part 5 - Regular vs Fast Builds (upcoming)

Part 6 - Benchmarking (upcoming)

Introduction

In the previous post, we learnt how to build a custom index for Kraken2.

FDA-ARGOS¹ is a popular database with quality reference genomes for diagnostic usage. Let's build an index for FDA-ARGOS.

FDA-ARGOS Kraken2 Index

FDA-ARGOS db is available at NCBI² from which we can download the assembly file.

We can extract accession numbers from the assembly file and then download the genomes from these accession ids.

$ grep -e "^#" -v PRJNA231221_AssemblyDetails.txt | cut -d$'\t' -f1 > accessions.txt

$ wc accessions.txt
 1428  1428 22848 accessions.txt

$ ncbi-genome-download --section genbank --assembly-accessions accessions.txt --progress-bar bacteria --parallel 40

It took ~8 minutes to download all the genomes, and the downloaded file size is ~4GB.

We can use kraken-db-builder³ tool to build index from these genbank genome files.

# kraken-db-builder needs this to convert gbff to fasta format
$ conda install -c bioconda any2fasta

$ kraken-db-builder --genomes-dir genbank --threads 36 --db-name k2_argos

It took ~30 minutes to build the index.

Conclusion

We have built a Kraken2 index for the FDA-ARGOS database on 2024-Aug-24.

FDA-ARGOS Library
Kraken2 Gzipped Index file (gzip size: 2.6GB, index size: 3.8GB, md5sum: 1dd946d2e405dfec35ed3e319e9dfeac)
Kraken2 Inspect file

In the next post, we will look at the differences between regular and fast builds.

Midnight Coding for Narendra Modi & Ivanka Trump

Anand Reddy Pandikunta — Sun, 18 Aug 2024 00:25:43 GMT

Introduction

In 2017, GES Event was held in Hyderabad, India. Narendra Modi (the Prime Minister of India) & Ivanka Trump (daughter of the then US President Donald Trump) were the chief guests.

At that time, I was part of Invento team, and we decided to develop a new version of Mitra robot for the event.

The Challenge

We had to develop the new version of Mitra robot in a short span of time. Entire team worked day and night to meet the deadlines and finish the new version.

We went to Hyderabad from Bangalore a few days before to prepare for the event. We have cleared multiple security checks, did some demos for various people before the event.

A day before the event, around 9 PM we discovered a critical bug in the software. Due to that bug, the Robot motors were running at full speed which was dangerous. If the robot hits someone at full speed, it could cause serious injuries.

I spent a few hours debugging the issue and even tried rolling back a few versions. Still, I couldn't pinpoint the issue.

Since we need only a small set of Robot features, we decided to create a new version of the software with only limited features. I spent the next few hours creating a new release.

After that, we spent the next few hours doing extensive testing to make sure there are no bugs in the new version.

It was almost morning by the time we were done with testing. We quickly went to hotel to have some rest and get back early for the event.

Conclusion

Mitra robot welcoming Modi & Trump went very well. You can read about Balaji Viswanathan's experience at GES 2017 on Quora¹.

Answer on Quora ↩

How (and when) to use systemd timer instead of cronjob

Anand Reddy Pandikunta — Mon, 05 Aug 2024 07:37:50 GMT

Introduction

* * * * * bash demo.sh

Just a single line of code is sufficient to schedule a cron job. However, there are some scenarios where I find systemd timer more useful than cronjob.

How to use systemd timer

We need to create a service file(contains the script to be run) and a timer(contains the schedule).

# demo.service
[Unit]
Description=Demo service

[Service]
ExecStart=bash demo.sh

# demo.timer
[Unit]
Description=Run myscript.service every 1 minutes

[Timer]
OnBootSec=1min
OnUnitActiveSec=1min

[Install]
WantedBy=multi-user.target

We can copy these files to /etc/systemd/system/ and enable the timer.

$ sudo cp demo.service demo.timer /etc/systemd/system/

$ sudo systemctl daemon-reload

$ sudo systemctl enable --now demo.timer

We can use systemctl to see when the task is executed last and when it will be executed next.

$ sudo systemctl list-timers --all

Use Cases

Singleton - In the above example, lets say demo.sh takes ~10 minutes to run. With cron job, in ten minutes we will have 10 instances of demo.sh running. This is not ideal. With systemd timer, it will ensure only one instance of demo.sh is running at a time.
On demand runs - If we want to test out the script/job, systemd allows us to immediately run it with usual systemctl start demo without needing to run the script manually.
Timer - With cron, we can run tasks upto a minute precision. Timer can run tasks till second level precision.

[Timer]
OnCalendar=*-*-* 15:30:15

In addition to that, we can run tasks based on system events. For example, we can run a script 15 minutes from reboot.

[Timer]
OnBootSec=15min

Conclusion

Systemd timer is a powerful tool that can replace cronjob in many scenarios. It provides more control and flexibility over cronjob. However, cronjob is still a good choice for simple scheduling tasks.

Mastering Kraken2 - Part 3 - Build Custom Database

Anand Reddy Pandikunta — Thu, 01 Aug 2024 05:22:30 GMT

Mastering Kraken2

Part 1 - Initial Runs

Part 2 - Classification Performance Optimisation

Part 3 - Build custom database indices (this post)

Part 4 - Build FDA-ARGOS index

Part 5 - Regular vs Fast Builds (upcoming)

Part 6 - Benchmarking (upcoming)

Introduction

In the previous post, we learned how to improve kraken2¹ classification performance. So far we have downloaded & used pre-built genome indices(databases).

In this post, let's build a custom database for kraken2. For simplicity, let's use only refseq archaea genomes² for building the index.

Building Custom Database

First, we need to download the taxonomy files. We can use the k2 script provided by kraken2.

$ k2 download-taxonomy --db custom_db

This takes ~30 minutes depending on the network speed. The taxonomy files are downloaded to the custom_db/taxonomy directory.

$ ls custom_db/taxonomy
citations.dmp  division.dmp  gencode.dmp  merged.dmp  nodes.dmp
nucl_wgs.accession2taxid delnodes.dmp  gc.prt 
images.dmp  names.dmp  nucl_gb.accession2taxid  readme.txt

$ du -hs custom_db/taxonomy
43G     custom_db/taxonomy

For simplicity, let's use the archaea refseq genomes. We can use kraken2-build to download the refseq genomes.

$ k2 download-library --library archaea --db custom_db

This runs on a single thread. Instead of using kraken2-build, we can use ncbi-genome-download³ tool to download the genomes. This provides much granular control over the download process. For example, we can download only --assembly-levels complete genomes. We can also download multiple genomes in parallel.

$ pip install ncbi-genome-download

$ conda install -c bioconda ncbi-genome-download

$ ncbi-genome-download -s refseq -F fasta --parallel 40 -P archaea
Checking assemblies: 100%|███| 2184/2184 [00:19<00:00, 111.60entries/s]
Downloading assemblies: 100%|███| 2184/2184  [02:04<00:00,  4.54s/files]
Downloading assemblies: 2184files [02:23, 2184files/s]

In just 2 minutes, it has downloaded all the files. Lets gunzip the files.

$ find refseq -name "*.gz" -print0 | parallel -0 gunzip

$ du -hs refseq
5.9G    refseq

Lets add all fasta genome files to the custom database

$ time find refseq -name "*.fna" -exec kraken2-build --add-to-library {} --db custom_db \;
667.46s user 90.78s system 106% cpu 12:54.80 total

kraken2-build doesn't use multiple threads for adding genomes to the database. In addition to that, it also doesn't check if the genome is already present in the database.

Let's use k2 for adding genomes to the database.

export KRAKEN_NUM_THREADS=40

$ find . -name "*.fna" -exec k2 add-to-library --files {} --db custom_db \;
668.37s user 88.44s system 159% cpu 7:54.40 total

This took only half the time compared to kraken2-build.

Let's build the index from the library.

$ time kraken2-build --db custom_db --build --threads 36
Creating sequence ID to taxonomy ID map (step 1)...
Found 0/125783 targets, searched through 60000000 accession IDs...
Found 59923/125783 targets, searched through 822105735 accession IDs, search complete.
lookup_accession_numbers: 65860/125783 accession numbers remain unmapped, see unmapped.txt in DB directory
Sequence ID to taxonomy ID map complete. [2m1.950s]
Estimating required capacity (step 2)...
Estimated hash table requirement: 5340021028 bytes
Capacity estimation complete. [23.875s]
Building database files (step 3)...
Taxonomy parsed and converted.
CHT created with 11 bits reserved for taxid.
Completed processing of 59911 sequences, 3572145823 bp
Writing data to disk...  complete.
Database files completed. [12m3.368s]
Database construction complete. [Total: 14m29.666s]
kraken2-build --db custom_db --build --threads 36  24534.98s user 90.50s system 2831% cpu 14:29.75 total

$ ls -ll
.rw-rw-r-- 5.3G anand  1 Aug 16:35 hash.k2d
drwxrwxr-x    - anand  1 Aug 12:32 library
.rw-rw-r--   64 anand  1 Aug 16:35 opts.k2d
.rw-rw-r-- 1.5M anand  1 Aug 16:22 seqid2taxid.map
.rw-rw-r-- 115k anand  1 Aug 16:23 taxo.k2d
lrwxrwxrwx   20 anand  1 Aug 12:31 taxonomy
.rw-rw-r-- 1.2M anand  1 Aug 16:22 unmapped.txt

We are able to build index for ~6GB input files in ~15 minutes.

Conclusion

We learnt some useful tips to speed up the custom database creation process. In the next post, we will learn about regular vs. fast builds.

Mastering Kraken2 - Part 2 - Performance Optimisation

Anand Reddy Pandikunta — Sun, 28 Jul 2024 05:21:30 GMT

Mastering Kraken2

Part 1 - Initial Runs

Part 2 - Classification Performance Optimisation (this post)

Part 3 - Build custom database indices

Part 4 - Build FDA-ARGOS index

Part 5 - Regular vs Fast Builds (upcoming)

Part 6 - Benchmarking (upcoming)

Introduction

In the previous post, we learned how to set up kraken2¹, download pre-built indices, and run kraken2. In this post, we will learn various ways to speed up the classification process.

Increasing RAM

Kraken2 standard database is ~80GB in size. It is recommended to have at least db size RAM to run kraken2 efficiently². Let's use 128GB RAM machine and run kraken2 with ERR10359977³ sample.

$ time kraken2 --db k2_standard --report report.txt ERR10359977.fastq.gz > output.txt
Loading database information... done.
95064 sequences (14.35 Mbp) processed in 2.142s (2662.9 Kseq/m, 402.02 Mbp/m).
  94816 sequences classified (99.74%)
  248 sequences unclassified (0.26%)
kraken2 --db k2_standard --report report.txt ERR10359977.fastq.gz >   1.68s user 152.19s system 35% cpu 7:17.55 total

Now the time taken has come down from 40 minutes to 7 minutes. The classification speed has also increased from 0.19 Mbp/m to 402.02 Mbp/m.

The previous sample had only a few reads, and the speed is not a good indicator. Let's run kraken2 with a larger sample.

$ time kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq.gz SRR6915097_2.fastq.gz > output.txt
Loading database information... done.
Processed 14980000 sequences (2972330207 bp) ...
17121245 sequences (3397.15 Mbp) processed in 797.424s (1288.2 Kseq/m, 255.61 Mbp/m).
  9826671 sequences classified (57.39%)
  7294574 sequences unclassified (42.61%)
kraken2 --db k2_standard --report report.txt --paired > output.txt  526.39s user 308.24s system 68% cpu 20:23.86 total

This took almost 20 minutes to classify ~3 Gbp of data. Out of 20 minutes, 13 minutes was spent in classification. The remaining time in loading the db into memory.

Let's use k2_plusPF⁴ db, which is twice the size of k2_standard and run kraken2.

$ time kraken2 --db k2_plusfp --report report.txt --paired SRR6915097_1.fastq.gz SRR6915097_2.fastq.gz > output.txt
Loading database information...done.
17121245 sequences (3397.15 Mbp) processed in 755.290s (1360.1 Kseq/m, 269.87 Mbp/m).
  9903824 sequences classified (57.85%)
  7217421 sequences unclassified (42.15%)
kraken2 --db k2_plusfp/ --report report.txt --paired SRR6915097_1.fastq.gz  >   509.71s user 509.51s system 55% cpu 30:35.49 total

This took ~30 minutes to complete, but the classification took only 13 minutes similar to k2_standard. The remaining time was spent in loading the db into memory.

Preloading db into RAM

We can use vmtouch⁵ to preload db into RAM. kraken2 provides --memory-mapping option to use preloaded db.

$ vmtouch -vt k2_standard/hash.k2d k2_standard/opts.k2d k2_standard/taxo.k2d
           Files: 3
     Directories: 0
   Touched Pages: 20382075 (77G)
         Elapsed: 434.77 seconds

When Linux requires RAM, it will incrementally evict the db from memory. To prevent this, we can copy the db to shared memory (/dev/shm) and then use vmtouch to preload the db.

$ cp -r k2_standard /dev/shm

$ vmtouch -t /dev/shm/*.k2d

Now, let's run kraken2 with --memory-mapping option.

$ time kraken2 --db k2_standard --report report.txt --memory-mapping --paired SRR6915097_1.fastq.gz SRR6915097_2.fastq.gz > output.txt
Loading database information... done.
17121245 sequences (3397.15 Mbp) processed in 532.486s (1929.2 Kseq/m, 382.79 Mbp/m).
  9826671 sequences classified (57.39%)
  7294574 sequences unclassified (42.61%)
  kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq.gz   >  424.20s user 11.76s system 81% cpu 8:54.98 total

Now the classification took only ~10 minutes.

Multi threading

kraken2 supports multiple threads. I am using a machine with 40 threads.

$ time kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq.gz SRR6915097_2.fastq.gz --memory-mapping --threads 32 > output.txt
Loading database information... done.
17121245 sequences (3397.15 Mbp) processed in 71.675s (14332.5 Kseq/m, 2843.81 Mbp/m).
  9826671 sequences classified (57.39%)
  7294574 sequences unclassified (42.61%)
kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq.gz      556.58s user 22.85s system 762% cpu 1:16.02 total

With 32 threads, the classification took only 1 minute. Beyond 32 threads, the classification time did not decrease significantly.

Optimising input files

So far we have used gzipped input files. Let's use unzipped input files and run kraken2.

$ gunzip SRR6915097_1.fastq.gz
$ gunzip SRR6915097_2.fastq.gz

$ time kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq SRR6915097_2.fastq --memory-mapping --threads 30 > output.txt
Loading database information... done.
17121245 sequences (3397.15 Mbp) processed in 34.809s (29512.0 Kseq/m, 5855.68 Mbp/m).
  9826671 sequences classified (57.39%)
  7294574 sequences unclassified (42.61%)
kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq    30   565.03s user 17.12s system 1530% cpu 38.047 total

Now the classification time has come down to 40 seconds.

Since the input fastq files are paired, interleaving the files also takes time. Lets interleave the files and run kraken2.

To interleave the files, lets use seqfu tool.

$ conda install -y -c conda-forge -c bioconda "seqfu>1.10"

$ seqfu interleave -1 SRR6915097_1.fastq.gz -2 SRR6915097_2.fastq.gz > SRR6915097.fastq

$ time kraken2 --db k2_standard --report report.txt --memory-mapping SRR6915097.fq --threads 32 > output.txt
Loading database information... done.
34242490 sequences (3397.15 Mbp) processed in 20.199s (101714.1 Kseq/m, 10090.91 Mbp/m).
  17983321 sequences classified (52.52%)
  16259169 sequences unclassified (47.48%)
kraken2 --db k2_standard --report report.txt --memory-mapping SRR6915097.fq  32  618.96s user 18.24s system 2653% cpu 24.013 total

Now the classification time has come down to 24 seconds.

Conclusion

In terms of classification speed, we have come a long way from 0.1 Mbp/m to 1200 Mbp/m. In the next post, we will learn how to optimise the creation of custom indices.

Mastering Kraken2 - Part 1 - Initial Runs

Anand Reddy Pandikunta — Sun, 28 Jul 2024 05:14:25 GMT

Mastering Kraken2

Part 1 - Initial Runs (this post)

Part 2 - Classification Performance Optimisation

Part 3 - Build custom database indices

Part 4 - Build FDA-ARGOS index (this post)

Part 5 - Regular vs Fast Builds (upcoming)

Part 6 - Benchmarking (upcoming)

Introduction

Kraken2¹ is widely used for metagenomics taxonomic classification, and it has pre-built indexes for many organisms. In this series, we will learn

How to set up kraken2, download pre-built indices
Run kraken2 (8GB RAM) at ~0.19 Mbp/m (million base pairs per minute)
Learn various ways to speed up the classification process
Run kraken2 (128GB RAM) at ~1200 Mbp/m
Build custom indices

Installation

We can install kraken2 from source using the install_kraken2.sh script as per the manual².

$ git clone https://github.com/DerrickWood/kraken2
$ cd kraken2
$ ./install_kraken2.sh /usr/local/bin
# ensure kraken2 is in the PATH
$ export PATH=$PATH:/usr/local/bin

If you already have conda installed, you can install kraken2 from conda as well.

$ conda install -c bioconda kraken2

If you have brew installed on Linux or Mac(including M1), you can install kraken2 using brew.

$ brew install brewsci/bio/kraken2

Download pre-built indices

Building kraken2 indices take a lot of time and resources. For now, let's download and use the pre-built indices. In the final post, we will learn how to build the indices.

Genomic Index Zone³ provides pre-built indices for kraken2. Let's download the standard database. It contains Refeq archaea, bacteria, viral, plasmid, human1, & UniVec_Core.

$ wget https://genome-idx.s3.amazonaws.com/kraken/k2_standard_20240605.tar.gz
$ mkdir k2_standard
$ tar -xvf k2_standard_20240605.tar.gz -C k2_standard

The extracted directory contains three files - hash.k2d, opts.k2d, taxo.k2d which are the kraken2 database files.

$ ls -l *.k2d
.rw-r--r--  83G anand 13 Jul 12:34 hash.k2d
.rw-r--r--   64 anand 13 Jul 12:34 opts.k2d
.rw-r--r-- 4.0M anand 13 Jul 12:34 taxo.k2d

Classification

To run the taxonomic classification, let's use ERR10359977 human gut meta genome from NCBI SRA.

$ wget https://ftp.sra.ebi.ac.uk/vol1/fastq/ERR103/077/ERR10359977/ERR10359977.fastq.gz
$ kraken2 --db k2_standard --report report.txt ERR10359977.fastq.gz > output.txt

By default, the machine I have used has 8GB RAM and an additioinal 8GB swap. Since kraken2 needs entire db(~80GB) in memory, when the process tries to consume more than 16GB memory, the kernel will kill the process.

$ time kraken2 --db k2_standard --paired SRR6915097_1.fastq.gz SRR6915097_2.fastq.gz > output.txt
Loading database information...Command terminated by signal 9
0.02user 275.83system 8:17.43elapsed 55%CPU

To prevent this, let's increase the swap space to 128 GB.

# Create an empty swapfile of 128GB
sudo dd if=/dev/zero of=/swapfile bs=1G count=128

# Turn swap off - It might take several minutes
sudo swapoff -a

# Set the permissions for swapfile
sudo chmod 0600 /swapfile

# make it a swap area
sudo mkswap /swapfile  

# Turn the swap on
sudo swapon /swapfile

We can time the classification process using the time command.

$ time kraken2 --db k2_standard --report report.txt ERR10359977.fastq.gz > output.txt

If you have a machine with large RAM, the same scenario can be simulated using systemd-run. This will limit the memory usage of kraken2 to 6.5GB.

$ time systemd-run --scope -p MemoryMax=6.5G --user time kraken2 --db k2_standard --report report.txt ERR10359977.fastq.gz > output.txt

Depending on the CPU performance, this will take around ~40 minutes to complete.

Loading database information... done.
95064 sequences (14.35 Mbp) processed in 1026.994s (5.6 Kseq/m, 0.84 Mbp/m).
  94939 sequences classified (99.87%)
  125 sequences unclassified (0.13%)
  4.24user 658.68system 38:26.78elapsed 28%CPU

If we try gut WGS(Whole Genome Sequence) sample like SRR6915097 ⁴ ⁵. which contains ~3.3 Gbp, it will take weeks to complete.

$ wget -c https://ftp.sra.ebi.ac.uk/vol1/fastq/SRR691/007/SRR6915097/SRR6915097_1.fastq.gz
$ wget -c https://ftp.sra.ebi.ac.uk/vol1/fastq/SRR691/007/SRR6915097/SRR6915097_2.fastq.gz

$ time systemd-run --scope -p MemoryMax=6G --user time kraken2 --db k2_standard --paired SRR6915097_1.fastq.gz SRR6915097_2.fastq.gz > output.txt

I tried running this on 8 GB machine. Even after 10 days, it processed only 10% of the data.

If we have to process a large number of such samples, it takes months and this is not a practical solution.

Conclusion

In this post, we ran kraken2 on an 8GB machine and learned that it is not feasible to run kraken2 on large samples.

In the next post, we will learn how to speed up the classification process and run classification at 1200 Mbp/m.

Next: Part 2 - Performance Optimisation

Headlamp - k8s Lens open source alternative

Anand Reddy Pandikunta — Sun, 23 Jun 2024 20:18:02 GMT

Since Lens is not open source, I tried out monokle, octant, k9s, and headlamp¹. Among them, headlamp UI & features are closest to Lens.

Headlamp

Headlamp is CNCF sandbox project that provides cross-platform desktop application to manage Kubernetes clusters. It auto-detects clusters and provides cluster wide resource usage by default.

It can also be installed inside the cluster and can be accessed using a web browser. This is useful when we want to access the cluster from a mobile device.

$ helm repo add headlamp https://headlamp-k8s.github.io/headlamp/

$ helm install headlamp headlamp/headlamp

Lets port-forward the service & copy the token to access it.

$ kubectl create token headlamp

# we can do this via headlamp UI as well
$ kubectl port-forward service/headlamp 8080:80

Now, we can access the headlamp UI at http://localhost:8080.

Conclusion

If you are looking for an open source alternative to Lens, headlamp is a good choice. It provides a similar UI & features as Lens, and it is accessible via mobile devices as well.

https://headlamp.dev/ ↩

macOS - Log & track historical CPU, RAM usage

Anand Reddy Pandikunta — Fri, 31 May 2024 20:18:02 GMT

In macOS, we can use inbuilt Activity Monitor or third party apps like Stats to check the live CPU/RAM usage. But, we can't track the historical CPU & memory usage. sar, atop can track the historical CPU & memory usage. But, they are not available for macOS.

Netdata

Netdata¹ is an open source observability tool that can monitor CPU, RAM, network, disk usage. It can also track the historical data.

Unfortunately, it is not stable on macOS. I tried installing it on multiple macbooks, but it didn't work. I raised an issue² on their GitHub repository and the team mentioned that macOS is a low priority for them.

Glances

Glances³ is a cross-platform monitoring tool that can monitor CPU, RAM, network, disk usage. It can also track the historical data.

We can install it using Brew or pip.

$ brew install glances

$ pip install glances

Once it is installed, we can monitor the resource usage using the below command.

$ glances

Glances can log historical data to a file using the below command.

$ glances --export-csv /tmp/glances.csv

In addition to that, it can log data to services like influxdb, prometheus, etc.

Let's install influxdb and export stats to it.

$ brew install influxdb
$ brew services start influxdb
$ influx setup

$ python -m pip install influxdb-client

$ cat glances.conf
[influxdb]
host=localhost
port=8086
protocol=http
org=avilpage
bucket=glances
token=secret_token

$ glances --export-influxdb -C glances.conf

We can view stats in the influxdb from Data Explorer web UI at http://localhost:8086.

Glances provides a prebuilt Grafana dashboard⁴ that we can import to visualize the stats.

From Grafana -> Dashboard -> Import, we can import the dashboard using the above URL.

Conclusion

In addition to InfluxDB, Glances can export data to ~20 services. So far, it is the best tool to log, track and view historical CPU, RAM, network and disk usage in macOS. The same method works for Linux and Windows as well.

Automating Zscaler Connectivity on macOS

Anand Reddy Pandikunta — Tue, 14 May 2024 02:19:53 GMT

Introduction

Zscaler is a cloud-based security service that provides secure internet access via VPN. Unfortunately, Zscaler does not provide a command-line interface to connect to the VPN. We can't use AppleScript to automate the connectivity as well.

Automating Zscaler Connectivity

Once Zscaler is installed on macOS, if we search for LaunchAgents & LaunchDaemons directories, we can find the Zscaler plist files.

$ sudo find /Library/LaunchAgents -name '*zscaler*'
/Library/LaunchAgents/com.zscaler.tray.plist


$ sudo find /Library/LaunchDaemons -name '*zscaler*'
/Library/LaunchDaemons/com.zscaler.service.plist
/Library/LaunchDaemons/com.zscaler.tunnel.plist
/Library/LaunchDaemons/com.zscaler.UPMServiceController.plist

To connect to Zscaler, we can load these services.

#!/bin/bash

/usr/bin/open -a /Applications/Zscaler/Zscaler.app --hide
sudo find /Library/LaunchAgents -name '*zscaler*' -exec launchctl load {} \;
sudo find /Library/LaunchDaemons -name '*zscaler*' -exec launchctl load {} \;

To disconnect from Zscaler, we can unload all of them.

#!/bin/bash

sudo find /Library/LaunchAgents -name '*zscaler*' -exec launchctl unload {} \;
sudo find /Library/LaunchDaemons -name '*zscaler*' -exec launchctl unload {} \;

To automatically toggle the connectivity, we can create a shell script.

#!/bin/bash

if [[ $(pgrep -x Zscaler) ]]; then
    echo "Disconnecting from Zscaler"
    sudo find /Library/LaunchAgents -name '*zscaler*' -exec launchctl unload {} \;
    sudo find /Library/LaunchDaemons -name '*zscaler*' -exec launchctl unload {} \;
else
    echo "Connecting to Zscaler"
    /usr/bin/open -a /Applications/Zscaler/Zscaler.app --hide
    sudo find /Library/LaunchAgents -name '*zscaler*' -exec launchctl load {} \;
    sudo find /Library/LaunchDaemons -name '*zscaler*' -exec launchctl load {} \;
fi

Raycast is an alternative to default spotlight search on macOS. We can create a script to toggle connectivity to Zscaler.

#!/bin/bash

# Required parameters:
# @raycast.schemaVersion 1
# @raycast.title toggle zscaler
# @raycast.mode silent

# Optional parameters:
# @raycast.icon ☁️

# Documentation:
# @raycast.author chillaranand
# @raycast.authorURL https://avilpage.com/

if [[ $(pgrep -x Zscaler) ]]; then
    echo "Disconnecting from Zscaler"
    sudo find /Library/LaunchAgents -name '*zscaler*' -exec launchctl unload {} \;
    sudo find /Library/LaunchDaemons -name '*zscaler*' -exec launchctl unload {} \;
else
    echo "Connecting to Zscaler"
    /usr/bin/open -a /Applications/Zscaler/Zscaler.app --hide
    sudo find /Library/LaunchAgents -name '*zscaler*' -exec launchctl load {} \;
    sudo find /Library/LaunchDaemons -name '*zscaler*' -exec launchctl load {} \;
fi

Save this script to a folder. From Raycast Settings -> Extensions -> Add Script Directory, we can select this folder, and the script will be available in Raycast.

We can assign a shortcut key to the script for quick access.

Conclusion

Even though Zscaler does not provide a command-line interface, we can automate the connectivity using the above scripts.

Screen Time Alerts from Activity Watch

Anand Reddy Pandikunta — Wed, 01 May 2024 01:29:27 GMT

Introduction

Activity Watch¹ is a cross-platform open-source time-tracking tool that helps us to track time spent on applications and websites.

At the moment, Activity Watch doesn't have any feature to show screen time alerts. In this post, we will see how to show screen time alerts using Activity Watch.

Python Script

Activity Watch provides an API to interact with the Activity Watch server. We can use the API to get the screen time data and show alerts.

import json
import os
from datetime import datetime

import requests


def get_nonafk_events(timeperiods=None):
    headers = {"Content-type": "application/json", "charset": "utf-8"}
    query = """afk_events = query_bucket(find_bucket('aw-watcher-afk_'));
window_events = query_bucket(find_bucket('aw-watcher-window_'));
window_events = filter_period_intersect(window_events, filter_keyvals(afk_events, 'status', ['not-afk']));
RETURN = merge_events_by_keys(window_events, ['app', 'title']);""".split("\n")
    data = {
        "timeperiods": timeperiods,
        "query": query,
    }
    r = requests.post(
        "http://localhost:5600/api/0/query/",
        data=bytes(json.dumps(data), "utf-8"),
        headers=headers,
        params={},
    )
    return json.loads(r.text)[0]


def main():
    now = datetime.now()
    timeperiods = [
        "/".join([now.replace(hour=0, minute=0, second=0).isoformat(), now.isoformat()])
    ]
    events = get_nonafk_events(timeperiods)

    total_time_secs = sum([event["duration"] for event in events])
    total_time_mins = total_time_secs / 60
    print(f"Total time: {total_time_mins} seconds")
    hours, minutes = divmod(total_time_mins, 60)
    minutes = round(minutes)
    print(f"Screen Time: {hours} hours {minutes} minutes")

    # show mac notification
    os.system(f"osascript -e 'display notification \"{hours} hours {minutes} minutes\" with title \"Screen TIme\"'")


if __name__ == "__main__":
    main()

This script² will show the screen time alerts using the Activity Watch API. We can run this script using the below command.

$ python screen_time_alerts.py

We can set up a cron job to run this script every hour to show screen time alerts.

$ crontab -e
0 * * * * python screen_time_alerts.py

We can also modify the script to show alerts only when the screen time exceeds a certain limit.

Conclusion

Since Actvity Watch is open-source and provides an API, we can extend its functionality to show screen time alerts. We can also use the API to create custom reports and dashboards.

Setup FTP server on Mac OS X

Anand Reddy Pandikunta — Tue, 30 Apr 2024 18:04:14 GMT

On Linux & Mac OS X, Python comes pre-installed. On Windows, we can install it from Windows store or from https://python.org website.

We can verify the Python version using the below command.

$ python --version
Python 3.11.6

We can use the pyftpdlib library to create an FTP server. We can install the library using the below command.

$ python -m pip install pyftpdlib
[I 11:28:21] concurrency model: async
[I 11:28:21] masquerade (NAT) address: None
[I 11:28:21] passive ports: None
[I 11:28:21] >>> starting FTP server on :::2121, pid=99951 <<<

Now, we can start the FTP server using the below command.

$ python -m pyftpdlib

It will start the FTP server on port 2121. We can connect to the FTP server using the below command.

$ ftp localhost 2121

Timestamp to Relative Time - Kibana Scripted fields

Anand Reddy Pandikunta — Wed, 10 Apr 2024 08:45:33 GMT

When browsing logs in Kibana, there will be a timestamp stamp field on the left for all the docs. It is difficult to read & comprehend the timestamp in the logs. It would be better if we can convert the timestamp to a human-readable relative time like 5 minutes ago, 1 hour ago, etc.

Kibana Scripted Fields

Kibana provides a feature called scripted fields to create new fields in the index pattern. We can use this feature to convert the timestamp to a relative time.

Go to Stack Management -> Index Patterns -> Create index pattern -> Select the index pattern -> Scripted fields, click on Add scripted field, add the below script.

long now = new Date().getTime();

long timestamp = doc['@timestamp'].value.toInstant().toEpochMilli();
long diff = now - timestamp;
if (diff > 7200000) {
  return Math.round(diff / 3600000) + " hours ago";
} else if (diff > 3600000) {
  return Math.round(diff / 3600000) + " hour ago";
} else if (diff > 120000) {
  return Math.round(diff / 60000) + " minutes ago";
} else if (diff > 60000) {
  return (Math.round(diff / 60000) + " minute ago");
} else {
  return Math.round(diff / 1000) + " seconds ago";
}

Once the field is saved, we can go back to Discover and see the new field in the logs. We can toggle the visibility of the Relative Time field to see the relative time.

Conclusion

Instead of looking at the timestamp and calculating the relative time in our head, we can use relative time in Kibana . This will make it easier to read & comprehend the logs.

The Strange Case of Dr. Linux and Mr. Mac

Anand Reddy Pandikunta — Sat, 30 Mar 2024 04:23:53 GMT

Few days back, some of the tests started failing on CI server. When I tried to run the tests locally, they were passing.

After debugging for a while, I found that the tests were failing because of the case sensitivity of the file system. One of the developer was using Linux and had committed 2 files with the same name but different case(config.json, Config.json).

Linux file system is case-sensitive. So these 2 files will be shown as 2 different files.

But Mac/Windows file system is case-insensitive. Out of these 2 files, only one file will be shown.

Due to this, the tests were failing on Linux but passing on Mac. Once the case of the file was corrected, the tests started passing on both the systems.

I have been using Mac for a long time and never faced this issue. Even though Mac's APFS is case-insensitive, we can create a case-sensitive volume using Disk Utility.

We have to be aware of these differences when working on a project with developers using different OS.

Archiving Option Chain Data

Anand Reddy Pandikunta — Thu, 29 Feb 2024 08:44:52 GMT

BSE & NSE are the prominent exchanges in India, and they provide option chain data for the stocks & indices listed in their exchange.

The option chain data is available for the current date. But it is not available for the past dates. This is a problem for traders who want to analyze the historical option chain data.

ArchiveBox

ArchiveBox¹ is a tool to archive web pages. It can be used to archive the option chain data for the stocks & indices.

Let's install ArchiveBox.

$ pip install archivebox
$ mkdir option_chain
$ cd option_chain
$ archivebox init
$ archivebox setup

We can start the server(defaults to https://localhost:8000) and add URLs manually to archive them.

$ archivebox server

There are 180+ stocks in FNO segment & 6 indices with weekly expiry. We can write a simple Python script to generate all combinations of URLs for the option chain data and archive them using ArchiveBox.

Once URLs is generated, we can use the following command to archive them.

$ archivebox add --input-file urls.txt

These URLs will be archived and stored in the archive directory. Since we want to archive the data regularly, we can use a schedule to archive daily.

$ archivebox schedule --every=day --depth=0 '{{url}}'

This will archive the option chain data for the stocks & indices on a daily basis.

Conclusion

Browsing the archived data of a single url is a bit difficult. Wayback machine provides a better interface to browse the archived data. I have raised an issue² regarding the same in the ArchiveBox repository. Once the UI issue is resolved, this will serve as the best tool to browse the historical option chain data.

Cross Platform File Explorer in 50 lines of code

Anand Reddy Pandikunta — Sat, 27 Jan 2024 12:29:05 GMT

In an earlier post, I wrote about why I need a "line count" column in file explorer and how I wrote a Lua script to see it in xplr file manager.

xplr has only terminal interface. It is hard for non-developers to use it. I wanted a small team to use this feature so that it will save several hours of their time. So I decided to write a cross-platform GUI app.

GUI app

Since I am familiar with PySimpleGUI, I decided to write a simple file explorer using it.

As seen in the above screenshot, the file explorer has a "Line Count" column. It is a simple Python script with ~50 lines of code.

The project is open source and source code is available at github.com/AvilPage/LCFileExplorer.

Cross Platform

A new user can't directly run this Python script on his machine unless Python is already installed. Even if Python is installed, he has to install the required packages and run it. This requires technical expertise.

To make it easy for non-tech users to run this program, I decided to use PyInstaller to create a single executable file for each platform.

I created a GitHub action to build the executable files for Windows, Linux, and macOS. The action is triggered on every push to the master branch. This will generate .exe file for Windows, .AppImage file for Linux, and .dmg file for macOS. The executable files are uploaded to the artifacts.

Conclusion

It is easy to create a cross-platform GUI app using Python and PySimpleGUI. It is also easy to distribute the apps built with Python using pyinstaller.

Running tests in parallel with pytest & xdist

Anand Reddy Pandikunta — Sat, 30 Dec 2023 14:46:31 GMT

When tests are taking too long to run, an easy way to speed them up is to run them in parallel.

When using pytest as test runner, pytest-xdist & pytest-parallel plugins makes it easy to run tests concurrently or in parallel.

pytest-parallel works better if tests are independent of each other. If tests are dependent on each other, pytest-xdist is a better choice.

If there are parameterised tests, pytest-xdist will fail as the order of the tests is not guaranteed.

$ pytest -n auto tests/

Different tests were collected between gw0 and gw1. The difference is: ...

To fix this, we have to make sure that the parameterised tests are executed in the same order on all workers. It can be achieved by sorting the parameterised tests by their name.

Alternatively, we can use pytest-randomly plugin to order the tests.

Remap F4 to Raycast, Alfred (cmd + space)

Anand Reddy Pandikunta — Thu, 30 Nov 2023 18:03:16 GMT

On Mac keyboard, there is F4 key which opens Spotlight¹ by default. I use Raycast² a lot instead of Spotlight and wanted to remap F4 to Raycast.

There is an app called Karabiner-Elements³ which can be used to remap keys. After the app is installed, we can use this rule⁴ called Map F4 to cmd+space.

You can import the rule from the above URL directly. Once the rule is imported & enabled, F4 will be remapped to cmd + space as shown in the video below.

Add "Line Count" Column in File Manager

Anand Reddy Pandikunta — Mon, 06 Nov 2023 20:43:13 GMT

While monitoring an ETL pipeline, I browse a lot of files and often need to know how many lines are there in a file. For that, I can switch to that directory from terminal and run wc -l for that.

To avoid the hassle of switching to the directory and running a command in the terminal, I wrote a simple lua script to show line count column in xplr¹ file manager.

Failed Attempts

Initially I set out to write a Finder² plugin to show the line count column. But I couldn't find a way to get the line count of a file in Finder plugin. I have explored other GUI file managers but none of them have a way to show custom columns with line count.

Finally, I stumbled upon xplr a TUI file manager, and it was a breeze to write a lua script to show the line count column.

xplr - line count

xplr can be installed via brew.

$ brew install xplr

$ xplr --version
xplr 0.21.3

xplr reads the default configuration from ~/.config/xplr/init.lua. The following configuration shows the line count column in xplr.

version = '0.21.3'

xplr.fn.custom.fmt_simple_column = function(m)
  return m.prefix .. m.relative_path .. m.suffix
end

xplr.fn.custom.row_count = function(app)
  if not app.is_file then
    return "---"
  end

  local file = io.open(app.absolute_path, "r")
  if file then
    local row_count = 0
    for _ in file:lines() do
      row_count = row_count + 1
    end
    file:close()
    return tostring(row_count)
  end
end


xplr.config.general.table.header.cols = {
  { format = "  path" },
  { format = "line_count" },
}

xplr.config.general.table.row.cols = {
  { format = "custom.fmt_simple_column" },
  { format = "custom.row_count" },
}

xplr.config.general.table.col_widths = {
  { Percentage = 30 },
  { Percentage = 20 },
}

This will show a row count on launch.

Conclusion

xplr is a very powerful file manager, and it is very easy to write lua scripts to create custom columns. I couldn't find a way to sort items based on the custom column. Need to explore more on that.

Guide to setting up GeoDjango on Mac M1

Anand Reddy Pandikunta — Mon, 30 Oct 2023 03:29:23 GMT

There are a lot of guides on setting up GeoDjango and PostGIS. But most of them are outdated and doesn't work on Mac M1. In this article, let us look at how to set up GeoDjango on Mac M1/M2.

Ensure you have already installed Postgres on your Mac.

Install GeoDjango

The default GDAL version available on brew fails to install on Mac M1.

$ brew install gdal
==> cmake --build build
Last 15 lines from /Users/chillaranand/Library/Logs/Homebrew/gdal/02.cmake:
    [javac] Compiling 82 source files to /tmp/gdal-20231029-31808-1wl9085/gdal-3.7.2/build/swig/java/build/classes
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 7
    [javac] error: Source option 7 is no longer supported. Use 8 or later.
    [javac] error: Target option 7 is no longer supported. Use 8 or later.

BUILD FAILED
/tmp/gdal-20231029-31808-1wl9085/gdal-3.7.2/swig/java/build.xml:25: Compile failed; see the compiler error output for details.

Total time: 0 seconds
gmake[2]: *** [swig/java/CMakeFiles/java_binding.dir/build.make:108: swig/java/gdal.jar] Error 1
gmake[2]: Leaving directory '/private/tmp/gdal-20231029-31808-1wl9085/gdal-3.7.2/build'
gmake[1]: *** [CMakeFiles/Makefile2:9108: swig/java/CMakeFiles/java_binding.dir/all] Error 2
gmake[1]: Leaving directory '/private/tmp/gdal-20231029-31808-1wl9085/gdal-3.7.2/build'
gmake: *** [Makefile:139: all] Error 2

We can use conda to install gdal. Create a new environment and install gdal in it.

$ conda create -n geodjango python=3.9
$ conda install -c conda-forge gdal
$ pip install django
$ pip install psycopg2-binary

Once installed, you can check the version using gdalinfo --version.

Remaining dependencies can be installed via brew.

$ brew install postgresql
$ brew install postgis
$ brew install libgeoip

Let's create a new django project and add spatial backends.

$ django-admin startproject geodjango

Add django.contrib.gis to INSTALLED_APPS in settings.py.

INSTALLED_APPS = [
    ...,
    'django.contrib.gis',
]

Add the following to DATABASES in settings.py.

DATABASES['default']['ENGINE'] = 'django.contrib.gis.db.backends.postgis'

Since we used conda to install gdal, we need to set the path to gdal in our django settings. Run locate libgdal.dylib to find the path to gdal.

GDAL_LIBRARY_PATH = '/opt/homebrew/anaconda3/envs/geodjango/lib/libgdal.dylib'

Similarly, we need to set GEOS_LIBRARY_PATH as well.

GEOS_LIBRARY_PATH = '/opt/homebrew/anaconda3/envs/geodjango/lib/libgeos_c.dylib'

Now, we can create a new app and add PointField or any other spatial fields to our models.

$ python manage.py startapp places

from django.contrib.gis.db import models

class Place(models.Model):
    name = models.CharField(max_length=100)
    location = models.PointField()

Conclusion

In this article, we looked at how to set up GeoDjango on Mac M1. We used conda to install gdal and brew to install other dependencies.

Remotely Access Raspberry Pi or Any Device

Anand Reddy Pandikunta — Mon, 25 Sep 2023 01:49:54 GMT

I recently started using Raspberry Pi and I wanted to access it when I am outside of home as well. After trying out few solutions, I stumbled upon Tailscale¹.

Tailscale is a mesh VPN that makes it easy to connect out devices, wherever they are. It is free for personal use and supports all major platforms like Linux, Windows, Mac, Android, iOS, etc.

Installation

I installed tailscale on Raspberry Pi using the following command.

$ curl -fsSL https://tailscale.com/install.sh | sh

Setup

Once the installation is done, I run tailscale up to start the daemon. This opened a browser window and asked me to log in with email address. After I logged in, I can see all the devices in the tailscale dashboard.

tailscale has CLI tool as well and status can be viewed with the following command.

$ tailscale status
100.81.13.75   m1                    avilpage@  macOS   -
100.12.12.92   rpi1.tailscale.ts.net avilpage@  linux   offline

I also set up a cron job to start tailscale daemon on boot.

$ crontab -e
@reboot tailscale up

Access

Now I can access the device from anywhere using the tailscale IP address. For example, if the IP address is 100.34.2.23. I can ssh into the device using the following command.

$ ssh pi@100.81.12.92

It also provides DNS names for each device. For example, I can ssh into the device using the following command as well.

$ ssh pi@raspberry3.tailscale.net

Conclusion

Tailscale is a great tool to access devices remotely. It is easy to set up and works well with Raspberry Pi, Mac & Linux as well.

https://en.wikipedia.org/wiki/Tailscale ↩

Create Telegram Bot To Post Messages to Group

Anand Reddy Pandikunta — Mon, 04 Sep 2023 13:14:32 GMT

Introduction

Recently I had to create a Telegram bot again to post updates to a group based on IoT events. This post is just a reference for future.

Create a Telegram Bot

First, create a bot using BotFather in the Telegram app and get the API token. Then, create a group and add the bot to the group. This will give the bot access to the group.

Post Messages to the Group

Now, we need to fetch the group id. For this, we can use the following curl API call.

curl is available by default on Mac and Linux terminals. On Windows, we can use curl from command prompt.

$ curl -X GET https://api.telegram.org/bot<API_TOKEN>/getUpdates

{
  "ok": true,
  "result": [
    {
      "update_id": 733724271,
      "message": {
        "message_id": 9,
        "from": {
          "id": 1122,
          "is_bot": false,
          "username": "ChillarAnand",
          "language_code": "en"
        },
        "chat": {
          "id": -114522,
          "title": "DailyPythonTips",
          "type": "group",
          "all_members_are_administrators": true
        },
        "date": 1694045795,
        "text": "@DailyPythonTipsBot hi",
        "entities": [
          {
            "offset": 0,
            "length": 19,
            "type": "mention"
          }
        ]
      }
    }
  ]
}

This will return a JSON response with the group id. It sends empty response if there are no recent conversations.

In that case, send a dummy message to the bot in the group and try again. It should return the group id in the response.

We can use this group id to post messages to the group.

$ curl -X POST https://api.telegram.org/bot<API_TOKEN>/sendMessage -d "chat_id=<GROUP_ID>&text=Hello"

{
  "ok": true,
  "result": {
    "message_id": 12,
    "from": {
      "id": 3349238234,
      "is_bot": true,
      "first_name": "DailyPythonTipsBot",
      "username": "DailyPythonTipsBot"
    },
    "chat": {
      "id": -114522,
      "title": "DailyPythonTips",
      "type": "group",
      "all_members_are_administrators": true
    },
    "date": 1694046381,
    "text": "Hello"
  }
}

Here is the message posted by the bot in the group.

Now, we can use this API to post messages to the group from our IoT devices or from any other devices where curl command is available.

Periodically Launch an App in Background

Anand Reddy Pandikunta — Wed, 23 Aug 2023 13:33:49 GMT

I recently started using Outlook app on my Mac. If the app is closed, it won't send any notifications. When I accidentally close the app, until I re-open it, I won't get any notifications.

I want to ensure that it starts periodically so that I don't miss any notifications for meetings.

After trying out various methods, I ended up using open command with cron to launch the app every 15 minutes.

$ crontab -e

*/15 * * * * /usr/bin/open -a "Microsoft Outlook"

This will launch the app every 15 minutes. This is inconvenient as it will bring Outlook to foreground every 15 minutes.

To avoid this, I passed -g option to run it in background.

$ crontab -e

*/15 * * * * /usr/bin/open -g -a "Microsoft Outlook"

This silently launches the app in background without causing any disturbance. Since the app is running the background, it will send notifications for any meetings.

This will ensure that I don't miss any meetings, even if I close outlook accidentally.

Rearrange CSV columns alphabetically from CLI

Anand Reddy Pandikunta — Fri, 04 Aug 2023 01:49:54 GMT

We can use tools like KDiff3 to compare two CSV files. But, it is difficult to identify the diff when the columns are not in the same order.

For example, look at the below output of 2 simple csv files.

Even though it highlights the diff, it is difficult to identify the diff because the columns are not in the same order. Here is the same diff after rearranging the columns alphabetically.

Now, it is easy to identify the diff.

Rearrange CSV columns alphabetically

We can write a simple python script using Pandas¹ as follows.

#! /usr/bin/env python3

"""
re-arrange columns in alphabetical order
"""
import sys

import pandas as pd


def colsort(df):
    cols = list(df.columns)
    cols.sort()
    return df[cols]


def main():
    input_file = sys.argv[1]
    try:
        output_file = sys.argv[2]
    except IndexError:
        output_file = input_file
    df = pd.read_csv(input_file)
    df = colsort(df)
    df.to_csv(output_file, index=False)


if __name__ == '__main__':
    main()

We can use this script as follows.

$ python3 rearrange_csv_columns.py input.csv output.csv

Instead of writing a script by ourselves, we can use miller² tool. Miller can perform various operations on CSV files. We can use sort-within-records to sort the columns.

$ mlr --csv sort-within-records -f input.csv > output.csv

Conclusion

We can use miller to sort the columns in a CSV file. This will help us to identify the diff easily when comparing two CSV files.

Train LLMs with Custom Dataset on Laptop

Anand Reddy Pandikunta — Thu, 06 Jul 2023 23:06:42 GMT

Problem Statement

I want to train a Large Language Model(LLM)¹ with some private documents and query various details.

Journey

There are open-source available LLMs like Vicuna, LLaMa, etc which can be trained on custom data. However, training these models on custom data is not a trivial task.

After trying out various methods, I ended up using privateGPT² which is quite easy to train on custom documents. There is no need to format or clean up the data as privateGPT can directly consume documents in many formats like txt, html, epub, pdf, etc.

Training

First, let's clone the repo, install requirements.txt and download the default model.

$ git clone https://github.com/imartinez/privateGPT
$ cd privateGPT
$ pip3 install -r requirements.txt
$ wget https://gpt4all.io/models/ggml-gpt4all-j-v1.3-groovy.bin

$ cp example.env .env
$ cat .env
MODEL_TYPE=GPT4All
MODEL_PATH=ggml-gpt4all-j-v1.3-groovy.bin

I have sourced all documents and kept them in a folder called docs. Let's ingest(train) the data.

$ cp ~/docs/* source_documents

$ python ingest.py

This will take a while depending on the number of documents we have. Once the ingestion is done, we can start querying the model.

$ python privateGPT.py
Enter a query: Summarise about Gaaliveedu

The default GPT4All-J v1.3-groovy³ model doesn't provide good results. We can easily swap it with LlamaCpp⁴. Lets download the model and convert it.

$ git clone https://huggingface.co/openlm-research/open_llama_13b

$ git clone https://github.com/ggerganov/llama.cpp.git
$ cd llama.cpp
$ python convert.py ../open_llama_13b
Wrote ../open_llama_13b/ggml-model-f16.bin

We can now update the .env file to use the new model and start querying again.

$ cat .env
MODEL_TYPE=LlamaCpp
MODEL_PATH=/path/to/ggml-model-f16.bin

$ python privateGPT.py
Enter a query: Summarise about Gaaliveedu

Conclusion

This makes it easy to build domain-specific LLMs and use them for various tasks. I have used this to build a chatbot for my internal docs and it is working well.

Remote Debug Docker Container with PyCharm

Anand Reddy Pandikunta — Sun, 11 Jun 2023 15:36:04 GMT

Problem Statement

How to debug a Python application running inside a Docker container that is launched by a third-party process using PyCharm?

Solution

Install the pydevd-pycharm package in the Docker image.

RUN pip install 'pydevd-pycharm~=222.4554.11'

Add the following lines to the Python script that you want to debug.

import pydevd_pycharm
pydevd_pycharm.settrace('host.docker.internal', port=12345, stdoutToServer=True, stderrToServer=True)

Create a new Python Remote Debug configuration in PyCharm with the following settings.

Run the Remote Debug configuration in PyCharm.
Run the Docker container with the following command or let a shell script or another package run the container.

$ docker build . -t flask_web
$ docker run --rm flask_web

Explanation

The pydevd-pycharm package is a Python debugger that can be used to debug a Python application running inside a Docker container. The pydevd_pycharm.settrace() function is used to connect the debugger to the PyCharm IDE. The host.docker.internal is the hostname of the host machine from inside the Docker container. The port is the port number that is used to connect to the PyCharm IDE. The stdoutToServer and stderrToServer are used to redirect the standard output and standard error to the PyCharm IDE.

Gotchas

You might face the following error depending on the version of the pydevd-pycharm package.

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 218, in locate_app
    __import__(module_name)
  File "/app/app.py", line 5, in <module>
    import pydevd_pycharm
  File "/usr/local/lib/python3.10/site-packages/pydevd_pycharm.py", line 3, in <module>
    from pydevd import settrace
  File "/usr/local/lib/python3.10/site-packages/pydevd.py", line 41, in <module>
    from _pydevd_bundle import pydevd_utils
  File "/usr/local/lib/python3.10/site-packages/_pydevd_bundle/pydevd_utils.py", line 24, in <module>
    from _pydevd_asyncio_util.pydevd_asyncio_utils import eval_async_expression_in_context
ModuleNotFoundError: No module named '_pydevd_asyncio_util'

There seems to be an issue with all 223.*.* versions. The solution is to use the 222.*.* version.

You might face ConnectionRefused error when running the docker container.

  File "/usr/local/lib/python3.10/site-packages/pydevd.py", line 1758, in _locked_settrace
    debugger.connect(host, port)  # Note: connect can raise error.
  File "/usr/local/lib/python3.10/site-packages/pydevd.py", line 660, in connect
    s = start_client(host, port)
  File "/usr/local/lib/python3.10/site-packages/_pydevd_bundle/pydevd_comm.py", line 463, in start_client
    s.connect((host, port))
ConnectionRefusedError: [Errno 111] Connection refused

Ensure that you have started the Remote Debug configuration in PyCharm before running the docker container.

Mastering HammerSpoon - Excel Automation

Anand Reddy Pandikunta — Sun, 21 May 2023 12:44:00 GMT

Introduction

Recently, I have been using Excel a lot. When opening a new Excel file, I have to do the following:

Maximize the window
Select all columns and fit them to its width
Apply filters to all columns
Freeze the first row

When opening and closing multiple Excel files, this becomes a tedious task. So, I decided to automate this and came across Hammerspoon.

HammerSpoon

Hammerspoon¹ is a powerful automation tool for macOS. It allows you to write Lua scripts to automate various tasks and make our keybindings.

First, let's install Hammerspoon using Homebrew.

$ brew install hammerspoon

We can write our automation script in ~/.hammerspoon/init.lua file. Let us see how we can automate the above tasks.

Automating Excel

-- excel
function excel(appObject)
   local win = hs.window.focusedWindow()
   if (not win) then
      return
   end
    win:maximize()

    appObject:selectMenuItem({"Edit", "Select all"})
    appObject:selectMenuItem({"Format", "Column", "Autofit Selection"})
    appObject:selectMenuItem({"Data", "Auto-filter"})

end)


function applicationWatcher(appName, eventType, appObject)
   local w = hs.application.watcher
   if (eventType == w.activated or eventType == w.launched) then
      if (appName == "Microsoft Excel") then
         excel(appObject)
      end
   end
end

This script will watch for application events and when Excel is launched or activated, it will call the excel function.

The excel function will maximize the window, select all columns and fit them to it's width, apply filters to all columns.

Free top row option is not available in the standard menu. So, I have added it to the quick access toolbar and click it via mouse event.

Conclusion

Hammerspoon is a powerful tool for various automation tasks. In addition to that it can replace a lot of utility apps like CheatSheet, BlueSnooze², Rectangle, ShiftIT³, HotKey etc. I have replaced most of the utility apps with Hammerspoon, and it is working great. I will be writing about it in detail in the upcoming posts.

Record Resource Usage of Single Process

Anand Reddy Pandikunta — Fri, 14 Apr 2023 00:48:37 GMT

Introduction

On Linux & Mac, we can use an inbuilt top command line tool to monitor the resource usage of a single process in real time.

# On Linux, for a given pid
$ top -p 1234

# On Mac, for a given pid
$ top -pid 1234

In this article, we will see how to record and plot resource usage of a single process using top and a Python package called psrecord¹.

Record Resource Usage

In some cases, we need to record the resource usage of a process to use it later. For example, we can use this data to find out the peak resource usage of a process. For this, we can use top to log resource usage into a text file.

# On Linux, for a given pid
$ top -p 1234 -b -d 1 > top.log

# On Mac, for a given pid
$ top -l 0 -s 1 -pid 32515 | awk 'NR%13==0; fflush(stdout)' > top.log

Once we have the log file, we can view the raw data or we can plot the resource usage by using tools like gnuplot or matplotlib.

Instead of using top command, we can use psrecord to record the resource usage of a process. psrecord is a Python package that can be installed all using pip.

$ python -m pip install psrecord

Once installed, we can use psrecord to record the resource usage of a process.

# record resource usage of a process with pid 1234
$ psrecord 1234 --log top.log

# start and record resource usage of a process
$ psrecord python script.py --plot graph.png

We can view the raw data in the log file.

# view raw data
$ head top.log
$ head a.txt
# Elapsed time   CPU (%)     Real (MB)   Virtual (MB)
       0.000        0.000        5.000   399461.438
       0.000       93.700        5.000   399461.438
       0.000       96.300        5.000   399461.438
       0.000       91.900        5.000   399461.438

Here is the generated graph.

Conclusion

In this article, we have seen how to record and plot resource usage of a single process using top(inbuilt tool), psrecord(3rd party package).

https://pypi.org/project/psrecord/ ↩

Reducing System Load With ChatGPT

Anand Reddy Pandikunta — Sat, 01 Apr 2023 02:25:49 GMT

Problem Statement

I am using M1 Macbook Air for Python development purposes. Since M1 uses ARM architecture, many Python packages don't have wheels for ARM64/aarch64. confluent-kafka-python is one of them.

I had to run AMD64 docker container to use confluent-kafka-python. Since it is a cross-architecture container, its CPU usage is too high and performance was too slow.

Solution

To reduce system load, I decided to build aarch64 wheels for confluent-kafka-python. I looked at open issues on GitHub and asked maintainers how to build aarch64 wheels. There was no response¹ from them.

As a workaround, I asked ChatGPT² on how to build confluent-kafka-python aarch64 wheels in a docker container.

This initial suggestion didn't work as confluent-kafka-python depends on librdkafka which is a C library. I had to build librdkafka from source for aarch64 and then build confluent-kafka-python from source.

To build librdkafka from the source, I again asked ChatGPT. After making minor changes to the snippet suggested by ChatGPT, I was able to build librdkafka from the source for aarch64.

Here is the final snippet:

FROM ubuntu:22.04

ARG DEBIAN_FRONTEND=noninteractive

RUN apt update && apt install -y \
  wget git curl g++ make postgresql-client \
  nano less shared-mime-info openjdk-17-jre-headless \
  libpq-dev vim tzdata python3 python3-dev

RUN apt install -y python3-pip
RUN python3 -m pip install setuptools

WORKDIR /
RUN git clone https://github.com/confluentinc/confluent-kafka-python
WORKDIR confluent-kafka-python

COPY . /app
WORKDIR /app
RUN ./configure --arch=aarch64 --prefix=/usr
RUN make
RUN make install

WORKDIR /confluent-kafka-python
RUN python3 setup.py install

Conclusion

By running native containers, I was able to reduce the system load by ~50%. With ChatGPT, it is easy to build/tweak programs in languages & environments that we are not familiar with.

Automator Quick Action for KDiff3 in Finder

Anand Reddy Pandikunta — Fri, 10 Mar 2023 02:54:11 GMT

The need for quick action

kdiff3¹ is a diff & merge tool that compares multiple files/directories and shows the difference line by line and character by character as shown below.

In Windows, when we select multiple files/directories and right click on them, it will show the option to compare selected items with kdiff3.

However, in Macbook, it doesn't show this option. In this tutorial, let us see how we can create the same quick action in the right-click menu when we right-click on the files/directories.

Creating Quick Action

Let us open Automator², create new file and select Quick Action.

On the left side select Utilities and then select Run Shell Script.

For Workflow receives current, select files or folders and then select in Finder.

Then select pass input as agruments and in the script section let us add the following command.

/path/to/kdiff3 $1 $2

After adding the command, save this Quick Action.

Now if we relaunch Finder app and then select multiple directories, and right click we can see Compare with KDiff3 in quick actions.

Conclusion

Even though we can use the command line to compare the files/directories, it is always good to have a quick action in the right-click menu.

Setup Kubernetes Anywhere with Single Command

Anand Reddy Pandikunta — Fri, 03 Mar 2023 21:25:27 GMT

Introduction

In an earlier article, we have seen how to set up Kubernetes on M1 Mac. That involved spinning up a VM and installing Kubernetes¹ on it. In this article, we will see how to set up Kubernetes directly on Docker so that we can use the same set-up on any operating system.

Prerequisites

Ensure you have Docker installed on your system. If you are on a Mac or Windows, you can install Docker Desktop².

k3s/k3d

k3s³ is a lightweight Kubernetes distribution by Rancher. It is a single binary that can be run on any Linux machine. But it doesn't work on Mac or Windows.

k3d⁴ is a wrapper around k3s that allows you to run k3s on Docker. It is a great option for running Kubernetes on your local machine.

Installation

k3d can be installed using the following command:

$ brew install k3d  # mac
$ chocolatey install k3d  # windows
$ curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash # linux

Once it is installed, we can create a cluster using the following command:

$ k3d cluster create demo

This will launch a cluster with a single node. We can also setup a multi-node cluster using the following command:

$ k3d cluster create demo --servers 3 --agents 2

We can verify the cluster is up and running using the following command:

$ kubectl get nodes

We can also use GUI tools like Lens to manage and navigate the cluster. In the above video we have used Lens to create a Jenkins deployment as well.

Conclusion

In this article, we have seen how to set up Kubernetes on Docker. This is a great option for running Kubernetes on your local machine. We can also use this to run production setup for small applications.

Using Conda/Mamba with Python Pip on M1 Mac

Anand Reddy Pandikunta — Mon, 27 Feb 2023 19:31:01 GMT

Introduction

From 2020, all Apple MacBooks are powered by Apple Silicone(M1) chips. This chip uses Aarch64 architecture which is different from x86 architecture which was used by Intel chips earlier.

Python is a cross-platform language. It can run on any platform. However, Python packages are compiled for specific platforms. For example, a package compiled for x86 will not work on Aarch64 platform. Also, many Python packages are not yet available for ARM64/Aarch64 platform.

M1 Mac and Python

If we want to run a python package on M1 Mac which doesn't have ARM64 support, we need to use an emulator(or a cross-architecture Docker image). This will significantly slow down the application.

An alternate solution is to build packages for ARM64 platform. Building binary packages from the source code requires a lot of time and effort. Also, we need to build the package for each Python version.

Instead of building from source, we can use Conda/Mamba to install Python packages as well as other system packages. Conda/Mamba will automatically install the correct binary for the package.

For example, python-confluent-kafka³ package doesn't have Linux aarch64 support. To run it on aarch64 platform, we have to build from source which takes a lot of time. Instead, we can simply install it using Conda/Mamba with a single command.

$ conda install -c conda-forge python-confluent-kafka

Similar to pip, Conda can also install all the packages mentioned in a file like requirements.txt.

$ conda install --file requirements.txt

Conclusion

In data science ecosystem, Conda¹/Mamba² are widely used as package managers. In web development ecosystem, they are not as widely used as pip.

Conda/Mamba is a great cross-platform system package manager, and it doesn't have all the Python packages available on PyPi. However, we can use it along with pip for easy package management on M1 Macbook.

Hot Module Reload In Python With Reloadium

Anand Reddy Pandikunta — Thu, 16 Feb 2023 06:28:58 GMT

Introduction

Hot module reloading is a feature that allows you to reload a module without restarting the whole application. This is very useful when we are developing/debugging an application, and we want to see the changes instantaneously.

Reloadium

Reloadium¹ is an advanced hot reloading library for python.

Instead of writing an article, I thought it would be much easier to show a live demo of Reloadium. In the below video, we can see how reloadium greatly improves developer experience.

Currently, reloadium can be used as a standalone tool. We can install it from PyPi and run any arbitrary python script with reloadium.

$ pip install reloadium
$ reloadium run myscript.py

Alternatively, it is available as a plugin for PyCharm as shown in the above video. VS Code support is also in the works.

Reloadium is capable of profiling too. Without writing a single line of code, we can profile Python code. But that's a topic for another article.

Conclusion

I have been using Reloadium from a few months, and it has become an essential part of my development workflow. These days I always run all the scripts or apps in debug mode with reloadium directly.

https://github.com/reloadware/reloadium ↩

Best Pay After Placement Courses In India

Anand Reddy Pandikunta — Mon, 23 Jan 2023 12:31:14 GMT

Introduction

In India, huge number of students are graduating every year. Most of them are not able to get a job right after graduation. In order to get a job in IT industry, students need to have some technical skills.

There are thousands of institutes in India that are providing paid technical courses. Depending on the course, the fees can be anywhere between 5,000 to 5 lakhs. The percentage of students who are getting a job after doing these courses is extremely low. In addition to that quite a few students are not able to afford fees to join these courses.

Pay After Placement Courses

To combat this problem, some institutes are providing pay after placement(PAP) courses. In these courses, students will pay the fees only after getting a job with a desired package. This is a win-win situation for both the students and the institutes. This is a far better option than paying the fees upfront and not getting a job. These courses are also called as income share agreement(ISA) courses.

Here is a list of top pay after placement courses in India for front end developers, back end developers, full stack developers, data scientists, machine learning engineers, and data engineers.

Site-Rank	Institute	Fee(Approx INR)
134,988	Sharpener Tech	68,000
59,928	AccioJob	177,000
321,989	Placewit	0 (Upto 10L)
37,294	Masai School	350,000
1,482,058	Digikul	234,000
1,554,412	10xAcademy	295,000
295,708	Function Up	295,000
84,513	AlmaBetter	Not known

Conclusion

Most of these courses have an entrance test that candidates have to clear before joining the course. However taking these courses is far better than paying the fees upfront and not getting a job. If you are interested in any of these courses, you can apply for the entrance test and join the course.

Pipe tail output into column

Anand Reddy Pandikunta — Mon, 02 Jan 2023 00:56:28 GMT

column command-line utility formats its input into multiple columns and aligns it nicely. It is useful for formatting output of csv files, or other commands.

$ cat users.csv
id,user,active
1,John Doe,true
2,Will Smith,false

$ column -s, -t < users.csv
id  user        active
1   John Doe    true
2   Will Smith  false

tail command-line utility prints the last 10 lines of a file. It can be used with -f option to follow the file as it grows.

$ tail -f users.csv
id,user,active
1,John Doe,true
2,Will Smith,false

To format the output of tail -f command, we can't use column command directly. column command can't produce output until it receives all the input. It needs all the input beforehand to calculate the column widths.

$ tail -f users.csv | column -s, -t

So, the above command won't work.

As the goal is to follow the output of the file, we can use watch command for this. watch command executes a command periodically, and displays its output.

$ watch -n 1 "tail -n 20 users.csv | column -s, -t"

This command will fetch the last 20 lines of the file, pipe it to column command, and display the output. It will repeat the command every 1 second.

As the file grows beyond 20 lines, the headers will be truncated. To preserve the headers, we can use head command in addition to tail command.

$ watch -n 1 "(head -n1 && tail -n20) < users.csv| column -s, -t"

This command will print the first line of the file, and then the last 20 lines of the file. The output will be piped to column command, and displayed.

Here is a screenshot of the output of a demo csv.

This makes it easy to watch the output of a file as it grows.

Change Kafka Log Directory & Format It

Anand Reddy Pandikunta — Sat, 24 Dec 2022 06:49:41 GMT

Problem Statement

On my local Mac, I was using Kafka to pass messages between various applications. Due to some reason, when I tried to start Kafka recently, it was failing to start and here are the relevant error logs.

[2022-12-23 11:57:06,217] WARN [Controller 1] writeNoOpRecord: failed with unknown server exception RuntimeException at epoch 139 in 5198 us.  Renouncing leadership and reverting to the last committed offset 927938. (org.apache.kafka.controller.QuorumController)

[2022-12-23 11:57:06,536] ERROR [Controller 1] registerBroker: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)

[2022-12-23 12:23:35,834] ERROR [RaftManager nodeId=1] Had an error during log cleaning (org.apache.kafka.raft.KafkaRaftClient)
org.apache.kafka.common.errors.OffsetOutOfRangeException: Cannot increment the log start offset to 927939 of partition __cluster_metadata-0 since it is larger than the high watermark 926507
[2022-12-23 12:23:36,035] WARN [Controller 1] writeNoOpRecord: failed with unknown server exception RuntimeException at epoch 294 in 137 us.  Renouncing leadership and reverting to the last committed offset 927938. (org.apache.kafka.controller.QuorumController)
java.lang.RuntimeException: Cant create a new in-memory snapshot at epoch 926507 because there is already a snapshot with epoch 927938

[2022-12-23 12:23:36,252] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)

Debugging

I tried to figure out the exact root cause. After multiple failed attempts, I decided to change the log directory temporarily and go ahead for now.

Solution

I create a new temporary directory and set the log directory to that.

$ mkdir /tmp/kafka-logs

# inside server.properties
log.dirs=/tmp/kafka-logs

When I started the Kafka server, it failed.

$ kafka-server-start server.properties

[2022-12-23 12:30:50,018] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
org.apache.kafka.common.KafkaException: No `meta.properties` found in /tmp/ (have you run `kafka-storage.sh` to format the directory?)

I ran the kafka-storage script to format the directory. First, we need to get the cluster-id. Since we already know the old kafa-logs directory, we can get the cluster-id from there.

$ cat ~/homebrew/var/lib/kraft-combined-logs/meta.properties 
#
#Thu Oct 20 11:48:12 IST 2022
cluster.id=5MB5lq-XT-6JzQqJeIuhWQ
node.id=1
version=1

Now, we can format the new directory.

$ kafka-storage format --config server.properties --cluster-id 5MB5lq-XT-6JzQqJeIuhWQ

Formatting /tmp/kafka-logs/ with metadata.version 3.3-IV3.

After changing log directory, Kafka has started working.

$ kafka-start-server /path/to/server.properties

Since I have changed log directory all older messages are lost. Since I am doing this on my local machine, it is fine. Need to revisit it to debug further.

Hands-on RabbitMQ Tutorial

Anand Reddy Pandikunta — Tue, 20 Dec 2022 23:42:21 GMT

A short hands-on guide to get started with RabbitMQ for people who are in a hurry.

What is RabbitMQ?

Image Credit: CloudAMQP

RabbitMQ¹ is an open-source message broker software that implements the Advanced Message Queuing Protocol (AMQP). With RabbitMQ, producer and consumer applications can communicate asynchronously, and they will be completely decoupled.

RabbitMQ Terminology

Producer: A producer is a client that publishes messages to the RabbitMQ broker. Producers write data to exchanges.

Consumer: A consumer is a client that subscribes to queues and processes the messages. Consumers read data from queues.

Queue: A queue is a buffer that stores messages. A queue is bound to an exchange and receives messages from it.

Exchange: An exchange is a message routing agent that receives messages from producers and routes them to queues.

Binding: A binding is a link between an exchange and a queue. It is created with a routing key. The producer sends messages to the exchange with a routing key. The exchange routes the message to the queues that are bound with a matching routing key.

RabbitMQ Setup

We can use the official RabbitMQ docker image to run RabbitMQ locally. We can run the following command to start a RabbitMQ container:

$ docker run --rm --name=rabbitmq -p 15672:15672 -p 5672:5672 rabbitmq:3-management

This image has rabbitmq management plugin enabled. We can access the management UI at http://localhost:15672. The default username and password are both guest.

It also has rabbitmqadmin command line tool installed, which can manage RabbitMQ.

Passing Messages from UI

We can use the management UI to send and receive messages. We can create a new queue and exchange from the Queues section.

Once a queue is created, we can publish and consume messages from that queue.

Passing Messages from CLI

Instead of using web UI, we can use rabbitmqadmin CLI tool² to send and receive messages. Let's create a topic exchange and a queue.

$ docker exec rabbitmq rabbitmqadmin declare exchange type=direct name=orders
# => exchange declared

$ docker exec rabbitmq rabbitmqadmin declare queue name=orders
# => queue declared

Let's publish a message to the exchange:

$ docker exec rabbitmq rabbitmqadmin publish routing_key=orders payload='dummy message'
# => Message published

To receive messages from the queue, we can use the following command:

$ docker exec rabbitmq rabbitmqadmin get queue=orders

Passing Messages from REST API

We can also use REST API to send and receive messages. Let's create a new exchange and queue:

$ curl -u guest:guest -X PUT -H "content-type:application/json" -d '{"type":"direct"}' http://localhost:15672/api/exchanges/%2f/orders

$ curl -u guest:guest -X PUT -H "content-type:application/json" -d '{"type":"topic", "durable": true}' http://localhost:15672/api/queues/%2f/orders

We can publish a message to the exchange:

$ curl -u guest:guest -X POST -H "content-type:application/json" -d '{"routing_key":"orders","payload":"dummy message","payload_encoding":"string", "properties": {} }' http://localhost:15672/api/exchanges/%2f/orders/publish

To receive messages from the queue, we can use the following command:

$ curl -u guest:guest -X GET http://localhost:15672/api/queues/%2f/orders/get

Conclusion

In this post, we have seen how to get started with RabbitMQ. We have seen how to use the management UI, CLI and REST API to send and receive messages.

Hands-on Apache Kafka | ZenTen Tutorials

Anand Reddy Pandikunta — Sat, 17 Dec 2022 01:42:21 GMT

A short hands-on guide to get started with Apache Kafka for people who are in a hurry.

In this guide, we will learn what is Apache Kafka, how to install and run it. We will also learn how to create/modify a topic and produce/consume messages from it.

What is Apache Kafka?

Apache Kafka¹ is a distributed event store and streaming-processing platform. It is used to build real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, and has high throughput.

Kafka Terminology

Topic: A topic is a category or feed name to which records are published/consumed. It is configured with a set of key-value pairs called topic configuration.

Producer: A producer is a client that publishes records to the Kafka cluster. Producers write data to topics and partitions.

Consumer: A consumer is a client that subscribes to topics and processes the records. Consumers read data from topics and partitions.

Consumer Group: A consumer group is a group of consumers that share a common purpose. Consumer groups enable a pool of processes to divide the work of consuming and processing records.

Broker: A broker is a server that hosts a set of topics/partitions. It receives data from producers and sends data to consumers.

ZooKeeper: ZooKeeper is used to store the cluster configuration and the state of the cluster. All Kafka brokers connect to ZooKeeper.

Kraft: Kraft(Apache Kafka Raft) is a consensus protocol that is used to manage the metadata of the Kafka cluster. It is introduced to remove dependency on ZooKeeper.

Setting up Apache Kafka

We can use cp-all-in-one² docker compose files to run Apache Kafka locally. This image contains all the components of Confluent Platform including Apache Kafka, Apache Zookeeper, Confluent Schema Registry, Confluent REST Proxy, Confluent Control Center, and others.

$ git clone https://github.com/confluentinc/cp-all-in-one
$ cd cp-all-in-one/cp-all-in-one
$ docker-compose up

Kafka CLI Tools

Kafka stores messages in topics. A topic is a category or feed name to which messages are published/consumed.

Let us create a topic called demo with kafka-topics command.

$ docker-compose exec broker kafka-topics --bootstrap-server localhost:9092 --topic demo --create

This will create a topic called demo with a single partition and a replication factor of 1. In multi-node cluster, we can use --replication-factor, --partitions to specify the number of replicas/partitions for the topic.

$ docker-compose exec broker kafka-topics --bootstrap-server localhost:9092 --topic demo --partitions 3 --replication-factor 2 --create --if-not-exists

To produce messages to a topic named demo, we can use kafka-console-producer and add messages to the topic:

$ docker-compose exec broker kafka-console-producer --broker-list localhost:9092 --topic demo

>order received
>order updated
>order shipped
>order delivered
>{"status": "completed"}

To consume messages from the same topic:

$ docker-compose exec broker kafka-console-consumer --bootstrap-server localhost:9092 --topic demo --from-beginning

order received
order updated
order shipped
order delivered
{"status": "completed"}

Since we have not defined schema for the messages, Kafka will store the messages as byte arrays. We can explicitly define the schema for the messages using Confluent Schema Registry if required.

We can list all the topics in cluster using kafka-topics:

$ docker-compose exec broker kafka-topics --bootstrap-server localhost:9092 --list

default_ksql_processing_log
docker-connect-configs
docker-connect-offsets
docker-connect-status
demo

To show details of a topic:

$ docker-compose exec broker kafka-topics --bootstrap-server localhost:9092 --describe --topic demo

Topic: demo TopicId: 7CckqkXsQXCNY0MNHYRv2w PartitionCount: 1   ReplicationFactor: 1    Configs: 
    Topic: demo Partition: 0    Leader: 1   Replicas: 1 Isr: 1  Offline:

By default, all messages are stored in the topic for 7 days. We can change this retention period using retention.ms configuration:

$ docker-compose exec broker kafka-topics --bootstrap-server localhost:9092 --alter --topic demo --config retention.ms=10000

To see all the available consumer groups, we can use kafka-consumer-groups:

$ docker-compose exec broker kafka-consumer-groups --bootstrap-server localhost:9092 --list

Confluent Control Center

Confluent Control Center is a web UI to manage and monitor Apache Kafka.

We can visit it http://localhost:9021 and access the cluster from this UI.

We can change topic configuration, view consumer groups, and monitor the cluster from this UI.

Kafka Rest Proxy

Kafka Rest Proxy³ is a RESTful interface to Apache Kafka. It provides a RESTful interface to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients.

To list all topics in the cluster using Kafka Rest Proxy:

$ curl -X GET "http://localhost:8082/topics"

To get config of a topic:

$ curl -X GET "http://localhost:8082/topics/demo"

To produce messages to a demo topic with curl:

$ curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" \
    --data '{"records":[{"value":{"status": "completed"}}]}' \
    "http://localhost:8082/topics/demo"

To consume messages from the same topic:

$ curl -X GET -H "Accept: application/vnd.kafka.json.v2+json" \
    "http://localhost:8082/topics/demo"

We can dynamically configure Kafka cluster settings as well.

To change log level of various components of Kafka cluster using Kafka Rest Proxy.

$ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" \
    --data '{"log4j.logger.kafka.server":"DEBUG"}' \
    "http://localhost:8082/config"

We can update the log level of various components of Kafka cluster and check the logs.

Kafka Connect

Kafka Connect is a framework to stream data between Apache Kafka and other systems. It is used to import/export data from/to Kafka.

Lets use the DataGenConnector to generate random data and write it to a topic.

curl --location 'http://localhost:8083/connectors' \
--header 'Content-Type: application/json' \
--data '{
  "name": "datagen-pageviews",
  "config": {
    "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
    "kafka.topic": "datagen-pageviews",
    "quickstart": "pageviews",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "max.interval": 100,
    "iterations": 10000000,
    "tasks.max": "1"
  }
}
'

This will create a connector called datagen-pageviews which will generate random data and write it to a topic called datagen-pageviews.

KsqlDB

KsqlDB is a streaming SQL engine for Apache Kafka. It provides a SQL interface to query, transform, and analyze data in Kafka.

Lets create a stream called pageviews from the topic datagen-pageviews:

$ curl -X "POST" "http://localhost:8088/ksql" \
     -H "Content-Type: application/vnd.ksql.v1+json; charset=utf-8" \
     -d $'{
  "ksql": "CREATE STREAM pageviews (viewtime BIGINT, userid VARCHAR, pageid VARCHAR) WITH (KAFKA_TOPIC=\'datagen-pageviews\', VALUE_FORMAT=\'JSON\');",
  "streamsProperties": {}
}'

We can create a table to generate viewcount per pageid:

$ curl -X "POST" "http://localhost:8088/ksql" \
     -H "Content-Type: application/vnd.ksql.v1+json; charset=utf-8" \
     -d $'{
  "ksql": "CREATE TABLE pageviews_count AS
SELECT 
  pageid,
  COUNT(*) AS page_views_count
FROM pageviews_stream
GROUP BY pageid
EMIT CHANGES;
",
  "streamsProperties": {}
}'

We can query the table to get the viewcount per pageid:

$ curl -X "POST" "http://localhost:8088/ksql" \
     -H "Content-Type: application/vnd.ksql.v1+json; charset=utf-8" \
     -d $'{
  "ksql": "SELECT * FROM pageviews_count EMIT CHANGES;",
  "streamsProperties": {}
}'

Conclusion

In this article, we have seen how to install Apache Kafka locally using Docker. We have also seen how to produce and consume messages using Kafka console commands and Kafka Rest Proxy.

Common Crawl on Laptop - Building Web Directory

Anand Reddy Pandikunta — Thu, 08 Dec 2022 02:11:39 GMT

This series of posts discuss processing of common crawl dataset on laptop.

Extracting Subset of Common Crawl
Building web directory (this post)

Introduction

In the earlier post, we have extracted all telugu web page urls to a csv file. In this post, let's explore these urls and build a web directory from it.

Explore Data

Let's see how many urls are present in the extracted subset of data.

$ wc -l telugu.csv
  852025 telugu.csv

In the earlier post, we have installed duckdb and used it for processing parquet files. duckdb can execute SQL queries directly on csv file. Let's use it to explore the data stored in telugu.csv.

Let's see how many unique domains are present in the data.

$ duckdb -c """
    SELECT COUNT(DISTINCT url_host_name_reversed) as unique_sites
    FROM read_csv('telugu.csv', auto_detect = TRUE);
"""
┌──────────────┐
│ unique_sites │
├──────────────┤
│ 13632        │
└──────────────┘

There ~14k unique domains. Let's see page density across these domains.

$ duckdb -c """
SELECT count    AS page_count,
COUNT(*) AS sites
FROM (SELECT url_host_name_reversed, COUNT(*) AS count
FROM read_csv('te.csv', auto_detect = TRUE)
GROUP BY url_host_name_reversed) AS t
GROUP BY page_count
ORDER BY page_count;
"""
┌────────────┬───────┐
│ page_count │ sites │
├────────────┼───────┤
│ 1          │ 6326  │
│ 2          │ 1904  │
│ 3          │ 733   │
│ 4          │ 459   │
│ 5          │ 315   │

About ~75% of the sites have less than 5 pages. It is highly unlikely that these sites complete content is in Telugu language. After manually checking a few of these sites, I found that there are a lot of false positives.

In the earlier post, we have extracted all pages where there is Telugu language content. Let's filter out pages where Telugu is primary language.

$ duckdb -c """
  COPY (
    SELECT * FROM read_csv('cct.csv', auto_detect=true) 
    WHERE content_languages like 'tel%'
  ) TO 'te_primary.csv' (DELIMITER ',', HEADER TRUE);
"""

$ wc -l te_primary.csv
  573130 te_primary.csv

$ duckdb -c "SELECT COUNT(DISTINCT url_host_name_reversed) as unique_sites FROM read_csv('te_primary.csv', auto_detect = TRUE)"                           
┌──────────────┐
│ unique_sites │
├──────────────┤
│ 5666         │
└──────────────┘

Let's see how page density per domain has changed.

$ duckdb -c """
SELECT count    AS page_count,
COUNT(*) AS sites
FROM (SELECT url_host_name_reversed, COUNT(*) AS count
FROM read_csv('te_primary.csv', auto_detect = TRUE)
GROUP BY url_host_name_reversed) AS t
GROUP BY page_count
ORDER BY page_count
;
"""
┌────────────┬───────┐
│ page_count │ sites │
├────────────┼───────┤
│ 1          │ 2183  │
│ 2          │ 843   │
│ 3          │ 235   │
│ 4          │ 146   │
│ 5          │ 98    │

Page density remains almost the same.

Let's filter out sites which have at least 5 pages in Telugu. This will eliminate a lot of false positives. Let's look at the most popular sites from the results.

   1   │ Rank,Domain,Open Page Rank
   2   │ 25,support.google.com,8.55
   3   │ 57,t.me,7.76
   4   │ 76,chrome.google.com,7.49
   5   │ 163,support.mozilla.org,6.99
   6   │ 170,groups.google.com,6.94

A lot of unrelated domains are present here because there might be 10+ pages in telugu in these domains as well. But we don't need these.

Let's look at only home page(or translated home page) where primary content language is telugu.

$ duckdb -c """
  SELECT COUNT(distinct url) 
  FROM read_csv('te_primary.csv', auto_detect=true) 
  WHERE (url_path = '/' or url_path = '/te/') and url_query is null;
"""

Now the domain count has reduced to 6k. Let's export these domains to csv file.

To categorize these domains, Common-crawl doesn't yet provide any kind of categorisation. For now, we can use Open PageRank to sort these domains based on rank.

We can download top 10 million domains from Open PageRank³. Here is a simple python script to extract telugu domains from the list.

import pandas as pd

domains_file = 'domains.csv'
with open(domains_file, 'r') as f:
    telugu_domains = [line.strip() for line in f.readlines()]

telugu_domains = ['.'.join(reversed(domain.split('.'))) for domain in telugu_domains]

df = pd.read_csv('t10m.csv')
df = df[df['Domain'].isin(telugu_domains)]

df.to_csv('t10m_telugu.csv', index=False)

Now, we have list of all telugu domains sorted by rank. In the next post, we will use this list to categorize the domains.

Common Crawl On Laptop - Extracting Subset Of Data

Anand Reddy Pandikunta — Thu, 17 Nov 2022 01:11:39 GMT

This series of posts discuss processing of common crawl dataset on laptop.

Extracting Subset of Common Crawl (this post)
Building web directory

Introduction

Common Crawl(CC)¹ is an open repository of web containing peta bytes of data since 2008. As the dataset is huge, most of the tutorials use AWS EMR/Athena to process the data.

In this post, let's learn how to extract a subset of data(entire telugu language web pages) and process it on our local machine.

Exploring Common Crawl

CC provides monthly data dumps in WARC format. Each crawl consists of about ~3 billion web pages with a compressed size of ~100 TB.

In addition to WARC files, CC provides index files as well as columnar index² files so that users can easily search, filter and download the data.

Common Crawl Index

Each crawl index is spread over 300 files consisting of ~250 GB of data. For this post, let use the latest crawl which is CC-MAIN-2022-40.

The index files can be accessed from AWS S3 or https. We can use aws cli to list all the files along with the sizes.

$ aws s3 ls --recursive --human-readable --summarize s3://commoncrawl/cc-index/collections/CC-MAIN-2022-40
2022-10-08 16:07:59  621.9 MiB cc-index/collections/CC-MAIN-2022-40/indexes/cdx-00000.gz
2022-10-08 16:08:26  721.6 MiB cc-index/collections/CC-MAIN-2022-40/indexes/cdx-00001.gz
...
2022-10-08 16:42:39  146.6 MiB cc-index/collections/CC-MAIN-2022-40/indexes/cluster.idx
2022-10-08 16:42:33   30 Bytes cc-index/collections/CC-MAIN-2022-40/metadata.yaml

Total Objects: 302
   Total Size: 236.1 GiB

Let's download an index file to our local machine and see how the data is arranged. We can use aws cli to download the data from s3 bucket or use wget to download it from https endpoint.

# from s3
$ aws s3 cp s3://commoncrawl/cc-index/collections/CC-MAIN-2022-40/indexes/cdx-00000.gz .

# from https
$ wget https://data.commoncrawl.org/cc-index/collections/CC-MAIN-2022-40/indexes/cdx-00000.gz

Let's print top five lines of the file.

$ zcat < cdx-00000.gz | head -n 5
0,1,184,137)/1klikbet 20221005193707 {"url": "http://137.184.1.0/1klikbet/", "mime": "text/html", "mime-detected": "text/html", "status": "200", "digest": "XTKGORHKLZCHDBBOMYCYYIZVRPMXNRII", "length": "7065", "offset": "83437", "filename": "crawl-data/CC-MAIN-2022-40/segments/1664030337663.75/warc/CC-MAIN-20221005172112-20221005202112-00011.warc.gz", "charset": "UTF-8", "languages": "ind"}
0,1,184,137)/7meter 20221005192131 {"url": "http://137.184.1.0/7meter/", "mime": "text/html", "mime-detected": "text/html", "status": "200", "digest": "KUJAMRT6MXYR3RTWRJTIWJ5T2ZUB3EBH", "length": "7456", "offset": "142680", "filename": "crawl-data/CC-MAIN-2022-40/segments/1664030337663.75/warc/CC-MAIN-20221005172112-20221005202112-00182.warc.gz", "charset": "UTF-8", "languages": "ind"}
...

The last column of each line contains the language information. We can use these index files, and we can extract all the lines containing tel language code.

Columnar Index

We can also use columnar index to filter out telugu language web pages. Let's download a single file from the index.

# from s3
$ aws s3 cp s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2022-40/subset=warc/part-00000-26160df0-1827-4787-a515-95ecaa2c9688.c000.gz.parquet .

# from https
$ wget https://data.commoncrawl.org/cc-index/table/cc-main/warc/crawl=CC-MAIN-2022-40/subset=warc/part-00000-26160df0-1827-4787-a515-95ecaa2c9688.c000.gz.parquet

We can use Python pandas to read the parquet file and filter out telugu language web pages. Columnar index has content_languages column which can be used to filter out telugu pages as shown below.

$ python -c """
import pandas as pd
filename = 'part-00000-26160df0-1827-4787-a515-95ecaa2c9688.c000.gz.parquet'
df = pd.read_parquet(filename)
df = df[df['content_languages'].str.startswith('tel', na=False)]
df.to_csv('telugu.csv')
"""

I have used Macbook M1 with local ISP(Internet Service Provider) to download and extract the index. It took around 7 minutes to download a single file and 2 minutes to extract the data. To process 300 index files, it takes ~2 days.

Let's see how we can speed it up.

Improving Performance

Faster Downloads

My Wi-Fi speed is ~4MBps when downloading the index file. To download faster, I have created t2.micro(free-tier) EC2 instance on AWS. In this machine, download speed is ~10MBps. We can use other instances, but I am trying to use only free resources. In this machine, single file download is taking ~3 minutes.

CC dataset is hosted in us-east-1 region. So, I have created a new t2.micro instance in us-east-1 region. This instance is taking <20 seconds to download a single file. We can download entire index in less than 2 hours.

Faster Performance

To extract data from index files, we have used Python pandas without specifying the engine. By default, it uses pyarrow which is a bit slow. To improve speed we can use fastparquet as engine which is ~5x faster than pyarrow.

import pandas as pd

filename = 'part-00000-26160df0-1827-4787-a515-95ecaa2c9688.c000.gz.parquet'
df = pd.read_parquet(filename, engine='fastparquet')

To get better performance, we can use duckdb. Duckdb is an in-process SQL OLAP DBMS and it can execute SQL queries directly on parquet files with parquet extension.

$ brew install duckdb

$ duckdb -c 'INSTALL parquet;'

We can write a simple SQL query to filter out the required rows.

$ duckdb -c """
LOAD parquet;
COPY (select * from PARQUET_SCAN('part-00000-26160df0-1827-4787-a515-95ecaa2c9688.c000.gz.parquet') where content_languages ilike '%tel%') TO 'telugu.csv' (DELIMITER ',', HEADER TRUE);
"""

Duckdb can execute SQL queries on remote files as well with httpfs extension.

$ duckdb -c 'INSTALL httpfs;'

$ duckdb -c """
    LOAD httpfs;
    LOAD parquet;

    COPY (select * from PARQUET_SCAN('s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2022-40/subset=warc/part-00001-26160df0-1827-4787-a515-95ecaa2c9688.c000.gz.parquet') where content_languages ilike '%tel%') TO 'telugu.csv' (DELIMITER ',', HEADER TRUE);"""
"""

Duckdb can also read series of parquet files and treat them as a single table. We can use this feature to process all the index files in a single command.

$ duckdb -c """
    LOAD httpfs;
    LOAD parquet;

    SET s3_region='us-east-1';
    SET s3_access_key_id='s3_secret_access_key';
    SET s3_secret_access_key='s3_secret_access_key';

    COPY (select * from PARQUET_SCAN('s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2022-40/subset=warc/*.parquet') where content_languages ilike '%tel%') TO 'telugu.csv' (DELIMITER ',', HEADER TRUE);
"""

Depending on the file size, duckdb takes 10-15 seconds to process a single file. Since we don't need all the columns for further data processing, we can limit columns to required 5 columns.

$ duckdb -c """
    COPY (select url, content_languages, warc_filename, warc_record_offset, warc_record_length from PARQUET_SCAN('s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2022-40/subset=warc/*.parquet') where content_languages ilike '%tel%') TO 'telugu.csv' (DELIMITER ',', HEADER TRUE);
"""

By limiting columns³ there is another 65% improvement in performance. Now duckdb can process a file in 3 to 8 seconds depending on the size of the file. We can process entire index in ~20 minutes.

Conclusion

With a single command, we can extract a subset of index from CC in ~2 hours. So far we have processed all files in a single process. We can also parallelize the process using parallel to get faster results.

In the upcoming posts, let's see how we can fetch the data from WARC files using this index and do further data processing.

Build & Distribute a Python C Extension Module

Anand Reddy Pandikunta — Tue, 01 Nov 2022 16:01:29 GMT

Introduction

Python is a great language for prototyping and building applications. Python is an interpreted language, and it is not compiled. This means that the code is not optimized for the machine it is running on. This is where C comes in.

C is a compiled language, and it is much faster than Python. So, if you want to write a Python module that is fast, you can write it in C and compile it. This is called a C extension module. In this article, we will see how to build and distribute a Python C extension module using wheels.

Building a C extension module

Let's start by creating a simple C extension module called maths. In this, we will create a square function that takes a number and returns its square.

First, create a directory called maths and create a file called maths.c inside it. This is where we will write our C code.

#include <Python.h>


int square(int num) {
    return num * num;
}


static PyObject *py_square(PyObject *self, PyObject *args) {
  int n_num, result;
  if (!PyArg_ParseTuple(args, "i", &n_num)) {
    return NULL;
  }
  result = square(n_num);

  return Py_BuildValue("i", result);
}


static PyMethodDef mathsMethods[] = {
  {"square", py_square, METH_VARARGS, "Function for calculating square in C"},
  {NULL, NULL, 0, NULL}
};


static struct PyModuleDef maths = {
  PyModuleDef_HEAD_INIT,
  "maths",
  "Custom maths module",
  -1,
  mathsMethods
};


PyMODINIT_FUNC PyInit_maths(void)
{
    return PyModule_Create(&maths);
}

We need to create a setup.py file to build our module. This file tells Python how to build our module.

from setuptools import setup, Extension

setup(
    name="maths",
    version="0.1",
    ext_modules=[Extension("maths", ["maths.c"])]
)

Now, we can build our module by running python setup.py build. This will create a build directory with a lib directory inside it. This lib directory contains our compiled module. We can import this module in Python and use it.

>>> import maths
>>> maths.square(5)
25

Instead of testing our module by importing it in Python, we can also test it by running python setup.py test. This will run the tests in the test directory. We can create a test directory and create a file called test_maths.py inside it. This is where we will write our tests.

import unittest

import maths

class TestMaths(unittest.TestCase):
    def test_square(self):
        self.assertEqual(maths.square(5), 25)

Distributing a C extension module

Now that we have built our module, we can distribute it. We can distribute it as a source distribution or a binary distribution. A source distribution is a zip file that contains the source code of our module. We can distribute our module as a source distribution by running python setup.py sdist. This will create a dist directory with a zip file inside it. This zip file contains our source code.

However, source distribution of C extension modules is not recommended. This is because the user needs to have a C compiler installed on their machine to build the module. Most users just want to pip install the module and use it. So, we need to distribute our module as a binary distribution.

We can use cibuildwheel package to build wheels across all platforms. We can install it by running pip install cibuildwheel.

To build a wheel for a specific platform and a specific architecture, we can run cibuildwheel --platform <platform> --architecture <architecture>. For example, to build a wheel for Linux x86_64, we can run cibuildwheel --platform linux --architecture x86_64. This will create a wheelhouse directory with a wheel file inside it. This wheel file contains our compiled module.

cibuildwheel runs on most CI servers. With proper workflows, we can easily get wheels for all platforms and architectures. We can then upload these wheels to PyPI and users can easily install these wheels.

Conclusion

In this article, we saw how to build and distribute a Python C extension module using wheels. We saw how to build a C extension module and how to distribute it as a binary distribution. We also saw how to use cibuildwheel to build wheels across all platforms and architectures.

Speed Up AMD64(Intel) VMs on ARM(M1 Mac) Host

Anand Reddy Pandikunta — Thu, 20 Oct 2022 17:08:56 GMT

Introduction

From 2020, Apple has transitioned from Intel to ARM based Apple Silicon M1. If we run uname -mp on these devices, we can see the CPU architecture details.

$ uname -mp
arm64 arm

Let's run the same command on a device using Intel x86 processor.

$ uname -mp
x86_64 x86_64

Many popular docker images¹ doesn't have ARM64 support yet. When setting up a dev environment in M1 Mac, there are high chances that we stumble on these containers if we are using plain docker or ARM64 VM. So, there is a need to spin up x86_64 VMs.

In this article, lets see how the performance affects when running a cross architecture containers and how to speed it up.

Setup

Lima² can run foreign architecture(x6_64) VMs on Mac. Let's install lima, start a AMD64 VM & ARM64 VM and install k3s[^k3s] in them. k3s will run multiple process in the background and let's see how resource consumption varies in these VMs.

$ brew install lima

$ limactl start linux_arm64
$ limactl start linux_amd64

When starting a VM, we can edit arch parameter in the configuration. Once VM starts, we can see the details by running limactl list.

$ limactl list
NAME                ARCH
linux_amd64         x86_64
linux_arm64         aarch64

Lets login to each VM & install k3s.

$ limactl shell linux_arm64

$ curl -sfL https://get.k3s.io | sh -

$ limactl shell linux_amd64

$ curl -sfL https://get.k3s.io | sh -

If we look at resource consumption on the host machine, x86_84 VM is using way more resources than ARM64 VM. This is because of the emulation layer that is running on top of the VM.

We can login to individual VMs, run top to see the load average as well.

Fast Mode

lima provides fast-mode option for cross architecture VMs which will speed up the performance.

For that, we need to log in to VMs and install emulators.

$ sudo systemctl start containerd
$ sudo nerdctl run --privileged --rm tonistiigi/binfmt --install all

After that we can restart the VMs and monitor the resource consumption. On an average, we can see that the resource consumption is reduced by 50%.

Conclusion

In this article, we saw how to run cross architecture VMs on M1 Mac and how to speed up the performance. We can use this technique to run cross-architecture containers on Linux as well.

Local Kubernetes Cluster with K3s on MacBook M1

Anand Reddy Pandikunta — Tue, 11 Oct 2022 02:38:36 GMT

Update: This article is outdated. I wrote an updated article on setting up k8s anywhere with single command.

Introduction

Kubernetes(k8s)¹ is an open-source system for managing large scale containerized applications. K3s² is lightweight K8s in a single binary file. However, K3s won't work directly on Macbook as it needs systemd/OpenRC.

$ curl -sfL https://get.k3s.io | sh -

[ERROR]  Can not find systemd or openrc to use as a process supervisor for k3s

To setup k8s/k3s on Mac, we need to setup a Linux layer on top of Mac. An easy way to spin up Linux VMs on Macbook M1(Apple Silicon) is to use multipass³. In this article, lets see how to setup K3s on Mac using multipass

K3s Setup

Install multipass with brew by running the following command.

$ brew install --cask multipass

Once it is installed, spin up a new VM by specifying memory and disk space.

$ multipass launch --name k3s --mem 4G --disk 40G

Once VM is launched, we can see VM details.

$ multipass info k3s
Name:           k3s
State:          Running
IPv4:           192.168.64.4
                10.42.0.0
                10.42.0.1
Release:        Ubuntu 22.04.1 LTS
Image hash:     78b5ca0da456 (Ubuntu 22.04 LTS)
Load:           1.34 2.10 1.70
Disk usage:     3.7G out of 38.6G
Memory usage:   1.2G out of 3.8G
Mounts:         /Users/chillaranand/test/k8s => ~/k8s
                    UID map: 503:default
                    GID map: 20:default

We can even mount Mac directories on the VM.

$ multipass mount ~/test/k8s k3s:~/k8s

This will be useful when we are making changes on host directories and want to apply changes on the cluster which is inside VM.

Now, we can install k3s by running the install script inside the VM.

$ multipass shell k3s

ubuntu@k3s:~$ curl -sfL https://get.k3s.io | sh -

This will setup a k3s cluster on the VM. We can use kubectl and deploy applications on this cluster.

By default, k3s config file will be located at /etc/rancher/k3s/k3s.yaml. With this config file, we can use Lens⁴ to manage k8s cluster.

Lets find out IP of the VM & k8s token so that we can spin up a new VM and add it to this cluster.

# get token & ip of k3s
$ multipass exec k3s sudo cat /var/lib/rancher/k3s/server/node-token
$ multipass info k3s | grep -i ip

$ multipass launch --name k3s-worker --mem 2G --disk 20G

$ multipass shell k3s-worker

ubuntu@k3s-worker:~$ curl -sfL https://get.k3s.io | K3S_URL=https://192.168.64.4:6443 K3S_TOKEN="hs48af...947fh4::server:3tfkwjd...4jed73" sh -

We can verify if the node is added correctly from k3s VM.

ubuntu@k3s:~$ kubectl get nodes
NAME         STATUS   ROLES                  AGE     VERSION
k3s          Ready    control-plane,master   15h     v1.24.6+k3s1
k3s-worker   Ready    <none>                 7m15s   v1.24.6+k3s1

Once we are done with experimenting k3s, we can delete the VMs.

$ multipass delete k3s k3s-worker
$ multipass purge

Conclusion

multipass is a great tool to spin up Linux VMs on Mac with single command. K3s is better tool to setup k8s cluster locally for development and testing.

Even though we have mentioned this tutorial is meant for Mac M1, it should work fine on any Linux distribution as well.

How To Root Xiamo Redmi 9 Prime Without TWRP

Anand Reddy Pandikunta — Wed, 28 Sep 2022 00:44:45 GMT

Introduction

Rooting¹ an android device voids warranty but gives a great control over the device. For most of the popular devices, TWRP² recovery is available. Once a device bootloader is unlocked, we can install TWRP recovery. After that we can flash Magisk to gain root access.

For some devices like Redmi 9 Prime(codename: lancelot), TWRP recovery is not available officially. There are couple of unofficial images but they are not working as expected and are causing bootloop.

In this article, lets see how to root lancelot device.

Rooting Lancelot

First ensure that the device bootloader is unlocked and your system has adb & fastboot installed. To root without TWRP, we need to obtain patched boot.img & vbmeta.img and flash them in fastboot mode.

First we need to download the stock ROM of the device to extract boot.img file. We can go to manufacturers site and download the same stock ROM³ that is running on the device. Once the ROM is downloaded, we can unzip it. There we can find boot.img file.

We need to patch this file. To patch this, download magisk app on the device. Click on install and select the boot.img file to patch. After a few minutes, it will generate a patched boot file.

We can download this file to system by running the following command.

$ adb pull -p /storage/emulated/0/Download/magisk_patched-25200_cU1ws.img .

Now we need to download patched vbmeta file. This is available in XDA⁴ forum. Click this link to download it.

Once we have both patched files, we can reboot the device in fastboot mode by using the following commands.

$ adb devices

$ adb reboot bootloader

When the device is in fastboot mode, run the following commands to root it.

$ fastboot --disable-verity --disable-verification flash vbmeta vbmeta_redmi9.img

$ fastboot flash boot magisk_patched-25200_cU1ws.img

$ fastboot reboot

Once the deivce is rebooted, we can install root checker app and verify that the device is rooted successfully.

Final Thoughts

When we buy a Mac or PC(Linux/Windows), it is rooted by default. For Linux/Mac, we can run programs as sudo. For windows, we can just run a program as an administrator. There are no extra steps to root/jailbreak these devices.

But most mobile companies make rooting hard and only tech savy users can root the device. It would be great if mobile phones are rooted by default.

Integrating Frappe Health with SNOMED CT

Anand Reddy Pandikunta — Wed, 31 Aug 2022 10:59:13 GMT

Introduction

Frappe Health¹ is an open-source Healthcare Information System(HIS), to efficiently manage clinics, hospitals, and other healthcare organizations. Frappe Health is built on the Frappe Framework², a low code highly customizable framework.

Frappe Health provides support for integrating various medical coding standards³. In the patient encounter doctype, doctors can search and add pre-configured medical codes.

In this article, let’s see how to integrate Frappe Health with SNOMED CT.

SNOMED CT Integration

SNOMED CT⁴ is a comprehensive collection of medical terms which helps consistent data exchange between systems. It can also cross-map to other standards like ICD-10, LOINC, etc.

Since SNOMED CT is a huge dataset, it takes a lot of effort to import the entire dataset into Frappe Health. It also provides REST API to query SNOMED terms. Also, if your healthcare organization is focusing on only a specific domain, it doesn’t make sense to import the entire dataset.

In such scenarios, it is better to map only the required diagnosis, symptoms, and other clinical objects.

Frappe Health has a Diagnosis doctype where practitioners can enter diagnosis. We can add an additional field called Snomed Code to link diagnosis to relevant SNOMED code.

Frappe framework provides server script⁵ to dynamically run python script on any document event. We can write a simple python script to fetch relevant SNOMED code using SNOMED REST API. This script can be executed whenever the clinical object gets modified.

Here is a simple python server script that adds relevant snomed codes to diagnosis.

diagnosis = doc.diagnosis

url = "https://browser.ihtsdotools.org/snowstorm/snomed-ct/browser/MAIN/descriptions?&limit=50&term=" + diagnosis + "&conceptActive=true&lang=english&skipTo=0&returnLimit=100"
data = frappe.make_get_request(url)
code = data['items'][0]['concept']['id']
description = data['items'][0]['concept']['fsn']['term']

mc = frappe.get_doc({
    'doctype': 'Medical Code',
    'code': code,
    'medical_code_standard': 'SNOMED',
    'description': description,
})
try:
    mc.insert()
except:
    pass

doc.medical_code = medical_code.name

After saving this script, if we go ahead and create or modify any diagnosis, it will automatically add relevant Snomed code to the diagnosis as shown below.

The server script makes sure all the diagnosis objects are codified automatically without any manual effort.

Since Frappe Framework & Frappe Health are low code, extremely customizable, we are able to integrate it with SNOMED in just a few minutes. Similarly, we can codify other clinical objects like Symptoms, Procedures, Medications, etc.

Mastering DICOM - #3 Setup Modality Worklist SCP

Anand Reddy Pandikunta — Sun, 31 Jul 2022 12:49:29 GMT

Introduction

In the earlier article, we have learnt how to setup DICOM for digging deeper into DICOM protocol. In this article, let us learn how to setup a modality worklist(WML) SCP. Modalities can send C-FIND queries to this SCP and retrieve worklist information

Using Orthanc Worklist Plugin

Orthanc server has worklist plugin¹ which will serve worklist files that are stored in a particular directory. Let us download sample worklist files from Orthanc repository and keep in "WorklistDatabase" directory.

Generate default configuration by running the following command.

$ ./Orthanc --config=config.json

In the orthanc configuration file, enable worklist plugin, specify the worklist database directory so that Orthanc can locate relevant worklist files, add required modalities and restart the server.

  "Plugins" : [
    "libModalityWorklists.dylib"
  ],

  "Worklists" : {
    "Enable": true,
    "Database": "./WorklistsDatabase",
    "FilterIssuerAet": false,
    "LimitAnswers": 0
  },

  "DicomModalities" : {
      "PYNETDICOM" : ["PYNETDICOM", "127.0.0.1", 4243],
      "FINDSCU" : ["FINDSCU", "127.0.0.1", 4244]
  }

Once the plugin is enabled, we can use findscu to send C-FIND query.

$ findscu -W -k "ScheduledProcedureStepSequence" 127.0.0.1 4242

This will retrieve all worklist files from the server.

Using wlmscpfs

dcmtk ² is a collection of utilities for DICOM standard. It has wlmscpfs application which implements basic Service Class Provider(SCP). We can start the service by running the following command.

wlmscpfs --debug --data-files-path WorklistsDatabase 4242

Once the service is started modalities can send C-FIND query to this service.

Conclusion

We have seen how to setup MWL SCP using Orthanc & wmlscpfs. Now that we have PACS & WML SCP up and running, in the next article lets see how to dig deeper in to the dicom standard.

Twilio Integration With Frappe Framework

Anand Reddy Pandikunta — Thu, 30 Jun 2022 06:16:02 GMT

Frappe¹ is a low code framework for rapidly developing web applications. Twilio² is a SAAS platform for SMS, Video etc with APIs.

In this post, lets see how to setup Twilio Integration with Frappe.

Sending SMS

Frappe has inbuilt SMS manager³ where users can confgiure SMS gateway and send SMS to mobiles directly.

To send out messages/SMS with Twilio, we just need to configure Twilio API keys in SMS settings.

First, create an account in Twilio and collect the following information from the account.

Twilio account SID
Gateway URL
Auth Token
"From" Phone number

These details need to be added to SMS settings in the following format.

For authorization parameter, we need to enter base64 encoded value of account_sid:auth_token.

Once these values are set, we can go to SMS Center and send out dummy messages to ensure all settings are configured properly.

Twilio App

If we want to manage incoming/outgoing voice calls or send messages via WhatsApp, we need to install twilio-integration⁴ app. We can install the app on our site by running the following commands.

bench get-app https://github.com/frappe/twilio-integration.git
bench --site example.com install-app twilio_integration

Once the app is installed, we can go to Twilio Settings and configure the keys as shown below.

After that we can setup Voice Call Settings to manage incoming/outgoing calls.

To send messages via Whatsapp, we can set the channel as Whatsapp in Notification doctype.

This is how we can send SMS, Whatsapp messages & manage calls via Twilio using Frappe Framework.

Why DMART is not in FNO category?

Anand Reddy Pandikunta — Sun, 15 May 2022 07:14:24 GMT

FNO Segment

There are 4000+ companies¹ that are traded in NSE/BSE. Of these, 198 stocks are included in FNO segement². For these stocks, Futures & Options(FNO) contracts will be available and these stocks won't have any fixed ciruit limits.

FNO Reviews

On 11th April 2018, SEBI released a circular(SEBI/HO/MRD/DP/CIR/P/2018/67) on a framework³ for reviewing stocks in derivatives segement. Based on this framework, a stock has to meet the below criteria be added in FNO segement.

The stock shall be chosen from amongst the top 500 stocks in terms of average daily market capitalization and average daily traded value in the previous six months on a rolling basis.
The stock’s median quarter-sigma order size over the last six months, on a rolling basis, shall not be less than ₹25 Lakh.
The market wide position limit(MWPL) in the stock shall not be less than ₹500 crore on a rolling basis.
Average daily delivery value in the cash market shallnot be less than ₹10 crore in the previous six months on a rolling basis.

If a stock is in FNO category and fails to meet this criteria, it will be removed from FNO segment.

In 2021 alone, SEBI released 6 circulars thrugh which 32 new stocks are added in the FNO segment.

DMART

DMART has met the above FNO criteria long back. Even after it has met the criteria, there were more than 6 reviews⁴⁵⁶ and suprisingly DMART was not added to the FNO segement.

There is a long discussion on Zerodha TradingQ&A forum⁷ on why DMART was not added to FNO category but no one could give any explaination.

After that, I have sent an email to SEBI & NSE seeking clarification for the same. It has been more than 6 months and they haven't responded yet. I have reached out to few people in the trading community privately to get clarification on the same. But nobody I know has any clue on this.

I am still looking for an answer. If you can shed some light, please send out a message to me. I would like to have a quick chat with you regarding the same.

Hoping to solve the mystery soon.

Using Frappe Framework As REST API Generator

Anand Reddy Pandikunta — Sat, 30 Apr 2022 13:31:53 GMT

Introduction

When a company plans to build a mobile application and/or a web application, they need to create REST APIs to store data. This is a common requirement for CRUD applications.

In the Python ecosystem, there are several projects like Django Rest Framework, Flask-RESTful, FastApi which does the heavy lifting of implementing REST APIs. In other ecosystems, there are similar projects/frameworks for this job.

REST API Generators

By using the above mentioned frameworks, developers can build REST APIs at a faster rate. Still, developers have to develop and maintain code for these APIs.

To avoid even this work, there are REST API generators like postgrest¹, prest which can instantly generate REST APIs by inspecting database schema. With these, developers just have to design the DB schema and these tools take care of generating APIs without writing a single line of code.

In this post, let us see how Frappe framework can be used as a REST API generator and what are the advantages of using Frappe.

Frappe Framework

Frappe framework is meta data driven low code, web framework written in Python and Javascript.

Web UI

Frappe framework provides web UI to create models(called doctypes in Frappe) and it provides REST API² for all the models out of the box.

There is no need to write manual SQL queries to manage schema. With some training even non-developers can even manage models in Frappe.

Roles & Permissions

With traditional API generators, managing roles & permissions involves additional development and maintenance costs. Frappes comes with an authentication system and it has support for role based permissions out of the box. From the web UI, users can manage roles & permissions.

Hooks

Even though REST API generators give API out of the box, there will be scenarios where custom business logic needs to be hooked in for various events. In such scenarios, developers end up using an alternate framework/tool to manage hooks and business logic.

Frappe provides server scripts by which arbitrary python code can be executed dynamically based on model events. There is no need to set up another framework for these things.

Utilities

Frappe framework comes with a lot of utilities like Job Queues, Schedulers, Admin interface, socket.io etc. As the project grows and the need evolves, Frappe has all the common utilities that are required for a web application development.

Conclusion

When a company wants to build a solution to a problem, it should focus most of the time in solving that problem instead of wasting their time on building CRUD interfaces or REST APIs.

Frappe framework was designed to rapidly build web applications with low code. If you need a REST API generator and some additional functionality for the REST APIs, Frappe framework fits the bill and reduces a lot of development time.

Adding Fuzzy Search To Frappe Framework

Anand Reddy Pandikunta — Thu, 31 Mar 2022 16:00:33 GMT

Introduction

As software developers, we use fuzzy search a lot especially when using Emacs or any other editors/IDEs. For example to search a file called patient_history.js, in the editor, we can just type pah and editor will narrow it down.

This is quite handy as we can open any file with just few characters.

FF is a low code, open source, web framework in Python and Javascript. All sites built with FF will have a global search bar(aka awesome bar) as shown below. Here, we can search for doctypes, reports, pages etc.

To open Patient History, we have to type almost the entire text in search bar. If we type pah like we have typed in the editor, it won't show any results.

Instead, we can add fuzzy search here so that we can search for any item with just a frew key strokes.

Fuzzy Search

There are many third party packages which implement fuzzy search in programming languages. However we can't use any of these fuzzy search packages directly. These editors internally use a scoring algorithm to rank the results and display results based on score.

It internally considers many factors mentioned below for scoring.

Matched letters
CamelCase letters
snake_case letters
Consecutive matching letters

We can come up with a scoring mechanism for these factors and based on the matches, we can rank the results. I have implemented a custom fuzzy search alogirthm based on the above factors but it was slow and results were not good in some cases.

Then I stumbled up this fts_fuzzy_match implementation. This is a reverse engineered implementation of sublime text fuzzy search and it has a detailed scoring mechanism as well. It assigns negative ranking to mismatched letters and bonus points for consecutive matches.

This works well and is as effective as most IDEs search. Now that there is a solid fuzzy search, all we need to do is hook this up in FF.

FF internally has a fuzzy search function and we can directly hook it up here as shown here.

After that, we can search for anything in just few key strokes. For example to open patient history, we can just type pah and it will show results like this.

This change is now merged into Frappe framework and will be available in the next release.

Conclusion

Fuzzy search in editors/IDEs is quite handy and when we bring to other places like FF or any other search bar, it improves search experience a lot.

Using Frappe Framework As An Integration Engine

Anand Reddy Pandikunta — Mon, 28 Feb 2022 16:01:40 GMT

Introduction

In healthcare orgainsations, data exchange between systems is complicated and has interopolabilty issues. Integration engines are widely used in healthcare industry for bi-directional data transfer.

In this article, let us look at the advantages of using interface engines and how Frappe Framework can be used as an interface engine.

Integration Engines

In a traditional agile development approach, building a new interface might take weeks/months. With an integration engine, a new interface can be replicated in a matter of hours with little or no scripting at all.

Creating a REST API, listening to a webhook, transforming a data between channels, broadcasting a message, sending/receiving HL7 messages or any other commonly performed task can be implemented in integration engine without much effort.

Due to this integration engines like Mirth Connect are widely used in healthcare.

The above diagram shows usage of integration engine in a healthcare orgainsation.

Frappe Framework

Frappe Framework is a low code web application framework with batteries included. Even though Frappe is lablled as a framework, it can be used as an integration engine as well.

It provides REST APIs out of the box for all the models(called doctypes in Frappe). Users can create custom APIs using server scripts and it has support for webhooks as well.

Users can schedule custom cron jobs, setup emails, enable data transformations and other tasks without much coding.

One feature Frappe Framework lacks when compared to integration engines is sending/receiving HL7 messages on ad-hoc ports. This feature is not available out of the box. Users need to develop a custom app or use any 3rd party app for it.

Frappe Healthcare is an HIS(Healthcare Information System) built on top of Frappe/ERPNext. If a hospital is using Frappe Healthcare, then there won't be a need to use integration engine as Frappe framework can take care of most of these things.

Conclusion

In healthcare, integration engines are used extensively to tackle data exchange between systems. Due to its low-code development and batteries included approach, even though Frappe is a web application framework, it can be used as an integration engine as well.

On Resuming Writing Challenge

Anand Reddy Pandikunta — Wed, 19 Jan 2022 00:36:06 GMT

Photo by Kaboompics on Pexels

In 2018, I decided to write at least one blog post per month throughout the year. Even though I tried to write posts every month, I couldn't publish anything in few months.

In 2019, I went a step ahead and made a legal(?) agreement with a friend. I paid him 1,00,000 rupees (~1,200$) and told him that he could keep the money as a reward if I failed to write a blog post every month.

This agreement kept me on my toes. I didn't miss writing a single month in 2019. I stayed awake on the last days of the month to finish and publish the post before midnight.

In 2020, I took up the challenge again and I was able to write at least one post every month.

In 2021, I didn't take up the challenge. I wrote just three posts in the entire year.

In the two years when I took the challenge, even though I wrote a few mediocre articles, I wrote a few good articles. In the other two years when I didn't take the challenge, my writing quality and quantity declined.

Due to this, I decided to take up the writing challenge again this year.

Instead of limiting the 1,00,000 reward to my friend, I decided to extend it to all the readers.

The first person who calls out that there is no new blog post in a month will get the 1,00,000 reward. The next three people will get a small gift as a token of appreciation.

I will try my best to write at least one post every month. Let's wait till the end of the year and see how it goes.

A Typo Got Me $100 Bug Bounty

Anand Reddy Pandikunta — Mon, 25 Oct 2021 03:22:19 GMT

Introduction

On a lazy evening, while on a call with a friend, I made a typo while entering the url. Instead of typing http://app-00421.on-aptible.com, I typed http://app-00412.on-aptible.com¹.

In this article, lets see how this typing mistake got me a bug bounty.

Vulnerability

A bug bounty program² is a deal offered by companies by which individuals can receive recognition and compensation for reporting bugs, security exploits and vulnerabilities.

Aptible provides HIPAA³ compliant PAAS platform so that healthcare companies can deploy their apps without compliance hassle.

After deploying an application on aptible, users can create an endpoint for public access. For this purpose, atpible generates domain names in sequential order.

Due to this, a set of publicly exposed servers will have incremental domain names. A lot of companies use these sequentially generated domain names for staging & testing purposes. In general, many companies don't bother about implementing security best practices on non-production servers.

When I was trying to access a demo site at http://app-00421.on-aptible.com, I made a typo and visited http://app-00412.on-aptible.com. This site was a staging site of some other company without any authentication. The company's source code, AWS keys and a lot of sensitive information was publicly accessible.

I quickly emailed that company regarding this issue and they took their site offline. As per Aptible disclosure policy⁴, this bug is out of scope. Howev er, I sent an email to their team regarding the severity of the issue. Since sequential domain names are generating additional target surface for attackers, I suggested to move to random urls.

For this disclosure, they have provided a bounty of 100$ and Aptible decided to move away from sequential domain names.

URL has been changed for anonymity. ↩
https://en.wikipedia.org/wiki/Bug_bounty_program ↩
HIPAA - Wikipedia ↩
https://www.aptible.com/legal/responsible-disclosure/ ↩

MacBook Productivity Tools For Developers

Anand Reddy Pandikunta — Fri, 11 Jun 2021 02:30:00 GMT

Introduction

When using Mac, there are few utilities which come in handy for day to day operations and also aid in productivity.

Here are some of the useful but lesser know utilities for mac.

Alfred

Alfred is a productivity app for Mac which helps you to search and launch apps, files, bookmarks, and more. You can also search the web and do calculations.

Bandwidth+

Bandwidth+ tracks network usage on Mac. If there are multiple networks, it gives detailed information about the network consumed on all the networks.

CheatSheet

Ever wondered what are the keybindings when using any application? With CheatSheet, we can just hold ⌘ key bit longer, and it will show all the available shortcuts in the application.

Debokee Tools

Wondering which network your Mac connected to? If you use multiple wireless networks, then Debokee Tools can show the connected wireless network name directly in the menu bar.

Espanso

Espanso is a text expanding tool that improves productivity across the system. We can set up shortcuts for frequently typed things like email, phone number etc., so that we don't have to type them again and again.

Flycut

Flycut is a simple clipboard manager, stores history. When you want to copy/paste frequently, this comes in handy.

Grand Perspective

If Mac is running low on disk space, Grand Perspective shows a graphical view of the disk usage. It will be much easier to pinpoint large files that are consuming the disk and then clean them up.

Hotkey

Hotkey is a simple app that allows you to set up global shortcuts for frequently used actions. It can be used to open applications, folders, websites, and more.

Karabiner-Elements

Karabiner Elements allows users to customize keyboard via simple modifications, complex modifications, function key modifications etc.

We can use space bar as space and control as well with a simple modification rule.

Stats

Stats is a simple app that shows the CPU, memory, disk, and network usage in the menu bar. It also shows the temperature of the CPU.

Conclusion

These are some useful utilities for day to day usage. In the upcoming articles, lets learn about useful command line utilities that improve productivity on a daily basis.

Mastering DICOM - #2 Setup Orthanc DICOM Server

Anand Reddy Pandikunta — Fri, 26 Mar 2021 00:30:00 GMT

This is a series of articles on mastering Dicom. In the earlier article, we have learnt how PACS/DICOM simplifies the clinical work flow.

In this article, lets setup a dicom server so that we have a server to play around with Dicom files.

Orthanc Server

There are several Dicom servers like Orthanc, Dicoogle etc. Orthanc is a lightweight open source dicom server and is widely used by many Health care organisations.

Sébastien Jodogne, original author of Orthanc maintains docker images. We can use these images to run Orthanc server locally.

Ensure docker is installed on the machine and then run the following command to start Orthanc server.

$ docker run -p 4242:4242 -p 8042:8042 --rm \
    jodogne/orthanc-python

Once the server is started, we can visit http://localhost:8042 and explore Orthanc server.

Heroku Deployment

Heroku is PAAS platform which supports docker deployments. Lets deploy Orthac server to Heroku for testing.

By default, Orthanc server runs on 8042 port as defined in the config file. Heroku dynamically assigns port for the deployed process.

We can write a shell script which will read port number from environment variable, replace it in Orthanc configuration file and then start Orthanc server.

#! /bin/sh

set -x

echo $PORT

sed 's/ : 8042/ : '$PORT'/g' -i /etc/orthanc/orthanc.json

Orthanc /etc/orthanc/

We can use this shell script as entry point in docker as follows.

FROM jodogne/orthanc-python

EXPOSE $PORT

WORKDIR /app
ADD . /app

ENTRYPOINT [ "./run.sh" ]

We can create a new app in heroku and we can deploy this container.

$ heroku apps:create orthanc-demo

$ heroku container:push web
$ heroku container:release web

Once the deployment is completed, we can access our app from the endpoint provided by heroku. Here is a orthanc demo server running on heroku.

Conclusion

In this article, we have learnt how to setup Orthanc server and deployed it to Heroku. In the next article, let dig deeper into dicom protocol by upload/accessing dicom files to the server.

Minimum Viable Testing - Get Maximum Stability With Minimum Effort

Anand Reddy Pandikunta — Thu, 31 Dec 2020 13:53:11 GMT

Introduction

Even though Test Driven Development(TDD)¹ saves time & money in the long run, there are many excuses why developers don't test the software. In this article, lets look at Minimum Viable Testing(aka Risk-Based Testing)² and how it helps to achieve maximum stability with minimum effort.

Minimum Viable Testing

Pareto principle states that 80% of consequences come from 20% of the causes. In software proucts, 80% of the users use 20% of the features. A bug in these 20% features is likely to cause higher impact than the rest. It makes sense to prioritize testing of these features than the rest.

Assessing the importance of a feature or risk of a bug depends on the product that we are testing. For example, in a project a paid feature gets more importance than free feature.

In TDD, we start with writing tests and then writing code. Compared to TDD, MVT consumes less time. When it comes to testing, there are unit tests, integration tests, snapshot tests, ui tests and so on.

When getting started with testing, it is important to have integration tests in place to make sure if something is working. Also the cost of integration tests is much cheaper compared to unit tests.

Most of the SAAS products have a web/mobile application and an API server to handle requests for the front end applications. Having UI tests for the applications and integration tests for APIs for the most crucial functionality should cover the ground. This will make sure any new code that is being pushed doesnt break the core functionality.

Conclusion

Even though RBT helps with building a test suite quicker that TDD, it should be seen as an alternate option to TDD. We should see RBT as a starting point for testing from which we can take next step towards achieving full stability for the product.

Find Performance Issues In Web Apps with Sentry

Anand Reddy Pandikunta — Mon, 30 Nov 2020 18:29:54 GMT

Introduction

Earlier, we have seen couple of articles here on finding performance issues¹ and how to go about optimizing them². In this article, lets see how to use Sentry Performance to find bottlenecks in Python web applications.

The Pitfalls

A common pitfall while identifying performance issues is to do profiling in development environment. Performance in development environment will be quite different from production environment due to difference in system requirements, database size, network latency etc.

In some cases, performance issues could be happening only for certain users and in specific scenarios.

Replicating production performance on development machine will be costly. To avoid these, we can use APM tool to monitor performance in production.

Sentry Performance

Sentry is widely used Open source error tracking tool. Recently, it has introduced Performance to track performance also. Sentry doesn't need any agent running on the host machine to track performance. Enabling performance monitoring is just a single line change in Sentry³ setup.

import sentry_sdk


sentry_sdk.init(
    dsn="dummy_dsn",
    # Trace half the requests
    traces_sample_rate=0.5,
)

Tracing performance will have additional overhead⁴ on the web application response time. Depending on the traffic, server capacity, acceptable overhead, we can decide what percentage of the requests we need to trace.

Once performance is enabled, we can head over to Sentry web application and see traces for the transactions along with operation breakdown as shown below.

At a glance, we can see percentage of time spent across each component which will pinpoint where the performance problem lies.

If the app server is taking most of the time, we can explore the spans in detail to pinpoint the exact line where it is taking most time. If database is taking most of the time, we can look out for the number of queries it is running and slowest queries to pinpoint the problem.

Sentry also provides option to set alerts when there are performance. For example, when the response time for a duration are less than a limit for a specified duration, Sentry can alert developers via email, slack or any other integration channels.

Conclusion

There are paid APM tools like New Relic, AppDynamics which requires an agent to be installed on the server. As mentioned in earlier articles, there are open source packages like django-silk to monitor performance. It will take time to set up these tools and pinpoint the issue.

Sentry is the only agentless APM tool⁵ available for Python applications. Setting up Sentry performance is quite easy and performance issues can be pinpointed without much hassle.

Finding High-impact Performance Bottlenecks ↩
Optimizing Bottlenecks In Django ↩
Sentry - Performance Monitoring ↩
Sentry has 8% overhead in Django test suite ↩
Earlier there was Opbeat APM which is now shutdown due to acquistion by Elastic. ↩

Make Python Docker Builds Slim & Fast

Anand Reddy Pandikunta — Sat, 31 Oct 2020 12:53:13 GMT

Introduction

When using Docker, if the build is taking time or the build image is huge, it will waste system resources as well as our time. In this article, let's see how to reduce build time as well as image size when using Docker for Python projects.

Project

Let us take a hello world application written in flask.

import flask


app = flask.Flask(__name__)


@app.route('/')
def home():
    return 'hello world - v1.0.0'

Let's create a requirements.txt file to list out python packages required for the project.

flask==1.1.2
pandas==1.1.2

Pandas binary wheel size is ~10MB. It is included in requirements to see how python packages affect docker image size.

Here is our Dockerfile to run the flask application.

FROM python:3.7

ADD . /app

WORKDIR /app

RUN python -m pip install -r requirements.txt

EXPOSE 5000

ENTRYPOINT [ "python" ]

CMD [ "-m" "flask" "run" ]

Let's use the following commands to measure the image size & build time with/without cache.

$ docker build . -t flask:0.0 --pull --no-cache
[+] Building 45.3s (9/9) FINISHED

$ touch app.py  # modify app.py file

$ docker build . -t flask:0.1
[+] Building 15.3s (9/9) FINISHED

$ docker images | grep flask
flask               0.1     06d3e985f12e    1.01GB

With the current docker, here are the results.

1. Install requirements first

FROM python:3.7

WORKDIR /app

ADD ./requirements.txt /app/requirements.txt

RUN python -m pip install -r requirements.txt

ADD . /app

EXPOSE 5000

ENTRYPOINT [ "python" ]

CMD [ "-m" "flask" "run" ]

Let us modify the docker file to install requirements first and then add code to the docker image.

Now, build without cache took almost the same time. With cache, the build is completed in a second. Since docker caches step by step, it has cached python package installation step and thereby reducing the build time.

2. Disable Cache

FROM python:3.7

WORKDIR /app

ADD ./requirements.txt /app/requirements.txt

RUN python -m pip install -r requirements.txt --no-cache

ADD . /app

EXPOSE 5000

ENTRYPOINT [ "python" ]

CMD [ "-m" "flask" "run" ]

By default, pip will cache the downloaded packages. Since we don't need a cache inside docker, let's disable pip cache by passing --no-cache argument.

This reduced the docker image size by ~20MB. In real-world projects, where there are a good number of dependencies, the overall image size will be reduced a lot.

3. Use slim variant

Till now, we have been using defacto Python variant. It has a large number of common debian packages. There is a slim variant that doesn't contain all these common packages⁴. Since we don't need all these debian packages, let's use slim variant.

FROM python:3.7-slim

...

This reduced the docker image size by ~750 MB without affecting the build time.

4. Build from source

Python packages can be installed via wheels (.whl files) for a faster and smoother installation. We can also install them via source code. If we look at Pandas project files on PyPi¹, it provides both wheels as well as tar zip source files. Pip will prefer wheels over source code the installation process will be much smoother.

To reduce Docker image size, we can build from source instead of using the wheel. This will increase build time as the python package will take some time to compile while building.

Here build size is reduced by ~20MB but the build has increased to 15 minutes.

5. Use Alpine

Earlier we have used, python slim variant the base image. However, there is Alpine variant which is much smaller than slim. One caveat with using alpine is Python wheels won't work with this image².

We have to build all packages from source. For example, packages like TensorFlow provide only wheels for installation. To install this on Alpine, we have to install from the source which will take additional effort to figure out dependencies and install.

Using Alpine will reduce the image size by ~70 MB but it is not recomended to use Alpine as wheels won't work with this image.

All the docker files used in the article are available on github³.

Conclusion

We have started with a docker build of 1.01 GB and reduced it to 0.13 GB. We have also optimized build times using the docker caching mechanism. We can use appropriate steps to optimize build for size or speed or both.

How To Deploy Mirth Connect To Kubernetes

Anand Reddy Pandikunta — Wed, 30 Sep 2020 17:55:46 GMT

Introduction

NextGen Connect(previously Mirth Connect) is widely used integration engine for information exchange in health-care domain. In this article, let us see how to deploy Mirth Connect to a Kubernetes cluster.

Deployment To k8s

From version 3.8, NextGen has started providing official docker images for Connect¹. By default, Connect docker exposes 8080, 8443 ports. We can start a Connect instance locally, by running the following command.

$docker run -p 8080:8080 -p 8443:8443 nextgenhealthcare/connect

We can use this docker image and create a k8s deployment to start a container.

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: mirth-connect
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: mirth-connect
        image: docker.io/nextgenhealthcare/connect
        ports:
        - name: http
          containerPort: 8080
        - name: https
          containerPort: 8443
        - name: hl7-test
          containerPort: 9001
        env:
          - name: DATABASE
            value: postgres
          - name: DATABASE_URL
            value: jdbc:postgresql://avilpage.com:5432/mirth_db

This deployment file can be applied on a cluster using kubectl.

$ kubectl apply -f connect-deployment.yaml

To access this container, we can create a service to expose this deployment to public.

---
apiVersion: v1
kind: Service
metadata:
  name: mirth-connect
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:ap-south-1:foo
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    external-dns.alpha.kubernetes.io/hostname: connect.avilpage.com
spec:
  type: LoadBalancer
  selector:
    app: mirth-connect
  ports:
    - name: http
      port: 80
      targetPort: 8080
      protocol: TCP
    - name: https
      port: 443
      targetPort: 8443
      protocol: TCP
    - name: hl7-test
      port: 9001
      targetPort: 9001
      protocol: TCP

This will create a load balancer in AWS through which we can access mirth connect instance. If an ingress controller is present in the cluster, we can use it directly instead of using a seperate load balancer for this service.

Once Mirth Connect is up & running, we might have to create HL7 channels running on various ports. In the above configuration files, we have exposed 9001 HL7 port for testing of channel. Once we configure Mirth Channels, we need to expose appropriate ports in deployment as well as service similiar to this.

Conclusion

Earlier, there were no official docker images for Mirth Connect and it was bit diffucult to dockerize Mirth Connect and deploy it. With the release of official Docker images, deploying Mirth Connect to k8s or any other container orchestration platform has become much easier.

https://hub.docker.com/r/nextgenhealthcare/connect/ ↩

Serial Bluetooth Terminal With Python Asyncio

Anand Reddy Pandikunta — Mon, 31 Aug 2020 15:51:21 GMT

Introduction

PySerial package provides a tool called miniterm¹, which provides a terminal to interact with any serial ports.

However miniterm sends each and every character as we type instead of sending entire message at once. In addition to this, it doesn't provide any timestamps on the messages transferred.

In this article, lets write a simple terminal to address the above issues.

Bluetooth Receiver

pyserial-asyncio² package provides Async I/O interface for communicating with serial ports. We can write a simple function to read and print all the messages being received on a serial port as follows.

import sys
import asyncio
import datetime as dt

import serial_asyncio


async def receive(reader):
    while True:
        data = await reader.readuntil(b'\n')
        now = str(dt.datetime.now())
        print(f'{now} Rx <== {data.strip().decode()}')


async def main(port, baudrate):
    reader, _ = await serial_asyncio.open_serial_connection(url=port, baudrate=baudrate)
    receiver = receive(reader)
    await asyncio.wait([receiver])


port = sys.argv[1]
baudrate = sys.argv[2]

loop = asyncio.get_event_loop()
loop.run_until_complete(main(port, baudrate))
loop.close()

Now we can connect a phone's bluetooth to a laptop bluetooth. From phone we can send messages to laptop using bluetooth terminal app like Serial bluetooth terminal⁴.

Here a screenshot of messages being send from an Android device.

We can listen to these messages on laptop via serial port by running the following command.

$ python receiver.py /dev/cu.Bluetooth-Incoming-Port 9600
2020-08-31 10:44:50.995281 Rx <== ping from android
2020-08-31 10:44:57.702866 Rx <== test message

Bluetooth Sender

Now lets write a sender to send messages typed on the terminal to the bluetooth.

To read input from terminal, we need to use aioconsole ³. It provides async input equivalent function to read input typed on the terminal.

import sys
import asyncio
import datetime as dt

import serial_asyncio
import aioconsole


async def send(writer):
    stdin, stdout = await aioconsole.get_standard_streams()
    async for line in stdin:
        data = line.strip()
        if not data:
            continue
        now = str(dt.datetime.now())
        print(f'{now} Tx ==> {data.decode()}')
        writer.write(line)


async def main(port, baudrate):
    _, writer = await serial_asyncio.open_serial_connection(url=port, baudrate=baudrate)
    sender = send(writer)
    await asyncio.wait([sender])


port = sys.argv[1]
baudrate = sys.argv[2]

loop = asyncio.get_event_loop()
loop.run_until_complete(main(port, baudrate))
loop.close()

We can run the program with the following command and send messages to phone's bluetooth.

$ python sender.py /dev/cu.Bluetooth-Incoming-Port 9600

ping from mac
2020-08-31 10:46:52.222676 Tx ==> ping from mac
2020-08-31 10:46:58.423492 Tx ==> test message

Here a screenshot of messages received on Android device.

Conclusion

If we combine the above two programmes, we get a simple bluetooth client to interact with any bluetooth via serial interface. Here is the complete code ⁵ for the client.

In the next article, lets see how to interact with Bluetooth LE devices.

Set Default Date For Date Hierarchy In Django Admin

Anand Reddy Pandikunta — Fri, 31 Jul 2020 14:36:49 GMT

Introduction

When we monitor daily events from django admin, most of the time we are interested in events related to today. Django admin provides date based drill down navigation page via ModelAdmin.date_hierarchy¹ option. With this, we can navigate to any date to filter out events related to that date.

One problem with this drill down navigation is, we have to navigate to todays date every time we open a model in admin. Since we are interested in todays events most of the time, setting todays date as default filtered date will solve the problem.

Set Default Date For Date Hierarchy

Let us create an admin page to show all the users who logged in today. Since User model is already registered in admin by default, let us create a proxy model to register it again.

from django.contrib.auth.models import User


class DjangoUser(User):
    class Meta:
        proxy = True

Lets register this model in admin to show logged in users details along with date hierarchy.

from django.contrib import admin


@admin.register(DjangoUser)
class MetaUserAdmin(admin.ModelAdmin):
    list_display = ('username', 'is_active', 'last_login')
    date_hierarchy = 'last_login'

If we open DjangoUser model in admin page, it will show drill down navigation bar like this.

Now, if we drill down to a particular date, django adds additional query params to the admin url. For example, if we visit 2020-06-26 date, corresponding query params are /?last_login__day=26&last_login__month=6&last_login__year=2020.

We can override changelist view and set default params to todays date if there are no query params. If there are query params then render the original response.

@admin.register(DjangoUser)
class MetaUserAdmin(admin.ModelAdmin):
    list_display = ('username', 'is_active', 'last_login')
    date_hierarchy = 'last_login'

    def changelist_view(self, request, extra_context=None):
        if request.GET:
            return super().changelist_view(request, extra_context=extra_context)

        date = now().date()
        params = ['day', 'month', 'year']
        field_keys = ['{}__{}'.format(self.date_hierarchy, i) for i in params]
        field_values = [getattr(date, i) for i in params]
        query_params = dict(zip(field_keys, field_values))
        url = '{}?{}'.format(request.path, urlencode(query_params))
        return redirect(url)

Now if we open the same admin page, it will redirect to todays date by default.

Conclusion

In this article, we have seen how to set a default date for date_hierarchy in admin page. We can also achieve similar filtering by settiing default values for search_filter or list_filter which will filter items related to any specific date.

Django docs - ModelAdmin.date_hierarchy ↩

How Dart, Flutter Stateful Hot Reload Work? - Part 1

Anand Reddy Pandikunta — Tue, 30 Jun 2020 13:47:20 GMT

This will be a series of articles on exploring the internals of Dart & Flutter stateful hot reload. In the first article, lets write a simple dart program to see stateful hot reload in action. Then lets delve into details on what is happening.

Stateful Hot Reload

import 'dart:async';

int total = 0;

void adder(_) {
  int delta = 2;
  total += delta;

  print("Total is $total. Adding $delta");
}

void main() {
  Timer.periodic(Duration(seconds: 2), adder);
}

In the above program¹, we are using a Timer.periodic² to create a timer which calls adder function every 2 seconds.

We can run this program from command line using

$ dart --observe hot_reload.dart
Observatory listening on http://127.0.0.1:8181/d42KmW4LknU=/
Total is 2. Adding 2
Total is 4. Adding 2
Total is 6. Adding 2
Total is 8. Adding 2
...

This will start executing the program and will provide a link to observatory³, a tool to profile/debug Dart applications.

As the program is executing, lets open the program in an editor, change delta from 2 to 3.

  # change this
  # int delta = 2;

  # change to
  int delta = 3;

If we restart the program, it will start executing from the beginning and it will lose the state of the program.

$ dart --observe hot_reload.dart
Observatory listening on http://127.0.0.1:8181/eoP2lpC2ZWw=/
Total is 3. Adding 3
Total is 6. Adding 3
Total is 9. Adding 3

Instead of restart, we can open the observatory link in browser, open main isolate and click on Reload Source button.

As we can see from the below output, it did a stateful hot reload and state of the program is preserved instead of starting from the beginning.

$ dart --observe hot_reload.dart
Observatory listening on http://127.0.0.1:8181/n_GSAKsyr5s=/
Total is 2. Adding 2
Total is 4. Adding 2
Total is 6. Adding 2
Total is 8. Adding 2
Total is 11. Adding 3 # after hot reload
Total is 14. Adding 3
Total is 17. Adding 3
Total is 20. Adding 3

During a hot reload, Dart VM will apply changes to a live program⁴. If the source code of a method is changed, VM will replace the methods with the new updated methods. Next time, when the program looks up for a particular method, it will find the updated method and use it.

Conclusion

In this article, we have seen how hot reload works by writing a simple program in Dart. In the upcoming articles, lets dive into the Dart VM internals, Flutter architecture and other nitty gritties of hot reload.

Tips On Improving kubectl Productivity

Anand Reddy Pandikunta — Sun, 31 May 2020 13:12:00 GMT

kubectl is CLI tool to control Kubernetes clusters. As we start using kubectl to interact with mutliple clusters, we end up running lengthy commands and even running multiple commands for simple tasks like running a shell in a container.

In this article, lets learn few tips to improve our productivity when using kubectl.

Aliases

Aliases in general improve the productivity when using a shell.

kubectl provides shortcuts for commands. For example,

# instead of running full command
$ kubectl get services

# we can use short hand version
$ kubectl get svc

It also provides completion for commands.

# enable completion for zsh
$ source <(kubectl completion zsh)

# type `kubectl ` and hit `<TAB>` will show possible options
$ kubectl
annotate       attach         cluster-info
api-resources  auth           completion
api-versions   autoscale      config
apply          certificate    convert

# type `kubectl g`, and hit `<TAB>` will show possible options
$ kubectl get

Even though completions are helpful, setting up up aliases for most commanly used commands will save a lot of time.

alias k='kubectl'

alias kdp='kubectl describe pod'
alias kgp='kubectl get pods'
alias kgpa='kubectl get pods --all-namespaces'
alias ket='kubectl exec -it'
alias wkgp='watch -n1 kubectl getp pods'

alias kga='kubectl get all'
alias kgaa='kubectl get all --all-namespaces'

alias kaf='kubectl apply -f'

alias kcgc='kubectl config get-contexts'
alias kccc='kubectl config current-context'

If you don't write your own aliases, there is kubectl-aliases which provides exhuastive list of aliases. We can source this file in rc file and start using them.

Use Functions

Even though aliases help us to run lengthy commands with an alias, there are times where we have to run multiple commands to get things done for a single task.

For example, to view kubenetes dashboard, we have to get the token, start proxy server and then open the url in browser. We can write a simple function as shown below to do all of that.

kp() {
    kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}') | grep 'token:' | awk '{print $2}' | pbcopy
    open http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
    kubectl proxy
}

Now from the shell, when we run kp, it function will copy the token to clipboard, open kubernetes dashboard in browser and will start the proxy server.

Use Labels

To describe a pod or tail logs from a pod, we can use pod names.

$ kubectl get pods
NAME                             READY   STATUS
hello-world-79d794c659-tpfv2     1/1     Running


$ kubectl describe pod hello-world-79d794c659-tpfv2

$ kubectl logs -f pod/hello-world-79d794c659-tpfv2

When the app gets updated, the name of pod also updates. If your shell has auto completion feature, it will autocomplete to previous name.

So, instead of using pod name, we can use pod labels as mentioned below.

$ kubectl describe pod -l=hello-world

$ kubectl logs -f -l=pod/hello-world

We run the command once and next time shell will show autocomplete and we can use that directly.

Kubectl Tools

k8s has a good ecosystem and the following packages are aimed to make certain k8s tasks easier.

kubectl-debug - Debug pod by a new container with all troubleshooting tools pre-installed.

kube-forwarder - Easy to use port forwarding manager.

stern - Multi pod and container log tailing.

kubectx - Quick way to switch between clusters and namespaces.

kubebox - Terminal and Web console for Kubernetes.

k9s - Interactive terminal UI.

kui - Hybrid CLI/UI tool for k8s.

click - Interactive controller for k8s.

lens - Stand alone corss platform k8s IDE.

Conclusion

In this article we have seen some useful methods as well as some tools to improve productivity with kubectl. If you spend a lot of time interacting with kubernetes cluster, it is important to notice your workflows and find better tools or ways to improve productivity.

Continuous Deployment To Kubernetes With Skaffold

Anand Reddy Pandikunta — Thu, 30 Apr 2020 11:15:36 GMT

In this article, let us see how to set up a continuous deployment pipeline to Kubernetes in CircleCI using Skaffold.

Prerequisites

You should have a kubernetes cluster in a cloud environment or in your local machine. Check your cluster status with the following commands.

$ kubectl cluster-info
$ kubectl config get-contexts

You should know how to manually deploy your application to kubernetes.

# push latest docker image to container registry
$ docker push chillaranand/library

# deploy latest image to k8s
$ kubectl apply -f app/deployment.yaml
$ kubectl apply -f app/service.yaml

Skaffold

Skaffold is a CLI tool to facilitate continuous development and deployment workflows for Kubernetes applications.

Skaffold binaries are available for all platforms. Download the binary file for your OS and move it to bin folder.

$ curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-darwin-amd64
$ chmod +x skaffold
$ sudo mv skaffold /usr/local/bin

Inside your project root, run init command to generate a config file. If your project has k8s manifests, it will detect them and include it in the configuration file.

$ skaffold init
Configuration skaffold.yaml was written

$ cat skaffold.yaml
apiVersion: skaffold/v2beta1
kind: Config
metadata:
  name: library
build:
  artifacts:
  - image: docker.io/chillaranand/library
deploy:
  kubectl:
    manifests:
    - kubernetes/deployment.yaml
    - kubernetes/service.yaml

To deploy latest changes to your cluster, run

$ skaffold run

This will build the docker image, push to registry and will apply the manifests in the clusters. Now, k8s will pull the latest image from the registry and create a new deployment.

CircleCI Workflow

version: 2.1

orbs:
  aws-cli: circleci/aws-cli@0.1.19
  kubernetes: circleci/kubernetes@0.11.0

commands:
  kubernetes-deploy:

    steps:
      - setup_remote_docker

      - aws-cli/setup:
          profile-name: default

      - kubernetes/install-kubectl:
          kubectl-version: v1.15.10

      - checkout

      - run:
          name: container registry log in
          command: |
            sudo $(aws ecr get-login --region ap-south-1 --no-include-email)

      - run:
          name: install skaffold
          command: |
            curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64
            chmod +x skaffold
            sudo mv skaffold /usr/local/bin

      - run:
          name: update kube config to connect to the required cluster
          command: |
            aws eks --region ap-south-1 update-kubeconfig --name demo-cluster

      - run:
          name: deploy to k8s
          command: |
            skaffold run

CircleCI orbs are shareable packages to speed up CI setup. Here we are using aws-cli, kubernetes orbs to easily install/setup them inside the CI environment.

Since CircleCI builds run in a docker container, to run docker commands inside container, we have to specify setup_remote_docker key so that a separate environment is created for it.

Remaining steps are self explainatory.

Conclusion

Here we have seen how to set up CD to kubernetes in CircleCI. If we want to set up this another CI like Jenkins or Travis, instead of using orbs, we have to use system package mangers like apt-get to install them. All others steps will remain same.

Work From Home Tips For Non-remote Workers

Anand Reddy Pandikunta — Tue, 31 Mar 2020 13:09:58 GMT

Remote-first and remote-friendly companies have a different work culture & communication process compared to non-remote companies. Due to COVID-19¹ world wide pandemic, majority of workers who didn't had prior remote experience are WFH(working from home). This article is intended to provide some helpful tips for such people.

Work Desk

It is important to have a dedicated room or at least a desk for work. This creates a virtual boundary between your office work and personal work. Otherwise, you will end up working from bed, dining tables, kitchen etc which will result in body pains due to bad postures.

Get Ready

Image Credit: raywenderlich ²

Start your daily routine as if you are going to office. It is easy to stop caring about personal grooming and attire when WFH. Your attire can influence your focus and productivity.

If you find getting ready is hard, schedule video calls for all meetings with your colleagues. This might give some additional motivation for you to get ready early in the morning. When the work time, go to your work desk and start working.

Self Discipline

Schedule your work time. Whenever possible try to stick to office working hours. Without proper schedule, you will either end up under working or over working as your personal work and office work gets mixed up.

Take regular breaks during work hours. Without any distractions, it is easy to get lost in the pixels for longer durations. Taking short breaks for a quick walk and getting a fresh air outside will freshen up.

With unlimited access to kitchen and snacks, it is hard to avoid binge eating at home. But atleast avoid binge eating during office hours.

Exercise. Since WFH involves only sitting in a chair through out the day, staying physically active is challenging especially during this pandemic. Exercise few minutes every morning, help yourself in the kitchen by making a meal or doing dishes, clean your house etc., should help in staying physically active.

Seek Help

WFH can be lonely at times as the social interactions are quite less. Schedule 1 to 1 meetings or virtual coffe meetings with your colleagues to increase social interactions. Discuss WFH problems with your colleagues, friends and remote communities to see how they are tacking those problems.

How To Reduce Python Package Footprint?

Anand Reddy Pandikunta — Sat, 29 Feb 2020 14:50:38 GMT

PyPi¹ hosts over 210K+ projects and the average size of Python package is less than 1MB. However some of the most used packages in scientific computing like NumPy, SciPy has large footprint as they bundle shared libraries² along with the package.

Build From Source

If a project needs to be deployed in AWS Lambda, the total size of deployment package should be less than 250MB³.

$ pip install numpy

$ du -hs ~/.virtualenvs/py37/lib/python3.7/site-packages/numpy/
 85M    /Users/avilpage/.virtualenvs/all3/lib/python3.7/site-packages/numpy/

Just numpy occupies 85MB space on Mac machine. If we include a couple of other packages like scipy & pandas, overall size of the package crosses 250MB.

An easy way reduce the size of python packages is to build from source instead of use pre-compiled wheels.

$ CLFAGS='-g0 -Wl -I/usr/include:/usr/local/include -L/usr/lib:/usr/local/lib' pip install numpy --global-option=build_ext

$ du -hs ~/.virtualenvs/py37/lib/python3.7/site-packages/numpy/
 23M    /Users/avilpage/.virtualenvs/all3/lib/python3.7/site-packages/numpy/

We can see the footprint has reduced by ~70% when using sdist instead of wheel. This⁴ article provides more details about these CFLAG optimization when installing a package from source.

Shared Packages

When using a laptop with low storage for multiple projects with conflicting dependencies, a seperate virtual environment is needed for each project. This will lead to installing same version of the package in multiple places which increases the footprint.

To avoid this, we can create a shared virtual environment which has most commonly used packages and share it across all the enviroments. For example, we can create a shared virtual enviroment with all the packages required for scientific computing.

For each project, we can create a virtual enviroment and share all packages of the common enviroment. If any project requires a specific version of the package, the same package can be install in project enviroment.

$ cat common-requirements.txt  # shared across all enviroments
numpy==1.18.1
pandas==1.0.1
scipy==1.4.1

$ cat project1-requirements.txt  # project1 requirements
numpy==1.18.1
pandas==1.0.0
scipy==1.4.1

$ cat project2-requirements.txt  # project2 enviroments
numpy==1.17
pandas==1.0.0
scipy==1.4.1

After creating a virtual enviroment for a project, we can create a .pth file with the path of site-packages of common virtual enviroment so that all those packages are readily available in the new project.

$ echo '/users/avilpage/.virtualenvs/common/lib/python3.7/site-packages' >
 ~/.virtualenvs/project1/lib/python3.7/site-packages/common.pth

Then we can install the project requirements which will install only missing packages.

$ pip install -r project1-requirements.txt

Global Store

The above shared packages solution has couple issues.

User has to manually create and track shared packages for each Python version and needs to bootstrap it in every project.
When there is an incompatible version of package in multiple projects, user will end up with duplicate installations of the same version.

To solve this⁵, we can have a global store of packages in a single location segregated by python and package version. Whenever a user tries to install a package, check if the package is in global store. If not install it in global store. If present, just link the package to virtualenvs.

For example, numpy1.17 for Python 3.7 and numpy1.18 for Python 3.6 can be stored in the global store as follows.

$ python3.6 -m pip install --target ~/.mpip/numpy/3.6_1.18 numpy

$ python3.7 -m pip install --target ~/.mpip/numpy/3.7_1.17 numpy

# in project venv
echo '~/.mpip/numpy/3.7_1.17' > PATH_TO_ENV/lib/python3.7/site-packages/numpy.pth

With this, we can ensure one version of the package is stored in the disk only once. I have created a simple package manager called mpip⁶ as a POC to test this and it seems to work as expected.

These are couple of ways to reduce to footprint of Python packages in a single environment as well as muliple enviroments.

Disabling Atomic Transactions In Django Test Cases

Anand Reddy Pandikunta — Fri, 31 Jan 2020 15:51:21 GMT

TestCase is the most used class for writing tests in Django. To make tests faster, it wraps all the tests in 2 nested atomic blocks.

In this article, we will see where these atomic blocks create problems and find ways to disable it.

Select for update

Django provides select_for_update() method on model managers which returns a queryset that will lock rows till the end of transaction.

def update_book(pk, name):
    with transaction.atomic():
        book = Book.objects.select_for_update().get(pk=pk)
        book.name = name
        book.save()

When writing test case for a piece of code that uses select_for_update, Django recomends not to use TestCase as it might not raise TransactionManagementError.

Threads

Let us take a view which uses threads to get data from database.

def get_books(*args):
    queryset = Book.objects.all()
    serializer = BookSerializer(queryset, many=True)
    response = serializer.data
    return response

class BookViewSet(ViewSet):

    def list(self, request):
        with ThreadPoolExecutor() as executor:
            future = executor.submit(get_books, ())
            return_value = future.result()
        return Response(return_value)

A test which writes some data to db and then calls this API will fail to fetch the data.

class LibraryPaidUserTestCase(TestCase):
    def test_get_books(self):
        Book.objects.create(name='test book')

        self.client = APIClient()
        url = reverse('books-list')
        response = self.client.post(url, data=data)
        assert response.json()

Threads in the view create a new connection to the database and they don't see the created test data as the transaction is not yet commited.

Transaction Test Case

To handle above 2 scenarios or other scenarios where database transaction behaviour needs to be tested, Django recommends to use TransactionTestCase instead of TestCase.

from django.test import TransactionTestCase

Class LibraryPaidUserTestCase(TransactionTestCase):
    def test_get_books(self):
        ...

With TransactionTestCase, db will be in auto commit mode and threads will be able to fetch the data commited earlier.

Consider a scenario, where there are other utility classes which are subclassed from TestCase.

class LibraryTestCase(TestCase):
    ...

class LibraryUserTestCase(LibraryTestCase):
    ...

class LibraryPaidUserTestCase(LibraryTestCase):
    ...

If we subclass LibraryTestCase with TransactionTestCase, it will slow down the entire test suite as all the tests run in autocommit mode.

If we subclass LibraryUserTestCase with TransactionTestCase, we will miss the functionality in LibraryTestCase. To prevent this, we can override the custom methods to call TransactionTestCase.

If we look at the source code of TestCase, it has 4 methods to handle atomic transactions. We can override them to prevent creation of atomic transactions.

class LibraryPaidUserTestCase(LibraryTestCase):
    @classmethod
    def setUpClass(cls):
        super(TestCase, cls).setUpClass()

    @classmethod
    def tearDownClass(cls):
        super(TestCase, cls).tearDownClass()

    def _fixture_setup(self):
        return super(TestCase, self)._fixture_setup()

    def _fixture_teardown(self):
        return super(TestCase, self)._fixture_teardown()

We can also create a mixin with the above methods and subclass it wherever this functionality is needed.

Conclusion

Django wraps tests in TestCase inside atomic transactions to speed up the run time. When we are testing for db transaction behaviours, we have to disable this using appropriate methods.

Mastering DICOM - #1 Clinical Workflows 101

Anand Reddy Pandikunta — Tue, 31 Dec 2019 13:29:09 GMT

Introduction

In hospitals, PACS simplifies the clinical workflow by reducing physical and time barriers. A typical radiology workflow looks like this.

Credit: Wikipedia

A patient as per doctor's request will visit a radiology center to undergo CT/MRI/X-RAY. Data captured from modality(medical imaging equipments like CT/MRI machine) will be sent to QA for verfication and then sent to PACS for archiving.

After this when patient visits doctor, doctor can see this study on his workstation(which has DICOM viewer) by entering patient details.

In this series of articles, we will how to achieve this seamless transfer of medical data digitally with DICOM.

DICOM standard

DICOM modalities create files in DICOM format. This file has dicom header which contains meta data and dicom data set which has modality info(equipment information, equipment configuration etc), patient information(name, sex etc) and the image data.

Storing and retreiving DICOM files from PACS servers is generally achieved through DIMSE DICOM for desktop applications and DICOMWeb for web applications.

All the machines which transfer/receive DICOM data must follow DICOM standard. With this all the DICOM machines which are in a network can store and retrieve DICOM files from PACS.

When writing software to handle DICOM data, there are third party packages to handle most of the these things for us.

Python(pydicom, pynetdicom)
Ruby (ruby-dicom)
R (oro.dicom)
C/C++ (dcmtoolkit)

Conclusion

In this article, we have learnt the clinical radiology workflow and how DICOM standard is useful in digitally transferring data between DICOM modalities.

In the next article, we will dig into DICOM file formats and learn about the structure of DICOM data.

Verifying TLS Certificate Chain With OpenSSL

Anand Reddy Pandikunta — Sat, 30 Nov 2019 04:45:14 GMT

Introduction

To communicate securely over the internet, HTTPS (HTTP over TLS) is used. A key component of HTTPS is Certificate authority (CA), which by issuing digital certificates acts as a trusted 3rd party between server(eg: google.com) and others(eg: mobiles, laptops).

In this article, we will learn how to obtain certificates from a server and manually verify them on a laptop to establish a chain of trust.

Chain of Trust

TLS certificate chain typically consists of server certificate which is signed by intermediate certificate of CA which is inturn signed with CA root certificate.

Using OpenSSL, we can gather the server and intermediate certificates sent by a server using the following command.

$ openssl s_client -showcerts -connect avilpage.com:443

CONNECTED(00000006)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = www.github.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=www.github.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
-----BEGIN CERTIFICATE-----
MIIHMTCCBhmgAwIBAgIQDf56dauo4GsS0tOc8
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlna
0wGjIChBWUMo0oHjqvbsezt3tkBigAVBRQHvF
aTrrJ67dru040my
-----END CERTIFICATE-----
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
-----BEGIN CERTIFICATE-----
MIIEsTCCA5mgAwIBAgIQBOHnpNxc8vNtwCtC
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGln
0wGjIChBWUMo0oHjqvbsezt3tkBigAVBRQHv
cPUeybQ=
-----END CERTIFICATE-----

    Verify return code: 0 (ok)

This command internally verfies if the certificate chain is valid. The output contains the server certificate and the intermediate certificate along with their issuer and subject. Copy both the certificates into server.pem and intermediate.pem files.

We can decode these pem files and see the information in these certificates using

$ openssl x509 -noout -text -in server.crt

Certificate:
    Data:
        Version: 3 (0x2)
    Signature Algorithm: sha256WithRSAEncryption
    ----

We can also get only the subject and issuer of the certificate with

$ openssl x509 -noout -subject -noout -issuer -in server.pem

subject= CN=www.github.com
issuer= CN=DigiCert SHA2 High Assurance Server CA

$ openssl x509 -noout -subject -noout -issuer -in intermediate.pem

subject= CN=DigiCert SHA2 High Assurance Server CA
issuer= CN=DigiCert High Assurance EV Root CA

Now that we have both server and intermediate certificates at hand, we need to look for the relevant root certificate (in this case DigiCert High Assurance EV Root CA) in our system to verify these.

If you are using a Linux machine, all the root certificate will readily available in .pem format in /etc/ssl/certs directory.

If you are using a Mac, open Keychain Access, search and export the relevant root certificate in .pem format.

We have all the 3 certificates in the chain of trust and we can validate them with

$ openssl verify -verbose -CAfile root.pem -untrusted intermediate.pem server.pem
server.pem: OK

If there is some issue with validation OpenSSL will throw an error with relevant information.

Conclusion

In this article, we learnt how to get certificates from the server and validate them with the root certificate using OpenSSL.

Writing & Editing Code With Code - Part 1

Anand Reddy Pandikunta — Thu, 31 Oct 2019 15:51:21 GMT

In Python community, metaprogramming is often used in conjunction with metaclasses. In this article, we will learn about metaprogramming, where programs have the ability to treat other programs as data.

Metaprogramming

When we start writing programs that write programs, it opens up a lot of possibilities. For example, here is a metaprogramme that generates a program to print numbers from 1 to 100.

with open('num.py', 'w') as fh:
    for i in range(100):
        fh.write('print({})'.format(i))

This 3 lines of program generates a hundred line of program which produces the desired output on executing it.

This is a trivial example and is not of much use. Let us see practical examples where metaprogramming is used in Django for admin, ORM, inspectdb and other places.

Metaprogramming In Django

Django provides a management command called inspectdb which generates Python code based on SQL schema of the database.

$ ./manage.py inspectdb

from django.db import models

class Book(models.Model):
    name = models.CharField(max_length=100)
    slug = models.SlugField(max_length=100)
    ...

In django admin, models can be registered like this.

from django.contrib import admin

from book.models import Book


admin.site.register(Book)

Eventhough, we have not written any HTML, Django will generate entire CRUD interface for the model in the admin. Django Admin interface is a kind of metaprogramme which inspects a model and generates a CRUD interface.

Django ORM generates SQL statements for given ORM statements in Python.

In [1]: User.objects.last()
SELECT "auth_user"."id",
       "auth_user"."password",
       "auth_user"."last_login",
       "auth_user"."is_superuser",
       "auth_user"."username",
       "auth_user"."first_name",
       "auth_user"."last_name",
       "auth_user"."email",
       "auth_user"."is_staff",
       "auth_user"."is_active",
       "auth_user"."date_joined"
  FROM "auth_user"
 ORDER BY "auth_user"."id" DESC
 LIMIT 1


Execution time: 0.050304s [Database: default]

Out[1]: <User: anand>

Some frameworks/libraries use metaprogramming to solve problems realted to generating, modifying and transforming code.

We can also use these techniques in everyday programming. Here are some use cases.

Generate REST API automatically.
Automatically generate unit test cases based on a template.
Generate integration tests automatically from the network traffic.

These are a some of the things related to web development where we can use metaprogramming techniques to generate/modify code. We will learn more about this in the next part of the article.

Tips On Writing Data Migrations in Django Application

Anand Reddy Pandikunta — Mon, 30 Sep 2019 15:51:21 GMT

Introduction

In a Django application, when schema changes Django automatically generates a migration file for the schema changes. We can write additional migrations to change data.

In this article, we will learn some tips on writing data migrations in Django applications.

Use Management Commands

Applications can register custom actions with manage.py by creating a file in management/commands directory of the application. This makes it easy to (re)run and test data migrations.

Here is a management command which migrates the status column of a Task model.

from django.core.management.base import BaseCommand
from library.tasks import Task

class Command(BaseCommand):

    def handle(self, *args, **options):
        status_map = {
            'valid': 'ACTIVE',
            'invalid': 'ERROR',
            'unknown': 'UKNOWN',
        }
        tasks = Task.objects.all()
        for tasks in tasks:
            task.status = status_map[task.status]
            task.save()

If the migration is included in Django migration files directly, we have to rollback and re-apply the entire migration which becomes cubersome.

Link Data Migrations & Schema Migrations

If a data migration needs to happen before/after a specific schema migration, include the migration command using RunPython in the same schema migration or create seperate schema migration file and add schema migration as a dependency.

def run_migrate_task_status(apps, schema_editor):
    from library.core.management.commands import migrate_task_status
    cmd = migrate_task_status.Command()
    cmd.handle()


class Migration(migrations.Migration):

    dependencies = [
    ]

    operations = [
        migrations.RunPython(run_migrate_task_status, RunSQL.noop),
    ]

Watch Out For DB Queries

When working on a major feature that involves a series of migrations, we have to be careful with data migrations(which use ORM) coming in between schema migrations.

For example, if we write a data migration script and then make schema changes to the same table in one go, then the migration script fails as Django ORM will be in invalid state for that data migration.

To overcome this, we can explicitly select only required fields and process them while ignoring all other fields.

# instead of
User.objects.all()

# use
User.objects.only('id', 'is_active')

As an alternative, we can use raw SQL queries for data migrations.

Conclusion

In this article, we have seen some of the problems which occur during data migrations in Django applications and tips to alleviate them.

Profiling & Optimizing Bottlenecks In Django

Anand Reddy Pandikunta — Sat, 31 Aug 2019 15:51:21 GMT

In the previous article, we have learnt where to start with performance optimization in django application and find out which APIs to optimize first. In this article, we will learn how to optimize those selected APIs from the application.

Profling APIs With django-silk

django-silk provides silk_profile function which can be used to profile selected view or a snippet of code. Let's take a slow view to profile and see the results.

from silk.profiling.profiler import silk_profile


@silk_profile()
def slow_api(request):
    time.sleep(2)
    return JsonResponse({'data': 'slow_response'})

We need to add relevant silk settings to django settings so that required profile data files are generated and stored in specified locations.

SILKY_PYTHON_PROFILER = True
SILKY_PYTHON_PROFILER_BINARY = True
SILKY_PYTHON_PROFILER_RESULT_PATH = '/tmp/'

Once the above view is loaded, we can see the profile information in silk profiling page.

In profile page, silk shows a profile graph and highlights the path where more time is taken.

It also shows cprofile stats in the same page. This profile data file can be downloaded and used with other visualization tools like snakeviz.

By looking at the above data, we can see most of the time is spent is time.sleep in our view.

Profling APIs With django-extensions

If you don't want to use silk, an alternate way to profile django views is to use runprofileserver command provided by django-extensions package. Install django-extensions package and then start server with the following command.

$ ./manage.py runprofileserver --use-cprofile --nostatic --prof-path /tmp/prof/

This command starts runserver with profiling tools enabled. For each request made to the server, it will save a corresponding .prof profile data file in /tmp/prof/ folder.

After profile data is generated, we can use profile data viewing tools like snakeviz, cprofilev visualize or browse the profile data.

Install snakeviz using pip

$ pip install snakeviz

Open the profile data file using snakeviz.

$ snakeviz /tmp/prof/api.book.list.4212ms.1566922008.prof

It shows icicles graph view and table view of profile data of that view.

These will help to pinpoint which line of code is slowing down the view. Once it is identified, we can take appropriate action like optimize that code, setting up a cache or moving it to a task queue if it is not required to be performed in the request-response cycle.

Versioning & Retrieving Files From AWS S3 With Boto

Anand Reddy Pandikunta — Wed, 24 Jul 2019 15:51:21 GMT

Introduction

Amazon S3 (Amazon Simple Storage Service) is an object storage service offered by Amazon Web Services. For S3 buckets, if versioning is enabled, users can preserve, retrieve, and restore every version of the object stored in the bucket.

In this article, we will understand how to enable versioning for a bucket and retrieve all versions of an object from AWS web interface as well as Python boto library.

Versioning of Bucket

Bucket versioning can be changed with a toggle button from the AWS web console in the bucket properties.

We can do the same with Python boto3 library.

import boto3


bucket_name = 'avilpage'

s3 = boto3.resource('s3')
versioning = s3.BucketVersioning(bucket_name)

# check status
print(versioning.status)

# enable versioning
versioning.enable()

# disable versioning
versioning.suspend()

Retrieving Objects

Once versioning is enabled, we can store multiple versions of an object by uploading an object multiple times with the same key.

We can write a simple script to generate a text file with a random text and upload it to S3.

import random
import string

import boto3

file_name = 'test.txt'
key = file_name
s3 = boto3.client('s3')

with open(file_name, 'w') as fh:
    data = ''.join(random.choice(string.ascii_letters) for _ in range(10))
    fh.write(data)

s3.upload_file(key, bucket_name, file_name)

If this script is executed multiple times, the same file gets overridden with a different version id with the same key in the bucket.

We can see all the versions of the file from the bucket by selecting the file and then clicking drop-down at Latest version.

We can write a script to retrieve and show contents of all the versions of the test.txt file with the following script.

import boto3


bucket_name = 'avilpage'
s3_client = boto3.client('s3')

versions = s3_client.list_object_versions(Bucket=bucket_name)

for version in versions:
    version_id = versions['Versions'][0]['VersionId']
    file_key = versions['Versions'][0]['Key']

    response = s3.get_object(
        Bucket=bucket_name,
        Key=file_key,
        VersionId=version_id,
    )
    data = response['Body'].read()
    print(data)

Conclusion

Object versioning is useful to protect data from unintended overwrites. In this article, we learnt how to change bucket versioning, upload multiple versions of same file and retrieving all versions of the file using AWS web console as well as boto3.

Why My Grandma Can Recall 100+ Phone Numbers, But We Can't

Anand Reddy Pandikunta — Thu, 20 Jun 2019 15:51:21 GMT

On a leisure evening, as I was chit chatting with my grandma, my phone started ringing. Some one who is not in my contacts is calling me. As I was wondering who the heck is calling, my grandma just glanced at my screen and said, "Its you uncle Somu, pick up the phone". I was dumbstruck by this.

Later that evening, I asked my grandma to recall the phone numbers she remembers. She recalled 30+ phone numbers and she was able to recognize 100+ phone numbers based on the last 4 digits of the mobile.

That came as a surprise for me as I don't even remember 10+ phone numbers now. Most of the smart phone users don't remember family and friends phone numbers anymore.

A decade back, I used to remember most of my relatives and friends phone numbers even though I didn't had a phone. My grandma used to use a mini notebook to write all the phone numbers. I was worried about this mini notebook as it can get lost easily and it is always hard to find when required.

Since my grandma didn't have any contacts in her phone, she gets a glimpse of number every time someone calls her. She also dials the number every time she has to call someone. With this habit she is able to memorize all the numbers.

I, on the other hand started using a smart phone which has all the contacts. I search my contacts by name when I have to dial someone and there is no need to dial the number. Also whenever someone call me, their name gets displayed in large letters and I never get to focus on the number. Due to this, I don't remember any of the phone numbers

After this revelation, I started an experiment by disabling contact permissions for dialer app. With this, I am forced to type the number or select appropriate number from the call history and dial it. This was a bit uncomfortable at first. Soon I got used to it as recognized more and more numbers.

This might seem unnecessary in the smart phone age. But when you are traveling or when your phone gets switched off, it's hard to contact people. Even if someone gives their phone, it is of no use if I don't remember any numbers.

Also it is important to remember phone numbers of family and friends which might be needed in case of emergencies.

Switching Hosts With Bookmarklets - Web Development Tips

Anand Reddy Pandikunta — Wed, 08 May 2019 15:40:52 GMT

When debugging an issue related to web development projects, which is inconsistent between environments (local, development, QA, staging and production), we have to frequently switch between them.

If we are debugging something on the home page, then we can just bookmark the host URLs. We can switch between them by clicking on the relevant bookmark. Some browsers provide autocompletion for bookmarks. So we can type a few characters and then select the relevant URL from suggestions.

When debugging an issue on some other page like https://avilpage.com/validate/books/?name=darwin&year=2019, which has URL path and query param, switching between enviroment becomes tedious. To switch to local environment, we have to manually replace the hostname with localhost.

To avoid this, we can use a bookmarklet to switch the hosts. A bookmarklet is a bookmark which contains a JavaScript code snippet in its URL. This code snippet will be executed when the bookmarklet is clicked.

Lets create a bookmarklet to replace host in the URL with http://localhost:8000. Create new bookmark called To Local and in the URL add the following snippet.

javascript:(function() { window.location.replace("http://localhost:8000" + window.location.pathname + window.location.search); }())

If we click on To Local bookmarklet, it will redirect the current active page to localhost URL.

We can create one more bookmarklet to switch to production. Create a bookmarklet called To Production and add the following snippet in the URL.

javascript:(function() { window.location.replace("http://avilpage.com" + window.location.pathname + window.location.search); }())

We can create similar bookmarklets to switch to other environments. Now, switching between environments on any page is as easy as clicking a button.

A Short Guide To Debugging PostgreSQL Triggers

Anand Reddy Pandikunta — Tue, 30 Apr 2019 15:51:21 GMT

Introduction

PostgreSQL triggers will associate a function to a table for an event. If multiple triggers of the same kind are defined for the same event, they will be executed in alphabetical order by name.

In this article, we will see how to debug PostgreSQL triggers.

Triggers

First, ensure triggers are enabled on the required tables for INSERT/UPDATE/DELETE events. We can check available triggers by running the following query.

SELECT * FROM information_schema.triggers;

We can also use EXPLAIN to show triggers that are executed for an event by running relevant queries.

PostgreSQL Logging

After ensuring triggers are applied correctly, set the logging level for postgresql server and client in postgres.conf file.

# let the server log all queries
log_statement = 'all'

# set client message to log level
client_min_messages = log

Restart PostgreSQL to reflect configuration changes.

# Linux
sudo service postgres restart

# Mac
brew services restart postgres

Tail the logs and check if queries are executing correctly with appropriate values.

Triggers Logging

After enabling logging for PostgreSQL, we can raise messages/errors in triggers so that we can see if any unexpected things are happening at any point in the trigger.

RAISE 'Updating row with ID: %', id;
RAISE division_by_zero;
RAISE WARNING 'Unable to delete record';

This ensures triggers are executing as expected and if there are any warnings/errors, it will log a message.

SQL/PostgreSQL Gotchas

Even though queries and triggers are executing correctly, we might not see the desired result because of the potentially surprising behavior of PostgreSQL. There are some scenarios where PostgreSQL seems to be not working at first but it actually is the expected behavior.

Unquoted object names will be treated as lowercase. SELECT FOO FROM bar will become SELECT foo FROM bar.
Comparing nullable fields. This might yield strange results as NULL != NULL.
PostgreSQL uses POSIX offsets. For 04:21:42 UTC+01, +1 means the timezone is west of Greenwich.

Conclusion

By being aware of common PostgreSQL gotchas and enabling logging for PostgreSQL clients, servers & triggers, pinpointing the bug in triggers becomes easy. Once the bug is identified, appropriate action can be taken to fix the issue.

Essential PyCharm (Intellij) Plugins To Improve Productivity

Anand Reddy Pandikunta — Sun, 31 Mar 2019 15:51:21 GMT

As per 2019 JetBrains survery, PyCharm is the most widely used(36%) IDE for Python development. Even though PyCharm comes with a lot of built-in features, there are a lot of plugins available for PyCharm and other Intellij IDEs. In this article, we will see some plugins which will boost our productivity during development.

Ace Jump

To move caret to a particular position in the editor without mouse, AceJump plugin will be useful. It allows to quickly navigate the caret to any position in the editor.

GitToolBox

GitToolBox provides a lot of additional features like inblame support, status bar widget, commit dialog, branch name completion, etc.

Grep Console

When running a django/flask server or any Python script which generates a lot of output, it is hard to filter specific output on console. Grep Console can filter or highlight output based on specific conditions which makes it easier to debug the code.

Highlight Bracket Pair

Instead of manually scanning where a bracket starts/ends, Highlight Bracket Pair will automatically highlight the bracket pairs based on cursor position.

Key Promoter

If you are new to PyCharm or an experienced user who is using mouse instead of key board shortcuts, Key Promoter will show relevant key board shortcut when mouse is used inside IDE. This provides an easy way to learn keyboard shortcuts faster.

Rainbow Brackets

Highlight Bracket Pair will hightlight the bracket pair around the cursor. When there are multiple bracket pairs deeply nested, Rainbow Brackets will highlight matching bracket pairs with matching rainbow colors.

Save Actions

Instead of manually optimizing imports or reformatting code when changes are made, we can use Save Actions which will automatically run a set of actions on every file save.

String Manipulation

To convert lower case letters to upper case letter, String manipulation plugin will be useful. In addition to lower/upper case conversion, it also provides options to convert to cameCase, kebab-case, PascalCase etc.

These are some plugins which will boost developers productivity while writing and debugging code in PyCharm or other JetBrains IDE.

Django Tips & Tricks #12 - Automatically Set CSRF Token in Postman

Anand Reddy Pandikunta — Thu, 28 Feb 2019 15:51:21 GMT

Introduction

Django has inbuilt CSRF protection mechanism for requests via unsafe methods to prevent Cross Site Request Forgeries. When CSRF protection is enabled on AJAX POST methods, X-CSRFToken header should be sent in the request.

Postman is one of the widely used tool for testing APIs. In this article, we will see how to set csrf token and update it automatically in Postman.

CSRF Token In Postman

Django sets csrftoken cookie on login. After logging in, we can see the csrf token from cookies in the Postman.

We can grab this token and set it in headers manually.

But this token has to be manually changed when it expires. This process becomes tedious to do it on an expiration basis.

Instead, we can use Postman scripting feature to extract token from cookie and set it to an environment variable. In Test section of postman, add these lines.

var xsrfCookie = postman.getResponseCookie("csrftoken");
postman.setEnvironmentVariable('csrftoken', xsrfCookie.value);

This extracts csrf token and sets it to an environment variable called csrftoken in the current environment.

Now in our requests, we can use this variable to set the header.

When the token expires, we just need to login again and csrf token gets updated automatically.

Conclusion

In this article we have seen how to set and renew csrftoken automatically in Postman. We can follow similar techniques on other API clients like CURL or httpie to set csrf token.

How To Install Private Python Packages With Pip

Anand Reddy Pandikunta — Thu, 31 Jan 2019 06:42:12 GMT

Introduction

To distribute python code, we need to package it and host it somewhere, so that users can install and use it. If the code is public, it can be published to PyPi or any public repository, so that anyone can access it. If the code is private, we need to provide proper authentication mechanism before allowing users to access it.

In this article, we will see how to use pip to install Python packages hosted on GitLab, GitHub, Bitbucket or any other services.

Packaging

To package python project, we need to create setup.py file which is build script for setuptools. Below is a sample setup file to create a package named library.

import setuptools


with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="library",
    version="0.0.1",
    author="chillaranand",
    author_email="foo@avilpage.com",
    description="A simple python package",
    long_description=long_description,
    url="https://github.com/chillaranand/library",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
)

Python provides detailed packaging documentation on structuring and building the package.

Installation

Once module(s) is packaged and pushed to hosting service, it can be installed with pip.

# using https
$ pip install git+https://github.com/chillaranand/library.git

# using ssh
pip install git+ssh://git@github.com/chillaranand/library.git

This usually requires authentication with usersname/password or ssh key. This setup works for developement machines. To use it in CI/CD pipelines or as a dependency, we can use tokens to simplify installation.

$ export GITHUB_TOKEN=foobar

$ pip install git+https://$GITHUB_TOKEN@github.com/chillaranand/library.git

Conclusion

In this article, we have seen how to package python code and install private packages with pip. This makes it easy to manage dependencies or install packages on multiple machines.

Find High-impact Performance Bottlenecks in Django

Anand Reddy Pandikunta — Tue, 11 Dec 2018 02:38:08 GMT

Introduction

When optimizing performance of web application, a common mistake is to start with optimizing the slowest page(or API). In addition to considering response time, we should also consider the traffic it is receving to priorotize the order of optimization.

In this article we will profile a django webapp, find high-impact performance bottlenecks and then start optimization them to yield better performance.

Profiling

django-silk is an open source profiling tool which intercepts and stores HTTP requests data. Install it with pip.

pip install django-silk

Add silk to installed apps and include silk middleware in django settings.

MIDDLEWARE = [
    ...
    'silk.middleware.SilkyMiddleware',
    ...
]

INSTALLED_APPS = (
    ...
    'silk'
)

Run migrations so that Silk can create required database tables to store profile data.

$ python manage.py makemigrations
$ python manage.py migrate
$ python manage.py collectstatic

Include silk urls in root urlconf to view the profile data.

urlpatterns += [url(r'^silk/', include('silk.urls', namespace='silk'))]

On silk requests page(http://host/silk/requests/), we can see all requests and sort them by overall time or time spent in database.

High Impact Bottlenecks

Silk creates silk_request table which contains information about the requests processed by django.

$ pgcli

library> \d silk_request;

+--------------------+--------------------------+-------------+
| Column             | Type                     | Modifiers   |
|--------------------+--------------------------+-------------|
| id                 | character varying(36)    |  not null   |
| path               | character varying(190)   |  not null   |
| time_taken         | double precision         |  not null   |
...

We can group these requests data by path, calculate number of requests, average time taken and impact factor of each path. Since we are considering response time and traffic, impact factor will be product of average response time and number of requests for that path.

library> SELECT
     s.*, round((s.avg_time * s.count)/max(s.avg_time*s.count) over ()::NUMERIC,2) as impact
 FROM
     (select path, round(avg(time_taken)::numeric,2) as avg_time, count(path) as count from silk_request group by PATH)
     s
 ORDER BY impact DESC;

+-------------------------+------------+---------+----------+
| path                    | avg_time   | count   | impact   |
|-------------------------+------------+---------+----------|
| /point/book/book/       | 239.90     | 1400    | 1.00     |
| /point/book/data/       | 94.81      | 1900    | 0.54     |
| /point/                 | 152.49     | 900     | 0.41     |
| /point/login/           | 307.03     | 400     | 0.37     |
| /                       | 106.51     | 1000    | 0.32     |
| /point/auth/user/       | 494.11     | 200     | 0.29     |
...

We can see /point/book/book/ has highest impact even though it is neighter most visited nor slowest view. Optimizing this view first yields in overall better performance of webapp.

Conclusion

In this article, we learnt how to profile django webapp and identify bottlenecks to improve performance. In the next article, we wil learn how to optimize these bottlenecks by taking an in-depth look at them.

Archive Million Pages With wget In Minutes

Anand Reddy Pandikunta — Sun, 18 Nov 2018 11:51:21 GMT

Introduction

webrecorder, heritrix, nutch, scrapy, colly, frontera are popular tools for large scale web crawling and archiving.

These tools require some learning curve and some of them don't have inbuilt support for warc(Web ARChive) output format.

wget comes bundled with most *nix systems and has inbuilt support for warc output. In this article we will see how to quickly archive web pages with wget.

Archiving with wget

In previous article we have extracted a superset of top 1 million domains. We can use that list or urls to archive. Save this list to a file called urls.txt.

This can be archived with the following command.

file=urls.txt
wget -i $file --warc-file=$file -t 3 --timeout=4 -q -o /dev/null -O /dev/null

wget has the ability to continue partially downloaded files. But this option won't work with warc output. So, it is better to split this list into small chunks and process them. One added advantage of this approach is we can parallely download multiple chunks with wget.

mkdir -p chunks
split -l 1000 urls.txt chunks/ -d --additional-suffix=.txt -a 3

This will split the file into several chunks each containing 1000 urls. wget doesn't have multithreading support. We can write a for loop to schedule a seperate process for each chunk.

for file in `ls -r chunks/*.txt`
do
   wget -i $file --warc-file=$file -t 3 --timeout=4 -q -o /dev/null -O /dev/null &
done

To archive 1000 urls, it takes ~15 minutes. In less than 20 minutes, it will download entire million pages.

Also, each process takes ~8MB of memory. To run 1000 process, a system needs 8GB+ memory. Otherwise, number of parallel processes should be reduced which increases overall run time.

Each archive chunk will be ~150MB and consume lot of storage. All downloaded acrhives can be zipped to reduce storage.

gzip *.warc

Here is an idempotent shell script to download and archive files in batches.

#! /bin/sh

set -x

batch=1000
size=`expr ${#batch} - 1`
maxproc=50
file=urls.txt
dir=$HOME'/projects/chunks'$batch


mkdir -p $dir
split -l $batch $file $dir'/' -d --additional-suffix=.txt -a $size
sleep 1

useragent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'


for file in `ls -r $dir/*.txt`
do
    warcfile=$file'.warc'
    warczip=$warcfile'.gz'
    if [ -f $warczip ] || [ -f $warcfile ]; then
        continue
    fi

    if [ $(pgrep wget -c) -lt $maxproc ]; then
        echo $file
        wget -H "user-agent: $useragent" -i $file --warc-file=$file -t 3 --timeout=4 -q -o /dev/null -O /dev/null &
        sleep 2
    else
        sleep 300
        for filename in `find $dir -name '*.warc' -mmin +5`
        do
            gzip $filename -9
        done
    fi
done

Conclusion

In this article, we have seen how to archive million pages with wget in few minutes.

wget2 has multithreading support and it might have warc output soon. With that, archiving with wget becomes much easier.

Alexa vs Domcop vs Majestic - Top Million Sites

Anand Reddy Pandikunta — Fri, 02 Nov 2018 06:34:58 GMT

Introduction

Alexa¹, Domcop²(based on CommonCrawl³ data) Majestic⁴ & provide top 1 million popular websites based on their analytics. In this article we will download this data and compare them using Linux command line tools.

Collecting data

Let's download data from above sources and extract domain names. The data format is different for each source. We can use awk tool to extract domains column from the source. After extracting data, sort it and save it to a file.

Extracting domains from alexa.

# alexa

$ wget http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

$ unzip top-1m.csv.zip

# data sorted by ranking
$ head -n 5 top-1m.csv
1,google.com
2,youtube.com
3,facebook.com
4,baidu.com
5,wikipedia.org

$ awk -F "," '{print $2}' top-1m.csv | sort > alexa

# domains after sorting alphabetically
$ head -n 5 alexa
00000.life
00-000.pl
00004.tel
00008888.tumblr.com
0002rick.tumblr.com

Extracting domain names from domcop.

# Domcop

$ wget https://www.domcop.com/files/top/top10milliondomains.csv.zip

$ unzip top10milliondomains.csv.zip

# data sorted by ranking
$ head -n 5 top10milliondomains.csv
"Rank","Domain","Open Page Rank"
"1","fonts.googleapis.com","10.00"
"2","facebook.com","10.00"
"3","youtube.com","10.00"
"4","twitter.com","10.00"

$ awk -F "\"*,\"*" '{if(NR>1)print $2}' top10milliondomains.csv.zip | sort > domcop

# domains after sorting alphabetically
$ head -n 5 domcop
00000000b.com
000000book.com
0000180.fortunecity.com
000139418.wixsite.com
000fashions.blogspot.com

Extracting domain names from majestic.

# Majestic

$ wget http://downloads.majestic.com/majestic_million.csv

# data sorted by ranking
$ head -n 5 majestic_million.csv
GlobalRank,TldRank,Domain,TLD,RefSubNets,RefIPs,IDN_Domain,IDN_TLD,PrevGlobalRank,PrevTldRank,PrevRefSubNets,PrevRefIPs
1,1,google.com,com,474277,3016409,google.com,com,1,1,474577,3012875
2,2,facebook.com,com,462854,3093315,facebook.com,com,2,2,462860,3090006
3,3,youtube.com,com,422434,2504924,youtube.com,com,3,3,422377,2501555
4,4,twitter.com,com,412950,2497935,twitter.com,com,4,4,413220,2495261

$ awk -F "\"*,\"*" '{if(NR>1)print $2}' majestic_million.csv | sort > majestic

# domains after sorting alphabetically
$ head -n 5 majestic
00000.xn--p1ai
0000666.com
0000.jp
0000www.com
0000.xn--p1ai

Comparing Data

We have collected and extracted domains from above sources. Let's compare the domains to see how similar they are using comm tool.

$ comm -123 alexa domcop --total
871851  871851  128149  total

$ comm -123 alexa majestic --total
788454  788454  211546  total

$ comm -123 domcop majestic --total
784388  784388  215612  total

$ comm -12 alexa domcop | comm -123 - majestic --total
31314   903165  96835   total

So, only 96,835(9.6%) domains are common between all the datasets and the overlap between any two sources is ~20%. Here is a venn diagram showing the overlap between them.

Conclusion

We have collected data from alexa, domcorp & majestic, extracted domains from it and observed that there is only a small overlap between them.

Setup Continous Deployment For Python Chalice

Anand Reddy Pandikunta — Tue, 30 Oct 2018 15:51:21 GMT

Outline

Chalice is a microframework developed by Amazon for quickly creating and deploying serverless applications in Python.

In this article, we will see how to setup continous deployment with GitHub and AWS CodePipeline.

CD Setup

Chalice provides cli command deploy to deploy from local system.

Chalice also provides cli command generate-pipeline command to generate CloudFormation template. This template is useful to automatically generate several resources required for AWS pipeline.

This by default uses CodeCommit repository for hosting code. We can use GitHub repo as a source instead of CodeCommit.

Chalice by default provides a build file to package code and push it to S3. In the deploy step, it uses this artifact to deploy the code.

We can use a custom buildpsec file to directly deploy the code from build step.

version: 0.1

phases:
  install:
    commands:
      - echo Entering the install phase
      - echo Installing dependencies
      - sudo pip install --upgrade awscli
      - aws --version
      - sudo pip install chalice
      - sudo pip install -r requirements.txt

  build:
    commands:
      - echo entered the build phase
      - echo Build started on `date`
      - chalice deploy --stage staging

This buildspec file install requirements and deploys chalice app to staging. We can add one more build step to deploy it production after manual intervention.

Conclusion

We have seen how to setup continous deployment for chalice application with GitHub and AWS CodePipeline.

Parsing & Transforming mitmproxy Request Flows

Anand Reddy Pandikunta — Sat, 11 Aug 2018 15:51:21 GMT

mitmproxy is a free and open source interactive HTTPS proxy. It provides command-line interface, web interface and Python API for interaction and customizing it for our needs.

mitmproxy provides an option to export web request flows to curl/httpie/raw formats. From mitmproxy, we can press e(export) and then we can select format for exporting.

Exporting multiple requests with this interface becomes tedious. Instead we can save all requests to a file and write a python script to export them.

Start mitmproxy with this command so that all request flows are appended to requests.mitm file for later use.

$ mitmproxy -w +requests.mitm

Here is a python script to parse this dump file and print request URLs.

from mitmproxy.io import FlowReader


filename = 'requests.mitm'

with open(filename, 'rb') as fp:
    reader = FlowReader(fp)

    for flow in reader.stream():
        print(flow.request.url)

flow.request object has more attributes to provide information about the request.

In [31]: dir(flow.request)
Out[31]:
[...
 'host',
 'host_header',
 'http_version',
 'method',
 'multipart_form',
 'path',
 'raw_content',
 ...
 'wrap']

We can use the mitmproxy export utilities to transform mitm flows to other formats.

In [32]: flow = next(reader.stream())

In [33]: from mitmproxy.addons import export

In [34]: export.curl_command(flow)
Out[34]: "curl -H 'Host:mitm.it' -H 'Proxy-Connection:keep-alive' -H 'User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36' -H 'DNT:1' -H 'Accept:image/webp,image/apng,image/*,*/*;q=0.8' -H 'Referer:http://mitm.it/' -H 'Accept-Encoding:gzip, deflate' -H 'Accept-Language:en-US,en;q=0.9,ms;q=0.8,te;q=0.7' -H 'content-length:0' 'http://mitm.it/favicon.ico'"

In [35]: export.raw(flow)
Out[35]: b'GET /favicon.ico HTTP/1.1\r\nHost: mitm.it\r\nProxy-Connection: keep-alive\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36\r\nDNT: 1\r\nAccept: image/webp,image/apng,image/*,*/*;q=0.8\r\nReferer: http://mitm.it/\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-US,en;q=0.9,ms;q=0.8,te;q=0.7\r\n\r\n'

In [36]: export.httpie_command(flow)
Out[36]: "http GET http://mitm.it/favicon.ico 'Host:mitm.it' 'Proxy-Connection:keep-alive' 'User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36' 'DNT:1' 'Accept:image/webp,image/apng,image/*,*/*;q=0.8' 'Referer:http://mitm.it/' 'Accept-Encoding:gzip, deflate' 'Accept-Language:en-US,en;q=0.9,ms;q=0.8,te;q=0.7' 'content-length:0'"

With these utilities we can transform mitmproxy request flow to curl command or any other custom form to fit our needs.

Linux Performance Analysis In Less Than 10 Seconds

Anand Reddy Pandikunta — Tue, 24 Jul 2018 15:51:21 GMT

If you are using a Linux System or managing a Linux server, you might come across a situation where a process is taking too long to complete. In this article we will see how to track down such performance issues in Linux.

Netflix TechBlog has an article on how to anlyze Linux performance in 60 seconds. This article provides 10+ tools to use in order to see the resource usage and pinpoint the bottleneck.

It is strenuous to remember all those tools/options and laborious to run all those commands when working on multiple systems.

Instead, we can use atop, a tool for one stop solution for performance analysis. Here is a comparision of atop with other tools from LWN.

atop shows live & historical data measurement at system level as well as process level. To get the glimpse of system resource(CPU, memory, network, disk) usage install and run atop with

$ sudo apt install --yes atop

$ atop

By default, atop shows resources used in the last interval only and sorts them by CPU usage. We can use

$ atop -A -f 4

-A sorts the processes automatically in the order of the most busy system resource.

-f shows both active as well as inactive system resources in the ouput.

4 sets refresh interval to 4 seconds.

Just by looking at the output of atop, we get a glimpse of overall system resource usage as well as individual processes resource usage.

Django Tips & Tricks #10 - Log SQL Queries To Console

Anand Reddy Pandikunta — Thu, 14 Jun 2018 15:51:21 GMT

Django ORM makes easy to interact with database. To understand what is happening behing the scenes or to see SQL performance, we can log all the SQL queries that be being executed. In this article, we will see various ways to achieve this.

Using debug-toolbar

Django debug toolbar provides panels to show debug information about requests. It has SQL panel which shows all executed SQL queries and time taken for them.

When building REST APIs or micro services where django templating engine is not used, this method won't work. In these situations, we have to log SQL queries to console.

Using django-extensions

Django-extensions provides lot of utilities for productive development. For runserver_plus and shell_plus commands, it accepts and optional --print-sql argument, which prints all the SQL queries that are being executed.

./manage.py runserver_plus --print-sql
./manage.py shell_plus --print-sql

Whenever an SQL query gets executed, it prints the query and time taken for it in console.

In [42]: User.objects.filter(is_staff=True)
Out[42]: SELECT "auth_user"."id",
       "auth_user"."password",
       "auth_user"."last_login",
       "auth_user"."is_superuser",
       "auth_user"."username",
       "auth_user"."first_name",
       "auth_user"."last_name",
       "auth_user"."email",
       "auth_user"."is_staff",
       "auth_user"."is_active",
       "auth_user"."date_joined"
  FROM "auth_user"
 WHERE "auth_user"."is_staff" = true
 LIMIT 21


Execution time: 0.002107s [Database: default]

<QuerySet [<User: anand>, <User: chillar>]>

Using django-querycount

Django-querycount provides a middleware to show SQL query count and show duplicate queries on console.

|------|-----------|----------|----------|----------|------------|
| Type | Database  |   Reads  |  Writes  |  Totals  | Duplicates |
|------|-----------|----------|----------|----------|------------|
| RESP |  default  |    3     |    0     |    3     |     1      |
|------|-----------|----------|----------|----------|------------|
Total queries: 3 in 1.7738s


Repeated 1 times.
SELECT "django_session"."session_key",
"django_session"."session_data", "django_session"."expire_date" FROM
"django_session" WHERE ("django_session"."session_key" =
'dummy_key AND "django_session"."expire_date"
> '2018-05-31T09:38:56.369469+00:00'::timestamptz)

This package provides additional settings to customize output.

Django logging

Instead of using any 3rd party package, we can use django.db.backends logger to print all the SQL queries.

Add django.db.backends to loggers list and set log level and handlers.

    'loggers': {
        'django.db.backends': {
            'level': 'DEBUG',
            'handlers': ['console', ],
        },

In runserver console, we can see all SQL queries that are being executed.

(0.001) SELECT "django_admin_log"."id", "django_admin_log"."action_time", "django_admin_log"."user_id", "django_admin_log"."content_type_id", "django_admin_log"."object_id", "django_admin_log"."object_repr", "django_admin_log"."action_flag", "django_admin_log"."change_message", "auth_user"."id", "auth_user"."password", "auth_user"."last_login", "auth_user"."is_superuser", "auth_user"."username", "auth_user"."first_name", "auth_user"."last_name", "auth_user"."email", "auth_user"."is_staff", "auth_user"."is_active", "auth_user"."date_joined", "django_content_type"."id", "django_content_type"."app_label", "django_content_type"."model" FROM "django_admin_log" INNER JOIN "auth_user" ON ("django_admin_log"."user_id" = "auth_user"."id") LEFT OUTER JOIN "django_content_type" ON ("django_admin_log"."content_type_id" = "django_content_type"."id") WHERE "django_admin_log"."user_id" = 4 ORDER BY "django_admin_log"."action_time" DESC LIMIT 10; args=(4,)
[2018/06/03 15:06:59] HTTP GET /admin/ 200 [1.69, 127.0.0.1:47734]

These are few ways to log all SQL queries to console. We can also write a custom middleware for better logging of these queries and get some insights.

How To Deploy Django Channels To Production

Anand Reddy Pandikunta — Fri, 18 May 2018 15:51:21 GMT

In this article, we will see how to deploy django channels to production and how we can scale it to handle more load. We will be using nginx as proxy server, daphne as ASGI server, gunicorn as WSGI server and redis for channel back-end.

Daphne can serve HTTP requests as well as WebSocket requests. For stability and performance, we will use uwsgi/gunicorn to serve HTTP requests and daphne to serve websocket requests.

We will be using systemd to create and manage processes instead of depending on third party process managers like supervisor or circus. We will be using ansible for managing deployments. If you don't want to use ansible, you can just replace template variables in the following files with actual values.

Nginx Setup

Nginx will be routing requests to WSGI server and ASGI server based on URL. Here is nginx configuration for server.

server {
    listen {{ server_name }}:80;
    server_name {{ server_name }} www.{{ server_name }};

    return 301 https://avilpage.com$request_uri;
}


server {
    listen {{ server_name }}:443 ssl;
    server_name {{ server_name }} www.{{ server_name }};

    ssl_certificate     /root/certs/avilpage.com.chain.crt;
    ssl_certificate_key /root/certs/avilpage.com.key;

    access_log /var/log/nginx/avilpage.com.access.log;
    error_log /var/log/nginx/avilpage.com.error.log;

    location / {
            proxy_pass http://0.0.0.0:8000;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_redirect off;
    }

    location /ws/ {
            proxy_pass http://0.0.0.0:9000;
            proxy_http_version 1.1;

            proxy_read_timeout 86400;
            proxy_redirect     off;

            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $server_name;
    }

    location /static {
        alias {{ project_root }}/static;
    }

    location  /favicon.ico {
        alias {{ project_root }}//static/img/favicon.ico;
    }

    location  /robots.txt {
        alias {{ project_root }}/static/txt/robots.txt;
    }

}

WSGI Server Setup

We will use gunicorn for wsgi server. We can run gunicorn with

$ gunicorn avilpage.wsgi --bind 0.0.0.0:8000 --log-level error --log-file=- --settings avilpage.production_settings

We can create a systemd unit file to make it as a service.

[Unit]
Description=gunicorn
After=network.target


[Service]
PIDFile=/run/gunicorn/pid
User=root
Group=root
WorkingDirectory={{ project_root }}
Environment="DJANGO_SETTINGS_MODULE={{ project_name }}.production_settings"
ExecStart={{ venv_bin }}/gunicorn {{ project_name}}.wsgi --bind 0.0.0.0:8000 --log-level error --log-file=- --workers 5 --preload


ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
Restart=on-abort
PrivateTmp=true


[Install]
WantedBy=multi-user.target

Whenever server restarts, systemd will automatically start gunicorn service. We can also restart gunicorn manually with

$ sudo service gunicorn restart

ASGI Server Setup

We will use daphne for ASGI server and it can be started with

$ daphne avilpage.asgi:application --bind 0.0.0.0 --port 9000 --verbosity 1

We can create a systemd unit file like the previous one to create a service.

[Unit]
Description=daphne daemon
After=network.target


[Service]
PIDFile=/run/daphne/pid
User=root
Group=root
WorkingDirectory={{ project_root }}
Environment="DJANGO_SETTINGS_MODULE={{ project_name }}.production_settings"
ExecStart={{ venv_bin }}/daphne --bind 0.0.0.0 --port 9000 --verbosity 0 {{project_name}}.asgi:application
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
Restart=on-abort
PrivateTmp=true


[Install]
WantedBy=multi-user.target

Deployment

Here is an ansible playbook which is used to deploy these config files to our server. To run the playbook on server avilpage.com, execute

$ ansible-playbook -i avilpage.com, django_setup.yml

Scaling

Now that we have deployed channels to production, we can do performance test to see how our server performs under load.

For WebSockets, we can use Thor to run performance test.

thor -C 100 -A 1000 wss://avilpage.com/ws/books/

Our server is able to handle 100 requests per second with a latency of 800ms. This is good enough for low traffic website.

To improve performance, we can use unix sockets instead of rip/port for gunicorn and daphne. Also, daphne has support for multiprocessing using shared file descriptors. Unfortunately, it doesn't work as expected. As mentioned here, we can use systemd templates and spawn multiple daphne process.

An alternate way is to use uvicorn to start multiple workers. Install uvicorn using pip

$ pip install uvicorn

Start uvicorn ASGI server with

$ uvicorn avilpage.asgi --log-level critical --workers 4

This will spin up 4 workers which should be able to handle more load. If this performance is not sufficient, we have to setup a load balancer and spin up multiple servers(just like scaling any other web application).

Reliable Way To Test External APIs Without Mocking

Anand Reddy Pandikunta — Sun, 08 Apr 2018 15:51:21 GMT

Let us write a function which retrieves user information from GitHub API.

import requests


def get_github_user_info(username):
    url = f'https://api.github.com/users/{username}'
    response = requests.get(url)
    if response.ok:
        return response.json()
    else:
        return None

To test this function, we can write a test case to call the external API and check if it is returning valid data.

def test_get_github_user_info():
    username = 'ChillarAnand'
    info = get_github_user_info(username)
    assert info is not None
    assert username == info['login']

Even though this test case is reliable, this won't be efficient when we have many APIs to test as it sends unwanted requests to external API and makes tests slower due to I/O.

A widely used solution to avoid external API calls is mocking. Instead of getting the response from external API, use a mock object which returns similar data.

from unittest import mock


def test_get_github_user_info_with_mock():
    with mock.patch('requests.get') as mock_get:
        username = 'ChillarAnand'

        mock_get.return_value.ok = True
        json_response = {"login": username}
        mock_get.return_value.json.return_value = json_response

        info = get_github_user_info(username)

        assert info is not None
        assert username == info['login']

This solves above problems but creates additional problems.

Unreliable. Even though test cases pass, we are not sure if API is up and is returning a valid response.
Maintenance. We need to ensure mock responses are up to date with API.

To avoid this, we can cache the responses using requests-cache.

import requests_cache

requests_cache.install_cache('github_cache')


def test_get_github_user_info_without_mock():
    username = 'ChillarAnand'
    info = get_github_user_info(username)
    assert info is not None
    assert username == info['login']

When running tests from developer machine, it will call the API for the first time and uses the cached response for subsequent API calls. On CI pipeline, it will hit the external API as there won't be any cache.

When the response from external API changes, we need to invalidate the cache. Even if we miss cache invalidation, test cases will fail in CI pipeline before going into production.

Convert Browser Requests To Python Requests For Scraping

Anand Reddy Pandikunta — Mon, 26 Mar 2018 15:51:21 GMT

Scraping content behind a login page is bit difficult as there are wide variety of authentication mechanisms and web server needs correct headers, session, cookies to authenticate the request.

If we need a crawler which runs everyday to scrape content, then we have to implement authentication mechanism. If we need to quickly scrape content just for once, implementing authentication is an overhead.

Instead, we can manually login to the website, capture an authenticated request and use it for scraping other pages by changing url/form parameters.

From browser developer options, we can capture curl equivalent command for any request from Network tab with copy as cURL option.

Here is one such request.

curl 'http://avilpage.com/dummy' -H 'Cookie: ASPSESSIONIDSABAAQDA=FKOHHAGAFODIIGNNNDFKNGLM' -H 'Origin: http://avilpage.com' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.9,ms;q=0.8,te;q=0.7' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Cache-Control: max-age=0' -H 'Referer: http://avilpage.com/' -H 'Connection: keep-alive' -H 'DNT: 1' --data 'page=2&category=python' --compressed

Once we get curl command, we can directly convert it to python requests using uncurl.

$ pip install uncurl

Since the copied curl request is in clipboard, we can pipe it to uncurl.

$ clipit -c | uncurl

requests.post("http://avilpage.com/dummy",
    data='page=2&category=python',
    headers={
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
        "Accept-Encoding": "gzip, deflate",
        "Accept-Language": "en-US,en;q=0.9,ms;q=0.8,te;q=0.7",
        "Cache-Control": "max-age=0",
        "Content-Type": "application/x-www-form-urlencoded",
        "Origin": "http://avilpage.com",
        "Referer": "http://avilpage.com/",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"
    },
    cookies={
        "ASPSESSIONIDSABAAQDA": "FKOHHAGAFODIIGNNNDFKNGLM"
    },
)

If we have to use some other programming language, we can use curlconverter to convert curl command to Go or Node.js equivalent code.

Now, we can use this code to get contents of current page and then continue scraping from the urls in it.

Running Django Web Apps On Android Devices

Anand Reddy Pandikunta — Mon, 19 Feb 2018 12:10:56 GMT

When deploying a django webapp to Linux servers, Nginx/Apache as server, PostgreSQL/MySQL as database are preferred. For this tutorial, we will be using django development server with SQLite database.

First install SSHDroid app on Android. It will start ssh server on port 2222. If android phone is rooted, we can run ssh on port 22.

Now install QPython. This comes bundled with pip, which will install required python packages.

Instead of installing these two apps, we can use Termux, GNURoot Debian or some other app which provides Linux environment in Android. These apps will provide apt package manager, which can install python and openssh-server packages.

I have used django-bookmarks, a simple CRUD app to test this setup. We can use rsync or adb shell to copy django project to android.

rsync -razP django-bookmarks :$USER@$HOST:/data/local/

Now ssh into android, install django and start django server.

$ ssh -v $USER@$HOST

$ python -m pip install django
$ cd /data/local/django-bookmarks
$ python manage.py runvserver

This will start development server on port 8000. To share this webapp with others, we will expose it with serveo.

$ ssh -R 80:localhost:8000 serveo.net

Forwarding HTTP traffic from https://incepro.serveo.net
Press g to start a GUI session and ctrl-c to quit.

Now we can share our django app with anyone.

I have used Moto G4 Plus phone to run this app. I have done a quick load test with Apache Bench.

ab -k -c 50 -n 1000  \
-H "Accept-Encoding: gzip, deflate" \
http://incepro.serveo/list/

It is able to server 15+ requests concurrently with an average response time of 800ms.

We can write a simple shell script or ansible playbook to automate this deployment process and we can host a low traffic website on an android phone if required.

Load Testing Celery With Different Brokers

Anand Reddy Pandikunta — Fri, 09 Feb 2018 13:49:18 GMT

Celery is mainly used to offload work from request/response cycle in web applications and to build pipelines in data processing applications. Lets run a load test on celery to see how well it queues the tasks with various brokers.

Let us take a simple add task and measure queueing time.

import timeit

from celery import Celery

broker = 'memory://'


app = Celery(broker=broker)


@app.task
def add(x, y):
    return x + y


tasks = 1000
start_time = timeit.default_timer()
results = [add.delay(1, 2) for i in range(tasks)]
duration = timeit.default_timer() - start_time
rate = tasks//duration
print("{} tasks/sec".format(str(rate))

On development machine, with AMD A4-5000 CPU, queueing time is as follows

memory ---> 400 tasks/sec
rabbitmq ---> 300 tasks/sec
redis ---> 250 tasks/sec
postgres ---> 30 tasks/sec

On production machine, with Intel(R) Xeon(R) CPU E5-2676, queueing time is as follows

memory ---> 2000 tasks/sec
rabbitmq ---> 1400 tasks/sec
redis ---> 1200 tasks/sec
postgres ---> 200 tasks/sec

For low/medium traffic webistes and applications, 1000 tasks/second should be fine. For high traffic webistes, there will be multiple servers queueing up the tasks.

Incase if we need to queue the tasks at a higher rate and if we have task arguments before hand, we can chunk the tasks.

tasks = add.chunks(zip(range(1000), range(1000)), 10)

This will divide 1000 tasks into 10 groups of 100 tasks each. As there is no messaging overhead, it can queue any number of tasks in less than a second.

How To Plot Renko Charts With Python?

Anand Reddy Pandikunta — Wed, 10 Jan 2018 09:27:23 GMT

Renko charts are time independent and are efficient to trade as they eliminate noise. In this article we see how to plot renko charts of any instrument with OHLC data using Python.

To plot renko charts, we can choose a fixed price as brick value or calculate it based on ATR(Average True Range) of the instrument.

There are two types of Renko charts based on which bricks are calculated.

Renko chart - Price movement

First one is based on price movement. In this, we will divide the price movement of current duration by brick size to get the bricks.

Once bricks are obtained, we need to assign the brick colors based on the direction of price movement and then plot rectangles for each available brick.

import pandas as pd
from matplotlib.patches import Rectangle
import matplotlib.pyplot as plt


brick_size = 2


def plot_renko(data, brick_size):
    fig = plt.figure(1)
    fig.clf()
    axes = fig.gca()
    y_max = max(data)

    prev_num = 0

    bricks = []

    for delta in data:
        if delta > 0:
            bricks.extend([1]*delta)
        else:
            bricks.extend([-1]*abs(delta))

    for index, number in enumerate(bricks):
        if number == 1:
            facecolor='green'
        else:
            facecolor='red'

        prev_num += number

        renko = Rectangle(
            (index, prev_num * brick_size), 1, brick_size,
            facecolor=facecolor, alpha=0.5
        )
        axes.add_patch(renko)

    plt.show()


df = pd.read_csv(file)

df['cdiff'] = df['close'] - df['close'].shift(1)
df.dropna(inplace=True)
df['bricks'] = df.loc[:, ('cdiff', )] / brick_size

bricks = df[df['bricks'] != 0]['bricks'].values
plot_renko(bricks, brick_size)

Here is a sample renko chart plotted using the above code.

Renko chart - Period close

In this bricks are calculated based on the close price of the instrument. Calculation of bricks is sligtly complex compared to price movement chart. I have created a seperate package called stocktrends which has this calculation.

from stocktrends import Renko

renko = Renko(df)
renko.brick_size = 2
data = renko.get_ohlc_data()
print(data.tail())

This will give OHLC data for the renko chart. Now we can use this values to plot the charts as mentioned above.

Automatic Magnetometer Calibration With Arduino

Anand Reddy Pandikunta — Sat, 02 Dec 2017 21:21:21 GMT

If we take readings from a 3-axis magnetometers like HMC5883L, AK8963C (used in MPU9250) or LSM303DLHC and plot them, its response should be a sphere with ceter at origin.

In practice, due to the presence of hard and soft iron distortions, the response will be an ellipsiod with its center shifted away from origin. We need to calibrate the magnetometer to nullify the distortions.

First we need to get sample readings of magnetometer in various positions. Depending on the magnetometer, we need to connect it to arduino and take readings by rotating it in 8 shape.

Calibration

Hard iron biases shifts center away from origin. We can eliminate this error by calculating the offsets and shifting the readings.

int mx, my, mz;

int mx_min, my_min, mz_min;
int mx_max, my_max, mz_max;
int mx_offset, my_offset, mz_offset;

int mx_calibrated, my_calibrated, mz_calibrated;

// get min/max values by taking readings
// from magnetometer of your choice

mx_offset = (mx_min + mx_max)/2;
my_offset = (my_min + my_max)/2;
mz_offset = (mz_min + mz_max)/2;

mx_calibrated = mx - mx_offset;
my_calibrated = my - my_offset;
mz_calibrated = mz - mz_offset;

Soft iron biases makes the axial responses uneven which results in ellipsiod shape. An easier way to correct this is to rescale the axial readings to an average value.

int mx_scale, my_scale, mz_scale;

mx_scale = (mx_max - mx_min)/2;
my_scale = (my_max - my_min)/2;
mz_scale = (mz_max - mz_min)/2;

float avg_scale = (mx_scale + my_scale + mz_scale)/3;

mx_calibrated = avg_scale/(mx - mx_offset);
my_calibrated = avg_scale/(my - my_offset);
mz_calibrated = avg_scale/(mz - mz_offset);

We can caclulate these biases once and store them in our code so that we don't need to calibrate it everytime. We can also write an auto update function which will recalibrate offsets & scale for every new reading.

Django Tips & Tricks #9 - Auto Register Models In Admin

Anand Reddy Pandikunta — Fri, 24 Nov 2017 21:21:21 GMT

Inbuilt admin interface is one the most powerful & popular feature of Django. Once we create the models, we need to register them with admin, so that it can read metadata and populate interface for it.

If the django project has too many models or if it has a legacy database, then adding all those models to admin becomes a tedious task. To automate this process, we can programatically fetch all the models in the project and register them with admin.

from django.apps import apps


models = apps.get_models()

for model in models:
    admin.site.register(model)

This works well if we are just auto registering all the models. However if we try some customisations and try to register them in admin.py files in our apps, there will be conflicts as Django doesn't allow registering the same model twice.

So, we need to make sure this piece of code runs after all admin.py files are loaded and it should ignore models which are already registered. We can safely hook this code in appconfig.

from django.apps import apps, AppConfig
from django.contrib import admin


class CustomApp(AppConfig):
    name = 'foo'

    def ready(self):
        models = apps.get_models()
        for model in models:
            try:
                admin.site.register(model)
            except admin.sites.AlreadyRegistered:
                pass

Now all models will get registed automatically. If we go to a model page in admin, it will just show 1 column like this.

This is not informative for the users who want to see the data. We can create a ListAdminMixin, which will populate list_display with all the fields in the model. We can create a new admin class which will subclass ListAdminMixin & ModelAdmin. We can use this admin class when we are registering the model so that all the fields in the model will show up in the admin.

from django.apps import apps, AppConfig
from django.contrib import admin


class ListAdminMixin(object):
    def __init__(self, model, admin_site):
        self.list_display = [field.name for field in model._meta.fields if field.name != "id"]
        super(ListAdminMixin, self).__init__(model, admin_site)


class CustomApp(AppConfig):
    name = 'foo'

    def ready(self):
        models = apps.get_models()
        for model in models:
            admin_class = type('AdminClass', (ListAdminMixin, admin.ModelAdmin), {})
            try:
                admin.site.register(model, admin_class)
            except admin.sites.AlreadyRegistered:
                pass

Now whenever we create a new model or add a new field to an existing model, it will get reflected in the admin automatically.

How To Install Custom ROMs In Xiamo MiPad?

Anand Reddy Pandikunta — Sun, 19 Nov 2017 10:39:27 GMT

Mi Pad and other Xiamo devices has Mi UI OS which is a dual boot system. A major problem with this system is it has only ~600 MB of space in 1st partition. Because of this, we cannot install some custom ROMs as they need more space. In this article we will see how to merge both partitions to get more free space and install any custom ROM.

Install TWRP

Dowload latest recovery from twrp.me. Copy it to android device or push it to android using adb.

adb push -p twrp-3.1.1-0-mocha.img /sdcard/

Now put the device into fastboot mode by pressing Volume down & Power button simultanesouly when you switch it on. When in fastboot mode, flash the downloaded recovery file

sudo fastboot flash recovery twrp-3.1.1-0-mocha.img

Now we can go to recovery mode using adb.

adb reboot recovery

Merge partitions

Before installing custom ROM, we need to merge partitions so that we will have enough space to install ROM. If you are familiar with parted command, you can directly merge the partitions from terminal in TWRP recovery.

There is also a script which you can flash to do the partition. You can read this guide on mi forum for more information.

After partition is completed, from TWRP go to Wipe -> Advanced wipe -> Select System -> Click on Repair or change file system. Here it should show that free space in system is more than 1GB.

Install ROM

After partitions are merged, it is straight forward to install any custom ROM. Download a custom ROM like lineage or RR, push it to device and then install it from TWRP. After reboot, you will see the custom ROM booting.

Django Tips & Tricks #8 - Hyperlink Foreignkey Fields In Admin

Anand Reddy Pandikunta — Tue, 14 Nov 2017 21:21:21 GMT

Consider Book model which has Author as foreignkey.

from django.db import models


class Author(models.Model):
    name = models.CharField(max_length=100)

class Book(models.Model):
    title = models.CharField(max_length=100)
    author = models.ForeignKey(Author)

We can register these models with admin interface as follows.

from django.contrib import admin

from .models import Author, Book

class BookAdmin(admin.ModelAdmin):
    list_display = ('name', 'author', )

admin.site.register(Author)
admin.site.register(Book, BookAdmin)

Once they are registed, admin page shows Book model like this.

While browsing books, we can see name and author. Here, name field is liked to change view of book. But author field is shown as plain text. If we have to modify author name, we have to go back to authors admin page, search for relevant author and then change name.

This becomes tedious if we spend lot of time in admin for tasks like this. Instead, if author field is hyperlinked to its change view, we can directly go to that page.

Django provides an option to access admin views by its URL reversing system. For example, we can get change view of author model in book app using reverse("admin:book_author_change", args=id). Now we can use this url to hyperlink author field in book admin.

from django.contrib import admin
from django.utils.safestring import mark_safe


class BookAdmin(admin.ModelAdmin):
    list_display = ('name', 'author_link', )

    def author_link(self, book):
        url = reverse("admin:book_author_change", args=[book.author.id])
        link = '<a href="%s">%s</a>' % (url, book.author.name)
        return mark_safe(link)
    author_link.short_description = 'Author'

Now in the book admin view, author field will be hyperlinked to its change view and we can visit just by clicking it.

Depending on requirements, we can link any field in django to other fields or add custom fields to improve productivity.

Remove Clock From LockScreen/StatusBar On Android RR

Anand Reddy Pandikunta — Mon, 13 Nov 2017 20:24:38 GMT

Last year, I wrote a blog post on how to remove clock from lock screen and status bar if xposed is installed on your android device. You can also do this without xposed if you are using RR(Resurrection Remix) as it comes with a lot of inbuilt customization.

Remove Clock From LockScreen

To remove clock from lockscreen, go to Settings -> Configurations -> Lock screen -> Show lock screen lock.

Remove Time From StatusBar

To remove clock from status bar, go to Settings -> Configurations -> Status bar -> System UI tuner -> Time -> Dont show this icon.

Once you do this, you will have a clean lockscreen and statusbar without any date or time them.

Bluetooth Serial Communication Between Linux & Android

Anand Reddy Pandikunta — Tue, 03 Oct 2017 14:53:04 GMT

Most laptops and smart phones(Android/iPhone) have builtin Bluetooth modules. We can use this bluetooth module to communicate with other devices or with other bluetooth modules like HC-05 or HM-10.

In this article, we will learn how to send data between laptop and android bluetooth.

First, we need to pair with a bluetooth device to send information. From Ubuntu, we can pair to a Bluetooth device from Bluetooth settings. Alternatively, we can also use CLI to do the same.

$ bluetoothctl
[NEW] Controller 24:0A:64:D7:99:AC asus [default]
[NEW] Device 94:E9:79:BB:F8:3A DESKTOP-C4ECO3K
[NEW] Device 88:79:7E:7B:4C:87 athene
[NEW] Device 94:65:2D:8C:2E:10 OnePlus 5
[NEW] Device 98:0C:A5:61:D5:64 Lenovo VIBE K5 Plus
[NEW] Device AC:C3:3A:A0:CE:EF Galaxy J2
[NEW] Device 98:D3:35:71:02:B3 HC-05

[bluetooth]# power on
Changing power on succeeded

[bluetooth]# agent on
Agent registered

[bluetooth]# default-agent
Default agent request successful

[bluetooth]# scan on
Discovery started
[CHG] Controller 24:0A:64:D7:99:AC Discovering: yes
[CHG] Device 94:E9:79:BB:F8:3A RSSI: -88
[CHG] Device 88:79:7E:7B:4C:87 RSSI: -66

[bluetooth]# pair 88:79:7E:7B:4C:87
Attempting to pair with 88:79:7E:7B:4C:87
[CHG] Device 88:79:7E:7B:4C:87 Paired: yes
Pairing successful

To communicate with paired devices, we will use RFCOMM protocol. RFCOMM is just a serial port emulation and provides reliable data tranfer like TCP.

From ubuntu, lets open a port for communication.

$ sudo rfcomm listen /dev/rfcomm0 3

From Android, we have to connect to ubuntu. For this, we can use Roboremo app which supports RFCOMM.

$ sudo rfcomm listen /dev/rfcomm0 3
Waiting for connection on channel 3
Connection from 88:79:7E:7B:4C:87 to /dev/rfcomm0
Press CTRL-C for hangup

Once the connection is established, we can communicate between devices.

In Unix like systems, OS provides a device file as an interface for device driver. To send and read messages from Linux or Mac is as easy as reading and writing to a file.

# to send message to bluetooth
$ echo 'hello from ubuntu' > /dev/rfcomm0

We can see the received messages on Android

We can also send messages from android and read from ubuntu.

# to read messages from bluetooth
$ cat /dev/rfcomm0
hello from android

This way, we can communicate with any bluetooth module using a laptop or a smart phone.

Arduino Programming From Text Editor & Command Line

Anand Reddy Pandikunta — Thu, 24 Aug 2017 13:29:59 GMT

To program Arduino, we can use Arduino IDE which facilitates writing and uploading code to the board.

If we are using a text editor for programming, it will have lot of customisations which speed up development process. In such case, it is better to use same text editor for arduino programming too.

I use Emacs as IDE and there is arduino mode for emacs which provides syntax highlighting and some useful utilites to write arduino code. We can find such packages for other editors also.

Arduino also provides cli interface to upload code to arduino. To upload code, we need to specify port, board and the code to upload.

In Linux system, to upload a file called foo.ino, we can run

arduino --port /dev/ttyACM0 --board arduino:avr:mega
 \ --upload foo.ino

An alternate way is to use platformio, an opensource tool chain for IoT development.

It can be installed using pip.

pip install platformio

Once it is installed, code can be directly uploaded using ci command.

pio ci --board=megaatmega2560 --project-option="upload_port=/dev/ttyACM0" --project-option="targets=upload" foo.ino

By this we can use text editor to write code and arduino/platformio to upload code to arduino board.

Using LSTM-CTC For Complex Script Recognition

Anand Reddy Pandikunta — Sat, 22 Jul 2017 14:09:32 GMT

Most Indian languages have strong consonant-vowel structures which combine to give syllables. These syllables are written as one continuous ligature and they require complex text rendering (CTL) for type setting.

Writing OCR (Optical Character Recognition) software for CTL scripts is a challenging task as segmentation is hard. Because of this overall accuracy drops drastically.

A better approach is to use Connectionist Temporal Classification¹(CTC) which can identify unsegmented sequence directly as it has a one-to-one correspondence between input samples and output labels.

Here is a sample input and output of an RNN-CTC network which takes an unsegmented sequence and outputs labels.

Open-source OCR software ocorpy uses BLSTM-CTC for text recognition. Tesseract started using the same in its latest(4.0) version.

I have trained a model to recognize Telugu script using Ocropy and the accuracy is ~99% which is far better when compared to OCR software without CTC which are accurate to ~70%.

https://en.wikipedia.org/wiki/Connectionist_temporal_classification ↩

Auto Register Subclasses Without Metaclass in Python

Anand Reddy Pandikunta — Sat, 10 Jun 2017 15:30:27 GMT

In registry pattern, a registry maintains global association from keys to objects, so that objects can be reached from anywhere by simple identifier. This is useful for doing reverse lookups.

When building a registry, programmers have to explicitly register each object with registry. Manually building a registry is error prone and it is tedious if there are too many objects to register. It is better to auto register objects if possible.

A commonly used approach is to use inheritance as an organizing mechanism. Create a meta class which will auto register classes and then create base class with this meta class.

REGISTRY = {}


def register_class(target_class):
    REGISTRY[target_class.__name__] = target_class


class MetaRegistry(type):

    def __new__(meta, name, bases, class_dict):
        cls = type.__new__(meta, name, bases, class_dict)
        if name not in registry:
            register_class(cls)
        return cls


class BaseClass(metaclass=MetaRegistry):
    pass


class Foo(BaseClass):
    pass


class Bar(BaseClass):
    pass

Now whenever you subclass BaseClass, it gets registered in the global registry. In the above example, Foo, Bar gets registered automatically.

Eventhough it solves registration problem, it is hard to understand the code unless you know how metaclasses work.

A simple alternative for this is to use __subclasses__() to get subclasess and register them.

REGISTRY = {cls.__name__: cls for cls in BaseClass.__subclasses__()}

This will work only for direct subclasses and won't with indirect subclasses like this.

class Baz(Bar):
    pass

To solve this, we can use a function to recursively retrieve all subclasses of a class.

def subclasses(cls, registry=None):
    if registry is None:
        registry = set()

    subs = cls.__subclasses__()

    for sub in subs:
        if sub in registry:
            return
        registry.add(sub)
        yield sub
        for sub in subclasses(sub, registry):
            yield sub


REGISTRY = {cls.__name__: cls for cls in subclasses(BaseClass)}

PEP 487 provides __init_subclass__ hook in class body to customize class creation without the use of metaclass. We can our registration logic in this __init_subclass__ hook.

class BaseClass:
    def __init_subclass__(cls, **kwargs):
        if cls not in registry:
            register_class(cls)
        super().__init_subclass__(**kwargs)

print(registry)

This is available only in Python 3.6+. For older versions, we have to use the recursive function to get all subclasess. This code is easier to understand than metaclass example.

How To Auto Reload Celery Workers In Development?

Anand Reddy Pandikunta — Sun, 07 May 2017 04:22:47 GMT

We can pass --autoreload option when starting celery worker. This will restart worker when codebase changes.

celery worker -l info -A foo --autoreload

Unfortunately, it doesn't work as expected and it is deprecated.

During development, we will keep on changing the code base. Manually restarting celery worker everytime is a tedious process. It would be handy if workers can be auto reloaded whenever there is a change in the codebase.

Watchdog provides Python API and shell utilities to monitor file system events. We can install it with

pip install watchdog

Watchdog provides watchmedo a shell utilitiy to perform actions based on file events. It has auto-restart subcommand to start a long-running subprocess and restart it. So, celery workers can be auto restarted using this.

watchmedo auto-restart -- celery worker -l info -A foo

By default it will watch for all files in current directory. These can be changed by passing corresponding parameters.

watchmedo auto-restart -d . -p '*.py' -- celery worker -l info -A foo

If you are using django and don't want to depend on watchdog, there is a simple trick to achieve this. Django has autoreload utility which is used by runserver to restart WSGI server when code changes.

The same functionality can be used to reload celery workers. Create a seperate management command called celery. Write a function to kill existing worker and start new worker. Now hook this function to autoreload as follows.

import shlex
import subprocess

from django.core.management.base import BaseCommand
from django.utils import autoreload


def restart_celery():
    cmd = 'pkill -9 celery'
    subprocess.call(shlex.split(cmd))
    cmd = 'celery worker -l info -A foo'
    subprocess.call(shlex.split(cmd))


class Command(BaseCommand):

    def handle(self, *args, **options):
        print('Starting celery worker with autoreload...')
        autoreload.main(restart_celery)

Now you can run celery worker with python manage.py celery which will start a celery worker and autoreload it when codebase changes.

Django Tips & Tricks #7 - Django Auth Plugin For HTTPie

Anand Reddy Pandikunta — Thu, 20 Apr 2017 15:00:28 GMT

HTTPie is an alternative to curl for interacting with web services from CLI. It provides a simple and intuitive interface and it is handy to send arbitrary HTTP requests while testing/debugging APIs.

When working with web applications that require authentication, using httpie is difficult as authentication mechanism will be different for different applications. httpie has in built support for basic & digest authentication.

To authenticate with Django apps, a user needs to make a GET request to login page. Django sends login form with a CSRF token. User can submit this form with valid credentials and a session will be initiated.

Establish session manually is boring and it gets tedious when working with multiple apps in multiple environments(development, staging, production).

I have written a plugin called httpie-django-auth which automates django authentication. It can be installed with pip

pip install httpie-django-auth

By default, it uses /admin/login to login. If you need to use some other URL for logging, set HTTPIE_DJANGO_AUTH_URL environment variable.

export HTTPIE_DJANGO_AUTH_URL='/accounts/login/'

Now you can send authenticated requests to any URL as

http :8000/profile -A=django --auth='username:password'

Super Charge Your Shell For Python Development

Anand Reddy Pandikunta — Sun, 26 Mar 2017 02:40:31 GMT

Last month, I gave a lightning talk about supercharging your shell for python development at BangPypers meetup.

This is a detailed blog post on how to setup your laptop for the same.

Autojump

When working on terminal, cd is used to traverse directories.

cd ~/projects/python/django

cd is inefficient to quickly traverse directories which are in different paths and far away from each other.

cd /var/lib/elasticsearch/
cd ~/sandbox/channels/demo

z, a oh-my-zsh plugin is efficient for traversing directories. With z, directory can be changed by typing name of directory.

z junction

Instead of full name, just a substring would do.

z ju

z keeps a score of all visited directories and moves to most frecency(frequent+recent) directory that matches the substring.

To install z, install oh-my-zsh and add z to plugins in .zshrc file.

plugins=(git z)

Aliases

Read this old blog post on how aliases will improve your productivity.

Autoenv

When working on multiple projects, it becomes necessary to use virtualenvs so that multiple versions of same package can be used. In addition to that, it be necessary to set environment variables on a per project basis.

To automate all these things, autoenv provides directory based environments. Whenever user changes directory, it will help to automatically activate environment and set environment variables.

If you have file named .env in a directory, autoenv will automatically source that file whenever user enters into it.

autoenv is a python package. It can be installed with

pip install autoenv

It provides a shell script which needs to sourced.

echo "source `which activate.sh`" >> ~/.bashrc

You can create a .env file like this in project root.

source ~/.virtualenvs/exp/bin/activate
export SECRET_KEY=foobar

Next time, when your enter into that directory, autoenv finds .env file and it will source it automatically.

Autoreload

I have written a sepeate blog post on how to automagically reload imports long time back.

Autoimports

When you copy code and paste it in ipython interpreter, it might fail with ImportError if required modules aren't already imported by the interpreter.

Also when playing with code, having some predefined data would be handy. This avoids populating of data everytime shell starts.

You can write an init script which will do all these things and load it automatically when ipython starts.

Here is a simple init script which I use to auto import modules and data. This file can be auto loaded by specifying it in your config file.

c.InteractiveShellApp.exec_files = [os.path.join(directory, 'ipython_init.py')]

Autocall

When using python interpreter, to call a function, you have to type parenthesis.Typing parenthesis is not ergonomic as you have to move both hands far away from homerow.

IPython provides autocall option to make functions callable without typing parenthesis. This can be activate with %autocall magic.

In [6]: %autocall 1
Automatic calling is: Smart

Now functions can be called without parenthesis.

In [7]: range 5
------> range(5)
Out[7]: range(0, 5)

You can also enable this by default by activating it in ipython config file.

c.InteractiveShellApp.exec_lines = ['%autocall  1']

These are some tips to become more productive with your shell when working on python projects.

Provisioning Laptop(s) With Ansible

Anand Reddy Pandikunta — Sat, 11 Mar 2017 14:16:32 GMT

Setting up a new laptop manually takes a lot of time and there is a good chance of forgetting tweaks made to configuration files. It is good idea to automate it via a shell script or using configuration management tools like Ansible. It also makes easy to sync configuration across multiple systems.

Why Ansible?

Ansible is lightweight and provides only a thin layer of abstraction. It connects to hosts via ssh and pushes changes. So, there is no need to setup anything on remote hosts.

Writing A Playbook

You should check out Ansible documentation to get familiar with ansible and writing playbooks. Ansible uses yaml format for playbooks and it's human readable. Here is a simple playbook to install redis on ubuntu server.

hosts: all
sudo: True

tasks:
  - name: install redis
    apt: name=redis-server update_cache=yes

Here is a playbook which I use to configure my laptop. As the playbook needs to run locally, just run

ansible-playbook laptop-setup.yml -i localhost, -c local

Bootstrap Script

To automate provisioning, a bootstrap script is required to make sure python, ansible are installed, to download and execute playbook on the system.

sudo apt update --yes
sudo apt install --yes python python-pip

sudo apt install --yes libssl-dev
sudo -H pip install ansible

wget -c https://path/to/playbook.yml

sudo ansible-playbook setup.yml -i localhost, -c local

Now, to provision a laptop, just run the bootstrap script.

sh -c "$(wget https://path/to/bootstrap_script.sh"

You can use a git repo to track changes in playbook and bootstrap script. If you are using multiple laptops, running bootstrap script on them will make sure everything is synced across them.

How BMTC Is Exploiting Crores From Bangalore Citizens?

Anand Reddy Pandikunta — Sun, 05 Feb 2017 17:08:16 GMT

BMTC (Bengaluru Metropolitan Transport Corporation) is a government agency which operates the public transport bus service in Bangalore, India. It holds monopoly as it is the only choice for public transportation.

Fare Calculation

BMTC considers ~2KM distance as a stage and the fares for each stage are as follows.

Let's say, you want to travel from Indira Nagar to BTM Layout, it will cover 5 stages. So, BMTC charges 19₹ for that.

Exploitation?

If you travel from Indira Nagar KFC signal to Doopanahalli Arch(1.2KM), which comes under 1 stage, you have to pay 5₹.

On the other hand, if you travel from Indira Nagar KFC signal to Doopanahalli bustop(1.4KM), you have to pay 12₹.

How can BMTC charge 5₹ for the first 1.2KM and 7₹ for the subsequent 0.2KM? If BMTC charges, 5₹ for 1st stage. Then it should charge 5₹ + extra for next stage. But it shouldn't be more than 10₹.

You can just take 1 ticket(5₹) for 1st stage(1.2KM) and one more ticket(5₹) for the next stage(0.2KM). You can just travel 2 stages with 2 tickets for 10₹.

Turnsout BMTC is charging 3₹ extra on every ticket which covers atleast 2 stages. As lakhs of people travel in BMTC buses daily, in a month, this 3₹ turns into crores of rupees.

I am not sure when BMTC has started charging like this. A month back, I sent them an email asking for an explaination of unfair bus tickets and they haven't replied yet.

Update:

After sending this to BMTC officials, they have reduced 2nd stage bus fare by 2 Rs. Thanks to Gopala Kallapura, Krace Kumar & Thegesh GN for supporting the issue.

Refactoring Django With Full Syntax Tree

Anand Reddy Pandikunta — Sun, 29 Jan 2017 17:51:37 GMT

Django developers decided to drop Python 2 compatability in Django 2.0. There are serveral things that should be refactored/removed.

For example, in Python 2, programmers has to explicitly specify the class & instance when invoking super.

class Foo:
    def __init__(self):
        super(Foo, self).__init__()

In Python 3, super can be invoked without arguments and it will choose right class & instance automatically.

class Foo:
    def __init__(self):
        super().__init__()

For this refactoring, a simple sed search/replace should suffice. But, there are several hacks in codebase where super calls the grandparent instead of the parent. So, sed won't work in such cases. It is hard to refactor them manually and much harder for reviewers as there are 1364 super calls in code base.

→ grep -rI "super(" | wc -l
   1364

So changes has to be scripted. A simple python script to replace super calls by class names will fail to capture classes with on top of them, classes with decorators and nested classes.

To handle all these cases, this python script gets more complicated and there is no guarantee that it can handle all edge cases. So, a better choice is to use syntax trees.

Python has ast module to convert code to AST but it can't convert AST back to code. There are 3rd party packages like astor which can do this.

# this is a comment
def foo():

   print(
    "hello world"
)

Converting above code to AST and then converting back gives this

def foo():
    print('hello world')

Code to AST is a lossy transformation as they cannot preserve empty lines, comments and code formatting.

ast_to_code(code_to_ast(source_code)) != source_code

For lossless transformation, FST(Full Syntax Trees) is needed.

fst_to_code(code_to_fst(source_code)) == source_code

RedBaron package provides FST for given piece of code. With this, just locate super calls, find nearest class node, check class name with super and replace accordingly. With RedBaron, this refactoring can be done in less than 10 lines of code.

RedBaron has good documentation with relveant examples and its API is similar to BeautifulSoup. To write code that modifies code RedBaron seems to be an apt choice.

Thanks to Tim Graham & Aymeric Augustin for reviewing the patch.

Why Blog Posts Are Better Than Scientific Papers?

Anand Reddy Pandikunta — Sun, 08 Jan 2017 17:03:38 GMT

Titus Brown wrote a blog post on why blog posts are better than scientific papers. Here are few more reasons for that.

Fast

Publishing blog posts is lightning fast(how fast can you type?). You can just publish it with a click after writing. However with scientific papers you have to wait months to get your work published.

Mutability

Blog posts are easy to edit. If you are using a version controlled system like git, any one can easily collaborate. You can also leave comments on (most) blog posts and can have discussion with the author and/or others. Scientific papers once published are hard to change.

No Gatekeepers

To publish blog posts, you can setup your own blog or use services like medium. There are no gatekeepers who says that you can't publish a blog post because they thinks its not worth it.

Organic

Blog posts tend to be casual and organic. You can crack a joke and readers will enjoy it. Scientific papers are formal. You spray some sarcasm and your paper never gets published.

What Is The Meaning Of Life?

Anand Reddy Pandikunta — Sun, 25 Dec 2016 18:29:23 GMT

The meaning of life is

the condition that distinguishes animals and plants from inorganic matter, including the capacity for growth, reproduction, functional activity, and continual change preceding death.

Source: Oxford Dictionary

Detect & Correct Skew In Images Using Python

Anand Reddy Pandikunta — Sat, 12 Nov 2016 12:11:17 GMT

When scanning a document, a slight skew gets into the scanned image. If you are using the scanned image to extract information from it, detecting and correcting skew is crucial.

There are several techniques that are used to skew correction.

Projection profile method
Hough transform
Topline method
Scanline method

However, projection profile method is the simplest and easiest way to determine skew in documents in the range ±5°. Lets take a part of scanned image and see how to correct skew.

In this method, we will convert image to black (absence of pixel) & white (presence of pixel). Now image is projected vertically to get a histogram of pixels. Now image is rotated at various angles and above process is repeated. Wherver we find maximum diffrence between peaks, that will be the best angle.

import sys

import matplotlib.pyplot as plt
import numpy as np
from PIL import Image as im
from scipy.ndimage import interpolation as inter

input_file = sys.argv[1]

img = im.open(input_file)

# convert to binary
wd, ht = img.size
pix = np.array(img.convert('1').getdata(), np.uint8)
bin_img = 1 - (pix.reshape((ht, wd)) / 255.0)
plt.imshow(bin_img, cmap='gray')
plt.savefig('binary.png')


def find_score(arr, angle):
    data = inter.rotate(arr, angle, reshape=False, order=0)
    hist = np.sum(data, axis=1)
    score = np.sum((hist[1:] - hist[:-1]) ** 2)
    return hist, score


delta = 1
limit = 5
angles = np.arange(-limit, limit+delta, delta)
scores = []
for angle in angles:
    hist, score = find_score(bin_img, angle)
    scores.append(score)

best_score = max(scores)
best_angle = angles[scores.index(best_score)]
print('Best angle: {}'.formate(best_angle))

# correct skew
data = inter.rotate(bin_img, best_angle, reshape=False, order=0)
img = im.fromarray((255 * data).astype("uint8")).convert("RGB")
img.save('skew_corrected.png')

Results:

Original Image

Black & white image

Histogram of image

Scores at various angles

Histogram at 2° (best angle)

Skew corrected image

Here we have done only one iteration to find best angle. To get better accuracy, we can search over at (2 ± 0.5)°. This process can be repeated until we find a suitable level of accuracy.

Why I Use CyanogenMod?

Anand Reddy Pandikunta — Mon, 31 Oct 2016 14:05:09 GMT

When listening to music, to switch tracks I want to use volume buttons. For this I have to use Pocket Skip Track app. CyanogenMod has inbuilt support for that.

To interact with mobile from computer over wifi, wifi adb is required. CyanogenMod provides an option to enable this in Developer Options.

To put mobile in recovery mode, Gravity Box is required. This should be used only after installing Xposed framework. CyanogenMod has Advanced reboot which provides option to put phone in recovery mode.

Battery icon in status bar is very small. Having a text percentage indicator will be more useful. For this battery percentage is required. CyanogenMod has inbuilt circle/text indicators.

Auto brighness stopped working after I have rooted Moto G4 plus. I had to use twilight for auto brighness adjustment. With CyanogenMod, it is working fine. It also has a night mode option.

With stock android, auto rotate won't work in all directions. Set orientation app is required for that. CyanogenMod has inbuilt support for that.

To adapt mobile display to the time of the day, flux app is required on stock android. CyanogenMod has inbuilt option for this.

To get superuser permissions, I have to root stock android. CyanogenMod comes with root permissions. To run adb as root on stock android, adbd insecure app is required. CyanogenMod works without it.

Stock android also doesn't provide any options to customize status bar. CyanogenMod has several options to customize it.

There are several other things where CyanogenMod excels when compared to stock android like themes, privacy guard e.t.c. However, CyanogenMod comes with a lot of inbuilt features(batteries included) and that is the main reason for me to use it.

Remove Clock From LockScreen/StatusBar On Android

Anand Reddy Pandikunta — Thu, 15 Sep 2016 13:24:38 GMT

I have been living without time from a few years and it is a bliss. I have removed time from my laptop. For android mobile, I couldn't find a reliable way to remove time and I used to change timezone to a random zone. With Xposed framework we can remove clock from lock screen and status bar.

Before proceeding, make sure your phone is rooted and Xposed framework is installed.

Remove Clock From LockScreen

To remove clock from lockscreen, download Lock screen widgets module and activate it.

If you open it, it has an option to hide date and time.

Remove Time From StatusBar

To remove clock from status bar, you have to install GravityBox. Open GravityBox and goto statusbar settings

If you are already using CyanogenMod, there is an option to hide time in settings.

Once you do this, you will have a neat lockscreen without any time on it.

Detecting USB Events In Ubuntu With Python

Anand Reddy Pandikunta — Fri, 02 Sep 2016 13:24:38 GMT

udev is responsible for managing devices on Linux. It provides udevadm, a CLI utility to monitor and control devices.

If we run udevadm monitor on terminal and connect a USB drive or hard disk or a mobile via USB to your computer, it will show that kernel has detected the device and it will send a signal to udev about it.

→ udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[4336.899091] add      /devices/pci0000:00/0000:00:14.0/usb1/1-2 (usb)
KERNEL[4336.899774] add      /devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0 (usb)
KERNEL[4336.902553] add      /devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.1 (usb)
UDEV  [4336.911201] add      /devices/pci0000:00/0000:00:14.0/usb1/1-2 (usb)
UDEV  [4336.936453] add      /devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.1 (usb)
UDEV  [4337.947174] add      /devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0 (usb)

Pyudev, a third party package provides python bindings for udev. It can be installed with pip install pyudev.

With this we can create a simple monitor to detect USB events.

import pyudev

context = pyudev.Context()
monitor = pyudev.Monitor.from_netlink(context)
monitor.filter_by(subsystem='usb')

for device in iter(monitor.poll, None):
    if device.action == 'add':
        print('{} connected'.format(device))
        # do something

Save this to a file say monitor.py and run it with python monitor.py and connect a mobile or pendrive and it will show something like this.

Device('/sys/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6:1.0') connected
Device('/sys/devices/pci0000:00/0000:00:14.0/usb1/1-1/1-1:1.0') connected

Here we are just detecting add event. Similary we can detect other events like delete.

This is useful for automatically running shell scripts once the device gets plugged or unplugged.

Migrate From Blogger To Static Site (Nikola)

Anand Reddy Pandikunta — Sun, 07 Aug 2016 11:32:15 GMT

Long back I have started this blog on blogger platform. Now I choose to migrate to a static because of limitations of blogger.

Blogger has a simple editor in browser to write posts in html. With static site, you can write post in your editor in markdown and keep them in version control system like git.

Jekyll, Pelican, Lektor were my initial choices for static site generators. Unfortunately, they didn't had any useful plugins for smooth migration of data from blogger. Then I stumbled across import_blogger plugin of Nikola and it is able to migrate text content, images without any issues. So, I decided to use Nikola.

Nikola has great documentation and setup was simpler. I used import_blogger plugin and imported all the data. All the posts and pages were working well with exact same urls.

Blogger tags all posts of particular label under /search/label/foo. I have used these urls in some posts. Nikola shows posts under /categories/foo.html.

This is where nikola REDIRECTIONS come in handy. Using this, a list of from & to urls can be mapped. I wrote a simple function for redirections and then enabled STRIP_INDEXES which removes trailing .html.

labels = ['python', 'emacs', 'zen', 'django', 'chrome']
LABEL_REDIRECTIONS = []

for label in labels:
    LABEL_REDIRECTIONS.append((
        '/search/label/{}/index.html'.format(label),
        '/categories/{}'.format(label),
    ))

Since I stripped .html from urls, old urls broke. So I generated a list of old urls and wrote another function to redirect them to new urls.

Nikola has few builtin themes. I wanted something minimalistic and so I rolled out custom theme which you are seeing right now.

Thanks to Puneeth Chaganti, Chris Warrick, Roberto Alsina for helping me with migration.

emblem-mode for Emacs

Anand Reddy Pandikunta — Mon, 25 Jul 2016 17:32:35 GMT

Emblem.js is a ember friendly templating engine used as an alternative for handlebars.js.

Emblem.js doesn't have any official plugins for emacs. However they recommend to use slim plugins as they are similar.

slim-mode is available on melpa. To install it, run

M-x install-package slim-mode

After installing it, activate slim-mode by running M-x slim-mode.

Instead of activating manually, auto-mode-alist can be used to set major modes that needs to activated for files with specific extension. To activate slim-mode for emblem files which will be ending with .em or .emblem, use

(add-to-list 'auto-mode-alist '("\\.\\(em\\|emblem\\)" . slim-mode))

By adding this line to emacs configuration file, slim-mode gets activated for emblem.js files.

Auto Completion For Custom Search Engines In Chrome

Anand Reddy Pandikunta — Sat, 11 Jun 2016 12:07:00 GMT

If we have to search for a video on YouTube or a product on Amazon, we will open up that site first and then we will search for it.

To avoid this, Chrome allows us to search using custom search engines without visiting the site first. In chrome search settings, we can assign a single letter to a custom search engine.

Now if we go to browser, type y and hit Tab, it prompts for YouTube search instead of default search.

This is handy. But the major problem with this approach is there won't be autocompletion in chrome address bar.

This is where vimium comes into picture. Vimium provides auto completion for custom search engines.

You can install it from chrome webstore. Once it is installed, go to Vimium options and add youtube search to custom search engines.


y: http://www.youtube.com/results?search_query=%s Youtube

Also add a custom key map for this search.


map y Vomnibar.activateInNewTab keyword=y

If we just press y, it open youtube custom search with autocompletion.

Now in every page, we have autocompletion for custom search engines.

Concurrent Downloads - Bash Vs Python

Anand Reddy Pandikunta — Sun, 29 May 2016 11:55:00 GMT

I just found one more free telugu book

Graded
      readings in modern literary Telugu

by Golla Narayanaswami Reddy and Dan M Matson in Digital South Asia Library.

Unfortunately they didn't provide it as an ebook but as a set of 221 tif images.

I wrote a simple for loop in shell which downloaded all images one by one using wget.

      $ base_url="http://dsal.uchicago.edu"
$ url="$base_url/digbooks/images/PL4775.R4_1967/PL4775.R4_1967_%03g.gif"
$ time -p sh -c 'for i in $(seq -f $url 1 221); do; wget $i; done;'

I took 375 seconds for that. This was too slow. So I tried to download them parallelly using xargs.

      $ time echo $(seq -f $url 1 221) | xargs -n 1 -P 36 wget

My laptop has a quad core processor. So I tried with 20, 24, 28, 32 process at a time.

With wget+xargs, the best timing is 13 seconds (CPU: 15%, Process: 28).

Again I tried downloading them parallelly but with GNU parallel.

      $ time seq -f $url 1 221 | parallel -j36 wget {}

With wget+parallel, the best timing is 12 seconds (CPU: 48%, Process: 24).

Here is cpu consumption and time taken at each step.

Once I have done with bash, I decided to try the same things with Python and see how it goes.

I wrote a simple script using requests to download images.

                    import shutil
import sys
from concurrent import futures

import requests


def download_image(url):
    r = requests.get(url)
    file_name = url.split('/')[-1]
    with open(file_name, 'wb') as fh:
        fh.write(r.content)


base_url = 'http://dsal.uchicago.edu'
book_url = base_url + '/digbooks/images/PL4775.R4_1967/PL4775.R4_1967_{}.gif'
urls = [book_url.format(str(i).zfill(3)) for i in range(1, 221)]

def download_serially():
    for url in urls:
        download_image(url)

download_serially()

This took 244 seconds.

To download images parallelly, I have used Threadpoolexecutor from concurrent module.

                    def download_parallely():
    workers = int(sys.argv[1])

    with futures.ThreadPoolExecutor(max_workers=workers) as executor:
        result = executor.map(download_image, urls)

download_parallely()

I used previous script but just added one more function which queues tasks. Then I have executed the script with several options.

      $ time python down.py 28

Threadpoolexecutor documentation uses 5 times number of processors as max_workers by default. I tried same options which I have used for bash. Here is the overall comparision.

With requests+ThreadPoolExecutor, the best timing is 12 seconds (CPU: 36%, Process: 28).

Here is the overall comparision.

For a simple concurrent download, xargs+wget seems to be the best option.

Emacs - Browsing Projects With Etags

Anand Reddy Pandikunta — Sun, 17 Apr 2016 10:41:00 GMT

A tag is a reference to a sub unit in a program. A tag references syntactic elements like functions, classes e.t.c. A tags table consists of list of tag names and their positions in corresponding files.

Creating Tags Table

A tags table can be generated by etags program. If you are using Ubuntu machine, you can install it with

sudo apt-get install exuberant-ctags

Now, you can generate tags for a file with

etags input_file

Depending on the project, we can also generate tags for a particular type. For example, we can generate tags for all python files using

find . -name '*.py' -exec etags -a {} \;

Browsing Tags

This tag tables enables you to jump to definition of any tag. You can place cursor on a tag and pressM-.(find-tag) which takes you to definition of that tag. If you want to go back to previous location, you can use M-*(pop-tag -mark). There are several functions which help to navigate tags.

Tips

ctags-update is a third party package which automatically updates TAGS file when new files are created.

In addition to them, helm provides a function called helm-semantic which shows all available tags in the current file. This works without TAGS file. Helm also has helm-etags-select which provides helm interface to select tags.

If you are working with a python projects, you can use elpy which navigates tags (using Jedi in the backend) without any TAGS file.

Django Tips & Tricks #6 - Make Deleting Easy In Admin

Anand Reddy Pandikunta — Mon, 07 Mar 2016 12:08:00 GMT

Lets say we have a model as shown in some project.

from django.db import models

class Resource(models.Model):
    """
    Model to hold resources for books.
    """
    book = models.ForeignKey('Book')
    book_type = models.IntegerField()
    url = models.URLField()

This model can be registered in admin as follows.

from django.contrib import admin

from . import models

class ResourceAdmin(admin.ModelAdmin):
    list_display = ('book', 'book_type', 'url')


admin.site.register(models.Resource, ResourceAdmin)

This will generate a nice admin interface to modify the data.

Now if we want to delete a record, we need to follow these steps.

First, we have to select that record
Next, we have to click on the action dropdown
Next, we have to select delete action
Next, we have to click Go button.
In the next page we have to confirm that we have to delete.

Just to delete a single record, we have to click 5 times. That's too many clicks for a single action.

To simplify the process, we can have simple delete button for the objects as shown. This can be achieved by writing a function which will insert delete button for every record.

from django.contrib import admin

from . import models


class ResourceAdmin(admin.ModelAdmin):
    def delete(self, obj):
        return '<input type="button" value="Delete" onclick="location.href=\'%s/delete/\'" />'.format(obj.pk)

    delete.allow_tags = True
    delete.short_description = 'Delete object'

    list_display = ('book',  'book_type', 'url', 'delete')


admin.site.register(models.Book)

Now we have an admin with delete button for the records.

To delete an object, just click on delete button and then confirm to delete it. Now, we are deleting objects with just 2 clicks.

Read more django tips.
Read more posts about django.

Auto Convert & Upload Books To Kindle

Anand Reddy Pandikunta — Sun, 28 Feb 2016 10:32:00 GMT

I have ebooks in various formats like pdf, epub, mobi e.t.c. If I want to read books in laptop, I could use ebook reader like Calibre which can open supports most of the formats.

If I have to read them on Kindle paperwhite, it becomes a problem as it supports only few formats. So whenever I get an ebook, I have to convert it to mobi format and send it to kindle.

I can connect kindle to laptop via USB cable and I can copy books. I can also send books to kindle via email.

I found an IFTTT recipe, to automate all this process. I just need to create a folder to store all the books in dropbox. Then use IFTTT recipe to link dropbox folder which has books and gmail to send books to kindle email.

Now when I move required books into dropbox folder in my laptop, they will automatically appear in my kinde.

Automate Boring Stuff - Accepting Waitlist RSVPs

Anand Reddy Pandikunta — Mon, 01 Feb 2016 11:26:00 GMT

Meetup.com makes it easier to organize events. Before meetup you can specify how many people you can accomodate. Then you send mail to all people so that interested people can RSVP.

If RSVP is full, people can join waitlist. So if somebody changes RSVP to No, we can accomodate people in waitlist.

On the otherhand, if you find a place where you can accomodate more number than you have planned, you can accept all people in waitlist.

Unfortunately, there is no option in meetup for that. You have to accept people one by one which is a boring job if there are ~50 people in waitlist.

This where Python comes to rescue. Python has an excellent package called pyautogui which helps you to automate all boring tasks.

Here is the simple script I have used for this task.

import time

import pyautogui


print('Goto meetup event page and place cursor...')
time.sleep(5)

x, y = pyautogui.position()

for i in range(100):
    pyautogui.click(x, y)
    time.sleep(5)

The code is self explanatory. First we are importing pyautogui and then wait for 5 seconds so that we can go to meetup page and point mouse correctly. Get the cursor position and start clicking on that position with a gap of 5 seconds. This gap is needed as meetup shows a notification once you accept RSVP.

Pyautogui is an interesting package which makes your boring job easier.

Update:

Just found that increasing Attendee limit auto changes RSVP of waitlist people to Yes.

Dropbox As Wormhole Between Ubuntu & Google Drive!

Anand Reddy Pandikunta — Mon, 21 Dec 2015 11:13:00 GMT

I use Ubuntu on multiple computers everyday. It is important to keep various files synced across them. I keep most of the public data in github repositories and it is very easy to sync across multiple machines.

Google Drive gives 15GB of space with free account. I have multiple accounts and that space is enough to backup my data. I have been storing lots of files in Google Drive from past few years. The main problem is that Google Drive doesn't have a Linux client. To add files, I need to open a web page and then select files to upload and wait until its done. There are few 3rd party clients and none of them work well. So, it is very difficult to sync them across multiple computers.

Dropbox gives 5GB of space with free account. It has Linux client and it works very well. But 5GB of space is not sufficient to store my data.

It is quite difficult to pick one among them as I need benefits of both for free.

Even though I have lot of data, I dont want all of it to sync to all computers everyday. Nearly 90% of data is just backup. Only 10% of data needs to be synced between multiple computers. So I decided to use dropbox as a frontend to google drive.

I have created 2 folders, say Daily & Backup in Dropbox.

Whatever I add to Daily folder will be in Dropbox and it will sync across multiple machines.

There is an amazing service called IFTTT which will help you to connect two products/apps. I found a IFTTT recipe which will add dropbox files in one folder to Google drive.

I have connected Backup folder in Dropbox to Google Drive. Whenever I add a file to Backup folder in Dropbox, it gets added to Google Drive. Once in a week, I will delete files in Backup directory so that Dropbox won't run out of space. So Backup folder will serve as a wormhole to transfer files into Google Drive from my system.

Since I have multiple Google accounts, I have created multiple folders in Dropbox and all folders will be synced to corresponding Google Drive folders.

This setup works well for me as I have Dropbox's Linux client and sufficient storage to backup data to Google Drive.

Book Review - A Psycho Path By Haris Ibrahim K. V.

Anand Reddy Pandikunta — Tue, 17 Nov 2015 10:50:00 GMT

I stumbled on haris blog and I am thrilled to find that he wrote a book called A Psycho Path. Since ebook was free, I downloaded it and started reading.

I skipped Foreward, Preface and skimmed through the first story Etsuko’s picnic.

The second story in the book is A surprising blood test. The story is quite impressive and it hooked me in. Before that story I was lying on the bed and reading. After reading it I got up, sat properly and started reading. Among all the stories, I liked it the most.

Sound relations is another article which I found very fascinating. I read the article many times... 10 times? 20 times? may be more. Long before reading the book, me & my friends had a similar discussion while discussing about impact of music. This article gave an interesting conclusion to our discussion.

Narration of Quito San is interesting.

The book has ~20 stories/articles. Most stories are short stories(2-4 pages). The Prophecy of Amelyah and Quito San are medium sized stories(~20 pages).

Overall the book is very good.

I got a print book after reading ebook. I would recommend getting a get print book from Pothi or Amazon. You can also download free ebook here.

PyCon India 2015 - First PyCon Experience!

Anand Reddy Pandikunta — Mon, 05 Oct 2015 14:03:00 GMT

PyCon India 2015 was my first PyCon as an attendee, speaker & volunteer.

When I first heard about PyCon in BangPypers meetup, I thought PyCon India is just like a big meetup with a little bit of planning & preparation. But by the end of conference, I have realised that organising PyCon is a very very huge task. It needs a great deal of preparation, planning up things and dedicated volunteers who are interested and patient enough to spend lots of time to run it.

Nicholas Tollervey was keynote speaker. He came to venue on October 1st(two days before conference) and helped volunteers with some tasks. He did the same on next day too. I was wondering whether he came for key note speaking or volunteering.

I have been talking to many people on mailing lists, github and other channels. Finally I was able to meet & talk with many of them in person.

I was not able to attend all sessions but many people said this

It was good to see some of 2nd & 3rd year B.Tech students, who were familiar with django & were contributing to many upstream packages. Some students were even using Emacs as their editor.

Toddlers were also interested to attend PyCon

I have planned to make Python logo mosaic with rubik cubes. I thought it will take ~10 hours to complete it. When I got there with cubes, Vignesh U, Apoorva, Ram Mohan, Sayan Chowday and many other volunteers came forward and completed it in an hour.

I was eagerly waiting for PyCon India from past few months. PyCon India was just like a cool breeze! Ah...!!

Update:

How the hell did I forget about swag kit marathon?

Day before conference, we had to prepare swag kits for participants. I don't know whose idea it was to prepare it like this, that but it was brilliant and most fun part.

Set Emacs As Default File Manager In Ubuntu!

Anand Reddy Pandikunta — Tue, 08 Sep 2015 12:42:00 GMT

In this post, lets see how to do this

I am using Ubuntu and it has nautilus as default file manager. You might need to use some other commands based on your OS.

File Manager:

If you are in terminal, to open file manager, you need to run nautilus. If you want to open file manager in specific folder, you can run nautilus /foo/bar

When you have downloaded some files using browser, if you click Show in folder in downloads page, it will open nautilus with that specific folder.

We can also make Emacs to act as file manager in dired mode. If you run

$ emacs -q --eval '(dired "~")'

In terminal, emacs starts with a dired buffer for loggined user home directory.

Now lets write a small bash script to do the same thing.

#!/bin/bash
 #!/usr/bin/bash
 if [ $# -gt 0 ]; then
     dirname=$1
 else
     dirname=$HOME
 fi
 emacsclient -c -eval "(dired \"$dirname\")" > /dev/null

Save this file as emacsfm in /usr/local/bin. Now you can run emacsfm in terminal which open home dired buffer or you can run emacsfm /foo/bar which will open /foo/bar dired buffer.

Now lets set, emacsfm as default file manager. For that first we need to install exo-utils.

$ sudo apt-get install exo-utils

and run it

$ exo-preferred-applications

It opens a new window like this.

Go to Utilities -> File manager, select Other and goto /usr/local/bin and select emacsfm from it.

Now if you go to browser downloads and click on show in folder, it will emacs dired.

References:

https://help.ubuntu.com/community/DefaultFileManager

[Python] Functions, Methods & Attributes!

Anand Reddy Pandikunta — Tue, 01 Sep 2015 14:17:00 GMT

First, lets take a look at functions & methods and then define attributes for them.

Functions Vs Methods:

In [2]: def f():
   ...:     pass
   ...: 

In [3]: class C:
   ...:     def m(self):
   ...:         pass
   ...:

We have just defined a function and a class(with a method in it).

In [18]: type(f)
Out[18]: function

In [19]: type(C.m)
Out[19]: function

In [20]: type(C().m)
Out[20]: method

In [21]: set(dir(C().m)) - set(dir(f))
Out[21]: {'__func__', '__self__'}

As seen above, a function binded to an instance of a class is method and an unbound method is a just a function. Also a method has __self__ and __func__ attributes in addition attributes of a function.

Attributes:

Lets add some attributes to function & method and see how they work.

In [21]: setattr(f, 'state', 1)

In [22]: hasattr(f, 'state')
Out[22]: True

In [24]: getattr(f, 'state')
Out[24]: 1

We can do the same thing with unbound method(which is nothing but a function) also

In [31]: setattr(C.m, 'state', 2)

In [32]: hasattr(C.m, 'state')
Out[32]: True

In [33]: getattr(C.m, 'state')
Out[33]: 2

But we cant do the same thing with bound methods.

In [34]: setattr(C().m, 'state', 3)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-34-1379e94f2f48> in <module>()
----> 1 setattr(C().m, 'state', 2)

AttributeError: 'method' object has no attribute 'state'

When users add attributes to a function, they are stored in its __dict__ attribute.

In [40]: f.__dict__
Out[40]: {'state': 1}

In [49]: C.m.__dict__
Out[49]: {'state': 2}

As we have seen in the beginning, method objects just hold reference to its class(__self__) and function(__func__). But they don't have its own __dict__ to hold custom attributes. So we cannot set custom attributes to instance methods.

But we can get the function that is referenced by bound method and set attribute for it.

In [36]: setattr(C().m.__func__, 'state', 3)

In [39]: getattr(C().m.__func__, 'state')
Out[39]: 3

Also methods provide, special __getattr__ which forwards attribute access to function object. So, this will work

In [52]: hasattr(C().m, 'state')
Out[52]: True

In [53]: getattr(C().m, 'state')
Out[53]: 3

So we can just set attributes to functions & unbound methods just like classes but we can't do it for bound methods.

Django Tips & Tricks #5 - Use for...empty in Django templates

Anand Reddy Pandikunta — Tue, 04 Aug 2015 12:16:00 GMT

It is quite common to send a query set from a view to template, loop over it and display items in it. If no items are there, it is good if we display some message. For that we can do something like this.

{% if books %}
  {% for book in books %}
    <li>{{ book }}</li>
  {% endfor %}
{% else %}
  <li>Sorry, there are no books.</li>
{% endif %}

Django {% for %} tag takes an optional {% empty %} clause which will be rendered when queryset is empty. So we can rewrite the above code as

{% for book in books %}
  <li>{{ book }}</li>
{% empty %}
  <li>Sorry, there are no books.</li>
{% endfor %}

This code is much cleaner than previous one. I tried to profile both code blocks. The first block took 0.9ms and second block took 0.8ms which means it is 11% faster than previous code :)

Reference: Django docs

Random Life Quotes #2

Anand Reddy Pandikunta — Sun, 26 Jul 2015 11:13:00 GMT

Django Tips & Tricks #4 - Make Django Development Server Persistent

Anand Reddy Pandikunta — Mon, 20 Jul 2015 13:11:00 GMT

I use Emacs for writing Python code. I also use real-auto-save to save my files after 10 seconds of inactivity.

While coding, when I stop writing in the middle, after 10 seconds Emacs auto saves the file. Django recognizes this & reloads the development server.

Once I complete writing code, I go to browser & refresh the page. Since the code I was writing was incomplete somewhere in the middle and had some errors, Django development server was stopped and page won't reload. Now I have to go back to terminal and restart the server and again go back to browser to refresh page.

To overcome this, Django server must be made persistent. The easiest way to accomplish this is to use a simple bash script as described here.

$ while true; do python manage.py runserver; sleep 4; done

When Django development server stops, after 4 seconds it tries to start automatically and goes on forever.

Instead of typing that everytime, it better to write this as a shell script and put it in system path, so that it can be used in any project.

while true; do
  echo "Re-starting Django runserver"
  python manage.py runserver
  sleep 4
done

Save this & make it executable(chmod +x), use it is ./filename.sh and to stop use Ctrl-c Ctrl-c

Git Etiquette: Meaningful Messages & Linear Logs!

Anand Reddy Pandikunta — Wed, 17 Jun 2015 11:29:00 GMT

Why Elegant Git Logs Matter?

I will show You few commit messages & logs from two different projects. Based only on those messages/logs, You need to decide which project is the best.

Project 1:

Project 2:

I hope that explains why We need to maintain elegant git logs.

How To Maintain Elegant Logs?

First let's see how to write good commit messages. There are several blog posts & guides about this. Here is the summary of all those taken from this article.

1. Separate subject from body with a blank line
2. Limit the subject line to 50 characters
3. Capitalize the subject line
4. Do not end the subject line with a period
5. Use the imperative mood in the subject line
6. Wrap the body at 72 characters
7. Use the body to explain what and why vs. how

If You are doing something hacky, writing a little message in commit on why you did that will help other developers to understand that a lot better. Even if You don't follow all rules, some of them will help you in writing great commit message next time. If You are mainainter of a project, ask Your team members to write meaningful messages.

The next that need to be taken care is commits. If a team member or someone else has submitted a pull request, once you merge the commit, just rebase it to remove unnecessary merge commits from log. Also make sure to squash redundant commits and You have almost a linear log.

A side benefit of this is, You can see recent commit there instead of merge commit, if your project is hosted on github.

The two minutes You spent on writing a meaningful message or maintaining linear log will
will help You a lot few months down the line & will save countless hours and sleepless nights for future contributors.

Read more articles about Technology!

Automatically PEP8 & Format Your Python Code

Anand Reddy Pandikunta — Sat, 23 May 2015 13:44:00 GMT

Even if you don't intend anybody else to read your code, there's still a very good chance that somebody will have to stare at your code and 
figure out what it does: That person is probably going to be you, 
twelve months from now[1].

or as Zen of Python says "Readability Counts".

PEP8 Style Guide:

Since code is read more than it is written, it is very important to make sure that the code is readable. Python has coding convetions(PEP8 style guide) to maintain consistency in code and thereby making it more readable.

There are lot of tools (like pep8, flake8, pylint) to check if Your code is in compliance with PEP8. Today most of the IDE/text editors have plugins which check for these errors and report it on the fly. If You are writing code from scratch, You can make sure that You are following coding conventions.

However once in a while You have to read/bug-fix other people's code who don't follow these conventions(or You might even stumble across Your old piece of code). Once You are accustomed to style guide, it will be uneasy to read inconsistent code. Here is a badly formatted Python file opened in Emacs editor(with flymake & flake8 configured)

Custom Style Guide:

If You don't like any of PEP8 rules, You can set Your own style guide for Your project. Most developers find limiting line length to 79 characters annoying. So they set their own line limit. You can even configure PEP8 checkers to take You own guidelines and check the code according to it.

Auto PEP8:

Formatting code to PEP8 style is boring & time consuming process. So instead of manually formatting code, You can use autopep8 package which automatically formats Python code to conform to the PEP 8 style guide.

To install the package run

$ pip install autopep8

If You have a file foo.py as show above, You can run

$ autopep8 --in-place --aggressive --aggressive foo.py

which neatly formats the file like this

Emacs Integration:

As emacs is extremely configurable editor, with just couple of lines of lisp, we can integrate autopep8 to emacs. Just find where autopep8 is installed; write a function which runs autopep8 on current buffer; bind a key to that function; set a hook, so that when ever You save a file, it will be automatically run the function.

This works reasonably well. But I do recommend using Elpy for Python developemnt in emacs. I have just sent a pull request to integrate autopep8 with elpy.

Formatting:

PEP8 formatters like autopep8, pep8ify will remove only lint errors. But they can't beautify code.

little = more[3:   5]

x = {'a': 37, 'b': 42,
     'c': 927}

Above code remains same after pep8ifying also. But the code doesn't look good yet. You can use formatters like yapf, which will format the code even if the code is PEP8 compliant.

Above code will be formatted to

little = more[3:5]

x = {'a': 37, 'b': 42, 'c': 927}

Some times this even destroys Your manual formatting. For example

BAZ = {
    [1, 2, 3, 4],
    [5, 6, 7, 8],
    [9, 10, 11, 12]
}

will be converted to

BAZ = {[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]}

But You can tell it to ignore some parts.

BAZ = {
    [1, 2, 3, 4],
    [5, 6, 7, 8],
    [9, 10, 11, 12]
}  # yapf: disable

Beyond PEP8:

As we have learnt to auto PEP8 code, it is the time to look beyond PEP8. There is a good talk by Raymond Hettinger on that - Beyond PEP8, Best practices for beautiful intelligible code.

Let Machines take care of PEP8 & Let Humans look Beyond PEP8.

References:

[1] Code is read much more often than it is written, so plan accordingly.
Python - PEP8 style guide.
Autopep8 - formats Python code to conform to the PEP 8.
PEP8ify - modifies python source code to conform to PEP8.

yapf - A formatter for Python files.

Beyond PEP8 - Best practices for beautiful intelligible code.

Elpy, the Emacs Lisp Python Environment.

Read more articles about Python!
Read more articles about Emacs!

Whats So Good About Django Traceback?

Anand Reddy Pandikunta — Thu, 21 May 2015 09:29:00 GMT

When You are working on django project, if You make any errors, it will throw a simple traceback on the terminal where you started server.

If you go to browser, you will find a rich traceback like this.

Most Python developers, discover django-extensions within a few weeks after they start working with Django and start using Werkzeug debugger. Werkzeug has lot of advantages when compared to default Django traceback. I also used it for a while. For the same error Werkzeug throws traceback like this.

One thing I really like about Django traceback is, the distinction between user code and the internal Django code. Most of the time, developers were looking for the bug in their code instead of looking for a bug in Django. So, Django makes it easier to skip over the frames that doesn't matter and focus on the one which matters most.

It also shows local vars in that frame. With this You instantly look at the variables to find out why error has occured(see this Django Ticket #11834: for more discussion about this).

These two features make it very easy to track down most common errors.

Read more articles about Python!

The Intelligence Behind Python Slices

Anand Reddy Pandikunta — Sun, 22 Mar 2015 10:31:00 GMT

When getting items from a list, if no index is supplied or an invalid index is given, Python throws error.

In [4]: alpha =  ['a', 'b', 'c', 'd', 'e', 'f']

In [5]: alpha[21]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-5-3b7fb56a1ede> in <module>()
----> 1 alpha[21]

IndexError: list index out of range

In [6]: alpha[]
  File "<ipython-input-6-a17d5cfb2c2f>", line 1
    alpha[]
          ^
SyntaxError: invalid syntax

When slicing a list, if start/stop/step values are not supplied or invalid values are provided, Python figures out appropriate values and returns relevant items or an empty list depending on the values.

In [170]: alpha
Out[170]: ['a', 'b', 'c', 'd', 'e', 'f']

In [172]: alpha[:]
Out[172]: ['a', 'b', 'c', 'd', 'e', 'f']

In [173]: alpha[::]
Out[173]: ['a', 'b', 'c', 'd', 'e', 'f']

In [174]: alpha[:-1:]
Out[174]: ['a', 'b', 'c', 'd', 'e']

In [175]: alpha[:2:-1]
Out[175]: ['f', 'e', 'd']

In [176]: alpha[:100000000000000000000000000000000000000000000000]
Out[176]: ['a', 'b', 'c', 'd', 'e', 'f']

In [177]: alpha[::1000000000]
Out[177]: ['a']

In [178]: alpha[::-1]
Out[178]: ['f', 'e', 'd', 'c', 'b', 'a']

There is something interesting happening with python slicing. Lets see how python slicing works and what happens behind the scenes.

Slicing Is NOT Indexing:

Wiki Python has this amazing picture which clearly distinguishes indexing and slicing.

It is a list with 6 elements in it. To understand slicing better, consider that list as a set of six boxes placed together. Each box has an alphabet in it.

Indexing is like dealing with the contents of box. You can check contents of any box. But You can't check contents of multiple boxes at once. You can even replace contents of the box. But You can't place 2 balls in 1 box or replace 2 balls at a time.

In [122]: alpha = ['a', 'b', 'c', 'd', 'e', 'f']

In [123]: alpha
Out[123]: ['a', 'b', 'c', 'd', 'e', 'f']

In [124]: alpha[0]
Out[124]: 'a'

In [127]: alpha[0] = 'A'

In [128]: alpha
Out[128]: ['A', 'b', 'c', 'd', 'e', 'f']

In [129]: alpha[0, 1]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-129-c7eb16585371> in <module>()
----> 1 alpha[0,1]

TypeError: list indices must be integers, not tuple

Slicing is like dealing with boxes itself. You can pickup first box and place it on another table. To pickup the box all You need to know is the position of beginning & ending of the box.

You can even pickup first 3 boxes or last 2 boxes or all boxes between 1 & 4. So, You can pick any set of boxes if You know beginning & ending. This positions are called start & stop positions.

The interesting thing is that You can replace multiple boxes at once. Also You can place multiple boxes where ever You like.

In [130]: alpha[0:1]
Out[130]: ['A']

In [131]: alpha[0:1] = 'a'

In [132]: alpha
Out[132]: ['a', 'b', 'c', 'd', 'e', 'f']

In [133]: alpha[0:2] = ['A', 'B']

In [134]: alpha
Out[134]: ['A', 'B', 'c', 'd', 'e', 'f']

In [135]: alpha[2:2] = ['x', 'xx']

In [136]: alpha
Out[136]: ['A', 'B', 'x', 'xx', 'c', 'd', 'e', 'f']

Slicing With Step:

Till now You have picked boxes continuously. But some times You need to pickup discretely. For example You can pickup every second box. You can even pickup every third box from the end. This value is called step size. This represents the gap between Your successive pickups. The step size should be positive if You are picking boxes from the beginning to end and vice versa.

In [137]: alpha = ['a', 'b', 'c', 'd', 'e', 'f']

In [142]: alpha[1:5:2]
Out[142]: ['b', 'd']

In [143]: alpha[-1:-5:-2]
Out[143]: ['f', 'd']

In [144]: alpha[1:5:-2]
Out[144]: []

In [145]: alpha[-1:-5:2]
Out[145]: []

How Python Figures Out Missing Parameters:

When slicing if You leave out any parameter, Python tries to figure it out automatically.

If You check source code of CPython, You will find a function called PySlice_GetIndicesEx which figures out indices to a slice for any given parameters. Here is the logical equivalent code in Python.

This function takes a Python object & optional parameters for slicing and returns start, stop, step & slice length for the requested slice.

def py_slice_get_indices_ex(obj, start=None, stop=None, step=None):

    length = len(obj)

    if step is None:
        step = 1
    if step == 0:
        raise Exception("Step cannot be zero.")

    if start is None:
        start = 0 if step > 0 else length - 1
    else:
        if start < 0:
            start += length
        if start < 0:
            start = 0 if step > 0 else -1
        if start >= length:
            start = length if step > 0 else length - 1

    if stop is None:
        stop = length if step > 0 else -1
    else:
        if stop < 0:
            stop += length
        if stop < 0:
            stop = 0 if step > 0 else -1
        if stop >= length:
            stop = length if step > 0 else length - 1

    if (step < 0 and stop >= start) or (step > 0 and start >= stop):
        slice_length = 0
    elif step < 0:
        slice_length = (stop - start + 1)/(step) + 1
    else:
        slice_length = (stop - start - 1)/(step) + 1

    return (start, stop, step, slice_length)

This is the intelligence that is present behind slices. Since Python has inbuilt function called slice, You can pass some parameters & check how smartly it calculates missing parameters.

In [21]: alpha = ['a', 'b', 'c', 'd', 'e', 'f']

In [22]: s = slice(None, None, None)

In [23]: s
Out[23]: slice(None, None, None)

In [24]: s.indices(len(alpha))
Out[24]: (0, 6, 1)

In [25]: range(*s.indices(len(alpha)))
Out[25]: [0, 1, 2, 3, 4, 5]

In [26]: s = slice(None, None, -1)

In [27]: range(*s.indices(len(alpha)))
Out[27]: [5, 4, 3, 2, 1, 0]

In [28]: s = slice(None, 3, -1)

In [29]: range(*s.indices(len(alpha)))
Out[29]: [5, 4]

This is how Python is able to figure out missing parameters from slices.

I started digging this after attending a talk by Anand B. Pillai on python gotchas. Thanks to Anand B. Pillai & Krace Kumar for conducting BangPypers meetup & encouraging to checkout Python source code.

References:

Wiki Python: https://wiki.python.org/moin/

CPython repo: https://github.com/python/cpython

Slice on SO: http://stackoverflow.com/questions/509211/explain-pythons-slice-notation

Docs for slice: https://docs.python.org/3.4/library/functions.html#slice

Python/C API for slice: https://docs.python.org/3.4/c-api/slice.html

Django Tips & Tricks #3 - Dynamic Initial Values In Forms

Anand Reddy Pandikunta — Thu, 19 Mar 2015 13:22:00 GMT

Django form fields accept initial argument. So You can set a default value for a field.

In [1]: from django import forms

In [2]: class SampleForm(forms.Form):
   ...:     name = forms.CharField(max_length=10, initial='avil page')
   ...:

In [3]: f = SampleForm()

In [4]: f.as_p()
Out[4]: u'<p>Name: <input maxlength="10" name="name" type="text" value="avil page" /></p>'

Sometimes it is required to override init method in forms and set field initial arguments.

In [11]: from django import forms

In [12]: class AdvancedForm(forms.Form):
   ....:
   ....:    def __init__(self, *args, **kwargs):
   ....:        super().__init__(*args, **kwargs)
   ....:        self.fields['name'].initial =  'override'
   ....:
   ....:        name=forms.CharField(max_length=10)
   ....:

In [13]: f2 = AdvancedForm()

In [14]: f2.as_p()
Out[14]: '<p>Name: <input maxlength="10" name="name" type="text" value="override" /></p>'

Now let's pass some initial data to form and see what happens.

In [11]: from django import forms

In [12]: class AdvancedForm(forms.Form):
   ....:
   ....:    def __init__(self, *args, **kwargs):
   ....:        super().__init__(*args, **kwargs)
   ....:        self.fields['name'].initial = 'override'  # don't try this at home
   ....:
   ....:        name=forms.CharField(max_length=10)
   ....:

In [19]: f3 = AdvancedForm(initial={'name': 'precedence'})

In [20]: f3.as_p()
Out[20]: '<p>Name: <input maxlength="10" name="name" type="text" value="precedence" /></p>'

If You look at the value of input field, it's is NOT the overrided. It still has form initial value!

If You look into source code of django forms to find what is happening, You will find this.

data = self.field.bound_data(
       self.data,
       self.form.initial.get(self.name, self.field.initial)  # precedence matters!!!!
)

So form's initial value has precedence over fields initial values.

So You have to override form's initial value instead of fields's initial value to make it work as expected.

In [21]: from django import forms

In [22]: class AdvancedForm(forms.Form):
   ....:
   ....:    def __init__(self, *args, **kwargs):
   ....:        super().__init__(*args, **kwargs)
   ....:        self.initial['name'] = 'override'  # aha!!!!
   ....:
   ....:        name=forms.CharField(max_length=10)
   ....:

In [23]: f4 = AdvancedForm(initial={'name': 'precedence'})

In [24]: f4.as_p()
Out[24]: '<p>Name: <input maxlength="10" name="name" type="text" value="override" /></p>'

Read official docs about django forms.

WD My Cloud NAS Setup On Linux

Anand Reddy Pandikunta — Tue, 17 Mar 2015 08:50:00 GMT

Introduction

I recently bought a WD My Cloud NAS¹ device to store my personal data. I wanted to set it up on my Ubuntu machine. For WD My Cloud, there is no official support for Ubuntu or any other Linux distros. But setting up it is quite easy.

NAS Setup

Make sure You have connected power adapter & LAN cables to it. If You open Your router config, You will see WD My cloud in client list. Make note of its IP address. If You want, You can assign a static IP also in the router settings.

Next step is to install NFS client package. NFS(Network File System) allows a system to share directories and files with others over a network. By using NFS, users and programs can access files on remote systems almost as if they were local files. So, update your packages & install nfs-common package.

$ sudo apt-get update
$ sudo apt-get install nfs-common

Now we can list folders which are available to mount using showmount command.

$ showmount -e <ip-address>

Create an empty folder to mount any of the folder you wanted and mount it.

$ sudo mount -o rw,soft,intr,nfsvers=3 <ip>:<folder-to-mount> <path-to-mount>

Now You can start moving data into/out of WD My Cloud.

If You want to mount it automatically on boot, add following line to /etc/fstab file.

<ip>:<folder-to-mount> <path-to-mount> nfs rw,soft,intr,nfsvers=3 0 0

Conclusion

Even though there is no official support for Ubuntu, WD My Cloud works pretty well with Ubuntu and other Linux distros.

https://en.wikipedia.org/wiki/Network-attached_storage ↩

Install Oh-My-Zsh On Ubuntu!

Anand Reddy Pandikunta — Sun, 15 Mar 2015 12:24:00 GMT

I crashed my OS & I had to re install all things again. This is just to keep a track on how to install zsh.

For oh-my-zsh to work, You need to install zsh & git core first. To do that

apt-get install zsh
apt-get install git-core

Now, You can install it by

curl -L http://install.ohmyz.sh | sh

Next, change your shell to zsh

chsh -s `which zsh`

and then restart your system.

sudo shutdown -r 0

Random Life Quotes #1

Anand Reddy Pandikunta — Fri, 13 Mar 2015 11:30:00 GMT

How To Make Better Use Of Any Emacs Package?

Anand Reddy Pandikunta — Sat, 07 Mar 2015 12:53:00 GMT

Elpy is an Emacs package that brings powerful Python editing to Emacs. While going through elpy issues, I came across an interesting thread, where someone asked this

This lead to this post. This post gives a brief introduction on getting familiar with any Emacs package & start hacking it quickly.

Getting Started:

Most Emacs packages provide a Readme file & some of them have good documentation too. Also there is emacswiki where some other hacks will be available.

That will be help You to get started with any package. You can start using default functions/key bindings.

If we take elpy as a case study, it has Readme file which shows how to install elpy. It also has a good documentation about elpy & some of its important features.

Getting Under The Hood:

Now lets see some interesting stuff which Emacs packages are capable of but read me file or documentation wont contain.

One useful package I recommend for this purpose is helm-descbindings which shows key bindings of all available modes. You can also filter among them.

Here You will come to know about few useful functions an Emacs package is capable of doing.

In the case of elpy, You can see something like this.

But You cannot find all key bindings here as some of them might be shadowed (or taken away) by other packages.

To specify a key binding, Emacs packages use a define-key function. So if You open source code of the package and search for define-key, you will see all key bindings there.

If you search for define-key in elpy.el You will come across this

Now You know a few functions & their key bindings which are not present in the documentation.

If there are any key bindings that are not useful You can unbind them.
For example elpy has "C-c C-p" binded to (elpy-flymake-previous-error) which I won't use much. So I can unbind it

    (define-key elpy-mode-map (kbd "C-c C-p") nil)

This sets "C-c C-p" to nil.

Digging Deeper:

So far we have seen what an Emacs package is capable of doing. Now, lets start tweaking it a little bit.

The first thing You can do is start using interactive functions right away. Functions in Emacs package which has interactive form are callable commands. You can invoke them using M-x and function name. You will find a lot of them here.

If you want to find all interactive commands in an emacs package, You have to search through source code & find them. Instead if you have helm (If You are not using Helm, I highly recommend You to start using it.) installed, if you type M-x & few letters of the package name, it will narrow down all the interactive functions of that package.

Elpy has a lot of interactive commands, You can see a few of them here.

Now You can start bindings keys to any of the functions that You like. Previously We have unbinded a key. Now lets bind it to some function.

(global-set-key (kbd "C-c C-p") 'elpy-set-project-root)

Getting (in)sane With Lisp:

So far we have just used Emacs package as it is. If You know a bit of lisp, You can start tweaking existing functions or You can write Your own functions. This mostly depends on how You want to use & customize a package. You can do a lot of fun stuff. Enjoy lisping :)

BangPypers Talks - Emacs As Python IDE!

Anand Reddy Pandikunta — Mon, 23 Feb 2015 15:02:00 GMT

BangPypers is the Bangalore Python Usergroup. It is one of the largest and oldest Python user groups in India. On 3rd Saturday of every month there will be meetup. This month(Feb 2014) took place at IBM, Domlur. I gave a talk about setting up Emacs as Python IDE.

When I started using Emacs, I spent a lot of time to configure & make it an IDE. Recently I came across elpy(emacs lisp python enviroment) & liked it a lot. With just couple of lines You can turn emacs into Python IDE. So I stressed a lot about elpy during the talk.

I talked about other important packages I used & other things that can be done with emacs.

Meetup Report

Slides

My .emacs.d

Bangalore Django User Group - Beginners Workshop

Anand Reddy Pandikunta — Mon, 16 Feb 2015 15:10:00 GMT

During BangPypers december dev sprint Few weeks back, Krace & Siva asked me if I can take a Django workshop. I agreed. They have worked on logistics & scheduled for February at IBM, Domlur.

I reached there by 9.30 AM. 26 people attended the session. Due to some problems we started a bit late than what we have planned. Krace, Siva & Ankur assisted a lot during the session.

We planned to cover the basics of Django & CRUD operations by building a simple app. We covered basics & simple create view. Due to time constraint we were unable to cover the remaining.

Content: https://github.com/ChillarAnand/django-bookmarks

Slides: http://chillaranand.github.io/django-bookmarks/

14 Great Quotes About Python Programming Language

Anand Reddy Pandikunta — Tue, 30 Dec 2014 08:11:00 GMT

A collection of quotes about Python programming language!

The joy of coding Python should be in seeing short, concise, readable classes that express a lot of action in a small amount of clear code -- not in reams of trivial code that bores the reader to death.

- Guido van Rossum

My favorite language for maintainability is Python. It has simple, clean syntax, object encapsulation, good library support, and optional named parameters.

- Bram Cohen

As it seems to me, in Perl you have to be an expert to correctly make a nested data structure like, say, a list of hashes of instances. In Python, you have to be an idiot not to be able to do it, because you just write it down.

- Peter Norvig

In many ways, it's a dull language, borrowing solid old concepts from many other languages & styles: boring syntax, unsurprising semantics, few automatic coercions, etc etc. But that's one of the things I like about Python.

- Tim Peters

Everyone knows that any scripting language shootout that doesn't show Python as the best language is faulty by design.

- Max M

Every sufficiently advanced LISP application will eventually reimplement Python.

- Hodgson's Law

The canonical, "Python is a great first language", elicited, "Python is a great last language!"

- Noah Spurrier

Python is the "most powerful language you can still read".

- Paul Dubois

"Python tricks" is a tough one, cuz the language is so clean. E.g., C makes an art of confusing pointers with arrays and strings, which leads to lotsa neat pointer tricks; APL mistakes everything for an array, leading to neat one-liners; and Perl confuses everything period, making each line a joyous adventure ;-)

- Tim Peters

Python is a truly wonderful language. When somebody comes up with a good idea it takes about 1 minute and five lines to program something that almost does what you want. Then it takes only an hour to extend the script to 300 lines, after which it still does almost what you want.

- Jack Jansen

Python is an experiment in how much freedom programmers need. Too much freedom and nobody can read another's code; too little and expressiveness is endangered.

- Guido van Rossum

I suggested holding a "Python Object Oriented Programming Seminar", but the acronym was unpopular.

- Joseph Strout

PYTHON = (P)rogrammers (Y)earning (T)o (H)omestead (O)ur (N)oosphere.

- Sean McGrath

Abstraction is one of those notions that Python tosses out the window, yet expresses very well.

- Gordon McMillan,

Surprisingly enough, Python has taught me more about Lisp than Lisp ever did ;-).

- Glyph Lefkowitz

Sources:
Starship Python
Scott's space

Importing & Exporting JSON Data To mongoDB!

Anand Reddy Pandikunta — Sun, 28 Dec 2014 08:03:00 GMT

mongoimport imports JSON, CSV or TSV data to mongoDB. If You are importing JSON data from a file to db, the file should have only one document per line and there shouldn't be any delimiters at the end of file. If You have a file sample.json like this

{"num":1,"status":"ok"}
{"num":2,"status":"ok"}
{"num":3,"status":"ok"}
{"num":4,"status":"ok"}

You can directly import it into db using

mongoimport --db <db> --collections <collection> --file sample.json

This is straight forward and works well. But some third party services provide data in the form of JSON array like this

[
    {"num":1,"status":"ok"},
    {"num":2,"status":"ok"},
    {"num":3,"status":"ok"},
    {"num":4,"status":"ok"},
]

If You try to import it, it throws FailedToParse error. For this, You need to pass additional param jsonArray.

mongoimport --db <db> --collections <collection> --file sample.json --jsonArray

If You want to export Your collection as a json dump, You can use mongoexport command

mongoexport --db <db> --collections <collection> --out sample.json

This exports data into JSON format with one document in a row.

If You pass jsonArray as an optional parameter, it will export the collection as a jsonArray

mongoexport --db <db> --collections <collection> --out sample.json --jsonArray

If You are lazy to type, You can use

mongoexport --d <db> --c <collection> --o sample.json --jsonArray

References:: mongoDB docs - mongoimport, mongoexport

7 Easter Eggs Hiding In Your Emacs!

Anand Reddy Pandikunta — Sat, 27 Dec 2014 14:59:00 GMT

I just came across this xkcd comic about Real programmers.

I started emacs invoked butterfly command and that's my first easter egg. Later I digged around and found several games & a few other easter eggs. Here are a few of them.

M-x butterfly: Flip the desired bit on the drive platter.

M-x doctor: Switch to *doctor* buffer and start giving psychotherapy.

M-x zone: zone out completely.

M-x life: runs Conway's "Life" cellular automaton.

M-x studlify-buffer: stUdlIfY-RegioN stUdlIfY-CaSeS thE region.

M-x landmark: Start or resume an Landmark game.

M-x animate-birthday-present: Return a birthday present.

M-x yow: Return or display a random Zippy quotation.

Notes:
1. yow was removed in Emacs 22 due to copyright problems. But you can get it manually.

Download quotes file to your .emacs.d directory
M-! wget bit.ly/emacs-yow -O ~/.emacs.d/yow.txt.gz

Specify the path of file in Your init file
(setq yow-file "~/.emacs.d/yow.txt.gz")

2. To know about each of the easter eggs, you can use help
C-h f RET <function> RET

More articles about Emacs

BangPypers - Python Dev Sprint

Anand Reddy Pandikunta — Mon, 22 Dec 2014 07:47:00 GMT

Here is the definition of sprint from Wikipedia:

On 3rd Saturday of every month there will be a meetup for Python developers in Bangalore(BangPypers) to have talks and workshops. This month instead of talks/workshops, a dev sprint was held. The basic idea was to encourage Python developers to contribute to any open source projects or work on their open source projects.

I was working on Nidaba, a project to extract information from StackOverflow dumps. I went to the meetup and gave a brief description about it. Several people were interested to work. I helped them a bit on getting started and the issues to solve.

By the end of event, I got one pull request which fixed a typo in setup file. Few people helped in adding couple of utility functions for the project. A couple of them were interested to continue working on with it in the coming weekends. In the end PSSI sponsored T-shirt :)

Here are a few snapshots of event.

ది పైథాన్ పారడాక్స్ (The Python Paradox)

Anand Reddy Pandikunta — Fri, 19 Dec 2014 10:21:00 GMT

ఆగస్టు 2004 లో ఈ వ్యాసం పాల్ గ్రాహం ఆంగ్లం లో వ్రాసారు. ఇది తెలుగు అనువాదం.

"మీరు ఒక Java ప్రాజెక్ట్ కన్నా Python ప్రాజెక్ట్ మీద పని చేయడానికి తెలివైన ప్రోగ్రామర్లను పొందవచ్చు" అని నేను ఇటీవల చర్చ లో అన్న మాటలకు ప్రజలు చాలా అప్సెట్ అయ్యారు.

దానికి అర్థం Java ప్రోగ్రామర్లు మొద్దు అని కాదు. Python ప్రోగ్రామర్లు స్మార్ట్ గా ఉంటారు అని. ఒక కొత్త ప్రోగ్రామింగ్ భాష నేర్చుకోవడం కష్టమైన పని. ప్రజలు ఒక ఉద్యోగం పొందటానికి Python నేర్చుకోరు. వారు పైథాన్ ఎందుకు నేర్చుకుంటారంటే వారికి స్వచ్ఛంగా ప్రోగ్రామ్మింగ్ అంటే ఇష్టం మరియు వారికి ఇప్పటికే తెలిసిన భాషలు సంతృప్తి ఇవ్వకపోవడం వలన. అందువల్లే కంపెనీలు Python ప్రోగ్రామర్లను నియమించుకుంటాయి. ఈ concept కి ఒక మంచి పేరు ఏది తట్టలేదు, అందుకే నేను దీన్నిPython పారడాక్స్ అని పిలుస్తాను. ఒక సంస్థ software వ్రాయడానికి తులనాత్మకంగా esoteric భాషను ఎంచుకుంటే, వారు మంచి ప్రోగ్రామర్లను నియమించుకుంటారు. ఎందుకంటే అలాంటి బాష నేర్చుకోవాలని చాలా కొద్ది మందే కుతూహలంగా ఉంటారు. ప్రోగ్రామర్లకి ఈ పారడాక్స్ మరింత వర్తిస్తుంది. మీరు ఒక మంచి ఉద్యోగం పొందడానికి నేర్చుకునే భాష, ప్రజలు కేవలం ఉద్యోగం పొందడానికి మాత్రమే నేర్చుకోలేదు.

కేవలం కొన్ని తెలివైన కంపెనీలు మాత్రమె ఇప్పటివరకు ఈ విషయాన్ని గ్రహించాయి. కానీ ఇక్కడ ఇంకో రకమైన ఎంపిక జరగుతోంది. సరిగ్గా అటువంటి కంపెనీలలోనే  ప్రోగ్రామర్లు పని చేయాలనీ కోరుకుంటున్నారు. ఉదాహరణకు, గూగుల్ Java ప్రోగ్రామింగ్ ఉద్యోగాలు ప్రకటన ఇచ్చినప్పుడు, వారు Python అనుభవం తప్పనిసరిగా అడుగుతున్నారు.

అన్ని భాషలు విస్తృతంగా ఉపయోగించిన నా స్నేహితుడు, దాదాపు అన్ని ప్రాజెక్ట్ ల  కొరకు Python ఉపయోగిస్తాడు. అందుకు ప్రధాన కారణం source code చదవటానికి చాల అనువుగా ఉండటం. ఆ ఒక్క కారణానికి మరొక భాషను ఎంచుకోవడం ఒక అల్పమైన కారణం అని మీరు అనుకోవచ్చు. కాని అది అనుకున్నంత అల్పమైన కారణం కాదు. ఎందుకంటే source code ను రాయడం కంటే చదవడం లో ఎక్కువ సమయం వెచ్చిస్తారు. ఒక శిల్పి మట్టి ముద్దల్ని పేర్చుకు పోయినట్లు మీరు కూడా source code blobs ని పేర్చుకుంటూపోతారు. అస్తవ్యస్తంగా పేర్చిన మట్టి ముద్దలు శిల్పిని చికాకుపరిచినట్లే, చదవటానికి అనువుగా లేని ఒక భాష, ప్రోగ్రామర్ ని చికాకు పరుస్తుంది.

Source code అవలక్షణం ప్రస్తావన వద్ద, ప్రజలు Perl ను గుర్తు చేసుకుంటారు. కానీ నేను మాట్లాడేది Perl యొక్క ఉపరిభాగ సంబంధిత  అందం గురించి కాదు. నిజమైన అవలక్షణం కఠినంగ కనిపించే వాక్యనిర్మాణం కాదు, కానీ తప్పు patterns ని అనుసరించి ప్రోగ్రమ్స్  వ్రాయటం. Perl ఒక కార్టూన్ పాత్ర తిట్లు వల్లే ఉండవచ్చు, కానీ కొన్ని చోట్ల అది Python ను అధిగమించిన సందర్భాలు ఉన్నాయి.

ఇప్పటివరకు, ఏమైనప్పటికీ. రెండు భాషలు కదిలే లక్ష్యాలు .  కానీ ఈ రెండు భాషలు కుడా Ruby, (Icon, Joy, J , Lisp & Small Talk) మాదిరి, నిజంగా ప్రోగ్రామింగ్ పట్టించుకుని, ఉపయోగించే   ప్రజలచే రూపొందించబడినవి అన్న సత్యాన్ని గుర్తిస్తున్నాయి.

Generate Sphinx Friendly Docstrings For Python Functions In Emacs!

Anand Reddy Pandikunta — Sat, 13 Dec 2014 11:59:00 GMT

Today, I went to Django REST Framework Workshop. After the workshop is completed, and we were discussing about Emacs, Krace showed this cool plugin sphinx-doc, it inserts docstring skeleton for Python functions/methods as per the requirement of the Sphinx documentation generator.

Installation:

To install it from MELPA, run "M-x package-list-packages", search for "sphinx-doc" , mark it with i and press x to install.

Configuration:

Add these lines to Your emacs config to enable sphinx-doc-mode.

(add-hook 'python-mode-hook (lambda ()
                  (require 'sphinx-doc)
                  (sphinx-doc-mode t)))

If you document your code(infact you should), this plugin saves quite a few strokes!

Django Tips & Tricks #2 - Check If App Is Production Ready

Anand Reddy Pandikunta — Fri, 12 Dec 2014 08:24:00 GMT

Before deploying Your Django app, You need to make sure security, logging and other issues are taken care. Django provides a simple deployment checklist which helps a lot. In development version Django provides --deploy option, which does some security checks. You can run it with

python manage.py check --deploy --settings=production_settings

I have just created a new project and Django identified 6 security issues in it.

→ python manage.py check --deploy                               
System check identified some issues:
WARNINGS:
?: (security.W001) You do not have 'django.middleware.security.SecurityMiddleware' in your MIDDLEWARE_CLASSES so the SECURE_HSTS_SECONDS, SECURE_CONTENT_TYPE_NOSNIFF, SECURE_BROWSER_XSS_FILTER, and SECURE_SSL_REDIRECT settings will have no effect.?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W016) You have 'django.middleware.csrf.CsrfViewMiddleware' in your MIDDLEWARE_CLASSES, but you have not set CSRF_COOKIE_SECURE to True. Using a secure-only CSRF cookie makes it more difficult for network traffic sniffers to steal the CSRF token.
?: (security.W017) You have 'django.middleware.csrf.CsrfViewMiddleware' in your MIDDLEWARE_CLASSES, but you have not set CSRF_COOKIE_HTTPONLY to True. Using an HttpOnly CSRF cookie makes it more difficult for cross-site scripting attacks to steal the CSRF token.
?: (security.W018) You should not have DEBUG set to True in deployment.
?: (security.W019) You have 'django.middleware.clickjacking.XFrameOptionsMiddleware' in your MIDDLEWARE_CLASSES, but X_FRAME_OPTIONS is not set to 'DENY'. The default is 'SAMEORIGIN', but unless there is a good reason for your site to serve other parts of itself in a frame, you should change it to 'DENY'.

System check identified 6 issues (0 silenced).

However this doesn't identify issues like absence of 404.html, 500.html templates, email setup, admin emails, presence of print, pdb statements. There is a Django package called django-production-ready which checks for these issues.

To the new project just created, I have added a new file, which has a print & pdb statements. I have installed django-production ready, added prodready to installed apps and ran is_it_ready management command which produced a neat log of all issues.

→ python manage.py is_it_ready
--------------------
Production ready: No
--------------------
Possible errors:
    * Enter valid email address in ADMINS section
    * Enter valid email address in MANAGERS section
    * Template 404.html does not exist
    * Template 500.html does not exist
    * Setup E-mail host
    * Set a valid email for SERVER_EMAIL
    * Set a valid email for DEFAULT_FROM_EMAIL
    * You have one or more ipdb import  statements
    * You have one or more print statements

This comes in handy to quickly make sure a lot of issues are taken care and the app is ready for deployment!

Search, Play & Download YouTube Videos From Command Line

Anand Reddy Pandikunta — Tue, 02 Dec 2014 09:04:00 GMT

If You are using Linux, You might be familiar with youtube-dl, a small command-line program to download videos from YouTube and a few more sites. But You need to go to YouTube, grab the url and then give it to youtube-dl to download it.

mps-youtube, a terminal based YouTube player makes it possible to browse, play & download videos right from the terminal.

You can install it using pip

pip install mps-youtube

and start using in with

mpsyt

It is better to set an alias for this

alias yt="mpsyt"

By default it is configured to search music category and stream audio only. You can change it

set music_search false
set show_video true

To get help, just press h.
To search for videos about Katy Perry, type /katy perry.
To stream a song, enter the number of the song and enter.
To download a video, type d <number> and it shows all the available formats, then you can select any video of your choice and download it.

Here are a few screenshots of it.

Mirror A Website (Or Part Of Website) In Split Second!

Anand Reddy Pandikunta — Sat, 29 Nov 2014 02:47:00 GMT

Occasionally it is necessary for developers to mirror a website. It is super easy to do this in no time.

1. Getting A Copy Of Original Site:

You can use the powerful tool wget for this. If you are familiar with *nix, you might have used it to fetch a single page or files to your system. But wget is capable of doing lot more than that. To download an entire site, you can use

wget --recursive --page-requisites --convert-links www.avilpage.com
# if you are lazy to type use 
# wget -rpk www.avilpage.com

This downloads all components(sounds, images etc) of www.avilpage.com and make links in downloaded HTML/CSS to point to local files.

If you want to mirror only a part of website say 2014 November archives only, then you can pass no-parent argument.

wget --recursive --page-requisites --convert-links --no-parent www.avilpage.com/2014/11/
#wget -rpknp www.avilpage.com/2014/11

This downloads all the urls that follow 2014/11.

In addition to that there are a hell lot of options you can pass like number of retries, time to wait, lot of HTTP options, FTP options and so on.

2. Serving The Mirror Website:

If you are familiar with Python, you might have heard of http.server. With a single command it starts running a basic web server. If you run the following command,

python -m http.server 8000  #Python 3
or
python -m SimpleHTTPServer 8000  #Python 2

it starts serving files on port 8000. If you are on LAN, all others can access this mirrored website from their browser by specifying the IP & PORT.

However this is just a basic server and not suitable for deployment. For that you can use Apache or Nginx to serve website.

As wget fetches pages, start server. While wget is downloading the pages, you can browse through the downloaded pages.

[Telugu] యండమూరి అంతర్ముఖం సినిమా

Anand Reddy Pandikunta — Wed, 26 Nov 2014 13:38:00 GMT

తెలుగు పుస్తకాలు, నవలలు చదివేవారు యండమూరి రాసిన " అంతర్ముఖం " నవల పేరు విని వుంటారు. కాని యండమూరి ఆ పుస్తకాన్ని సినిమా గా తీసినట్లు చాల మందికి తెలియదు. ఈ సినిమా YouTube లో అప్లోడ్ చేసారు. IMHO, సినిమా క్వాలిటీ, ఆక్టర్స్ వగైరా వగైరా ఏమి బాగలెవు.

Split Video From Command Line Using FFmpeg

Anand Reddy Pandikunta — Sun, 23 Nov 2014 13:05:00 GMT

I was writing a post about You Dont Mess With Zohan movie and I had to split a 50 second video from it.

I have Vlc installed, so I enabled to view the advanced controls and recorded the required portion.

Once it is done, I played it back, but the audio and video are not syncing. I tried again and the same thing happened again.

This is the video splitted with Vlc.

Next, I splitted video with ffmpeg using this command.

ffmpeg -i input.mp4 -vcodec copy -acodec copy -ss 01:20:12 -t 00:50:00 out.mp4

I splitted correctly and audio/video are synced perfectly well.

This video is split with ffmpeg.

What Is The Area Code Of Amman?

Anand Reddy Pandikunta — Sun, 23 Nov 2014 12:57:00 GMT

In the movie, You Don't Mess With Zohan, these guys sitting at a dining table, try to call Phantom. One of the guy asks other for area code of Amman where Phantom lives. Here is the scene.

Here is the transcript:

What is area code of Aman?

sifr (0) wahid (1) wahid (1).
arba'a (4) khamsa (5) talaata (3).
khamsa (5) khamsa (5) sifr (0) sifr (0).
saba'a (7) saba'a (7).
sitta (6) talaata (3) arba'a (4) sitta (6).
wahid (1).
arba'a (4).
sifr (0).
saba'a (7).
talaata (3) arba'a (4) khamsa (5).

So the area code of amman is: 011453550077634561407345!

If you want to learn arabic numerals this will help.
sifr - 0
wahid- 1
itnayn - 2
talaata - 3
arba'a - 4
khamsa - 5
sitta - 6
saba'a - 7
tamaanya - 8
tisa'a - 9

Source: Yahoo Answers

Unix Timestamp, UTC And Their Conversions In Python

Anand Reddy Pandikunta — Sat, 22 Nov 2014 12:56:00 GMT

Coordinated Universal Time(UTC):

It is the primary time standard by which the world regulates clocks and time. To get current UTC time in Python, we can use datetime module.

In [5]: import datetime

In [6]: datetime.datetime.now(datetime.timezone.utc)
Out[6]: datetime.datetime(2014, 11, 22, 14, 42, 21, 34435, tzinfo=datetime.timezone.utc)

Unix time / POSIX time / Epoch time:

It is a system for describing instants in time, defined as the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970, not counting leap seconds. To get Unix timestamp, we can use time module.

In [8]: import time

In [9]: time.time()
Out[9]: 1416667432.5664258

In [10]: int(time.time())
Out[10]: 1416667766

With Pyhon 3.4, we can directly get timestamp from UTC.

In [13]: datetime.datetime.utcnow().timestamp()
Out[13]: 1416649608.58369

Conversions:

To convert Unix timestamp to UTC we can use utcfromtimestamp function.

In [21]: datetime.datetime.utcfromtimestamp(1416668401)
Out[21]: datetime.datetime(2014, 11, 22, 15, 0, 1)

To convert UTC time object to Unix time, we can use strftime function.

In [38]: dt = datetime.datetime.now()

In [39]: dt.strftime("%s")
Out[39]: '1416668938'

Alternatively, we can use calendar.timegen function.

In [46]: import calendar

In [47]: dt = datetime.datetime.utcnow()

In [48]: calendar.timegm(dt.utctimetuple())
Out[48]: 1416669150

[Git] Updating Cloned/Forked Repository On Local Machine and GitHub!

Anand Reddy Pandikunta — Sat, 22 Nov 2014 03:56:00 GMT

If You have forked a GitHub repo, after a few days/months later and the original master repo might change. So, it is essential to update your forked repo to reflect those changes. One simple solution is, you can delete it and fork again. But, if you have made any changes then you need some other solution.

Updating Cloned Repo On Local Machine:

If you have cloned the repo to your local machine, you can add the original GitHub repository as a "remote". Then you can fetch all the branches from that original repository, and rebase your work to continue working on the upstream version.

From command line you can do this

Add the remote, call it "original":

git remote add original https://github.com/whoever/whatever.git

Fetch all the branches of that remote into remote-tracking branches, such as original/master:

git fetch original

Make sure that you're on your master branch:

git checkout master

Rewrite your master branch so that any commits of yours that aren't already in upstream/master are replayed on top of that other branch:

git rebase original/master

If you don't want to rewrite the history of your master branch, (for example because other people may have cloned it) then you should merge it

git merge original/master

However, for making further pull requests that are as clean as possible, it's probably better to rebase.
If you've rebased your branch onto upstream/master you may need to force the push in order to push it to your own forked repository on GitHub. You'd do that with:

git push -f origin master

Updating Forked Repo On GitHub:

If you have forked the repo on GitHub, then you can update it with web interface

Go to your fork and issue a Pull Request.
By default this will be your fork on the right (head repo) requesting to push its commits and changes to the original repo (base repo) on the left.
Click the drop down for both base repo and head repo and select each other's repos. You want yours listed on the left (accepting changes) while the original repository is on the right (the one with changes to push). As illustrated in this image:

Send the pull request. If your fork has not had any changes, you should be able to automatically accept the merge.

If your code somehow conflicts or is not quite clean enough, then this will not work to update via the GitHub web interface and you will need grab the code and resolve any conflicts on your machine before pushing back to your fork.

Sources: StackOverflow, WebApps

Automagically Reload Imports In iPython!

Anand Reddy Pandikunta — Wed, 19 Nov 2014 11:51:00 GMT

When using iPython, users can import required modules to test them. After importing them, if they get modified either by user or some other process, users have to reload it for futher usage.

Depending on the Python version, appropriate reload function can reload modules.

# Python 2.x
In [15]: imp.reload(module)

# Python 3.0–3.3
In [15]: imp.reload(module)

# Python 3.4+
In [15]: importlib.reload(module)

Instead of manually reloading, ipython has autoreload extention which can auto reload modules. For that, load the extention and activate it.

In [15]: %load_ext autoreload

In [16]: %autoreload 2

This can be added to ipython config file so that autoreload gets activated, whenver it starts.

$ ipython profile create

This creates a default config file. Open config file which is present at ~/.ipython/profile_default/ipython_config.py and add these two lines to it.

c.InteractiveShellApp.extensions = ['autoreload']
c.InteractiveShellApp.exec_lines = ['%autoreload 2']

Note that it won't reloads c extensions automatically.

Converting Strings To Correct Types In Python

Anand Reddy Pandikunta — Mon, 17 Nov 2014 14:10:00 GMT

I was writing custom template tags for one of my Django package. I came across a situation where I had lists in string format. I need to convert them into lists.

temp_str = '[345, 3, 456, 45]'
required_obj = [345, 3, 456, 45]

There are two methods to get this job done.

ast.literal_eval:

This can be used to evaluate strings containing Python values from untrusted sources without parsing values. import ast ast.literal_eval(temp_str)

json.loads:

This is used to deserialize a string to python object using a conversion table.

import json
json.loads(temp_str)
These two functions come in handy whenever you want to convert a list of python objects to their correct types. For example, if you have list of python objects like this
obj_list =  ['hello', '3', '3.64', '-1']

You can convert them to their corresponding types using these functions.

def converter(l):
    for i in l:
        try:
            yield json.loads(i)
        except ValueError:
            yield i
def converter(l):
    for i in l:
        try:
            yield ast.literal_eval(i)
        except ValueError:
            yield i

They yield a objects of corresponding types new_obj_list = ['hello', 3, 3.64, -1]

Scaling Celery - Sending Tasks To Remote Machines

Anand Reddy Pandikunta — Sun, 09 Nov 2014 08:35:00 GMT

Celery:

Celery is a Python package which implements a task queue mechanism with a foucs on real-time processing, while also supporting task scheduling.

It has 3 main components.

Celery Application(or Client): It is responsible for adding tasks to the queue.

Celery Worker(or Server): It is responsible for executing the tasks given to it.

Broker: It is responsible for transporting messages between client and server.

What You Should Know:

You should know basics of Celery and you should be familiar with

creating celery tasks


from celery import Celery

app = Celery('tasks', backend='amqp',
                      broker='amqp://<user>:<password>@<ip>/<vhost>')

@app.task
def add(x, y):
    return x + y

adding tasks to a queue,


add.apply_async(args=[1,2])

and consuming tasks with a worker


celery worker -A my_app -l info

If your tasks doesn't need much system resources, you can setup all of them in the same machine. But, if you have a lot of jobs which consume resources, then you need to spread them out in several machines.

In this tutorial lets move our celery workers into a remote machine keeping client and broker in same machine.

Sending Tasks To Another Machine:

On Machine A:

Install Celery & RabbitMQ.
Configure RabbitMQ so that Machine B can connect to it.


# add new user
sudo rabbitmqctl add_user <user> <password>

# add new virtual host
sudo rabbitmqctl add_vhost <vhost_name>

# set permissions for user on vhost
sudo rabbitmqctl set_permissions -p <vhost_name> <user> ".*" ".*" ".*"

# restart rabbit
sudo rabbitmqctl restart

Create a new file remote.py with a simple task. Here we have broker installed in machine A. So give ip address of machine 1 in broker url option.


from celery import Celery

app = Celery('tasks', backend='amqp',
                      broker='amqp://<user>:<password>@<ip>/<vhost>')

def add(x, y):
     return x + y

Now we have everything setup on machine A. We can now put some tasks in queue.


In [1]: from remote import add

In [2]: add.delay(1, 2)
Out[2]: <AsyncResult: 3eb96a11-aa61-46d3-9b9d-e0e1703438d0>

In [3]: b.delay(2, 3)
Out[3]: <AsyncResult: ec40db1a-a43c-4486-9530-0a3153fe1380>

In [4]: b.delay(3, 4)
Out[4]: <AsyncResult: ca53a4c7-061b-408e-82ee-86c2d43d21a0>

Everything is setup on machine A. Now lets get into machine B.

On Machine B:

Install Celery.
Copy remote.py file from machine A to this machine.
Run a worker to consume the tasks


celery worker -l info -A remote

As soon as you launch the worker, you will receive the tasks you queued up and gets executed immediately.


[2014-11-09 00:01:19,168: INFO/MainProcess] Received task: remote.add[c2d2bb27-ff5f-47da-b2b9-6fb11669ee1a]
[2014-11-09 00:01:19,170: INFO/MainProcess] Received task: remote.add[8daa1a5c-17d0-46dc-9c93-faf7fbeccdd9]
[2014-11-09 00:01:19,172: INFO/MainProcess] Received task: remote.add[79603d15-24f1-43f8-b8b7-525b7cd4b9a2]
[2014-11-09 00:01:19,401: INFO/MainProcess] Task remote.add[8daa1a5c-17d0-46dc-9c93-faf7fbeccdd9] succeeded in 0.226168102003s: 3
[2014-11-09 00:01:19,462: INFO/MainProcess] Task remote.add[c2d2bb27-ff5f-47da-b2b9-6fb11669ee1a] succeeded in 0.286503815001s: 5
[2014-11-09 00:01:19,464: INFO/MainProcess] Task remote.add[79603d15-24f1-43f8-b8b7-525b7cd4b9a2] succeeded in 0.288741396998s: 7

This is just a simple guide on how to send tasks to remote machines.

Depending on your need, you might have to set up a cluster of servers and route tasks accordingly to scale.

Pandoc: Best Way To Convert Markdown to reStructuredText!

Anand Reddy Pandikunta — Tue, 04 Nov 2014 12:34:00 GMT

I wrote a simple readme to one of my Python package Fadapa. I built the package using setuptools. In setup.py file, for long_description field I just read README.md file and assigned it. After uploading the package to PyPi, I saw the page and found that PyPi doesn't support Markdown format. So I had to change it to reStructuredText.

Pandoc is the best tool available to convert markup formats. It supports a lot of formats and you can convert it to any format you want.

If you have a few documents to convert, you can convert them online. You can also install in in your system if you want to convert a lot files. Instructions on how to install on various operating systems is give here.

For ubuntu you can install by

sudo apt-get install pandoc

Then you can convert files using

pandoc readme.md --from markdown --to rst -s -o readme.rst

This converts readme.md to readme.rst.

Pypandoc is a simple python wrapper for pandoc. Using that also, you can convert files.

Install it using pip

pip install pypandoc

and then to convert files just do

import pypandoc
output = pypandoc.convert('somefile.md', 'rst')

Update: There is an feature request with pull request to add support for Markdown for PyPi.

Start Up Job Boards In India

Anand Reddy Pandikunta — Mon, 03 Nov 2014 10:53:00 GMT

Working with start ups is not only exciting & challenging but its a lot of fun and there are lot of reasons why working with a start up is the best thing you will ever do.

While hunting for a start up job in Bangalore, these are some good job boards I came across.

1. Has Job

2. Your Story Jobs

3. Startup Jobs @ The Morpheus

4. NextBigWhat Job Board

5. HeadStart Network

6. Venture Hire

7. Dream Starts

8. Bangalore Startups

9. f6s Jobs

Django Tips & Tricks #1 - Shell Aliases For Python/Django Developers

Anand Reddy Pandikunta — Sat, 25 Oct 2014 12:57:00 GMT

Developers and hackers prefer using terminal and spend a lot of time on it. Instead of typing long commands over and over, they can be aliased to shortnames. The shell builtin alias allows users to set aliases.

One of the most used command while setting up development environment is pip install -r requirements.txt This can be aliased to pir.

alias pir='pip install -r requirements.txt

Now to install requirements, type pir and pressing enter. Here are some other aliases related to python which will be useful on a daily basis.

alias py='python'
alias ipy='ipython'
alias py3='python3'
alias ipy3='ipython3'

alias jn='jupyter notebook'

alias wo='workon'
alias pf='pip freeze | sort'
alias pi='pip install'
alias pun='pip uninstall'

alias dj="python manage.py"
alias drs="python manage.py runserver"
alias drp="python manage.py runserverplus"
alias drps="python manage.py runserverplus --print-sql"
alias dsh="python manage.py shell"
alias dsp="python manage.py shell_plus"
alias dsps="python manage.py shell_plus --print-sql"
alias dsm="python manage.py schemamigration"
alias dm="python manage.py migrate"
alias dmm="python manage.py makemigrations"
alias ddd="python manage.py dumpdata"
alias dld="python manage.py loaddata"
alias dt="python manage.py test"

Just add the above aliases to your ~/.bashrc or ~/.zshrc. That's it. Hpy alsng!

A Better Way To Take Screenshots Of Movies With Subtitles

Anand Reddy Pandikunta — Fri, 17 Oct 2014 12:45:00 GMT

I take a lot of screenshots(just press SHIFT + S or select Take screenshot from Video drop down menu) when I am watching movies. Few days back I was watching The Matrix and I wanted a take screenshot of this quote.

However the entire quote is present in two different frames. I used to use image editor and join those two images and keep them in my screenshots folder.

Today as I was thinking about a better solution for this. I got this idea of editing subtitles. I opened the subtitles files and edited this part

753
01:06:45,251 --> 01:06:47,085


To deny our own impulses...




754


01:06:47,336 --> 01:06:50,463


... is to deny the very thing


that makes us human.

and changed to this


753


01:06:45,251 --> 01:06:47,085


To deny our own impulses is to deny the very thing


that makes us human.

which made it possible to capture the entire quote in a single frame.

Of course this is not the best method but it is far better than editing and joining couple of images.

I Found A Purse Near Dairy Circle, Bangalore!

Anand Reddy Pandikunta — Sun, 12 Oct 2014 12:03:00 GMT

I was going to Dairy circle yesterday evening. Near to the bus stop I found a purse. A lot of people were standing and passing by but no one noticed it. I just looked around to see if any one is searching for it. But no one was there. I checked the contents to see if any number or any information is available to call the owner of the purse.

Here are the contents of the purse:

1. Bus ticket from Chennai to Bangalore
2. A shop's visiting card
3. A hundred rupee note
4. An old ten rupee note
5. Some change.
6. 3 photos
7. A SIM card

There was number written on the purse. I tried to call that number but it was not reachable and the SIM card also not working.

How To Install Adobe Digital Editions On Ubuntu 14.04?

Anand Reddy Pandikunta — Wed, 08 Oct 2014 11:14:00 GMT

Few days back, I rented an e-book from Kinige, a retailer of telugu print & ebooks. I thought I could read that book immediately on their website. But they offered a .acsm & I had to install ADE(Adobe Digital Editions) to read that book. Since ADE is not available for Linux platform, I had to install wine first and then install ADE from it.

1. Installing Wine:

Wine is a compatibility layer capable of running Windows applications on several POSIX-compliant operating systems, such as Linux, Mac OSX, & BSD. The easiest way to install wine on Ubuntu 14.04 is from its PPA. Open the terminal and run these commands.

sudo add-apt-repository ppa:ubuntu-wine/ppa
sudo apt-get update
sudo apt-get install wine1.7 winetricks

You can check if wine is installed correctly by typing wine --version

2. Install ADE

Once you installed wine, you can easily install ADE. The current version is ADE 4.0 as of today(Oct 9th 2014). But only ADE 1.7 works on wine. So you have to download ADE 1.7. You can download it from here. Once you have downloaded that, you can just double click and install it just like windows.

తెలుగు సాహిత్యం పైన చలం ప్రభావం!

Anand Reddy Pandikunta — Tue, 07 Oct 2014 13:03:00 GMT

చలం వల్ల తెలుగు సాహిత్యం లోను రచయతల్లోను వ్యక్తుల జీవితాల్లోను ఏం మార్పులు జరిగాయో అర్థం చేసుకోవటానికి ఈ ఒక్క పేరా చాలు.

ఈ పేరా రంగనాయకమ్మ గారి చలం సాహిత్యం నుండి తీసుకోబడింది.

Illumina BaseSpace WWDC - Bangalore

Anand Reddy Pandikunta — Tue, 30 Sep 2014 07:10:00 GMT

Illumina¹ develops, manufactures and markets integrated systems for the analysis of genetic variation and biological function. BaseSpace² is Illumina’s genomics computing environment for next-generation sequencing (NGS) data analysis.

To teach developers, how to build and launch your own bioinformatics apps on BaseSpace, Illumina conducted a two day hands-on hackathon. Here is the summary of the event.

Raymond Tecotzky provided a good overview of basespace, plugged in missing information in various situations. Most importanatly super charged participants & the team.

James Hadfield & Ramesh Hariharan delivered good talks about their work on Genomics and their experience with Basespace.

Sucheth Koppa provided a good overview of BaseSpace Onsite & how you can keep all data on-premises.

Mayank Tyagi & his team stressing the importance of Docker, taught how to build apps on Basespace. Hands-on hackathon & very informative.

Along with talks, food, wine, t-shirt & USB Drive(16 GB) are also great.

Using Sourcegraph To Search Real Code!

Anand Reddy Pandikunta — Fri, 19 Sep 2014 13:23:00 GMT

What Is Sourcegraph Anyway?

" Sourcegraph is a code search engine that lets you search across hundreds of thousands of libraries and browse code in the same way you can do in a great IDE. "

Sourcegraph is not the first code search engine ever built, neither it will be last, but it offers is completely different from all other code search engines like Nullege, OpenHUB.

Why Should You Care?

I will explain a few situations where it makes developers life a little bit easier.

1. Search Real Code:

Sourcegraph helps you to quickly search open source projects & see realcode examples. Let's say I am using Celery to schedule some tasks. I know there is chord (a task that only executes after all of the tasks in a taskset has finished executing) but I don't know how to use it. I can use Sourcegraph to find chord, look at its api, check out some real world examples.

Here it doesn't sound Sourcegraph is much useful because you can go through the documentation and you can get all the information about it. But you won't find a lot of real world examples in documentation.

Also if you learning about a new module, it will take a while to wrap your head and get used to that module. What you can do is browse through sourcegraph to understand the api & immediately check out some real code where other programmers have used it for various purposes. I feel this helps a lot to start using a new module and get used to it quickly.

2. Bad Documented Libraries:

There are tons of open source libraries & some of them have very good documentation. There are some good libraries which are documented badly or don't have documentation at all. It's a huge pain to to use a library with bad documentation. Here sourcegraph once again comes to rescue us by quickly navigating around the code.

3. Chrome Extension:

I spend significant time on Github browsing code. Sometimes I clone the repo and browse it on Emacs. But sometimes I just want to browse on Github itself. Sourcegraph has chrome extension which lets you browse code quickly on GitHub by making every identifer a jump-to-definition link. By the way if you are a using it please write a review.

That's if for now. I came across Sourcegraph a couple of weeks back. From then on I am using it almost everyday. Currently it supports only Python, Go, Ruby, Node.js. I hope it will support for other languages soon. What I mentioned here is just a little introduction. Check out ProTips, Sourcegraph docs & srclib(a polyglot code analysis library, built with hackability in mind). If you are a Go programmer, read How do I use Sourcegraph with Go? (a short FREE ebook by Satish Talim).

========================================================================

More Chrome Extensions

Chrome Extensions For Efficient & Productive Web Surfing

Anand Reddy Pandikunta — Mon, 08 Sep 2014 12:20:00 GMT

I do use a lot of chrome extensions. Here are some of my favorite extensions which I use for everyday browsing.

1. AdBlock

Blocks annoying ads all over the web.

2. Vimium

I love browsing web with just keyboard. If you are familiar with Vim, install it without any second thoughts. Vimium provides keyboard shortcuts for navigation and control in the spirit of Vim.

3. Alexa Traffic Rank

If you are a webmaster or stataholic, you need this.

4. Google Quick Scroll

If you have searched for something on Google and landed on a page with lot of text, it helps you jump directly to the relevant bits of a search result.

5. Hover Zoom

Simply enlarge thumbnails by hovering your mouse.

6. HoverReader

While reading an article, if you encounter a hyperlink, peek into it by hovering the mouse.

7. Magic Actions for YouTube

Auto HD, Ad Block, Cinema Mode, Mouse Wheel Volume Control and more magic actions for YouTube.

8. Readable Wikipedia WikiWand

Previously I used Readable Wiikpedia to read wiki articles.
But now I use WikiWand. It provides amazing look for our beloved Wikipedia.

9. AutoPager Chrome

Scroll through paginated web pages.

10. DevTools Auto Save

Saves changes to your CSS and JS right from the browser. Comes in handy for web developers.

11. OneTab:
Pull all tabs into single tab & save up to 95% memory and reduce tab clutter.

12. Popup Blocker Pro:
Blocks all annoying popups and popunders the professional way. You will never see popups anymore. Permissions Visit website

13. Stylebot:
Change the appearance of websites instantly. Preview and install styles created by other users on stylebot.me

14. Tab Number:
Tab number show number on each tab. If you want to jump a particular tab, So you can jump around easily by pressing ALT + pagenum.

15. Thin Scroll Bar:
Thin Scroll Bar to increase screen space for little more visibility.

Use Space As Both Space & Control - Avoid Emacs Pinky!

Anand Reddy Pandikunta — Fri, 29 Aug 2014 11:57:00 GMT

Most of the commands in Emacs start with 'Control' & 'Meta'. Control key is present at the corners of the keyboard, and it's very uncomfortable to press it every-time to invoke a command. Most popular solution to this is to swap CAPS lock & CTRL key. But you have to press the key with Pinky which might cause Emacs Pinky.

A much better solution is to use space bar as control key. When you press the space bar, it will function as a normal space key. If you press it with any other key. it will function as control key. So, to run any commands, you can hold space with one thumb and press other key(s) with another hand which will be handy.

In Linux, we can achieve this with Space2Ctrl.

Install the dependencies, clone the repo, make and start the script.

sudo apt-get install libx11-dev libxtst-dev

git clone https://github.com/r0adrunner/Space2Ctrl

cd Space2Ctrl

make

./s2cctl start

Now you can use your space as space and control key.

If you are using Mac, you can use Karabiner with the following modification.

{
  "title": "Change spacebar to spacebar & control",
  "rules": [
    {
      "description": "SpaceControl",
      "manipulators": [
        {
          "from": {
            "key_code": "spacebar",
            "modifiers": {
              "optional": [
                "any"
              ]
            }
          },
          "to": [
            {
              "key_code": "right_control"
            }
          ],
          "to_if_alone": [
            {
              "key_code": "spacebar"
            }
          ],
          "type": "basic"
        }
      ]
    }
  ]
}

Auto Remove Python Unused Imports & Variables In Emacs

Anand Reddy Pandikunta — Sat, 16 Aug 2014 10:44:00 GMT

Unused code will distract you(and other people) while reading code. Also, a single unused import can cause out of memory error. So, it is considered as a bad practice. Lets try to get rid of unused imports & variables from Python code.

Autoflake removes unused imports and unused variables as reported by pyflakes. Install it\

$ pip install --upgrade autoflake

Now select a file which you want to clean and run this in terminal.

$ autoflake --in-place --remove-unused-variables example.py

It removes all unused imports & variables from the file.

Instead of running this command every time, I wrote a simple elisp function and assigned a F8 key to that function. So while writing the code itself it can clean it & it's much efficient.

;; Py-rm - Remove unused variables & imports from python

(defun pyrm ()
  (interactive)
  (setq command (concatenate 'string "autoflake --in-place --remove-unused-variables " buffer-file-name))
  (shell-command command)
  ;; Reload the modified file
  (revert-buffer t t)
  )

;; set a custom key for pyrm
(global-set-key [f8] 'pyrm)

Put the above code in your emacs configuration file, restart it & press F8 whenever you want to clean the code.

However, if you just need to highlight but not delete them, you can install flymake and then enter this.

M-x flymake-mode RET

It just highlights the other syntax errors and unused imports but won\'t delete them.

Screenshots

Simple Python Code

Unused code highlighted with flymake-mode

Cleaned with autoflake

Amazing Things To Do With Python In Everyday Life!

Anand Reddy Pandikunta — Fri, 15 Aug 2014 05:54:00 GMT

In Python, with a couple of imports and a few lines of code, we can do amazing things.

I am post here a list of amazing scripts/projects that I came across.

VLC With Subtitles: Automatically download and load correct subtitles in VLC.

Youtube-dl: Small command-line program to download videos from YouTube.com and other sites.

2048: 2048 Game with Kivy(Also available on Play Store, iTunes, and for terminal)

Rainbowstream: A smart and nice Twitter client on terminal wrote by Python.

Send Personal Messages: Thank all your friends who wished on your birthday on Facebook.

T: A command-line todo list manager for people that want to finish tasks, not organize them.

qblog: Build & test a blog with Django 1.7 in less than 17 minutes.

couresera: Script for downloading Coursera.org videos and naming them.

sh: Call any shell program as if it were a function.

AutoFlake: Remove unused imports and unused variables as reported by pyflakes.

Django-Extensions: Collection of global custom management extensions for the Django Framework.

Pattern: Scrape & Process Web easily.

PDFMiner: Exact a part(or entire) of text from pdf.

tqdm: Add a progress meter to your loops in a second

onionshare: Securely and anonymously share a file of any size.

q: - Run SQL directly on CSV or TSV files.

iPython Notebook: Combine code execution, text, mathematics, plots and rich media into a single web document & share with others.

How To Get Telugu Books If You Are Out Of Andhra!

Anand Reddy Pandikunta — Wed, 13 Aug 2014 13:05:00 GMT

If you are a (Telugu) bookworm, and you are living out of Andhra, then getting Telugu books is a huge problem. Thankfully a lot of online stores are coming forward to solve this problem. Let's look at a few sites where we can get Telugu print & e-books.

1. Digital Library of India¹: [Free]

Government of India is spending a crazy amounts of money to digitize public domain books. This is the best place to get telugu ebooks for free of cost. Currently there are 23,370 books in Telugu. Eventhough you won't new books there, you can find a lot of great classics.

2. Kinige²:

This is the most popular site(Alexa Traffic Rank: 118,798) for buying Telugu books online. They supply both print books & e-books. The good thing about Kinige is they rent the for a period of 30 days at a reduced price. Most books are available here and is a good place to get started.

3. Logili³:

One more popular (Alexa Traffic Rank: 138,459) website to buy books online. It's also has great collection. However they sell only print books but no e-books. At times when there are no print books on Kinige, this is a great alternative.

4. Flipkart⁴:

Most of you heard of Flipkart. You can't get all telugu books here but a few. If you are a regular shopper at flipkart, you might get available books at discount price.

5. Amazon⁵:

It is struggling & competing with other online stores in India. Here also you can find a reasonable number of telugu books are great discounts.

You can also buy telugu books at Supatha, BooksForYou, InfiBeam, EveningHour.

In addition to these sites, authors or publication houses might be running their own sites ( for example Yandamoori sells his books from his site also ), you can buy from them also.

Update:
AVKF:
Recently I came across AVKF. It also has a great collection of telugu books.

How To Fix Washedout Colors In VLC Media Player?

Anand Reddy Pandikunta — Tue, 12 Aug 2014 08:57:00 GMT

Have you ever played a video on VLC or any other media player and seen this type of washed out colors or distorted video like these??

To get rid of this problem, just open preferences from the Tools menu. Uncheck "Accelerated video output" from the video settings.

If you are using and old version of VLC, you might have an option to "Skip frames", you can select that option. Save your preferences and restart the VLC player. Now you can enjoy your videos. If the problem still persists make sure you have installed all your video drivers.

Django Meetup In Bangalore

Anand Reddy Pandikunta — Sun, 08 Jun 2014 01:46:00 GMT

"Krace played a key role in bringing back the almost dead Bangalore Python Users Group into action."
- Shreyansh Singhal on YourStory.

Few weeks back, while I was searching for Python I came across Krace's blog. Then I checked his Github, Stackoverflow, Quora profiles and I was amazed.

Last week when I found there was a Django users workshop by him & his friends, without a second thought I decided to go.

Since the organizers marked the location (ThoughtWorks Technologies) on Google Maps, its quite easy to reach there w/o any trouble.

Workshop took place from 10 A.M. to 5 P.M. The goal was to build a simple phone book app to add/view/edit contacts with Django & SQLite.

Before starting of workshop, organizers helped with installation of requirements(Django & South).

Krace showed the agenda. At each & every step, he first explained what he is going to do. Then he would do it and gives us time to complete the task. All organizers patiently helped in troubleshooting the errors at each and every step. In the end we had little Q & A session.

Overall it was a great hands on experience with great content & crew! Looking forward for more meetups!! Thank You very much!!!

If you are form Bangalore & love Python, don't miss Django & BangPypers meetup.

P.S: A snap of workshop..

A Weather Widget For Your Terminal!

Anand Reddy Pandikunta — Wed, 04 Jun 2014 06:16:00 GMT

Getting weather info right on the terminal..?? Sounds cool.. Isn't it? With Python, it's even easier to make one! Lets get started.

First we need to determine the location of the user. So let's find out ip address. There are a lot of sites which gives the ip address but this site is the simplest & very easy to get.

Once we have the ip address, we can use it to determine the location and then use any weather api to get the info. However Worldweatheronline api gives weather info directly based on the api, which makes it easier to fetch data.

We can get the data in JSON format and by parsing it we can make the weather widget.

I wrote a simple script for that and here is the sample output of the script.

Definition Of Infatuation From 100% Love Telugu Movie!

Anand Reddy Pandikunta — Mon, 12 May 2014 08:14:00 GMT

According to Freud psychology, Infatuation is only an attraction between male and female and it's a state of being completely carried away by unreasoned passion foolish examined feeling and unappreciated often completely unwarranted emotion.

The Mysterious Wall - Best Zen Story By Osho!

Anand Reddy Pandikunta — Thu, 08 May 2014 21:43:00 GMT

Osho is a great story teller. Whether he is writing a book, preaching his students, giving speeches, he tells interesting short stories. I have read a lot of them from his books. This is one of my favorite stories.

Osho I used present tense above because Osho believes that he is 'Never Born, Never Died. Only Visited this Planet Earth.'

The Mysterious Wall!

There was an ancient mysterious wall which stood at the edge of a village, and whenever anyone climbed the wall to look onto the other side, instead of coming back he or she smiled and would jump to the other side, never to return.

The inhabitants of the village became curious as to what could draw these people to the other side of the wall. After all, their village had all the necessities of living a comfortable life.They made an arrangement to where they would tie a person's feet, so that when he or she looked over and wished to jump, they could be pulled back.

The next time someone tried to climb the wall to see what was on the other side, they chained her feet so that she could not go over. She looked on the other side and was delighted at what she saw, and smiled.Those standing below grew curious to question her and pulled her back, but to their great disappointment she had lost the power of speech.

"Those who have Seen cannot say. That which has been Seen cannot be painted, cannot bereduced to words. But still each one has to give a try - and the world goes on becoming more and more beautiful because of these efforts."

Split/Merge PDF Files From Command Line

Anand Reddy Pandikunta — Wed, 07 May 2014 08:14:00 GMT

Introduction

Sometimes we come across a situation where we need to split or stitch the pdf document. The simplest and easiest way is to use PDF Toolkit on the command line.

Let's first install it using the following command:

$ sudo apt-get install pdftk

Now that pdftk is installed, lets take a smaple pdf file and split it into parts.

Splitting PDF

We can split the entire pdf into single pages using the following command:

$ pdftk myFile.pdf burst

We can also split only specific page by specifying the page number:

$ pdftk myFile.pdf cat 21 output page_21.pdf

You can also split specific set of pages by using the following command:

$ pdftk myFile.pdf cat  21-end output 21_to_last.pdf

Merging PDF

Now that we have scissored a lot of pages. Let's go ahead and stitch a few of them.

We can merge specific pages by using the following command:

$ pdftk page_21.pdf 21_to_end.pdf cat output stitched.pdf

This will stitch the two pdf's we created above.

If we have a huge list of single age pdf's in a folder and if we want to stitch them together, we can run the following command.

$ pdftk *.pdf cat output newFile.pdf

This will stitch all the pdf in the current directory to a single pdf.

Conclusion

That's it. We have successfully split and merged pdf files using pdftk. Hope you find this useful.

How To Use Python & Beautiful Soup To Scrap Web Pages On Ubuntu

Anand Reddy Pandikunta — Mon, 28 Apr 2014 13:43:00 GMT

In this tutorial, You will learn to install Beautiful Soup and parse any web page you like.

Python & Beautiful Soup Installation On Ubuntu 12.04:

1. Open your terminal ( Alt + Ctrl + T ). Install Python & Soup by using these commands.


sudo add-apt-repository ppa:fkrull/deadsnakes

sudo apt-get update

sudo apt-get install python2.7 

sudo apt-get install python-bs4

Scraping A Web Page:

Lets start our programm by importing Beautiful soup. Since we are going to open a web page, we need urllib2 ( This is for Python 2, For Python 3 see urllib.request ). So import that library also. Select any url to parse. I have selected the home page of this blog and opened that page with urlopen(). Pass the web page contents of 'page' variable to beautiful soup. Lets print all the links which are present in this page.
So, the final code is


from bs4 import BeautifulSoup
import urllib2
 
url = "https://www.goodreads.com/quotes/tag/love"
page = urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
 
for anchor in soup.find_all('a'):
    print(anchor.get('href', '/'))

Go ahead and run this file. I have name the file as scrap_web_page.py


python scrap_web_page.py

It will produce all the links in the terminal itself. I have added a screenshot of the output I have received.

anchor.get() is just one method to get all links, you can grab any element or class or name or anything from the page. For complete details, read the documentation of Beautiful Soup.

Custom Key Maps For Vimium To Speed Up Browsing

Anand Reddy Pandikunta — Sat, 26 Apr 2014 08:04:00 GMT

For keyboard ninjas, Vimium is a boon! Vimium made browsing with keyboard a cat walk and ditch the mouse completely! This post is specifically written for those who are good at Vim and installed Vimium on Chrome or Firefox!

Vimium is THE BEST Chrome extension for Keyboard navigation! Since it was inspired by Vim, most of the shortcuts resemble that of vim. In Vim, we can set custom maps, if we are not comfortable with a particular command. The same goes for Vimium also!

By having some custom mapping, we can make surfing the web much easier and productive! Lets look at some custom maps which will come handy in everyday browsing.

0. Characters Used for Link Hints!

By default some characters are set for link hints. But I am more comfortable with the following keys

"sdfercvjkluim". So I set the characters to "fjdkslrueiwvm". But You can choose the key which you are comfortable with and use them. So when you press "f" or "F", characters used to suggest links will be the ones which you are comfortable to type!

1. Navigating Your History!

I navigate back and forth a lot. To back in history, I need to press "H" ( which means SHIFT + h ). So I mapped this 'h'. Actually by default h is mapped to scroll left, which I never use at all. So, to go back 3 pages in history, U just press "3h". The same applies for "L" also.

2. Navigating to New pages

By default "o" is mapped to open URL ( or bookmarks or history ) in current tab and "O" to open them in new tab. Most of the time, I need to open URL in new tabs only so I swapped the above options. The same goes for "b" ( for bookmarks ) also.

You can get a list of all shortcuts at Vimium Github page or if You have installed Vimium, You can just press "?" You can set these custom maps in advanced settings of Vimium

Custom Key Mappings


                        

                            map h goBack

                            map l goForward

                            map o Vomnibar.activateInNewTab

                            map O Vomnibar.activate

                            map b Vomnibar.activateBookmarksInNewTab

                            map B Vomnibar.activateBookmarks

Characters used for Link hints:

fjdkslrueiwvm

With these custom key maps, browsing becomes much productive.

Access Clipboard From Linux Terminal

Anand Reddy Pandikunta — Sun, 20 Apr 2014 14:01:00 GMT

If You are going back and forth between terminal and any other application, accessing system clipboard contents from command line will be invaluable.

If You are using mac, there are inbuilt commands pbcopy & pbpaste. But in Ubuntu these are not available. You need to install a small utility called xclip. Go ahead and install it.

sudo apt-get install xclip

Now, You can copy any text ( or the output one command ) into the clipboard using xclip. To copy contents of fruits.txt to clipboard,

cat fruits.txt | xclip

If You want to see the contents of clipboard, You can use

xclip -o

This copy and paste will work only in the terminal, If You switch to another application and try to paste there, it wont work.

If You want to paste in another application, You need to copy like this

cat fruits.txt | xclip -selection clipboard

Now, You can switch to any other application & You can paste (CTRL + V) the contents.

Tip:

Instead of typing all of this everytime, You can setup alias in .bashrc file

alias c='xclip -selection clipboard'
alias v='xclip -o'

Now You can easily copy the contents like this

cat fruits.txt | c

I found this a lot helpful while switching between browser and terminal.

చలం వ్యాసాలు - ఆనందం (Aanandam - Gudipati VenkataChalam )

Anand Reddy Pandikunta — Sun, 20 Apr 2014 11:16:00 GMT

లోకం ఆనంద మయం. ఆనందం కోసమే ప్రయత్నిస్తుంది. ప్రతి జీవి యొక్క పరమావధి ఆనందమే... పశువులు పక్షులు అన్నిటికి బాధ నుంచి తప్పించుకోవాలని హాయి గా బతకాలని ఒక్కటే ప్రయత్నం. మనుష్యులు హాయి గా బతకటమే కాకుండా కొన్ని విధాలైన ఆనందాన్ని కూడా పొందాలని చూస్తారు. బాధ కావాలని కోరే వారెవరు లేరు. కోరి బాధ లు పడే వాళ్ళూ, సుఖాన్ని త్యజించే వాళ్ళూ లేక పోలేదు. చచ్చిన వాళ్ళను మర్చిపోక వూరికే జ్ఞాపకం చేసుకుని ఏడ్చే వాళ్ళూ , ఇతరుల సౌఖ్యం కోసం తమ ఆనందాన్ని వొదులుకునే వాళ్ళూ, నోట్లోంచి కడ్డీలు దూర్చుకునే వాళ్ళూ , పంచాగ్ను ల మద్య తపస్సు చేసే వాళ్ళూ వున్నారు. కాని , ఆ బాధ వాళ్లకు ఆనందం కనుక లేక ఆనందకరమైనది బాధ వల్ల సమకూరుతుంది అనుకుంటారు కనుక , ఆ బాధ పడతారు.

ఆనందం అనేక రకాలు. ముఖ్యం పంచేంద్రియాలవల్ల, సంపాయించేది. మనసు వల్ల పొందేది.. వున్నతమైన ఆలోచనలూ, సంభాషణా స్నేహమూ ప్రేమా మొదలైన వాటివల్లా సౌందర్యాల వల్ల కలిగే ఆనందం; అందమైన కధలూ, నాటకమూ, సంగీతమూ, బొమ్మలూ మొదలైన వాటివల్ల కలిగే ఆనందం. ఈ ఆనందం లోనే యెక్కువ తక్కువ లు వున్నాయి. తిని నిద్ర పోయి జంతువులవలే బతుకులో పొందే ఆనందం, గంతులేసి నవ్వి అల్లరి గా వుంటే వచ్చే ఆనందం, తృప్తి పడి నా కింకేమీ అక్కర్లేదని శాంతం గా వుండే ఆనందం, లోకమంతా తనకే కావాలనే అధికారాలకీ, ధనార్జనకీ కష్ట పడుతు పొందే ఆనందం, దేశాల్ని, సంఘాల్ని బాగు చెయ్యలనీ, కొత్త విజ్ఞానాన్ని సంపాయించాలని, కొత్త లోకాన్ని కనిపెట్టాలనీ ప్రయత్నిస్తో జీవితాలర్పించే ఆనందం.

ఇవేకాక మనకు తెలీనివీ, ఈ భూలోకానికే చెందనివీ, ఇంకా మనం చూసే ఆకాశానికీ ఇతర గోళాలకు చెందినవీ అనేక విధాలైన ఆనందాలు వుండవచ్చు. చీమ పొందే ఆనందం వుంది మనం పొందే ఆనందం వుంది. భేదం ఇంద్రియ భేదం వలన కలుగుతోంది. దాని కళ్ళు మన కళ్ళు ఒకే కాంతిని చూడవు. చెవులు ఒకటే ధ్వని ని వినవు. అట్లానే మనకే ఇంకా బలమైన చెవులు - కళ్ళు వుంటే, ఆకాశ ధ్వనులు, కాంతులు, -- ఆకులు చేసే రహస్య గీతాలు -- కీటకాల సంభాషణ, -- ఎన్ని వినగలమో, చూడగలమో.

అంతే కాదు ఈ పంచేంద్రియాలు వుండబట్టి ఈ ఆనందాన్ని అనుభవిస్తున్నాము. పది జ్ఞానేంద్రియాలుంటే - పది రకాలు ఈ ఐదూ కాక, ఇంకో విధమైనవి వుంటే - కొంచం ఆలోచిస్తే తెలుస్తుంది - అనంత మైన ఆనందాలు వుండటానికి వీలుందని. భూలోకం లో ఈ విధమైన ఆనందాలున్నాయి; చంద్ర లోకం లో, అనూరాధా లోకం లో, ఆరుద్ర లోకం లో -- చెప్పలేము

ఇవన్ని శరీరాలకు సంభందించిన ఆనందాలు. మతాలు ఇంకో ఆనందాన్ని గురించి చెపుతాయి. ఆ ఆనందాన్ని స్వర్గ లోకం లో పొందుతామంటారు. ఆ ఆనందం రెండు విధాలు. భూలోకపు ఆనందాల వంటివే శరీరానికి సంభందించినవి, రెండోది ఈశ్వరుడి సాన్నిధ్యం వల్ల కానీ, స్తోత్రం వల్ల కాని, లేక ఐక్యం వల్ల కాని, కలిగే ఆనందమూ, ఆ స్వర్గానందాలకు భూఆనందాలకు విరోధం అంటారు. అవి కావాలంటే వీటిని త్యజించాలి. ముఖ్యం గా ఆ స్వర్గం సంగతి తెలిసిన వాళ్ళు కొన్ని నీతులను చట్టాలను ఏర్పరిచారు. ముఖ్యం గా వాటిని అతిక్రమించకుండా సంచరిస్తే ఆ స్వర్గము, ఈశ్వరుడు దొరుకుతారు. కాని , ఆ ప్రకారం చేస్తున్న మనుష్యులెవ్వరు కనబడరు. ఎవరికి అది నిశ్చయం లేదనుకుంటా. ఒకవేళ అందరు వివేక వంతులై ఆ స్వర్గం కోసం క్షణ భంగురమైన ఈ భూలోక ఆనందాలని త్యజించి ఆ శాస్త్ర ప్రకారం ఆ చట్టాలు, ఆ ధర్మాలు అనుష్టిస్తారు అనుకోండి, ఆ స్వర్గం ఏట్లా వున్నా ఈ భూలోకం ధ్వంసమైపోతుంది.

ఇప్పుడున్న తమాషా, నవ్వు, ప్రేమా, స్నేహమూ అన్నీ నశించి ఈ లోకం నివాస యోగ్యం కాకుండా పోతుంది. ఎంత త్వరలో ఈ లోకాన్ని విడిచి ఆ లోకానికి వెళితే అంత వుత్తమం గా తోచాలి ఈ ప్రజలకు. మొత్తానికి ఎవరిని చూసినా ఈశ్వరుడు, ఆయన లోకమూ, దాని మార్గము గురించి మాట్లాడే వాళ్ళే కాని, ఈ లోకాన్ని విడిచి ఆ లోకానికి, దాని ఆనందాల కోసం త్వర గా ప్రయాణమవుతున్న వాళ్ళు ఎవ్వరు కనిపించరు.

ప్రతి వాళ్ళు బాధ నుంచి తప్పించుకుని ఆనందాన్ని పొందుదామని చూస్తారుగాని ఆ నేర్పు కలిగి ఆనందాన్ని పొందగలిగే వాళ్ళు కొద్దిమందే కనపడతారు. మన "ఇన్ స్టింక్ట్" ఆనందం కోసం బలీయం గా వుంది. తనకు బాధ కలిగించే పనుల నుంచి మన శరీరం తనంతట తానే తప్పుకుంటుంది. కాని మన మనసుకు మాత్రం ఇంకా ఆ "ఇన్ స్టింక్ట్" రాలేదు. జ్ఞానము , నేర్పు ఇంకా సంపాయించలేదు. ఏది తమకు ఆనందం ఇస్తుందో, బాధల నుంచి ఎట్లా తప్పుకోగలరో మనుష్యులు ఇంకా నేర్చుకోలేదు, కనుకనే ఎవడు వచ్చి "ఆనందం" అని కేకలు వేసినా వాడి వెంట పరుగెత్తుతారు.

ఆనందం విషయమై కొన్ని నిబంధనలు కనపడుతున్నాయి.
నాకు ఆనందం కావాలి, నేను ఆనందం అనుభవించాలి అని ప్రయత్నిస్తే వొచ్చేట్లు కనపడదు. ఒక కార్యం ద్వారానే కలుగుతుంది ఆనందం. మనసు ఆ కార్యం మీద వుండాలికాని ఆనందం మీద వుంటే ఆనందం చెదిరిపోతుంది. తీపి కావలన్న వాడూ తీపి కోసం ఎక్కడన్నా వెతుకుతుందా? తీపి నివ్వగల చెరుకు కోసం వెతకాలి. అట్లానే ఆనందం కావాలిస్తే ఒంటరిగా కూచుని ఎవరికి లేకుండా కొంత ఆనందంసంపాయించుకోవాలంటే అది క్షుద్రరూపాల ప్రసన్న మవుతుంది. ఒక్క నిమిషము లో నశిస్తుంది. మనం జీవిస్తున్నాం, మనకు ఆనందం కావాలి అనే ధ్యాస లేకుండా జీవితాన్ని గొప్ప వుద్యమాల్లో, లోక క్షేమానికి ప్రపంచానందానికి చేసే ఘన ప్రయత్నాలలో ఐక్యమయ్యే మనిషి పొందే ఆనందాన్ని స్వార్ధపరులు పొందనట్లు తోస్తుంది. మొదటి రకపు ఆనందం స్వభావమే వుత్తమమైనది గా తోస్తుంది.. ఒక ఆనందమైన కధనుగాని, బొమ్మను గాని సృష్టించే ఆనందం తలుచుకోండి...

ఈ ఆనందం ఎవ్వరికి ఇవ్వను; నేనే దాచుకుంటాను అన్న నిమిషాన ఆ ఆనందం మాయ మౌతుంది . చెట్టునున్న పువ్వును జేబులో దాచుకున్నట్లు, ఎవరెత్తుకుపోతారో అని భార్యలను దాచుకునే వాళ్ళు, ఈ విషయమై కొంచం ఆలోచిస్తే బాగుంటుంది.... చక్క గా యోచిస్తే ఇతరులనుంచి దాచుకున్నామన్న తృప్తి తప్ప ఏ మాత్రమూ వాళ్ళు ఆనందాన్ని పొందలేరు.

తనే ఆనందం పొందాలనే కార్యాలకన్నా ఇతరులకు ఆనందమిద్దామనే వూహ తో చేసే కార్యాలు ఎక్కువ ఆనందాన్ని ఇస్తాయి. తిండి తినటానికి సంగీతం పాడటానికి భేదం ఆలోచించండి. అందువల్లనే ప్రేమ వలన వచ్చే ఆనందం అద్వితీయమైనది. ఎందుకంటే ప్రేమ వున్నప్పుడు తనకి ఎంత ఆనందం వస్తుందనే ధ్యాస వుండదు. ఎంతసేపు ఇతరులకు ఎంత ఆనందం ఇవ్వగలమనే యోచనే బలం గా వుంటుంది..

ప్రపంచమంతా ఆనందం పొంగి పొర్లి పోతోంది. దాన్ని అందుకోగలిగిన హృదయాలు వుండాలి. లోకం అంతా కాంతి వుంది. కాని కన్ను వుంటే కాని ఆ కాంతి అర్ధం కాదు. ఎంత శక్తి కలిగిన కన్ను వుంటే అంత కాంతి వుపయోగపడుతుంది. లోకమంతా శక్తి నిండి వుంది. ఆ శక్తి ని వుపయోగపరచుకునే యంత్రాన్ని బట్టి ఆ శక్తి వ్యక్తమవుతుంది. అట్లానే ఆనందం. ఆకాశం, సముద్రం, గాలి, ఇసిక, స్నేహం, తోటలు, నదులు,, కీటకాలు, పసిపిల్లలు, నవ్వు అన్నీ ఆనందమే. తెలుసుకునే హృదయం వుండాలి. ఆ హృదయానికి ఎంత శక్తి వుంటే అంత ఆనందాన్ని తీసుకోగలదు.

వెన్నెల అందరికి కాస్తుంది. వెన్నెల రాకుండా కిటికీలు మూసుకునే వాళ్ళు వున్నారు. వెన్నెల చాలదని ఎలక్ట్రిక్ లైట్లు పెట్టుకునే వాళ్ళు వున్నారు. వెన్నెలలోని ఆనందాన్ని భరించలేక గీతాల్లోకి ఆ అనందాన్ని పొల్లేట్లు చేసే వాళ్ళు వున్నారు. ఆ ఆనందం అతీతమై వాళ్ళకే తెలియని పిచ్చి బాధ లో పడిపోయే ఆత్మ లూ వున్నాయి.

ప్రతి జీవికి కొంత శక్తి వుంది. కొంత కాలం వుంది. ఈ శక్తినీ కాలాన్నీ ఆనందం లో కూర్చుకునే తెలివితేటలు లేవు. జీవితం "ఆర్ట్". ఒక కళ. మనం వివేకవంతులమైతే మన శక్తి నంతా ఆనందం గా మార్చుకోగలం. మన కాలమంతా మనకు ఆనందాన్ని ఇచ్చేట్లు చేయగలం. ఈ మహా శక్తి మన పుటకవల్లనో సంస్కారం వల్లనో కొంత కలుగుతుంది. కొంత మనకు చిన్నప్పుడు ఇతరులు నేర్పిన "అలవాట్ల" వల్లా పెద్దయ్యి మనకు మనమిచ్చుకునే శిక్షణ వల్లా యేర్పడుతుంది.

లోకం లో మొత్తానికి ఎవరు ఆనందవంతులుగా కనపడుతున్నరంటే, వారి స్వభావాలకీ, గుణాలకి, వేటి వేటి మధ్య వున్నారో ఆ పరిస్తితులకు; సమత్వం కలిగించుకున్న వాళ్ళు.. చాపలు ఎంత మంచి మందిరాలలో వుంటే యేమి లాభం? ఏనుగులు సముద్రాల మధ్య తెల్లని ఇసుకల్లో పడుకుంటే ఏం సౌఖ్యం? తక్కువ స్థితిలో వున్నా, ఎక్కువ స్థితి లో వున్నా సరిపోవాలి. లోకానికి వ్యక్తి కి సంధి కుదరాలి. వ్యక్తి లోని శక్తులన్నీ చక్క గా వినియోగమయ్యే పరిస్తితి వుండాలి. చిత్రకారుణ్ణి చిత్రించ వద్దని ఆజ్ఞ పెట్టి మహారాజు గా చేసినా అతనికి తృప్తి వుండదు. తన కళ కోసం బాధ లోనే వుంటాడు.

లోకం లో అనేక విధాలైన ప్రాణాలున్నాయి. కంటి కగపడని సూక్ష్మ జీవుల నుంచి మనుష్యుల వరకు -- అనేక స్తితులలో అనేక ప్లేన్ ల లో బ్రతికేవి వున్నాయి. వేటి గుణాలు,ఆకారం,కష్టాలు, ఆనందం,ఇంద్రియాలు, బుద్ది వాటివే. కీటకాల్లో అనేక రకాలు. పక్ష్యులలో - చేపల్లో - జంతువుల్లో అనేక బేధాలు వున్నాయి. అట్లానే మనుష్యులలో వున్నాయి. దేశం వల్ల, రేస్ వల్లా పరిస్థితుల వల్ల కలిగే బేధాలు కాక మనిషికి మనిషికి ముఖం లో -- కంఠం లో దేహాకారం లో... రేఖలలో, రుచులలో, గుణాలలో బేధాలున్నాయి. ఈ బేధాల్ని మత ప్రవక్త లు నీతి ప్రవక్త లు ఒప్పుకోరు. సర్వ మానవులకు ఒకటే మత విశ్వాసం, ఒకటే నీతి, చట్టము విధిస్తారు. ఒకరికి కుదిరిన మందు ఇంకొకరికి కుదురుతుందని, ఒకరికి ఆనందమిచ్చిన విషయం ఇంకొకరికి ఆనందమిచ్చి తీరాలని మూఢాభిప్రాయం ప్రజలనికా వదల లేదు. అసంఖ్యాకాలైన ఆనందాలిని, అభిరుచులను, సౌందర్యాలను ఏర్పరిచిన సృష్టి కొల్లబోలేదు.... వాటినన్నిటిని వివిధ రీతుల అనుభవించేందుకే వివిధాలైన స్వభావాలను సృష్టించింది. ఏకత్వం, సమానత్వం సృష్టి సూత్రానికే విరుద్ధం.

ఒక కాలం లో దేశాలకీ, సంఘాలికి సరిపడిన నీతి, ఆచారం ఇంకో కాలం లో కూడా సరిపడి తీరాలని, ఒక దేశానికి జాతికి అనుకూలించిన పరిస్తితులు ఇంకో దేశానికి జాతికి అనుకూలించాలని వాదించే వారు అనేకులు. ఎట్లా మనిషి మనిషి కు రూపము, బుద్ది, మారుతుందో అట్లానే వారి స్వభావాన్ని, సంస్కారాన్నీ, బుద్ధి ని బట్టి నీతి ఆనందం మార తాయంటే అంగీకరించరు. థియరీ లో అంగీకరించినా ఆచరణ లో ఒప్పుకోరు.

ఈ మత సిద్ధాంతులు, నీతి ప్రవక్త లు,ప్రతి వారమూ సాయింత్రమూ వేదిక నించి, రోడ్డు మూలల నుంచి వుపన్యాసాలిచ్చే దేవ భృత్యులూ ఎవరికి తోచినట్లు ఆనందపడే విధాలు ప్రకటిస్తూ వుంటారు.

"సత్యం చెప్పండి బాధలన్నీ పోతాయి" " నా గీతాలను చదవండి", క్రీస్తు ను నమ్మండి", "యోగం చెయండి", "తొట్టి వైద్యం", కార్కు టిప్పు సిగిరెట్ట్లు," , " చచ్చిన వారితో సంభాషణ" " అపక్వాహారం", "సర్వ భాతృత్వం" --తలనూనెలు - స్వరాజ్యం, -- లైబ్రరీలు, " ఇట్లాంటి వాటి ద్వారా వేనవేలు చిరతరానందాన్ని ప్రపంచాలకు ప్రకటిస్తున్నారు.

ఇవన్నీ కూడా ఆనందాన్ని ఇచ్చే మాట నిజమే కాని కొన్ని కొన్ని కొందరికి మాత్రమే ఇస్తాయి. కాకి, చచ్చిన ఎలకను తింటే ఆనందమని ఏనుక్కి చెప్పి, చేపలా నీటిలో ఈదమని నక్క కు చెప్పినట్లు వుంటుంది. ప్రతి వాడు తన స్వభావానికి, తన గుణానికి ఏ ఆనందం కావాలో అవి కనుక్కోవాలి. ఆ స్వభావం ఆనందం వైపు ఈడుస్తుంది. కాని ఈ ప్రకటనలు చూసి మోస పోతారు. ఈ ప్రకటనలలో ముఖ్య మైనవి శాస్త్రాలు,, నీతులు, ఆచారం, ధనం, అధికారం మొదలైన భ్రాంతులు. ఒక యోగి వచ్చి ఆసనం చూపగానే అందరూ ఆసనాలు ప్రారంభిస్తారు. పండితులు వచ్చి జుట్టు గొరిగించమనగానే గొరిగిస్తారు. శాంతి లేక కొట్టూకునే ధనవంతుడు భార్య కావాలని యేడిచి పెళ్ళి చేసుకుని "ఎందుకు చేసుకున్నానని" యేడుస్తాడు. పిల్లలు లేరని ఏడ్చే వాళ్ళు,, వుద్యోగం చేస్తూ ఎన్నడు నవ్వని దురదృష్ట వంతుడు, ప్లీడరై కవుల వెంట పరుగెత్తే రసికుడు, పెళ్ళి చేసుకుని నమ్మకం గా నిలువలేని స్త్రీ, చదువుకుని ఆరోగ్యం పోగొట్టుకున్న రోగి,
కధలు చదివి, సినిమాలు చూసి భ్రమసి ఇల్లు వదిలి పరుగెత్తి తిరిగి వొచ్చిన జోగి, -- అందరూ, ఇట్లాంటి మూర్ఖులు - తమ ఆనందం తెలీక మాటలు నమ్మి మోసపోయిన ధౌర్భాగ్యులు,. తనకు లేని వస్తువు ఆనందం ఇస్తుందనుకోవటం ఈ భ్రమలలో ముఖ్య మైనది.

"ఈ పెళ్ళి నరకం " రా అని ఎంత మంది ఎంత చెప్పినా బాల బ్రహ్మచారి ససేమిరా నమ్మడు. " నీ పెళ్ళాన్ని వదలవేం?" అంటాడు. "ధనం వల్ల సౌఖ్యం లేదు" అని ధనికుడెంత అరిచినా బీదవాడు నమ్మడు. "ధనమంతా నాకియ్యవే" --అంటాడు. ఏ ఆనందం రాకపోయినా ఆ పెళ్ళాన్ని, ధనాన్ని వొదిలే ఆత్మబలం వాళ్ళకు లేదనే సంగతి వీళ్ళకు తెలీదు.

కొందరికి దేశ విప్లవం, కొందరికి తిండి, కొందరికి ప్రేమ, కవిత్వం, తగాదాలు, - నిద్ర - ఇట్లా అనేక ఆనందాలు వున్నాయి. తత్వాన్ని బట్టీ మారుతో, ఎవరు చెప్పిన మాట వినకుండా, తన స్వభావాన్ని ఇన్ స్టింక్ట్ ను నమ్ముకుంటే, ఏ మనిషికి అతని ఆనందమేదో స్పృష్టమవుతుంది. ఏ మృగానికి దాని ఆహారమేదో అర్ధమైనట్లు... కాని ఈ మనుష్యులలో జన్మమంతా మాంసం తినే ఆవులు, గడ్డి మేసే సింహాలూ వున్నాయి. -- పైగా అజీర్నమెందుకా అని ఆవు అనేక రకాల మాసం రుచి చూస్తుంది. సిం హం రక రకాల గడ్డి తెప్పించుకుని మేస్తుంది. కాని ఆ గడ్డిని ఇది ఆ మాంసాన్ని అది వొదిలే తెలివితేటలు కాని, ధైర్యం కాని, శక్తి కాని వాటికి వుండవు.

కవిత్వం చదివి " ఆహా సూర్యాస్తమయం యెంత అందమైనది ! నేను చూడనే లేదే " అని ఒకడు సాయింత్ర్రం గోదావరి గట్టున కూర్చుంటే ఆ ఆనందం వొస్తుందా? ఆవడలో, ఇంటి తగాదాలో, సినిమా కథలో, ఏదో తలుచుకుంటూ కూచుంటాడు అంత సేపు...!

అట్లానే కొందరికి త్యాగం ఆనందమిస్తుంది -- కొన్ని పరిస్తుతులలో. కాని ప్రతి వుపన్యాసకుడు, గ్రంధకర్తా త్యాగం చెయ్యమనే వాడే -- కవిత్వానికని, సత్రానికని, కొడుకు పెళ్ళికని, దేశానికనీ, పత్రిక్కి చందా కని త్యాగం రాదు. ఏ స్వభావానికి ఏ త్యాగం ఆనందమిస్తుందో అదే అవసరం, ఆరోగ్యమూ... కొందరికి అసలు త్యాగమే పనికి రాదు. చాలా మంది త్యాగాలు ఏదుస్తూ చేస్తారు. మతము ధర్మము డ్యూటి అని పేర్లు పెట్టి త్యాగాలు చేస్తారు. సాధారణం గా తల్లులు భార్యలు చేసే త్యాగాలు ఇట్లాంటివే. ఎవరికే త్యాగం అవసరమో వళ్ళ ఆనందమే నిర్ణయించాలి. ఒక స్త్రీ తన భర్త ను వదిలి పిల్లలను వదిలి పనికిమాలిన ప్రియుడి వెంట వెళుతుంది. ఇంకో స్త్రీ పనికి మాలిన భర్త కోసం ప్రియుణ్ణీ లోకాన్నీ, పిల్లలనీ వదులుకుంటుంది. ఇంకో ఆమె బిడ్డ కోసం భర్త ని, సంఘాన్ని, నీతి ని వదులుతుంది.. వీటిల్లో సంఘానికి నచ్చినవి మెచ్చుకుంటారు. తక్కినవాటిని ఖండిస్తారు. కాని ఏది ధర్మమో, ఏది ఆనందాన్ని ఇస్తుందో ఎవరికి వారు నిర్ణయించుకోవలసిందే కాని మనుష్యులు కాని శాశ్త్రాలు కాని నిర్ణయించలేవు... ఎందుకంటే ఆనందమివ్వని ధర్మం నీ ధర్మం కాదు, ఇంకొకడిదీ,

దేశానికి ప్రతివాడు త్యాగం చెయ్యాలని అంటారు. అక్కర్లేదు అంటే తిడతారు. ఎవరి హృదయం దేశభక్తి తో పరవశమౌతుందో వాళ్ళు సర్వము త్యాగం చెయ్యనే చేస్తారు. ఇది మన ధర్మం కనక చేయాలి అనే ప్రసక్తే వుండదు... "ఇది నా ధర్మం" అనుకునేప్పటికే ఇతరులు విధించిన ధర్మమని వ్యక్తమవుతుంటుంది... అట్లా దేశభక్తి వల్ల అనందం రాని వాళ్ళు ధర్మమని కీర్తికని, పక్కవాడు చేసేడని, చాలామంది చేసేరు... త్వరలోనే చింతించారు... వారి స్వభావం ఆ త్యాగానికి తగినది కాదు గనుక.

సుఖము బాధ పరిస్తితులు తెచ్చి పెడుతు వుంటాయి, చాలా మంది ఏమి ఆలోచించకుండా అనుభవిస్తూ బతుకుతారు. తాము ఎంత వరకు ఆ బాధలకు కారణం, తప్పించుకోవటానికి తమ ప్రయత్నమెంతవరకు వుపయోగపడుతుందో, యోచించరు. కర్మ లోనూ, జ్యోతిష్కం లోను నమ్మకం ఈ నిద్ర కు తోడ్పడి, జోల పాడుతుంది. జీవితం లో సంతోషం లేనప్పుడు కొత్త సంతోషము కల్పించుకోగలం. బాధలు తటస్తించినప్పుడు నిగ్రహించుకోగలం. బాధను ఎదిరించి పోట్లాడితే చాలా వరకు లోబడుతుంది. కాని ఆ జ్ఞానము శక్తీ చాలా కొద్ది మందికే వున్నాయి.

లోకం సౌందర్యాన్ని కల్పిస్తుంది., కాని అనుభవించమని బలవంతపెట్టగలదా? అట్లానే జీవితం కష్టాలను కల్పిస్తుంది, కాని అనుభవించమని బలవంత పెట్టలేదు.

వెయ్యకు తొమ్మిదివందల తొంభైమందికి ప్రత్యేకం గా ఏదో అందరాని ఆనందాన్ని సాధించాలనే ఆర్జి వుండదు. ఇంకా మృగాల స్తితి లోనే వున్నారు. సుఖం గా తినటమూ, కనడమూ , చావడమూ , వాళ్ళు చేయవలసిందల్లా. అదే వాళ్ళ ఆనందము. కాని ఆ సంగతి గమనించరు. ఘన కార్యాలు చేసిన వాళ్ళ గురించి విని కని, చదివీ, తమకు లేని శక్తులు వున్నయనుకుని అనుసరించ పోతారు. ప్రయత్నించి విఫలులైన బోసులూ, శాండో లు, శుక మహర్షులు, రవీంద్రులు, I.C.S లు ఎందరో వున్నారు మన మధ్య. నరకం, స్వర్గం సంగతులు విని, భయపడి, ఆశపడి భక్తి ని, వైరాగ్యాన్ని, నటిస్తారు. దాని వల్ల రోగాలు బాధలు అప్పులు ఇన్ని పడతారు. ఎంత తన్నుకున్న శాస్త్రాలు ఎంత ఘోషించినా ఎన్ని తత్వాలు పాడినా అందరు విరాగులు కాదు, ధర్మాత్ములు కారు. ఆ పాడే వాళ్ళకు తెలుసు "అన్నళ్ళీ ముచ్చటలు - తనువులు శాశ్వతమా" అని పాడి విరక్తి పుట్టిస్తారు. కాని ఒక్క గుప్పెడు బియ్యం మీదో, ఒక్క కానీ మీదో మనకు విరక్తి కలిగితే చాలు ఆ పూటకి, వాడికి తృప్తి. తమ జీవితం లో ఆనందం ఎట్లా కలుగుతుందో తెలిసినా, ఆ ఆనందం కోసమే ప్రయత్నం చేసి ఆటంకాలను వదులుచుకునే వాళ్ళూ చాలా కొద్ది మంది. ప్రతి పని విషయమై కూడా 'ఇది నా ఆత్మ కు ఆరోగ్యమా కాదా? నాకు లభ్యమైన కాలాన్ని, శక్తి ని, ధనాన్ని ఆనంద రూపం గా మారుస్తున్నానా లేదా?' అని విమర్శించరు. మొహమాటం, ప్రతిష్టా, భేషజం, కపటమూ ఇన్నీ అడ్డం వస్తాయి.

వొంటి నిండా చీము తో, చెయ్యి ఎత్తలేక వొణికే వాడికి కానీ ఇవ్వము. మనకి అక్కర్లేని పత్రికకు చందాగా అప్పుడే ఇంకొకడు నాలుగు రూపాయిలు పట్టుకుని పోతాడు.

మనం మీటింగులకు ప్రెసిడెంట్లు గా వుండటం, అనుష్టించే కర్మ కలాపాలు, తగాదాలు, అన్నీ ఏ మాత్రమైనా ఆరోగ్యాన్ని ఇస్తాయా? వుత్తమ లోకం లో నమ్మకం లేని వాడు బ్రాహ్మల కాళ్ళు కడిగి తద్దినం పెడతాడు. బ్రాహ్మణ్ణి తిట్టి కమ్మ బ్రాహ్మణ్ణి పూచ్చేసి దక్షిణలిస్తాడు.

మన వేషం, బట్టలు, జుట్టు, తిండి, మనం పెట్టే భోజనాలు, ఆడే మాటలు, అన్నీ ఏమీ ఆనందం ఇవ్వని శుష్కమైన ప్రదర్శనాలు --ఎన్ని!
జీవనమంతా వాటితోనే వృధాగా గడిచిపోతుంది. మనకు ఆనందం ఇచ్చే వాటిని అనుభవించటానికి జంకుతాము. ఎవరు చూస్తున్నారో అనే భయం తో, చక్కని మనిషి నడుస్తో వుంటే తేరి చూడ్డానికి భయం, భోగమాట చూడ్డానికి భయం, బట్టలు లేని బొమ్మలు, వెంకటాచలం కధ లు ఇవన్నీ రహస్యం గా ఆనందిస్తాము.

ఇట్లా ఏళ్ళకు ఏళ్ళు గడిచి పోతాయి. మనకు ఆనందం వుంది అని ఇతరులు అనుకోవటానికి ఆనందాన్ని ధార పోస్తాం. మనం కట్టే డాబైన ఇల్లు, మోసే బంగారు నగలు, చేసే గొప్ప పెళ్ళిళ్ళు, ఖర్చులు, కష్టాలు, ఏడిపించుకు తినే అల్లుళ్ళు, అన్నీ ఒక రవ్వ ఆనందాన్ని ఇవ్వవు. కొడుకుని ద్వేషిస్తాము; వొదిలే ధైర్యం లేదు, భార్యలు భర్తలను, భర్తలు భార్యల్నీ,వొదిలే ధైర్యం లేదు. ఇష్టం లేని వాడు వొస్తే వొద్దని చెప్పే ధైర్యం లేదు.

నీ జీవితం దేనికి, ఏ ఆనందానికి ఏర్పడిందో చూడు. ఆ ఆనందం నీతి గాని కాక పోని, ధర్మం కానీ, అధర్మం కానీ, వున్నతం కానీ కాక పోనీ .... అన్నిటినీ -- బంధువుల్నీ, కులాన్నీ, నీతిని, చివరకు నీ సౌఖ్యాన్ని, అన్నిటిని త్యజించి అవసరమైతే ఆ అధర్మాన్ని ఆశ్రయించు. ఆ ప్రయత్నమే, ఆ కష్టమే, ఆ రాపిడే ఆత్మకు ఆనందాన్ని ఇస్తుంది. నీ ఆత్మ లో కొత్త విస్తీర్ణం, నీ కళ్ళ ముందు కొత్త లోకాలు వెలుగుతాయి. తప్పు కానీ, ఒప్పు కాని వెనుక ముందు చూడకు. తప్పైతే దిద్దుకోవటానికి చాలా కాలముంది. వెనుకముందులు యోచించే చచ్చు బతుకు కంటే ధీరత్వంతో ముందుకు సాగి గోతిలో పడేవాడికి ఎక్కువ ఆనందముంది ఈ లోకం లో.

ఈ ఆనందాల్లో డిగ్రీలున్నాయి. పేడ తింటో సుఖించే పురుగూ, కాంతిలో ఎగురుతు అరిచే చిలుకా, డబ్బు లెక్క పెడుతో తన్మయత్వం లో పడే పిసినారి, దేశం కోసం గుండు దెబ్బ తిని చస్తో ధన్యుణ్ణనుకున్న యోధుడూ, స్త్రీ బొమ్మ ను చెక్కి తను కల్పించిన ఆనందాన్ని తాను చూసి మూర్చిల్లే శిల్పీ, అనంత విశ్వం లో తన ఆత్మ ను కలిపి సర్వ జీవుల సుఖ దుఖాలు తనలో అనుభవించగల యోగీ, అందరూ ఆనందాన్నే అనుభవిస్తున్నారు. కానీ అన్నీ ఒకటే డిగ్రీలోవి కావు. లోకం చూసిన కొద్దీ ఈ అనుభవం విషయం లో కూడా 'లైఫ్' లో గొప్ప 'ఎవల్యూషన్ ' కలుగుతున్నట్లు తోస్తుంది.

కొత్త ఆనందాలను సగం సగం తోచి తామందుకోలేని సౌందర్యాలనూ, అనుభవించే విధం తెలీక, తపన పడి, జీవులు తమ ఇంద్రియ శక్తి ని మార్చుకోవాలని తమకు తెలీకుండానే ప్రయత్నించి ఒక రూపం నుంచి ఇంకో రూపం పొందుతున్నట్లు తోస్తుంది. రెక్కల పురుగు ఇంకా విస్తీర్ణత ను కోరి కోరి చిలుక కావొచ్చు. పిల్లి ఇంకా బలాన్ని గభీరాన్నీ కోరి సిమ్హం కావొచ్చును. అట్లాగే ఒకడు తనలోలేని శక్తుల్నీ కోరి కోరి ఏ జన్మలోనైతేనేం, ఏ లోకంలోనైతేనేం, పాటకుడూ, వస్తాదూ, యోగీ, కిన్నరుడూ లేక రాక్షసుడుగా మార వచ్చు. అట్లాంటి మార్పు రాక పోతే, నా హృదయం లో చూచాయ గా తోచే ఈ వున్నతానందాలూ, నేను కలలు గనే ఈ లోకాలకి సంబంధించని సౌందర్యాలు, నేను పొందాలని కోరే మానవాతీత శక్తులూ ఇవన్నీ అర్ధ విహీన మవుతాయి. భోజనమూ, ఆరోగ్యమూ, ధనమూ వుండి కూడా ఆత్మలు ఎందుకిట్లా తమకే తెలీని ఆరాటాలతో బాధ పడాలి? పాకే బిడ్డ ఎన్ని సార్లు పడి దెబ్బ తిన్నా నడవాలని ఎందుకు ప్రయత్నం చెయ్యాలి? అదేననుకుంటా, ఈ సృష్టి సూత్రం. మనం అందుకోగలిగి నంతవరకు ఈ అభివృద్ధి ఒక్క జీవితం లోనే నిశ్చయం గా పొందగలుగుదుమనుకునే వాళ్ళు మూర్ఖులు, జన్మలనేవి వుంటే, ఎప్పటికో సాధించవలసిందే. ఏమైనా, ఏ ఆనందానికిగాని చేసే ప్రయత్నం కూడా ఆనందమివ్వాలి. ఇవ్వకపోతే నీ ఆనందం అది కాదు.

"నీకింకేం కావాలి, ఎందుకట్లా వెతుకుతా?" వంటారు మిత్రులు.

'కారణం లేని ఈ ఆరాటం -- ఈ సృష్టికి అర్ధమేమిటి, నీతి ఏమిటి, పాపమేమిటి, నేను నమ్మేవి చేసేవి సత్యమా -- అనే మీమాంస, కొత్త సౌందర్యాలకోసం, ఎండమావుల కోసం వలే, రెప్పలార్చుకుంటో పరుగులు: మనకు కానిది, మనం అందుకోలేనిది, మనం మిస్ అయింది, ఎంతో లోకం, ఎంతో జీవితం, కాలం, అందం వృధా పోతుందనే దిగులు -- ఇవేమి లేకుండా తక్కిన వారివలే బతికి కూచోకూడదా ?' అని అడుగుతుంది శ్రమపడ్డ ఆత్మ.

కాని ఏమి లాభం? ఎందుకా విధం గా ఆ రెక్కలు కొట్టుకుంటో పరుగెత్తుతావు? నేను నీకు పళ్ళు పెడతాను, ఆడ చిలుక ను తెస్తాను, నా ఇంట్లో వుండమని చిలక నడిగి ఏమి లాభం... ???

More stories & essays.... నాకు నచ్చిన చలం కథలు, వ్యాసాలు

4 Delightful Short Zen Stories About Women & Love!

Anand Reddy Pandikunta — Mon, 31 Mar 2014 07:27:00 GMT

Reading stories is my all time favorite hobbies. I read a lot of short stories and I am sharing a few best Zen stories I have been reading recently.

1. Carrying A Woman!

Tanzan and Ekido were once travelling together down a muddy road. A heavy rain was still falling.

Coming around a bend, they met a lovely girl in a silk kimono and sash, unable to cross the intersection.

"Come on, girl," said Tanzan at once. Lifting her in his arms, he carried her over the mud.

Ekido did not speak again until that night when they reached a lodging temple. Then he no longer could restrain himself.

"We monks don't do near females," he told Tanzan, "especially not young and lovely
ones. It is dangerous. Why did you do that?"

"I left the girl there," said Tanzan. "Are you still carrying her?"

2. If You Love, Love Openly

Twenty monks and one nun, who was named Eshun, were practicing meditation with a certain Zen master.

Eshun was very pretty even though her head was shaved and her dress plain. Several monks secretly fell in love with her. One of them wrote her a love letter, insisting upon a private meeting.

Eshun did not reply. The following day the master gave a lecture to the group, and when it was over, Eshun arose. Addressing the one who had written her, she said: "If you really love me so much, come and embrace me now."

3. Too Much Love

An aged monk, who had lived a long and active life, was assigned a chaplain's role at an academy for girls. In discussion groups he often found that the subject of love became a central topic.

This comprised his warning to the young women: "Understand the danger of anything-too-much in your lives. Too much anger in combat can lead to recklessness and death. Too much ador in religious beliefs can lead to close mindedness and persecution. Too much passion in love creates dream images of the beloved - images that ultimately prove false and generate anger. To love too much is to lick honey from the point of a knife."

"But as a celebate monk," asked one young woman, "how can you know of love between a man and a woman?"

"Sometime, dear children," replied the old teacher, "I will tell you why I became a monk."

4. Change!

A Chinese emperor known for his foul temper entered the bedroom of his soon-to-be-bride, who was one of the most beautiful women in all of China. She was being made to marry him against her will, as her parents were forcing her into it.

Little did the emperor know however that she had also been taught by wise sages as a child. She sat expressionless, staring at the wall. "Hello, pretty," he said to her, but she didn't respond.

"I said hello to you, and you will respond when I address you, do you understand me??" he snarled. But still, she didn't reply.

Most people would have answered him by now, so despite himself he grew curious, and gruffly asked, "What is it you are thinking?"

Finally she answered him. "Two things. One, that I do not wish to marry you because you are so callous and mean-spirited. And the other thing, is that I was wondering if you have it within your power to have a certain something changed."

"What?!" the emperor exclaimed with outrage. "You bitch! How dare you question my authority! ... But ... I admit I'm curious. Since I have it within my power to snap my fingers and whatever I command within my kingdom will be obeyed, what is it you are wondering if I could change?"

"Your attitude," she replied. And with that she got up and walked out of the room, leaving him in stunned silence.

2048 Game - Tips To Make a 2048 Tile Easily & Quickly

Anand Reddy Pandikunta — Fri, 28 Mar 2014 13:44:00 GMT

2048 Game:

So, You got addicted to 2048 and wondering how to make 2048 tile? As long as you don't know the strategy its a nightmare. Once you know that, its pretty easy and I bet you can make 2048 tile every time you play a game. Sounds good? Lets get started.

Warning: This is How I play the game and I can make 2048 tile every time I play this.

Strategy: Goal & The Path

Now, lets look at the game a bit differently. Its very tedious( and time consuming ) to look at all 16 tiles for each and every move we make while playing the game. Actually we don't have to. Take a look at the pictures below.

The Goal - The tile where 2048 will be formed.

The Path - A few tiles which we will use to make goal.

Now don't worry much about the remaining tiles.

How To Make 2048 Tile?

1. Start the game.

2. Move the first tile to the goal position and keep adding remaining tiles to this.

3. Do not move the goal tile unless you have no other choice.

4. DO NOT MOVE THE GOAL TILE UNLESS YOU HAVE NO OTHER CHOICE.

5. Move the next tiles to path in DECREASING ORDER ONLY.

6. Make sure the goal is always the biggest tile and tiles in the path must be in decreasing order.

Example: {Goal:16, Path: 8,4,2} or {Goal 1024; Path:512,256,256 as show in the pic}

7. Go on adding the tiles in the path to the goal.

That's it.

If you follow the above rules and play the game a few times You will make 2048 tile easily. Good luck.

Tips:

1. Instead of pushing tiles around stick to 2 swipes only.

While I play the game, I swipe from top to bottom and right to left only. If these two are not possible then I swipe from top to bottom or left to right depending up on the tiles.

2. Always keep the high valued tile in bottom left and keep adding the remaining tiles in descending order from bottom to top as shown in the figure.

For Geeks: If you want to solve 2048 programatically check this Stackoverflow Q & A.

Check this video on how to make 2048 tile!

My Tribute To Relecura!

Anand Reddy Pandikunta — Sun, 02 Mar 2014 07:54:00 GMT

tribute (ˈtrɪbjuːt) noun

A gift or statement made in acknowledgment, gratitude, or admiration.

" Feeling gratitude and not expressing it is like wrapping a present and not giving it. "

- William Arthur Ward

More than 100 days of have passed since I joined INDUS Techinnovations and I am pretty excited to be a part of this company. I thought of doing something for our company.

So, I brought some Rubik Cubes and made a mosaic.

Rubik Cube Mosaic of Relecura Logo!

*Thanks to Sreenivas, George, Nikhil and all our team.

Best Way To Restore USB Drive Back to Full Capacity!

Anand Reddy Pandikunta — Tue, 25 Feb 2014 22:00:00 GMT

If You have used USB Drive to install a Ubuntu or any Linux Distros on your PC, Your pen drive might lost the space. Whether Your pen drive is 4GB/8Gb/16GB or whatever it will reduce to the size of the ISO file You have used normally to 700MB. You can format this pendrive but still you will get only 700MB. So, what now?

There is a simple trick to recover pendrive to its full capacity.

What You Need?
1. A PC with Windows OS.
2. A corrupted USB Drive.

Process:
1. Open Run ( Windows + R ).

2. Type Diskpart and press Enter ( diskpart ).

3. Type list disk ( list disk ).

4. Type select disk 1 ( select disk 1 ).

5. Type clean and enter ( clean ).

6. Now you USB will show 0Bytes. Don't panic. Just unplug USB and insert again.

8. The system will prompt for fromat and do format.

Thats it. You have recovered Your USB drive to its full capacity from corruption!!

How To Find All Subsidiaries Of A Company?

Anand Reddy Pandikunta — Sun, 23 Feb 2014 23:58:00 GMT

Understanding Subsidiaries:

A subsidiary is a company that is completely or partly owned by another corporation that owns more than half of the subsidiary's stock.

A first-tier subsidiary means a subsidiary/daughter company of the ultimate parent company, while a second-tier subsidiary is a subsidiary of a first-tier subsidiary: a "granddaughter" of the main parent company.

Ford motor company with its tiered subsidiaries is shown below.

Ford Motor Company – Parent company

Ford International Capital LLC – First-tier subsidiary

Blue Oval Holdings – Second-tier subsidiary

Ford Motor Company Limited – Third-tier subsidiary

Ford Component Sales Limited – Fourth-tier subsidiary

Finding Subsidiaries:

1. Corporate Sites:

The best source to find subsidiaries of a company is its corporate sites itself. Press Releases provide information about recent acquisitions and other information. A lot of companies provide a lot of information about its subsidiaries in their annual Reports

2. SEC.gov

All companies, foreign and domestic, are required to file registration statements, periodic reports, and other forms electronically through EDGAR. Anyone can access and download this information for free. Here, subsidiaries list is available for most public companies.
Example: List of subsidiaries of General Electric at SEC.gov

3. Open Corporates

Opencorporates have information about 60+ million companies. You can search the database and find the parent company. There you can a huge list of subsidiaries for it.

Example: List of subsidiaries of General Electric at Opencorporates.

4. Wikipedia

Wikipedia is also most valuable source to find acquisitions and assets of companies. There are a lot of articles covering acquisitions of various companies. In addition to that there a few lists which contain subsidiaries of a company.
Example: List of assets owned by Berkshire Hathaway, List of mergers and acquisitions by IBM

చలం కథలు - సీత తల్లి (Seetha Thalli - Gudipati Venkatachalam )

Anand Reddy Pandikunta — Thu, 20 Feb 2014 03:28:00 GMT

ముందు మాట:

చలంగా ప్రసిద్ధుడైన గుడిపాటి వెంకట చలం సుప్రసిద్ధ తెలుగు రచయిత, వేదాంతి మరియు సంఘసంస్కర్త. హేతువాది .ఆధునిక తెలుగు సాహిత్యాన్ని ప్రభావితం చేసిన అతి ముఖ్య వ్యక్తుల్లో చలం ఒకడు. చలం రచనలు చాలా వరకు స్త్రీల జీవితాలను ఇతివృత్తంగా చేసుకుని ఉంటాయి. ముఖ్యంగా సమాజంలో వారికి ఎదురయ్యే శారీరక మరియు మానసిక హింసలు, వాటిని వారు ఎదుర్కొనే విధానాలను చర్చించాడు. చలం రచనలలో ఇతివృత్తమూ, తాత్వికతా, రచనాశైలీ ఆయనకు ఆధునిక తెలుగు రచనా రంగంలో అనన్యమైన స్థానాన్ని సంపాదించిపెట్టాయి. ( Source: వికీపీడియా )

కథ - సీత తల్లి

How To Screencast Android Device Without Root?

Anand Reddy Pandikunta — Tue, 28 Jan 2014 22:33:00 GMT

Screencasting Android Device:

If You want to screencast something on Anroid device, its a lot of headache. There are a few apps on Play Store by which you can screencast. But a lot of them require rooting. But if Your phone/tablet is new or if You are afraid of bricking, You don't want to root the phone.

There are some procedures to screencast Android, but they require SDK. But You can also screencast without SDK also. This will be simplest way to screencast.

There are only two apps on play store which offer screencasting, without rooting and without SDK.

1. ASC Screen Recorder

2. Free Screen Recorder No Root

Lets look at both of them, which is easy to install and which is easy to use!

ASC Screen Recorder:

To use ASC Screen Recorder, You NEED a PC ( must be Windows or Mac, no activator for Linux/Ubuntu ).

Download app ( free or paid ) from Play Store.
Download Activator from here.
Install activator and start recording.

With free app, you can record unlimited videos but they are restricted to only 20 seconds.

Free Screen Recorder No Root:

For Free Screen Recorder No Root, You can use any PC ( can be used with Ubuntu ).

Download app ( free or paid ) from Play Store.
Download Activator from here.
Install activator and start recording.

You can record unlimited videos, but their logo will be displayed on the recorded video and framerate is only upto 4fps which is not good.

I have tested both the apps on My Samsung Core. Without root and without SDK, these are best and these are highly use for dummies!

Wipro Campus Placement - My Interview!

Anand Reddy Pandikunta — Thu, 23 Jan 2014 11:22:00 GMT

It happened few months back in our Sri Venkateswara University College of Engineering!

First our placement officer R.V.S sent a notice about the campus placement! He arranged a session to get us prepared for campus placement. Students who have less than 2 backlogs were allowed to placement.

On the day, a few officials from Wipro came to our college. We all attended pre-placement talk in our formal dress. After the talk, they separated the students into two batches.

The first batch were the toppers who have more than 80% academic score. They don't have to take the written test and have to attend the interview directly.

I had around 78% score. So, I have to take the written test. Written test consisted of Aptitude questions and English vocabulary, passage writing. I was qualified in written test!

Interview:

First round:

I entered into the cell greeted the interviewer. He asked me to sit down. He spoke politely and asked a few questions casually to make me comfortable.

Second round:

In this round, the interviewer asked me about the Android app I have developed. I told that it was very basic app and told him how I developed it. He spoke a lot about Andoid apps and asked me to participate in App contest help by Wipro.

That's it. My interview was completed. Later I went out for a walk!

On return one of my friend told me that I got selected!

How I Got Into Alexa Top Million In A Few Days?

Anand Reddy Pandikunta — Thu, 23 Jan 2014 10:39:00 GMT

Just now ( 02:00 PM, 23 Jan 2014) I checked Alexa Rank of this blog and I found that it is 982,516! I just wrote a few posts and told about them to few online friends who are interested to read them. That's it!

Here I have added complete details you needed to know about my blog! Hope that helps!

Content:

As of today, this blog has 28 posts.

No. of Posts	Month
1	March 2013
1	October 2013
2	November 2013
4	December 2013
20	January 2014

Backlinks:

I logged into Google webmasters and downloaded this table. I have to 15 backlinks, some of them are do follow!

Domains	Links	Linked pages
google.com	43	8
youtube.com	43	2
blogspot.com	8	1
medium.com	1	1
medlibrary.org	1	1
stumbleupon.com	1	1
mashpedia.com	1	1
web-sniffer.me	1	1
justclicksearch.com	1	1
digplanet.com	1	1
ency.cl	1	1
statscrop.com	1	1
verkata.com	1	1
goo.ne.jp	1	1
wikipedia.org	1	1

Stats:

Little Alchemy Walkthrough 450 Elements Combinations!

Anand Reddy Pandikunta — Tue, 21 Jan 2014 03:14:00 GMT

About Little Alchemy:

Little Alchemy is a simple but addictive game. You start with four basic elements, and mix and match them to create more and more awesome things.

Combine the elements by simply dragging and dropping one on the other to make something new. See how many elements you can come up with. There are 480 elements to create.

Get Little Alchemy App for Chrome

Get Little Alchemy App for iOS

Get Little Alchemy App for Android

Little Alchemy Walkthrough:

I have recorded a few videos for few elements.

Little Alchemy Walkthrough First 20 Elements - Part 1!

Little Alchemy Walkthrough First 20 Elements - Part 2!

Little Alchemy All Cheats

You can make 450 elements with the four basic elements. Here is the complete solutions for Little Alchemy!

1. Water

2. Fire

3. Earth

4. Air

5. Acid rain = rain + smoke

6. Airplane = metal + bird, steel + bird

7. Alarm clock = clock + sound

8. Alcohol = time + fruit, juice + time

9. Algae = plant + ocean, plant + water, plant + sea

10. Alien = life + space

11. Allergy = dust + human

12. Alligator = swamp + lizard

13. Ambulance = car + hospital, car + doctor

14. Angel = bird + human

15. Antarctica = desert + ice, desert + snow

16. Aquarium = water + glass, glass + fish

17. Archipelago = isle + isle

18. Armadillo = wild animal + armor

19. Armor = metal + tool, steel + tool

20. Ash = energy + volcano

21. Astronaut = human + moon, rocket + human, human + spacestation

22. Atmosphere = air + pressure, sky + pressure

23. Atomic bomb = energy + explosion

24. Avalanche = energy + snow

25. Axe= blade + wood

26. Bacon = fire + pig

27. Bacteria = swamp + life

28. Baker = human + bread, human + flour

29. Banana = monkey + fruit

30. Bank = house + gold, house + money

31. Barn = house + cow, house + livestock

32. Bat = mouse + bird, sky + mouse

33. Batter = flour + milk

34. Batman = bat + human

35. Bayonet = blade + gun

36. Beach = water + sand, sand + ocean, sand + sea

37. Beaver = wild animal + wood

38. Bee = flower + wild animal

39. Beehive = house + bee, tree + bee

40. Beer = alcohol + wheat

41. Bicycle = wheel + wheel

42. Bird = air + egg, sky + egg, sky + life

43. Birdhouse = bird + house

44. Black hole = star + pressure

45. Blade = stone + metal

46. Blizzard = snow + wind, snow + storm

47. Blood = human + blade

48. Boat = water + wood

49. Boiler = metal + steam

50. Bone = time + corpse

51. Book = story + paper

52. Bread = fire + dough, fire + flour

53. Brick = fire + mud, fire + clay, sun + clay, sun + mud

54. Bridge = steel + river, metal + river, wood + river

55. Broom = hay + wood

56. Bullet = gun powder + metal

57. Butcher = human + meat

58. Butter = pressure + milk

59. Butterfly = flower + wild animal

60. Cactus = desert + plant, plant + sand

61. Camel = desert + horse, desert + wild animal

62. Campfire = fire + wood

63. Candle = fire + wax, wax + thread

64. Candy cane = sugar + Christmas tree

65. Car = metal + wheel

66. Carbon dioxide = human + oxygen, plant + night

67. Cart = wood + wheel

68. Castle = knight + house

69. Cat = wild animal + milk

70. Catnip = plant + cat

71. Caviar = human + hard roe

72. Centaur = horse + human

73. Cereal = milk + wheat

74. Chainsaw = electricity + axe

75. Chameleon = lizard + rainbow

76. Charcoal = fire + wood, fire + tree

77. Cheese = time + milk

78. Chicken = bird + livestock

79. Chimney = house + fireplace

80. Christmas stocking = wool + fireplace

81. Christmas tree = light bulb + tree

82. Cigarette = paper + tobacco

83. City = skyscraper + skyscraper, village + village

84. Clay = mud + sand

85. Clock = electricity + time, time + wheel

86. Cloud = air + steam

87. Coal = plant + pressure

88. Coconut = palm + fruit

89. Coconut milk = coconut + tool

90. Coffin = wood + corpse

91. Cold = rain + human

92. Computer = electricity + nerd, nerd + tool

93. Computer mouse = computer + mouse

94. Confetti = paper + blade, paper + scissors

95. Constellation = star + star

96. Cookie = sugar + dough

97. Corpse = human + gun, grim reaper + human

98. Cotton = cloud + plant

99. Cow = livestock + grass

100. Crystal ball = glass + wizard, glass + witch

101. Cuckoo = clock + bird

102. Cyborg = robot + human

103. Cyclist = human + bicycle

104. Dam = river + wall, beaver + river, beaver + tree, beaver + wood

105. Darth Vader = Jedi + lava, fire + Jedi

106. Day = sun + time, sun + night

107. Desert = sand + sand

108. Dew = fog + grass

109. Diamond = coal + pressure, coal + time

110. Dinosaur = time + lizard

111. Doctor = hospital + human

112. Dog = wild animal + human

113. Doghouse = dog + house

114. Donut = dough + oil

115. Double rainbow! = rainbow + rainbow

116. Dough = water + flour

117. Dragon = fire + lizard

118. Drum = leather + wood

119. Drunk = alcohol + human, beer + human

120. Dry ice = cold + carbon dioxide

121. Duck = water + bird

122. Dune = beach + wind, desert + wind, wind + sand

123. Dust = earth + air, sun + vampire

124. Dynamite = wire + gunpowder

125. Eagle = mountain + bird

126. Earthquake = energy + earth, wave + earth

127. Eclipse= sun + moon

128. Egg = stone + life, bird + bird, lizard + lizard, turtle + turtle

129. Egg timer = clock + egg

130. Electric car = car + electricity

131. Electric eel = electricity + fish

132. Electrician = electricity + human, electricity + flashlight

133. Electricity = energy + metal, sun + solar cell, wind + wind turbine

134. Email = computer + letter

135. Energy = fire + air, sun + plant

136. Engineer = human + tool

137. Eruption = energy + volcano

138. Excalibur = stone + sword

139. Explosion = fire + gunpowder

140. Fabric = thread + tool

141. Family = human + house

142. Farmer = human + field, plant + human

143. Faun = goat + human

144. Fence = wall + wood

145. Field = earth + tool

146. Fireman = fire + human

147. Fireplace = campfire + house

148. Fire truck = fireman + car

149. Fireworks = sky + explosion

150. Fish = time + hard roe

151. Fishing rod = tool + fish, wood + fish

152. Flashlight = light + tool

153. Flood = rain + time, rain + rain

154. Flour = stone + wheat, windmill + wheat

155. Flower = plant + garden

156. Flute = wind + wood

157. Flying fish = fish + bird

158. Flying squirrel = squirrel + bird, squirrel + wind, squirrel + airplane

159. Fog = cloud + earth

160. Forest = tree + tree

161. Fossil = dinosaur + earth, dinosaur + time

162. Fountain = statue + water

163. Fox = chicken + wild animal

164. Frankenstein = electricity + corpse, electricity + zombie

165. Fridge = cold + metal, cold + electricity

166. Fruit = tree + farmer, sun + tree

167. Fruit tree = tree + fruit

168. Galaxy = star + star

169. Garden = plant + grass, plant + plant

170. Geyser = earth + steam

171. Ghost = graveyard + night, castle + night

172. Gift = Santa + Christmas tree, Santa + chimney, fireplace + Santa, Santa + Christmas stocking, Santa + cookies

173. Glacier = ice + mountain

174. Glass = fire + sand, electricity + sand

175. Glasses = glass + glass, glass + metal

176. Glasshouse = plant + glass

177. Goat = mountain + livestock

178. Gold = sun + metal

179. Golem = life + clay

180. Grass = earth + plant

181. Grave = earth + coffin, earth + corpse

182. Gravestone = stone + grave

183. Graveyard = grave + grave

184. Grenade = explosion + metal

185. Grim reaper = scythe + corpse, scythe + human

186. Gun = metal + bullet

187. Gunpowder = fire + dust

188. Hail = cloud + ice, ice + storm, rain + ice

189. Ham = meat + smoke

190. Hamburger = bread + meat

191. Hard roe = water + egg, egg + ocean, egg + sea, fish + fish

192. Hay = farmer + grass, scythe + grass

193. Helicopter = blade + airplane

194. Hero = dragon + knight

195. Hippo = horse + river, horse + water

196. Honey = bee + flower

197. Horizon = earth + sky, sky + ocean, sky + sea

198. Horse = hay + livestock

199. Hospital = sickness + house

200. Hourglass = sand + glass

201. House = brick + tool, human + wood, brick + human, wall + wall

202. Human = earth + life

203. Hummingbird = bird + flower

204. Hurricane = energy + wind

205. Ice = water + cold

206. Ice cream = cold + milk, ice + milk

207. Iceberg = Antarctica + ocean, Antarctica + sea, ice + ocean, ice + sea

208. Idea = light bulb + human

209. Igloo = ice + house

210. Internet = computer + computer, wire + computer

211. Isle = volcano + ocean

212. Jack-o-lantern = pumpkin + fire

213. Jedi = light saber + human, knight + light saber

214. Juice = water + fruit, pressure + fruit

215. Kite = wind + paper

216. King = human + castle, human + Excalibur

217. Knight = armor + human

218. Lamp = light bulb + metal

219. Lava = fire + earth

220. Lava lamp = lamp + lava

221. Lawn mower = grass + tool

222. Leaf = tree + wind

223. Leather = blade + cow, pig + blade

224. Lemonade = water + fruit

225. Letter = paper + pencil

226. Life = energy + swamp, time + love

227. Light = electricity + light bulb

228. Light bulb = electricity + glass

229. Lighthouse = light + ocean, light + sea, beach + light, light + house

230. Light saber = light + sword, electricity + sword, energy + sword

231. Lion = wild animal + cat

232. Livestock = wild animal + human, life + farmer

233. Lizard = swamp + egg

234. Love = human + human

235. Lumberjack = human + axe

236. Map = paper + earth

237. Manatee = cow + sea

238. Marshmallows = sugar + campfire

239. Mayonnaise = egg + oil

240. Meat = tool + cow, pig + tool

241. Medusa = human + snake

242. Mermaid = human + fish

243. Metal = fire + stone

244. Meteor = atmosphere + meteoroid

245. Meteoroid = space + stone

246. Milk = human + cow, farmer + cow

247. Milk shake = milk + ice cream

248. Minotaur = cow + human

249. Mirror = glass + metal

250. Money = paper + gold

251. Monkey = wild animal + tree

252. Moon = sky + cheese

253. Moss = algae + stone, plant + stone

254. Motorcycle = energy + bicycle, steel + bicycle

255. Mountain = earthquake + earth

256. Mouse = wild animal + cheese

257. Mousetrap = wood + cheese, metal + cheese

258. Mud = water + earth

259. Mummy = pyramid + corpse

260. Music = flute + human, sound + human

261. Narwhal = water + unicorn, ocean + unicorn

262. Needle = hay + metal, metal + thread

263. Nerd = glasses + human

264. Nest = hay + bird, tree + bird

265. Newspaper = paper + paper

266. Night = time + moon

267. Ninja = human + shuriken

268. Oasis = water + desert

269. Obsidian = water + lava

270. Ocean = water + sea

271. Oil = sunflower + pressure

272. Omelette = fire + egg

273. Orchard = fruit tree + fruit tree

274. Origami = paper + bird

275. Ostrich = sand + bird, earth + bird

276. Owl = night + bird

277. Oxygen = sun + plant

278. Palm = isle + tree

279. Paper = wood + pressure

280. Paper airplane = airplane + paper

281. Parrot = pirate + bird

282. Pasta = egg + flour

283. Peacock = bird + rainbow

284. Pegasus = horse + bird, horse + sky

285. Pencil = charcoal + wood, coal + wood

286. Penguin = wild animal + ice, ice + bird

287. Petroleum = fossil + pressure, fossil + time

288. Phoenix = fire + bird

289. Picnic = grass + sandwich, beach + sandwich

290. Pie = dough + fruit

291. Pig = mud + livestock

292. Pigeon = bird + statue, bird + city, bird + letter

293. Piggy bank = pig + money

294. Pilot = airplane + human, seaplane + human

295. Pinocchio = life + wood

296. Pipe = wood + tobacco

297. Piranha = blood + fish

298. Pirate = sword + sailor

299. Pirate ship = pirate + boat

300. Pizza = cheese + dough

301. Planet = earth + space

302. Plankton = water + life

303. Plant = rain + earth

304. Platypus = beaver + duck

305. Pond = water + garden

306. Popsicle = ice + juice

307. Pottery = fire + clay

308. Pressure = earth + earth, air + air

309. Printer = paper + computer

310. Prism = rainbow + glass

311. Pumpkin = vegetable + fire, vegetable + light

312. Pyramid = desert + grave

313. Rain = water + air, water + cloud

314. Rainbow = sun + rain

315. Reindeer = wild animal + Santa

316. Ring = diamond + metal, diamond + gold

317. River = water + mountain

318. Robot = metal + life, steel + life

319. Rocket = airplane + space

320. Rose = flower + love, plant + love

321. Rust = air + metal, metal + oxygen

322. Saddle = leather + horse

323. Sailboat = wind + boat

324. Sailor = sailboat + human, human + boat

325. Salt = sun + ocean, sun + sea

326. Sand = stone + air

327. Sandcastle = sand + castle

328. Sandpaper = paper + sand

329. Sandstone = stone + sand

330. Sandstorm = hurricane + sand, energy + sand, storm + sand

331. Sandwich = cheese + bread, bread + ham

332. Santa = human + Christmas tree

333. Scalpel = doctor + blade

334. Scarecrow = human + hay

335. Scissors = blade + blade

336. Scorpion = wild animal + sand, wild animal + dune

337. Scythe = blade + grass, blade + wheat

338. Sea = water + water

339. Seagull = beach + bird, bird + ocean, bird + sea

340. Seahorse = horse + ocean, horse + sea

341. Seal = dog + sea

342. Seaplane = water + airplane, sea + airplane, ocean + airplane

343. Seasickness = sickness + ocean, sickness + sea

344. Seaweed = plant + ocean, plant + sea

345. Shark = blood + ocean, blood + sea, wild animal + ocean, wild animal + sea

346. Sheep = cloud + livestock

347. Sheet music = music + paper

348. Shuriken = star + blade

349. Sickness = rain + human, sickness + human

350. Skateboard = snowboard + wheel

351. Skeleton = corpse + time, bone + bone

352. Ski goggles = snow + glasses

353. Sky = cloud + air

354. Skyscraper = sky + house

355. Sledge = snow + wagon, snow + cart

356. Sloth = wild animal + time

357. Smog = fog + smoke

358. Smoke = fire + wood

359. Snake = wire + wild animal

360. Snow = rain + cold

361. Snowball = snow + human

362. Snowboard = snow + wood

363. Snowman = snow + human

364. Snowmobile = cold + car, snow + motorcycle

365. Soap = ash + oil

366. Solar cell = sun + tool

367. Solar system = sun + planet

368. Sound = wave + air

369. Space = star + moon, star + sky, sun + star

370. Space station = house + space

371. Spider = thread + wild animal

372. Squirrel = mouse + tree

373. Star = night + sky

374. Starfish = star + fish, star + ocean, star + sea

375. Statue = stone + tool

376. Steam = water + fire, water + energy

377. Steam engine = boiler + tool

378. Steamboat = steam engine + boat, water + steam engine

379. Steel = coal + metal

380. Stone = air + lava

381. Storm = electricity + cloud, energy + cloud

382. Story = human + campfire

383. Sugar = fire + juice, energy + fruit, energy + juice

384. Sun = fire + sky

385. Sundial = sun + clock

386. Sunflower = sun + plant, sun + flower

387. Sunglasses = sun + glasses

388. Surfer = wave + human

389. Sushi = seaweed + caviar, seaweed + fish

390. Swamp = mud + plant

391. Sweater = wool + tool

392. Swim goggles = water + glasses

393. Sword = metal + blade, steel + blade

394. Swordfish = sword + fish

395. Tank = armor + car, car + gun

396. Tea = water + leaf

397. Telescope =star +glass, sky + glass

398. Thread = cotton + tool

399. Tide = moon + ocean, moon + sea

400. Time = sand + glass

401. Titanic = iceberg + steamboat

402. Toast = fire + bread, fire + sandwich

403. Tobacco = fire + plant

404. Tool = metal + human

405. Train = metal + steam engine, steel + steam engine

406. Tree = plant + time

407. Treehouse = tree + house

408. Trojan horse = horse + wood

409. Tsunami = earthquake + ocean

410. Turtle = sand + egg

411. Twilight = day + night

412. Umbrella = rain + tool

413. Unicorn = double rainbow! + horse, double rainbow! + life, rainbow + horse, rainbow + life

414. Vampire = blood + human, vampire + human

415. Vase = plant + pottery

416. Vegetable = fruit + field

417. Village = house + house

418. Volcano = earth + lava

419. Vulture = bird + corpse

420. Wagon = horse + cart

421. Wall = brick + brick

422. Wand = wizard + wood

423. Warrior = sword + human

424. Watch = clock + human

425. Water gun = water + gun

426. Water lilly = flower + pond

427. Water pipe = water + pipe

428. Wave = wind + ocean, wind + sea

429. Wax = bee + beehive

430. Web = spider + thread

431. Werewolf = wolf + human, werewolf + human

432. Wheat = field + farmer

433. Wheel = tool + wood

434. Wild animal = life + forest

435. Wild boar = wild animal + pig, pig + forest

436. Wind = air + pressure

437. Wind turbine = electricity + windmill

438. Windmill = wind + house, wind + wheel

439. Wine = alcohol + fruit

440. Wire = electricity + metal

441. Witch = broom + wizard

442. Wizard = energy + human

443. Wolf = dog + forest, wild animal + dog, wild animal + moon

444. Wood = tree + tool

445. Woodpecker = bird + wood

446. Wool = sheep + tool

447. Wrapping paper = gift + paper

448. Yoda = Jedi + swamp

449. Yogurt = bacteria + milk

450. Zombie = life + corpse, human + zombie

Chrome Web App Pathuku Walk through All Levels!

Anand Reddy Pandikunta — Thu, 16 Jan 2014 11:46:00 GMT

As few of You know I have solved a lot of Unblock Me puzzles and posted cheats at Unblock Me Solutions.

From the past few months I haven't playing any games or puzzles. But just now I stumbled on this interesting and addictive chrome web app Pathuku!

Pathuku - Connecting The Lines!

It is much interesting for me and started solving a few of them. It is extremely simple and extremely tricky too! You can add this puzzle to Your chrome browser and you can play yourself for a few minutes.

I have added a few cheats that I have recorded. Check out my Youtube playlist Pathuku Walkthrough for more cheats!

నాకు తెలిసిన చలం! (The Chalam, I Know)

Anand Reddy Pandikunta — Thu, 16 Jan 2014 08:20:00 GMT

గొప్పగా అలోచించే వాళ్ళలో ఓ రకమైన విశ్రుంఖలత్వం వుంటుంది. వీళ్ళు ముఖ్యంగా మన సనాతన ధర్మాలని ప్రశ్నిస్తారు. కానీ వీళ్ళ వాక్పటిమ ఎంతగా వుంటుందంటే, వీళ్ళు మన ఆచారాలను ప్రశ్నించినా అందులో ఏదో నిజం ఉందనుకుంటాం. ఇలాంటివాళ్ళు వితండవాదం చేసినా అది అభ్యుదయ భావాలుగల వాళ్ళకు నచ్చుతుంది. కానీ వీళ్ళని సాంప్రదాయవాదులు వ్యతిరేకిస్తారు, సరైన కారణం లేకపోయినా, ఎందుకంటే వీళ్ళ తర్కానికి వాళ్ళు సమాధానం చెప్పలేరు. ఈ అభ్యుదయ భావాలు గల వాళ్ళు (వీళ్ళలో ఓ వర్గమే ఈ హేతువాదులు) ఇలాంటి వాళ్ళకోసమే ఎదురు చూస్తుంటారు. వీళ్ళు చెప్పేది నిజమా కాదా అని ఆలోచించరు.

ఇలాంటి విశ్రుంఖలత్వం ఉన్నవాళ్ళలో చలం ఒకరని నాకు అనిపించింది. నేను సుజాత అనే ఒక కథ చదివాను (చలం వ్రాసింది). సుజాత భర్త గొప్ప స్వాతంత్ర్యవాది(ఇది బ్రిటిష్ కాలంలో జరిగిన కథ). సుజాత భర్త ఆమెని చాలా ప్రేమగా చూసుకుంటుంటాడు. ఐనా ఆమె ఓ పోలీస్ తో అక్రమసంబంధం పెట్టుకుంటుంది. భర్తకి జబ్బు చేస్తుంది. అప్పుడు ఆమెకి భర్త మీడ జాలి కలుగుతుంది. భర్తకి సేవ చేస్తూ ఉంటుంది. కానీ ఈ పోలీస్ దగ్గరికి వెళ్ళడం మాత్రం మానదు. చివరగా తన తప్పు తెలుసుకుంటుంది (తెలుసుకున్నట్టు అనిపిస్తుంది), ఎలా అంటే నా భర్తని ఈ స్థితిలో వుండగా నీతో రాలేను అంటుంది. మళ్ళీ పోలీస్ మీదే మనసు ఉంటుంది ఆమెకు. ఇది మామూలుగా ఎవరైనా చెబితే ఈమె శీలం లేనిది అనే భావన కలుగుతుంది. కాని ఈ కథను చలం మలచిన తీరు చూస్తే మనకు వేరే అభిప్రాయం కలుగుతుంది. అరె ఈమె భర్తకోసం ఎంత త్యాగం చేసింది అని అనుకుంటాం చివరకి. ఇది చలం నైపుణ్యము.

చలం ఆదర్శవాదులకు నచ్చుతాడు. ఎందుకంటే వీళ్ళు సామాజిక కట్టుబాట్లగురించి ఆలోచించరు. కేవలం ఆ వ్యక్తి కోణం నుంచి ఆలోచిస్తారు. ఈమెకి కోరిక కలిగింది కాబట్టి అతనితో వెళ్ళింది అని అనుకుంటారు. ఇక్కడ శీలం అన్న పదానికి విలువ లేదు. మనకు నచ్చింది మనం చేస్తాం అనే పద్దతిలో ఉంటారు. కాని కాస్త వాస్తవికతతో ఆలోచించే సాంప్రదాయవాదులు (వీళ్ళు, ఈమె తప్పుచేసింది అనే కోణంలో ఆలోచిస్తారు ) చలాన్ని వ్యతిరేకిస్తారు. ఈ గొప్పోళ్ళంతా ఏదో సమాజం పాడైపోయింది, మనమే దీన్ని ఉద్దరించాలి అని ఉన్న సాంప్రదాయాలకి భిన్నంగా రాస్తారు. ఇలాంటివారు కొన్ని సామాజిక కట్టుబాట్లు సహేతుకంగా లేకపొయినా, వీటిపట్ల వీళ్ళు మరీ అతిగా స్పందిస్తారు, పర్యవసానాలగురించి అలోచించకుండా.

Credit: శశాంక

చలం కథలు

సీత తల్లి
ఒరెయ్ వెంకటచలం!

Get 8 High PR Dofollow Backlinks In 8 Minutes!

Anand Reddy Pandikunta — Wed, 15 Jan 2014 07:14:00 GMT

I just checked my Quotations blog ( PR #1, Backlinks are 110 as per Google Webmasters, 30 as per Alexa )and I found the number of back links ( both do follow and no follow ) increased a lot. I added a list of sites from where I got backlinks. So lets get started.

Assuming You can register and add a link 1 minute, You can get 8 good high quality do follow backlinks from these sites. I have already got them for my blog. You can check them on Alexa.

List Of Sites To Get High PR Dofollow Backlinks:

SITE	PAGE RANK	COMMENTS
ted.com	8	Register, add Your link in profile.
weheartit.com	6	Register, add Your link in profile.
bubblews.com	4	Register, add Your link in profile.
technorati.com	7	Register and verify Your site/blog.
plus.google.com/	9	Place Your link in profile.
youtube.com	9	Link Your site to Your channel.
pinterest.com	9	Link Your site to Your channel, upload photos from your site.

Here I have added 150+ other sites that I haven't tried!

nationalacademies.org

windowsmediaplayer.com

winzip.com

wisc.edu

Good luck with your backlinks and traffic!

List of Tollywood Movies Copied from Hollywood Movies

Anand Reddy Pandikunta — Tue, 14 Jan 2014 11:08:00 GMT

The first Telugu copycat I came across was Homam. I watched that movie and felt it was great until I watched The Departed. I also had good expectations about Action 3D movie. But after watching the first 15 minutes or so, I severely disappointed as I already watched the Hangover.

Later I came to know a lot about such movies. Here I presented a few movies which I came across.

List Of Telugu Movies Copied/Inspired From English Movies:

TOLLYWOOD MOVIE	HOLLYWOOD MOVIE
Maa nana chiranjeevi	Pursuit of Happiness.
Aatma balam	The Reincarnation of Peter Proud
Action 3d	Hangover
Ahana Pellanta (Allari Naresh)	Marrying the Mafia
Amaravathi	Silence of the lambs
Andaru Dongale Dorikithe	Excess Baggage.
Anjali	Son-Rise: A Miracle of Love
ANJI	Indiana Jones and the Last Crusade
Anthapuram	Not Without my Daughter.
Apoorva Sahodarulu	Corsican Brothers
Arunachalam	Brewster's Millions
Bachelors(2000)	100 girls
Bhairava dweepam	Conan the Barbarian (1982)
Bhamane Satyabhamane	Mrs. Doubtfire
Bharateeyudu	Falling Down
Chandralekha	While you were Sleeping.
Chiru Navvutho	Life is Beautiful (1997)
Chirutha	Swept Away
Criminal	The Fugitive.
Dookudu (2011)	Good bye Lenin
Endukante... Premanta	Just Like Heaven
February 14 Necklace Road	I Know What you did Last Summer (1997)
Gaayam 2	A History Of Violence.
Game (2006)	Changing lanes
Ghajini (2008)	Momento (2002)
Hasini	Late Autumn (2010)
Hello Brother	Twin Dragons( Jackie Chan).
Hero (2008)	Police Academy
Homam (2008)	The Departed (2006)
Indrudu Chandrudu	Moon Over Parador
Indumathi	Psycho
Keka	Classic
Key (2011)	Exam (2009)
Khaidi (1983)	Rambo First Blood
Kirathakudu	Escape from New York( entire 2nd half).
Manavudu-Danavudu	Face Off.
Manmadhudu (2002)	What a woman wants
Maryada Ramanna	Our Hospitality
Mee Shreyobhalashi(2007)	Japanese Film Ikinai
Money	Ruthless People.
Mosagallaku Mosagadu	The Good the Bad and the Ugly
Mrugaraju	Ghost and the Darkness
Nani	Big (1988)
Nee manasu naaku telusu	On the Line (2001)
Nenu meeku Telusa	Clean Slate (1994)
Neti Siddartha	The Godfather.
Nirnayam	Stakeout
Oka Vichitram	Bowfinger
Oosaravelli	Vengeance
Paape Na Praanam	Nick of Time.
Pilla Zamindar (2011)	A Millionaire's First Love
Pothuraju	The Life of David Gale (2003)
Raatri	The Exorcist.
Robo	Making Mr.Right
Sakhi	Barefoot in the Park
Sathi Leelavathi	She Devil
Satyabhama (2007)	50 First Dates
Satyame shivam	Planes, Trains and Automobiles
Sisindri (1995)	Baby's Day out
Siva Shankar	The Road to Perdition.
Stalin (2006)	Pay It Forward (2000)
Thenali	What about Bob?
Totti Gang	Saving Silverman.
Veta (1986)	The Count of Monte Cristo
Visakha Express (2008)	Strangers on a Train

Start Up Companies In Bangalore

Anand Reddy Pandikunta — Mon, 13 Jan 2014 11:22:00 GMT

After completion of my B.Tech in 2013, I started looking for start-up companies in Bangalore and here are a few of them I came to know.

Peppersquare

Founded in 2002, we are a happily independent digital agency from Bangalore. We work as an extended team to deliver digital services to enterprises, agencies and emerging companies. Our proprietary Digital Maturity Index measures and monitors where brands stand in the digital business future, and offers effective engagement to tap new markets.

Toptomato

Toptomato is a one-stop online solution for all household needs, spanning the FMCG portfolio of foods, personal care and household care products, thus ushering in a new era of convenient grocery shopping. Leveraging on new-technologies and business strategies, Toptomato is committed to bring in superior service and customer delight.

Foofys

We are a mobile app creation company that also provides digital design services.We see ourselves as crusaders of providing relevant information and this has been our constant striving, creating sustainable solutions that give you the information you need.

Hifives

HiFives is the flagship product of DealsNow Consumer Solutions Private Limited, an employee applications company. DealsNow develops corporate solutions for employee rewards, recognition, gamification, engagement, collaboration, productivity, work-life balance and other employee focused applications, that can be deployed on the cloud, corporate intranets and mobile platforms. It was founded by IIT-IIM alumni in 2010.

Frrole

Frrole's Social Intelligence Platform currently analyzes almost half a billion data units every month, using the derived insights to create solutions for brands, media and entertainment verticals. We take pride in our ability to answer fuzzy questions, when almost all traditional data analytics stops at providing statistical and keyword based answers.

Fungroo

A company started by IIM Bangalore Graduates having a customer engagement platform that captures customer feedback in real time from anywhere in this world and helps organizations gain customer insights to take actions at the right time.

Address: Fungaroo, C/O Alitum management solutions pvt. ltd, #32, Kormangala inner ring road Bangalore, Karnataka : 560047

Getmeenabled

GME Disablity Enablers is an online store for disability equipments and devices (Registration No:Firm/GNR/48/2013-14). GME Disability Enablers mission is to make available world class disability equipments to persons with disability at the door step. This online venture was setup to bridge the gap in availability of suitable devices that empower the disabled to lead independent lives.

Promptcloud

PromptCloud is a Bangalore-based firm dealing with Big Data solutions; where an integral part is web crawling and data extraction. We use cloud computing solutions to facilitate our Data as a Service platform. We work with clients from a range of domains including travel, finance, health-care, marketing, analytics and more.

Metaome

Metaome is a "Big Data" company for the life sciences. We have an agile, customer focused product development strategy and are committed to help scientists glean the best insights and value from data.

Qubole

At Qubole we’re developing the next generation cloud data platform. With an unparalleled understanding of the cloud and a can do attitude we’re addressing today’s challenges of storing and managing data to create the future: a fast, easy and reliable path to turning Big Data into business insight.

Dreamstarts

DreamStarts.in is a job search engine for Indian Startups where Startups post job vacancies and potential employees (Talents) apply to the jobs for free. DreamStarts.in targets elite professionals, Marketing Gurus, freelance designers / developers and talented students in India.

Relecura

Relecura is developed by INDUS TechInnovations, a technology company developing innovative products based on knowledge discovery. We provide software solutions to mine knowledge from relevant data. We have developed application-specific technology platforms to organize and deliver knowledge solutions. These enable customers in different verticals to better explore, comprehend, and glean insights from the large and rapidly growing data sets that are of importance in their domains.

EMSwebtech

EMS Webtech is a leading IT solutions company. We provide full-cycle services in the areas of software development, web-based enterprise solutions, web application, portal development, Ecommerce developement, Graphic Design and internet marketing services.

Mydeals247

MyDeals247 offers three unique business models:

A Real-Time MarketPlace (Buy / Sell Model) - No inventory on the website - everything starts from the consumer interest - system notifies the consumer requests to all the sellers whoever selling the particular product or service along with the consumer's budget - sellers make offers against the consumer request along with the offer expiry times - system picks only the top 5 best offers (by lowest price) - totally real-time and rank order changes based on the offers received by the sellers - consumers takes advantage of the best offer and purchase the same.
Volume Based Discounts - deals with only 5 star rated retailers - focus on general house-hold items, entertainment, education, local events, amusement parks, professional training, and services companies - consumer can share the vouchers with friends and colleagues - volume based purchase only - buy more save big!
Introduces personalized Ad display and ‘Pay Per Lead’ model - Displays the local Ads based on the consumer’s interest within the local city at any given point in time. For instance, if someone is looking for the footware and watches in San Francisco, system consolidates all the local Ads relevant to footwear and watches under "San Francisco Talks about..."

Yourstory

YourStory is India’s no.1 media platform for entrepreneurs, dedicated to passionately championing and promoting the entrepreneurial ecosystem in India.

YourStory was started in October 2008 by Shradha Sharma, who has held senior positions in leading media organizations like The Times of India and CNBC TV18. Today, YourStory is driven by Shradha’s vision to create a hub for thousands of entrepreneurs across India, by providing them with critical value-additions like visibility, business networking, corporate partnerships, growth opportunities and investor connections.

INKtalks

To create a community of like-minded peers on unconventional ‘career’ paths, from any field. The experience of being a change-maker is often isolating, as it can lack the financial security, clarity of direction and social network that come with traditional jobs. Being an INK Fellow provides the peer-support necessary to take brilliant work to the next level.
To leverage their impact, through mentorship, support, networks and specially designed opportunities.
To provide them a platform for discovery and exposure like no other, so that our change-makers gain access to potential customers, funding opportunities, accolades and other forms of support critical to their development and success.

QuickSearch

We are a local search service focusing on tier 4-5 cities in India. Currently focussing on Chhattisgarh, we are already handling enquiries for 12 odd cities in Chhattisgarh, MP and Jharkhand. With 100k+ data points, many of which are exclusive to us, we aim to put every neighbourhood shop and business online. Imagine searching for and finding online the number of your neighbourhood store even in relatively smaller cities in Jharkhand, Chhattisgarh etc! "

HousingMan

"HousingMan brings property buyers and sellers together in one transparent marketplace where both sides get equal opportunity to realize benefits.

Capabilities Tools for buyers and property developers/builders to list, browse, research, bid, place discount offers, and track/manage residential property information —all in one place.

Transparency A “more windows, fewer walls” approach to real estate: we believe in making information easy to find, keeping all parties clued in, and creating smooth transactions.

Choices Users have control, whether customizing listings, choosing to accept an offer, or deciding when to jump into an auction."

Mplentis

MPLENTIS is a WEB Development and SEO Company headquartered in Bangalore, Offering affordable website Design and Development, Responsive Web Design and Development, E-Commerce websites, Content Management System, Content Writing, SEO, Logo Design, Brochure Design and other printing solutions.

doodleblue

Right from mobile application development, web application development, UI/UX, to digital marketing, doodleblue provides a one-stop solution for all your digital needs.

Feel free to suggest any other companies to add to this list.

తప్పక చదవాల్సిన తెలుగు సామెతలు (Must Read Telugu Proverbs)

Anand Reddy Pandikunta — Fri, 10 Jan 2014 12:32:00 GMT

పల్లెలో ఉన్నప్పుడు సామెతలు ఎక్కువగా వినిపించేవి. మంచి సామెతలు విని చాలా రోజులైంది . అందుకే ఈ రోజు సామెతలు అన్ని గుర్తు చేసుకుంటున్నాను.

మొదట కొన్ని ఇంటరెస్టింగ్ సామెతలు...

అల్లం అంటే నాకు తెలీదా బెల్లంలా పుల్లగా ఉంటదన్నాడట!
[ He says he very well knew what ginger is and it tastes sour like jaggery! ]

అమాయకునికి అక్షింతలు ఇస్తే ఆవళికి వెళ్ళి నోట్లో వేసుకున్నాడట!
[ When a person was given rice grains to bless (the couple), he stepped aside and ate it; out of his innocence! ]

మొహమాటానికి పోయి ముండ కడుపు తెచ్చుకుందట
[ Being too submissive, a widow gets pregnant ]

సిగ్గు లేని వాడికి నవ్వే సింగారం
[ Laughter is the ornament for the shameless. ]

శుభం పలకరా పెళ్ళికొడుకు అంటే పెళ్ళికూతురు ముండ ఎక్కడ చాచింది అన్నాడట
[ used to describe a person with a negative attitude and when doing something is very pessimistic about it. ]

దమ్మిడి ముండకి ఏగాణి క్షవరం!
[ Wasting (shaving off) a lot of money on a petty thing! ]

ఆవలింతకు అన్న ఉన్నాడు కాని, తుమ్ముకు తమ్ముడు లేడంట!
[ It is to be understood that a yawn may lead to another one yawning, but a sneeze does not! ]

తగువెలా వస్తుంది జంగందేవరా అంటే బిచ్చం పెట్టవే బొచ్చుముండ అన్నాడుట!
[ The lady asks the tribal priest how in the world can a quarrel begin. The tribal priest replies back, "Get my alms, bitch! ]

అఆలు రావు గాని అగ్రతాంబూలం నాకే అన్నాడంట
[ Someone who doesnt know the ABCs of the event wants an important place in certain event. ]

అయ్యకు కోపం సంవత్సరానికి రెండు సార్లే వస్తుంది, వచ్చింది ఆరేసి నెలలు ఉంటుంది
[ Man gets angry only twice a year, each for six months ]

మన దీపమని ముద్దులాడితే మూతి కాలకుండా వుంటుందా?
[ If we kiss our candle, doesn't it burn? ]

అన్నీ తెలిసినమ్మ అమావాశ్య నాడు చస్తే, ఏమీ తెలియనమ్మ ఏకాదశి నాడు చచ్చిందిట!
[ Lady who knows all dies early and who knows nothing lives lately! ]

నిద్ర పోయే వాడిని నిద్ర లేప్పోచు కానీ; నిద్ర పోయిన్నాటు నటిచే వాడిని నిద్ర లేపలెం
[ You can wake up someone who is sleeping; It is impossible to wake up a person who is pretending to be asleep ]

పొమ్మనలేక పొగ పెట్టినట్లు
[ Instead of directly asking someone to leave, making them leave by starting some smoke ]

ఊరి కోక కోడి ఇస్తే, ఇంటి కోక ఈక అంట.
[ Distributing less for more people. ]

పోన్లే పాపమని పాత బట్ట ఇస్తే; గుడి వెనక పోయి ఉరి వేసుకుందట
[ If you give some clothes for a woman, she used that to hand herself. ]

పేనుకు పెత్తనం ఇస్తే తల అంత గోరికి పెట్టింది అంట.
[ Giving authority for wrong people. ]

తాటి చెట్టు కింద కూర్చొని పాలు తగిన అది కళ్ళే అనుకుంటారు
[ Even if you do right things in wrong place, people will think it as wrong. ]

ఇంకా కొన్ని రొటీన్ సామెతలు....

అందని ద్రాక్షలు పుల్లన
[ The grapes you can't eat are sour. ]

అడగందే అమ్మైనా అన్నం పెట్టదు
[ Even your mother won't give you food if you don't ask. ]

ఆలూలేదు చూలులేదు కొడుకుపేరు సోమలింగం
[ You don't have a wife and she's not pregnant, but your kid's name in Somalingam. ]

కుక్క కాటుకి చెప్పు దెబ్బ
[ Tit for tat ]

తిన్నింటి వాసాలు లెక్కపెట్టు
[ used when someone who is trusted and helped by a person , cheats his/her benefactor. ]

ఆలస్యం ఆమృతం విషం
[ "A stich in time saves nine". ]

పరిగెత్తి పాలు తాగే కంటే ణిల్చిఅని నీళ్ళు తాగటం మేలు
[ Make use of the available things first before you run for the unavailable ]

చేతులు కాలినాక ఆకులు పట్టుకున్నట్లు
[ (no point in) holding medicinal leaves after burning one's hands ]

తంతే గారెల బుట్టలో పడ్డాడుట!
[ He got kicked into a basket of donuts (meaning, prosperity)! ]

కూసే గాడిద వచ్చి మేసే గాడిదను చెరిచిందట!
[ A useless person (braying donkey) has distracted another useless person (grazing donkey), nothing really eventful! ]

ఆహారానికి ముందు వ్యవహారానికి వెనుక!
[ First for food, last (later) for work! ]

ఆకు యెగిరి ముల్లు మీద పడ్డా, ముల్లు వచ్చి ఆకు మీద పడ్డా, చిరిగేది ఆకే!
[ If a leave falls onto a thorn, or vice versa, it is only the leaf that will be torn! ]

ఆస్తి మూరెడు ఆశ బారెడు!
[ Has tiny bit of money, but wants to buy a lot! ]

అబద్దమైనా అతికినట్టు ఉండాలి!
[ Even if it's a lie, it has to stick together! ]

అద్దం అబద్ధం చెప్పదు!
[ A mirror does not lie! ]

అగ్నికి వాయువు తోడైనట్లు
[ Like Wind supporting fire.(Two enemies tied up to act against) ]

పిచుక మీద బ్రహ్మాస్త్రం
[ Refers to situations where one uses disproportionately excessive force on a weak opponent. ]

మింగటానికి మెతుకు లేదు కాని మీసానికి సంపెంగ నూనె
[ we don't have a single rice grain to eat,but we want the aromatic oil for the moustache ]

పొరుగింటి పుల్ల కూర రుచి
[ Grass on the other side is greener. ]

అందితే జుట్టు అందక పోతే కాలు
[ If possible (go for) hair, if not (go for) feet. ]

ఎంకి పెళ్లి సుబ్బి చావు కొచ్చింది
[ A person is suffering to the maximum because of somebody else's cause. ]

వీధిలో పులి ఇంట్లో పిల్లి
[ A Tiger at home, but a cat outdoors. ]

తూర్పు తిరిగి దండం పెట్టు
[ Tried everything, all we can do now is turn east and pray. ]

మొదటికే మోసం మొగుడా అంటే పెసరపప్పు పెళ్ళామా అన్నట్టు
[ asking people to eat cake when they have no bread ]

ఇల్లలకగానే పండగ కాదు
[ if you have finished painting your home, it does not mean that the festival is over. ]

ఆడది తిరిగి చెడుతుంది, మగవాడు తిరగక చెడతాడు
[ A woman gets corrupted with too much socializing, and Man gets corrupted with too little ]

మోసే వాడికి తెల్సు కావడి బరువు
[ person carrying the sedan chair knows the weight of it. ]

ఏరిగెటప్పుడు తినొద్దుర అంట్, అద్దుకు తింట అన్నాడట.
[ when A suggests to B not to eat while shitting, instead of listening B replies that he will eat with it. ]

అంగట్లో అన్ని ఉన్నా అల్లుడి నోట్లో శని
[ even though you have all you need, you are unable to make business as your son-in-law doesn't talk good about it ]

ఊళ్ళో పెళ్ళి కి కుక్కల హడావిడి
[ Dogs eagerness for village marriage! ]

Some Frequently Asked Questions About This Blog

Anand Reddy Pandikunta — Sun, 05 Jan 2014 11:47:00 GMT

Q: What is Avilpage?
A: Just an e-page to write my personal stuff.

Q: Why there are no comments on this blog?
A: Comments are a good way of interacting and there are many benefits with that but they take a lot of time and I will loose the focus on the topics, so I disabled comments on my blog.

Q: No sharing buttons?
A: I just removed them and I don't want them here. If you really liked an article, send a message or a mail to your friends about that. That will be great.

Q: No Ads?
A: Unlike my previous blogs, I am not placing any ads here. I am getting a bad feeling about the ads and soon I will remove the ads from my previous blogs also.

Q: No Search?
A: Yeah no. Lets say you want to search about "movies". You can go to List of Avil Pages, press "Ctrl + F" and search for movies. If You need a better search go to Google and search "site:avilpage.com movies"

Q: Why there is no subscribe button?
A: I don't want it to be there. I don't want a mail delivered to your inbox whenever I wrote some nonsense. When you have some free time, just check out here.

Q: I need to talk to you. How to contact you?
A: Ok, Lets Talk.

Q: Tech details of blog?
A: Domain registered with Godaddy and hosted on blogger platform. Using Simple Template.

Top 10 Most Visited Telugu Websites

Anand Reddy Pandikunta — Fri, 27 Dec 2013 03:09:00 GMT

Since my mother tongue is Telugu, I just wanted to look at top ten most famous Telugu websites. But to my vast surprise, no one has a compiled a list till now. So I took the chance and compiled it. The rankings are based on Alexa ( accurate and famous web traffic measurements ) ranks.

There are a few other English websites which are most visited by people of Andhra Pradesh ( considering it as united as of now - Seemandhra and Telangana ). I considered only Telugu websites ( or which have a Telugu translation )

Rank in AP	India Rank	All India Rank	World Rank
1	sakshi.com	112	1534
2	eenadu.net	177	1966
3	idlebrain.com	410	4413
4	andhrajyothy.com	593	6809
5	apherald.com	581	7394
6	namasthetelangaana.com	725	9377
7	ap7am.com	868	8104
8	teluguone.com	930	10594
9	ragalahari.com	1057	11567
10	andhraheadlines.com	---	15383

Using Webcam In Ubuntu - Cheese Installation

Anand Reddy Pandikunta — Thu, 10 Oct 2013 01:48:00 GMT

I started using Ubuntu recently and started exploring various features. For screen recording and few other apps, webcam is needed.

Here I am showing how to use inbuilt webcam in Ubuntu 12.04.

Open your terminal ( Shortcut: Ctrl + Alt + T ) and run the following command.

$ sudo apt-get install cheese

Once the above command is executed, you can find the Cheese app in your application menu. It will take around 3 minutes (~ 18MB)depending on your net speed.

We can open the app by searching "cheese".

Unblock Me Puzzle Solutions

Anand Reddy Pandikunta — Thu, 21 Mar 2013 06:42:00 GMT

I like to solve puzzles a lot. I started playing UnblockMe puzzle. I solved all levels. Since some people are asking for solutions, I have screencasted solutions and uploaded them to YouTube.

Here is a sample screencast.

Thanks to Divakar, Nagmohan, Chaitanya, Nagendra for helping with screencast.