David M. Chan - Blog

Migrating to 11ty

2024-07-26T00:00:00Z

Wow, so this is my new blog! And it's built with 11ty! I'm so excited to share this with you all. I've been working on this for a while now and I'm so happy with how it turned out. I hope you all enjoy it as much as I do!

What is 11ty?

11ty is a static site generator that is simple, fast, and flexible. It's built with JavaScript and it's super easy to use. You can use it to build blogs, portfolios, documentation sites, and more. It's a great tool for anyone who wants to build a static site without having to deal with a lot of complexity.

Why did I migrate to 11ty?

I decided to migrate to 11ty because I wanted a simpler, faster, and more flexible way to build my blog. Before I was using ghost, a headless CMS, but I found that it was too complex for my needs. I wanted something that was easier to use and that would allow me to have more control over my content. 11ty was the perfect solution for me. It's simple to use, it's fast, and it's super flexible. I can customize it to fit my needs and I can build my blog exactly the way I want it.

What's next?

I have a lot of exciting things planned for this blog. I'm going to be posting (probably not regularly, but sometimes), and I'm also going to be adding new features and functionality to the site, so stay tuned for updates. I hope you enjoy my new blog and I can't wait to share more with you soon!

A Kubernetes Jouney

2023-02-01T00:00:00Z

So, I realized that a bunch of the TIL's that I've posted on the Sister "blog" to this blog are based on my journey in Kubernetes (yes, my unused blog has another unused blog attached to it.... it's blogs all the way down) , so I thought that it might be interesting to sit down and write a bit of a longer piece about my 6 month process of transitioning all of my personal hosting infrastructure to Kubernetes.

What is Kubernetes

If anyone is visiting this page, they've probably heard the word "Kubernetes" or "K8s" thrown around. It's possible, though, that even though you've heard the term, you have no idea what it means. Kubernetes (or, since we love abbreviations in web development, K8s), is, in the terms of their homepage: "an open-source system for automating deployment, scaling, and management of containerized applications." In practice, for me it's a tool which helps to manage all of the infrastructure for anything I expose to the web. Famously used by Google to manage its deployment infrastructure, Kubernetes excels at managing huge deployments of containerized applications, and can scale to the size of most of the cloud giants. It's an impressive piece of work. For more information on what exactly Kubernetes is, you can check out the Google cloud page but, actually, while googling this, I didn't find a great resource for people who aren't deep into devops. I suspect this belies the complexity of getting this whole thing working – perhaps I should write something. But that's a blog post for another day.

Why on earth would you use Kubernetes for a personal project?

The first thing that anyone who has ever used Kubernetes before will say is "Why would you ever want to use this for a personal project?" Kubernetes is notorious for it's finicky deployments, impossible configuration files, dense documentation, steep learning curve, and complicated infrastructure.

Well, you see, I have a lot of personal projects. Currently, I manage over 20 different applications which are exposed to the web. One of these is this blog that you're reading right now. Another one is the sister project to this blog, til.dchan.cc which is a collection of much shorter "Today I learned" style posts. I run several services which backend data collection for my PhD. I run a system which automates checking each of the 15 servers that I manage for the lab at UC Berkeley for SSH accessibility (and makes that available in real-time on statuspage.io). I run DDNS servers with healthchecks.io (That's a whole different blog post). I run a web service that updates me every day with the latest papers published in my field. And so so so many more things. At a certain point, running each of these apps in a loose collection of VPS instances from sketchy VPS providers became dangerous, expensive, and it became super easy to forget that something was running (or where something was running), or even perhaps how much something was costing me to run. So I decided to move from this loose collection to a tight ship of dockerized containers.

Towards Kubernetes Everywhere

So, now that I've made this decision, what exactly did it take? What exactly was the price I paid...

Step 1: Dockerizing Every Application

So, the first problem that I had was that almost none of my projects were built into containers. The container infrastructure was a revolution in devops - you could package up all of your code and dependencies into a package, and deploy this package on pretty much anything, pretty much anywhere, with one command. For somebody who is running deployments, this is amazing, but for somebody building personal projects, it is a nightmare of additional complexity. It took probably a few long weekends to get every one of my personal projects to run on docker.

Step 2: Finding a Container Registry

Next, I needed somewhere to host all of these docker files. Because most of the files are for personal projects, that aren't open source, I didn't necessarily want to publish them to a public container registry like Docker hub. For me ghcr.io came to the rescue: It's tight integration with Github (which I already use to store all of my personal code), and the relatively relaxed pricing structure gave me an excellent place to start (and for public repos, it's free!). Fantastic. Now, I had done most of the hard work, I thought. I have all of my code running in containers, and uploaded to a container repository somewhere. All I have to do is create a cluster and press go, right? Right?

Step 3: Setting up the control plane

So Kubernetes needs a couple of things to run. It needs worker nodes which run each of your individual pods (read: servers that run containers), and it needs a server which organizes all of the work that the servers do. Because I'm nothing but cheap, I thought to myself "Ok, maybe I can provision a few VPS instances (at least three, they say in the docs) using my cheap (and sketchy) online VPS providers, and get this working". Turns out, that really doesn't work. I was about six hours deep into reading about how to link Kubernetes nodes together over the internet before I realized that this was a lost cause. Turns out, that Kubernetes is designed to run between nodes which live in a firewall-protected VPS. And getting around that is a huge no. So I did the numbers, and I caved, and set up the cluster through linode (I was influenced by all of the podcasts. So many podcasts). Each node in the cluster cost $10/month, and you needed at least three, so we're at 30$/month already. Yes, I know, not that much.... but I'm a PhD student. That's the cost of Dinner out in the bay area, for one person. I guess more hot dogs on rice for me. Ok, so now I had the control plane, just a few lines of YAML to get my first app up and running to the public right?

Step 4: My First Deployment (And Regret)

So, the first thing that I decided to do was to get a simple static page running, the landing page for a site I haven't yet built: starlight.dev. All I needed to do was serve a static file, so I had created a docker container with nginx, and I needed to get this exposed to the internet. So I followed a tutorial, and created a deployment. And nothing happened. So I added some more YAML code to make sure that my cluster was authorized to pull containers from ghcr. Well, that kinda worked. Now, I could see my pod running in the cluster (woohoo!), but I couldn't access it from the outside world. Or really, from the inside world. Or really, from anywhere. Then I read another tutorial, and I wrote some more YAML, and I had a service. But I still couldn't access it. And then I read another tutorial. And I realized that actually, I needed an "Ingress service", and without really knowing what that was, I somehow managed to get traefik installed on the cluster, and then finally, I still couldn't access anything. Not only that, but ingress services cost money, so we were up to $40/month, and I still couldn't even see my static page. After another few tutorials, I finally created an IngressRoute which apparently talks to Traefik, and Finally, finally something happened. I saw a static page. On the internet. After at least 30 hours of messing around with it. Turns out, the Kubernetes learning curve is no joke. And honestly, at that point I still didn't really understand what I had done. It was just a huge collection of YAML files I had applied (to the default namespace no less) to the cluster, and I had no idea what I was doing for real. To this day, I'm still not really sure that I know what I'm doing, but hey, maybe I'll figure it out one day.

Step 5: SSL

Well, there was only one little problem with my static web page. It had the red lock of doom (i.e. no SSL), so the next step was to figure out how to get my cluster to hand out the right SSL keys. Since Traefik was terminating the SSL connections, I had to figure out how to give Traefik the certs (from Cloudflare, which handles my DNS, in strict mode), that it needed to prove that yes, indeed, I am David (or at least, that my sites are trusted by me). This required a new thing - cert-manager, which, after a bunch of fiddling around with YAML files, I managed to get deployed in my cluster. But cert-manager only handles the soring fo the certificates, I still needed to generate them. Luckily, there was this tool origin-ca-issuer from Cloudflare which is supposed to interface between Cloudflare and my cluster, to get the origin certificates sorted out correctly. There were installation instructions, and boy did I run them. Like two or three times. But each time, the docker container would crash. Turns out, it just needed more memory. Ah. Good times. But it finally worked. And I got the green lock. But yeah, another set of hours sunk into something that I have not (to this day), touched in my cluster. Sometimes I worry that if something dies, I won't be able to get all of the origin certs working again.... but that's a problem for the David of the future.

Step 6: More Deployments

I'll skip over the hours sunk into each of my other deployments. From not adding the -n to echo when encoding secrets in base64, to figuring out how to attach Linode volumes (turns out, Linode block storage doesn't support ReadWriteMany), to having to pay at least $1/month for each 10GB block storage (since even though my app only needs 150Mb of persistent storage, I still have to attach a volume), to getting MySQL running since Ghost needs that... to getting OOM errors for my pods, and installing metrics-server to track that. Man, it was a journey. But honestly, at this point, it was starting to feel a bit smoother. I (somewhat) understood how Kubernetes was working at this point, and I was starting to get the muscle memory required to write the YAML files which became a huge chunk of my infrastructure. I started using namespaces for each of my projects, and I started storing all of the YAML files in a GitHub repository, so that the entire cluster would be reproducible if something failed... I'm still not sure that it is, but since Linode's control plane hasn't failed yet, it's not something that I've had to worry about. It also became much easier over time to add new deployments. It's no longer a pain to worry about the infrastructure for a new side project, all I have to do is just use a cookie-cutter template to create a new deployment/service, and it's all up and running pretty quickly. Exposed to the internet. With SSL. All with one or two commands. And I get some redundancy built in (but honestly, none of my projects are popular enough to need it).

Step 7: Cron Jobs

I have a bunch of things that run on their own, and honestly, this was one of the easiest things to get running. All I had to do was to create a YAML definition of the job, add a schedule, and it would pull and run the image every time it was scheduled. This was, by far, the easiest part of getting this whole thing to work, and to be honest, I would start here for anyone learning how Kubernetes works, or for anyone who wants to get their feet wet. Getting a CronJob running on Kubernetes is rewarding, and frankly, not that hard. Certainly beats SSL. And Reverse Proxies. And literally. Everything. Else.

Step 8: Figuring out how to get DNS to work

So, at this point, I had everything working, but every time I created a deployment, I needed to manually create the DNS entries in cloudflare. Hunh, I thought. I can do this automatically! No need to handle anything manually! I am a Kubernetes master! I can do this myself. So I found this tool external-dns, which did a great job of handling the DNS. Or so it claimed. Well, you see, because I was using Traefik IngressRoutes, there wasn't any built-in support. Oh no. I tried everything to get it working, but no dice. Finally, I stumbled across a tiny footnote in the docs – the custom resource directory, and the source code for the crd-source. By registering a CRD, and uploading it to the cluster, and pairing it with a working external-dns installation, I was finally able to create DNS Endpoints in YAML, and finally, my Kubernetes cluster was complete!

So... there's this thing called "helm"

So, up until this point, I had been building out my cluster *entirely from scratch. *This means that I wasn't using helm, or rancher, or any of those handy tools that actually make Kubernetes usable. What does this mean? To run installs, it was a combination of cloning source code, building libraries, and running kubectl apply -f a lot. Like a lot. But after setting all of this up, I realized that there's this tool called helm, which makes a lot of these things easier. Oh yeah, and there's a tool called rancher which makes this easier too. Oops. But I guess, learning experience?

So, is it worth it?

Now that I've been living with my (personal) Kubernetes cluster for a few months, I can honestly say... it's not worth it at all. The pain of setting things up, and getting the whole thing deployed, along with the costs that are associated with it (since all of the companies that have the useful tools assume that you're using it for corporate purposes – what even is $10/month for each ingress??), means that it's in no way worth it for a single user. Really, just get a beefy computer, and run docker swarm or something. So much easier.

But.

I love it. I love my new Kubernetes cluster. Not only is it so much easier to manage all of my personal applications, it feels like a pet that I've been tending to for several months. It's really fulfilling to say to myself - I can manage all of these projects using the best of the best tools that's out there. And frankly, the fact that in less than 30 seconds I can provision DNS, SSL, Hardware infrastructure, Routing, and Databases for a new application using a cookiecutter template is amazing. It really allows me to focus the time that I would have put into managing all of my code into new projects, and worry about what color the font on my home page should be. It's great.

What's Next?

So, what's next? I'll probably continue using this cluster for a while. It's a great tool, and it really helps amp up my productivity. As I find more things to run, I'll probably start figuring out how to scale the ingresses, and scale the underlying cluster... it'll probably break this blog at some point. But for now, my Kubernetes journey has reached a plateau. I'll end with one last meme, which I think sums everything up. Cheers! David

Quick Tricks: Downloading Files in Parallel with Curl and Parallel

2021-08-12T00:00:00Z

Recently, I've been doing a lot of downloading of data. The Kinetics dataset from DeepMind is about 600GB, and as hosted by the Common Visual Data Foundation, consists of about 1300 links to files, each about 1.5GB that need to be downloaded. It's quite a task. In this post, I'll talk a bit about how I got this done.

What didn't work - Aria2

Originally, I looked online for a good way to download large numbers of files from the internet. Aria2 seemed to me to be the best option, and fit the bill perfectly. The first time that I went about this download, I used the following option:

aria2c -x8 -j32 -i k600_train_path.txt

This allowed me to use 8 connections per file, with 32 unique processes to fetch the data from the list of S3 links in the k600_train_path.txt file. While the usual common courtesy on the internet is to make no more than 4 connections, with 16 parallel downloads, I figured that Amazon, the great S3 provider, could handle the additional traffic which would be able to saturate my 1Gbps conneciton.

While the files downloaded fine (and in remarkable time, I might add), I had an issue: The files were mostly incomplete, and thus, contained a truncated amount of training data. Because tar/gzip is capable of recoving from some corrruption, and is a linear file, I actually didn't notice until I realized that some of my training classes had no examples! Not only this, Aria2 didn't provide any warnings or errors, so I had figured everything was ok. I guess not.

Usually, when you're working with data at the research scale, you can run into issues that people hadn't thought much about in the initial design process. Files are too large, there are too many files, etc.

A better/old-school option: Curl + GNU Parallel

With Aria2 off the table, I went back to my bash roots, and dusted off two of my favorite command line programs: curl and parallel. Both of these programs are remarkably powerful, and I figured, it would be easy enough to stitch together the two to create a parallel downloader of my own. Not only this, but I probably could have done this faster than typing apt install aria2c which is the command to install the (somewhat flawed) data downloader above.

First, I had to think about how to download a single file. Easy, right?

curl -O https://s3.amazonaws.com/kinetics/600/test/part_59.tar.gz

Then, we just need to let parallel do it's magic. Since I have a pretty large number of files, I don't have to worry too much about the number of connections per file.

cat k600_train_path.txt | parallel --bar -j32 curl -sO {}

Let's break down the command above. First, this takes the list of URLs in the k600_train_path.txt file, and puts them into stdout. I then pipe these URLs into GNU parallel with two options: --bar to give me a nice download progress bar, and -j32 for 32 parallel threads of execution. What follows, is the command that I want to run for each of the URLs. curl -sO {}. The -sO says that curl should be silent (so it doesn't mess with my beautiful progress bar), and that the file should be dumped to disk. The {} is where each of the URLs is coming in from stdin should go.

This worked, and in fact, it was much cleaner, with tools that I already had installed on my system. I probably should have just gone with this from the start, but I am, reallly, a sucker for new tools. You can use this in your own work as well! Whenever you need a large number of files downloaded, or need to do a batch operation, I'd look into the wonders of GNU parallel, and what it can do! There's so much more to this program than meets the eye, and it's well worth your time to take a look. Check out the original docs here, and get it with sudo apt install parallel on ubuntu, or check here for any other system.

Staying Productive - Tools & Methods

2019-08-23T00:00:00Z

I'm on vacation. In lieu of another shallowish dive, I'm posting this quick look into what I do when I'm not enjoying the sunny weather of somewhere nice.

It may not seem like it from this blog post, but in reality, I'm actually a pretty productive person. A lot of this productivity stems from two things: Philosophies, and gadgets. Staying productive is something that we all struggle with from time to time - so to keep you motivated, here are some of the ways that I try to stay productive!

Getting Things Done

The core of being productive is getting things done - that is, taking in tasks, and producing results. In this section, I'm going to talk about some of the things that I do to get things done, and to motivate myself to finish work once I've begun.

Gadget: Make a list

There's nothing quite like the age old todo list, and there's a reason that this is on everybody's "5 things you can do to be more productive" list. As humans, we have very little ability to keep things in our head - and by moving our todo list from our heads onto a piece of paper (or an awesome app) can keep the mind free for actually thinking, instead of trying to remember what we have to do. The issue with the todo list is that if something doesn't make it onto the list, it flies into the garbage heap of forgotten things - so the only gotcha here is that you have to stick with it, and keep writing everything down!

The todo app that I use is Todoist. I have a lot of qualms with todo apps (which could easily be another blog post) - but this one hits my minimum qualifications:

I can view todos on all of my devices (there's a web app, along with mobile clients)
I can organize todos into projects
I can schedule todos on a particular day, prioritize them, and view my whole week from a top-down view.

Philosophy: Just Do It.

One philosophy that I love to use to keep quick things off of my todo list, and keep my time free, is a "Just do it" philosophy that I picked up somewhere (not Shia LeBoeuf). "Just do it" is the idea that if you have a task which is going to take less than five minutes, you should do it immediately instead of putting it off until later. This strategy of completing tasks has a number of immediate benefits:

It makes people think you are task superman (since you get most things done super fast)
It keeps your todo list free of small things, which can pile up rather quickly
It keeps your mind free to work on big projects, instead of small nagging things in the back of your head.

Unfortunately, sometimes the Todoist app and the Just Do It philosophy are at odds with one another. Todoist has a thing called "Karma", which you get for adding and completing tasks in the app. When you're "Just doing" things, you really quickly create and check off tasks, which is somewhat annoying, but hey, you can get that rush of finishing something without feeling like you did any work.

Philosophy: Frog Friday

While it might seem odd, this is a classic productivity technique. The idea stems from the Mark Twain strategy of "Eat Your Frogs", which while quoted in different ways, goes mostly like:

If it's your job to eat a frog, it's best to do it first thing in the morning. And If it's your job to eat two frogs, it's best to eat the biggest one first.

This roughly correlates to "Do the thing that you dread most early in the morning." While I would love to wake up every morning and get the day's rough work out of the way, there are always some things that I manage to push on and on and on until they end up costing me a lot of productivity. To this end, I use the idea of "Frog Friday."

Frog Friday is where I save everything that I know I don't want to do, or find tedious until Friday, and then take the whole day to work on these things. When I'm done with these things, I consider my week finished, and I pack it in for the day. This has the net result of leaving me with a clean desk/mind for the weekend, while accomplishing everything that I need to do that isn't so pleasant. Packing these things into a single day forces me to work on them - and provides some structure that I really need to get things done.

Philosophy: Fill Productive Time with Productivity

One of the hardest things that we can do as people is ignore all of the distractions around us and focus on what needs to be done. This starts by separating time into "productive time" and "relaxation time," and drawing strict boundaries between them.

During productive time, I expect to be getting things done - I take the highest priority items from the top of my todo list, and finish them. Then you take the next item, and do that. It's a shockingly simple philosophy - but it works really quite well. During productive time, you shouldn't be focusing on those notifications on facebook, or that funny link that somebody just sent you. You should be getting things done - trust me - those messages and emails can wait.

When you are using your time productively, it can be surprising how much you actually get done. One of the things that I can struggle with from time to time is not having enough things on my Todo list to keep working on during productive time periods. When you hit this point, it's always really easy to quit, and say "Well, I've finished everything." This isn't really what you should be doing, however. If this ever happens, you should take some time to replan what you are doing the next few days, and add some more tasks to today. If you have nothing for the rest of the week, then you should search out some things to do (Or take up a hobby which fills arbitrary amounts of time).

During relaxation time, I expect to be relaxing. I expect to be spending hours binging the latest episodes of Friends, or playing games. This one is the easy bit. Letting work spill into relaxation time is just as bad as letting relaxation spill into productivity time.

Staying Productive

When I'm working, I usually have more than one project, more than one task, and more than one thing to do. It's always hard to prioritize what comes next, and switch contexts from one project to another. These are some of my strategies for keeping my projects straight, and on track.

Gadget: Dropbox Paper

Recently, Dropbox released a tool called "Dropbox Paper", which is basically a notepad for the internet. There's nothing particularly special about it, but it provides a great place to keep your life organized at a high level. I have a paper doc for every project with a few sections:

Project Outline

In the project outline, I talk a bit about what the project is, who is involved, and some of the high level details. It's amazing how just writing these things down can clean up the details of a project that lives only in your head. For many of my projects, this is the "abstract" of the paper that I'm going to write. It's the high level overview of what I think I'm going to get done.

Project Goals

The project goals are targets/milestones that I want to hit. These can involve tasks, or they can be higher level goals, such as things we want to demonstrate by working on the project. By clarifying the goals of a project, I can generate todos that work towards these goals, and I can always take the product that I have so far, and keep re-evaluating to make sure that I'm staying true to what I want to get out of the project.

Project Timeline

Planning ahead is an amazing thing to do. By estimating the amount of time that I expect different parts of a project to take, as well as plotting out the different milestones, deadlines, and other bits and pieces, I can get a good feel for how urgent some things are. In addition, I can use the super handy Gantt chart feature in Dropbox paper to do this - which makes planning super easy.

Week 1...N progress

At the end of every week, I like to write a summary of what I accomplished for that project in the given week, and what the upcoming todos are for the next week. This helps me see if I'm still on track, and can be handy later when somebody asks "What have you been doing for the past week" - and you can go back to this section, and see your progress right there, without having to do too much thinking.

Philosophy: Stay on Target

While the name for this philosophy comes from Star Wars, it's actually a really interesting idea to keep in mind when planning long projects. Usually, at the beginning of each project, we have a set of goals that we want to get out of the project. Sometimes these are deliverables (and very clear) and sometimes these are much less clear, and much less obvious. It's always important to go back to these goals (preferably for every week/todo) and make sure that you're generating tasks which take you closer to a goal. A lot of times, we can avoid doing unnecessary work by making sure we're staying on track at a high level.

Philosophy: Shot for an MVP, then refine

This idea is one that I think a lot of people say they are going to do, and then they don't. When working on a problem, it's easy to get lost in "What if's...." and do preventative work that isn't exactly necessary. Since I'm a machine learning researcher, I tend to do this a lot, where I start working on something and then add some functionality or code that I don't exactly need - but could help solve a problem that I don't have yet. While it's important to keep potential problems in mind, and leave yourself enough flexibility to solve them later, it's important to not spend too much time solving problems that you don't have if you want to move fast.

Inbox Zero

This is a controversial one, but when it comes to email, I like to keep inbox zero. This means that the only things that I have in my inbox at any given time are emails that I have to respond to. When an email comes in, I archive it right away if it doesn't matter to my productivity flow. I use the "mute" function in gmail liberally, and I do my best to unsubscribe to promotional emails when I get them. My metric for unsubscribing is not a harsh one: "Have I ever used the content in this email?" If yes, then I keep them around, but it's amazing how many emails that I get fail this check.

What I Don't Use

For me, these two productivity tools are enough. I used to pay for Newton, an excellent email client, however I realized that I don't spend enough time responding to emails or handling email things to use the advanced functionality. I used to use Trello as well, but I never really could get moving the cards around to stick. Not only that, cards wouldn't archive automatically when they reached the end of my boards (unless I paid for a plugin), so that was right out. I used to use Toggl for time tracking, but I realized that I don't have the patience to start and stop productivity times every time I wanted to get work done.

Conclusion

I might not use many tools, but by sticking with my philosophies of getting work done, and using the tools that make sense, I'm able to be rather productive - I mean hey, I managed to write a blog post three weeks in a row. That's something! I hope that this was useful to somebody - but if not, it's nice to just get some of my thoughts and philosophies down so I can point people at this later if they ever have the question "How are you so productive?"

What's wrong with configuration management?

2019-08-09T00:00:00Z

Configuration management isn't a sexy topic. It's not AI. It's not Machine Learning. It's not the Cloud or Blockchain. But it's an essential part of what we do every day. When we write code, we always know that we will have to support multiple different functionalities, multiple "configurations." In machine learning, this can be different hyper-parameters/optimizations, input files, produced artifacts, etc.

Why Configuration Management Is Broken

For a long time I've thought that configuration management is broken. Current configuration tools suffer from a number of problems:

1. They're one dimensional (And pretty opinionated)

Most configuration tools focus on one mode of configuration, and allow you to configure in one way). You can use the command line OR you can use a database. You can access your variables using a namespace object, or you can automatically assign them to variables in a function scope.

A good configuration management tool should be able to handle many different kinds of configuration, from configuration files to command line, to even configuration from database tools, or HTML requests. Good configuration management allows you to load and combine configurations from different sources, defining priorities and defaults between these methods.

2. They don't leave a trace

I have written the following line of code 1000 times:

with open('config.json') as jf:
    json.dump(vars(args), jf)

Unfortunately, this means that I have to write paired loading code for the arguments later on, and I have to worry about all of the configuring myself. This isn't exactly an appealing use of my time.

3. They don't scale

When you start a project, you don't really think about scaling. You think about how quickly you can get an MVP up and running. Unfortunately, this means that a lot of the configuration tools that exist right now won't scale to larger projects, or scenarios that you never imagined would exist. Not only this, but they're inflexible when it comes to expanding it yourself. A good configuration management tool allows for expansion.

4. They require too much code to use

You shouldn't have to write hundreds of lines of code to get your code to respect your configurable variables. Configurations should be available where you need them, when you need them, without requiring a user to bend over backwards.

A good configuration management tool should be invisible in your code - if something is configurable, it shouldn't require hundreds of lines of code to understand and write. It shouldn't require you to do crazy backwards magic in order to get code to run in different configurable states. A good configuration tool has a "magical" interface which gives you what you need, when you need it.

A good example of an invisible configuration tool is Sacred which silently injects the value of configurable variables into your function scopes (If this sounds dangerous, you're right! it is! It's not a perfect tool, but it's just fun to use).

Designing the Perfect Configuration Tool

Over the next few weeks, I'm going to be working on developing the outline of a configuration management tool. Who knows, I might even build the tool. But this is going to be an exercise in discovering the world of configuration management - and building a new tool which I can use in my everyday work which is designed to be a powerful and flexible solution to our configuration woes.

From the above problems, I can already think of a set of overall design goals:

Be flexible
Seamlessly scale from small projects to large codebases
Leave footprints
Be "magical" to use, but safe enough to bet the farm on

There are some awesome tools out there like argparse, Sacred, dynaconf, click, docopt, gflags, PyInquirer, Clint, Cliff, Cement, Plac, and many more.

Installing CUDA on Linux the Right Way (in Ubuntu < 19.04)

2018-12-15T00:00:00Z

So, you've got a shiny new computer, or a shiny new graphics card, and you want to install CUDA. Not only this, but you're not going to play video games with your graphics card. Maybe you want to try some of this new "Deep Learning" hype. Maybe you just want to make your code a little bit faster (There's easier ways... try the -O3 compiler command). Regardless of the reason, you need to install CUDA on your machine.

For some reason, everyone that I talk to really struggles with this installation process. The reason for this is that it's not easy. It's not easy at all. Most people install CUDA with apt (and then it breaks their whole computer when they want the next version) or they try downloading the CUDA package file, and get a cryptic error about "X-Windows." In this tutorial, I'm going to walk you through the right way to download and install CUDA on your Linux box, in a way that's not so scary, and will actually make sense later when you want to go upgrade to the next version. So let's begin!

1. Read the Tutorial

Most people fail this step. They think to themselves that they really just need to skip ahead to the commands. They're not wrong. You can probably do everything in this tutorial without reading it. But it won't make you happy. I promise.

2. Installing the NVIDIA drivers

There is one prerequisite step to installing CUDA, and that's making sure that you have the latest NVIDIA drivers. This is actually a separate step to the CUDA installation that we're going to do below, since it's useful to know how to install the drivers on your own. Not only this, but installing the drivers is one of the more challenging things that you have to do.

Note: As of Ubuntu version 19.04, it is no longer necessary to obtain the CUDA drivers. They should already be pre-installed. If you need to update the drivers, you should do so by using your package manager.

The first thing to do is obtain the right Runfile for your GPU and Linux version. Go here: https://www.nvidia.com/Download/index.aspx?lang=en-us and use the manual search to select your GPU version, and the Linux version that you're using (Usually the system box is Linux 64-bit, unless you have something special, and if you have something that special, you're probably not in need of this tutorial). Clicking on "Start Search" will give us the driver download link: Agree to the TOS, and download. Or don't agree to the TOS, and give up. This will download a file, appropriately named NVIDA-Linux-<system>-<version>.run. This file is the one that you're going to use. If you're installing on a remote machine, or don't have direct access. Right click on the Download button, and copy the link. You can then use wget:

wget <copied link>

to get the file on the remote machine.

Now that we have the file we need, we need to install the drivers. Open up a terminal (Ctrl+Alt+T on Ubuntu), and navigate to your downloads folder. If you don't know how to use the terminal, I suggest a crash course: https://www.howtogeek.com/140679/beginner-geek-how-to-start-using-the-linux-terminal/

Running chmod +x NVIDA-Linux-<system>-<version>.run replacing the and options will make this file executable. If you try to run this file as is, you're going to get a myriad of issues. One, for example, might be this one: Most people give up here, and try to google a tutorial. But if we read these errors, they're actually pretty insightful. This one says that I'm already using my NVIDIA driver, and that it has to not be in use before we can upgrade it.

To deal with these issues, we're going to have to do something a little bit unique. It's at this point that you want to open this tutorial up on your phone, or another mobile device. We're going to be working in a TTY session for the remainder of this install, which means that you won't have access to your internet browser. The reason that we have to enter a TTY session is that we're going to have to kill the X-server which runs the GUI. This means that we can't be using any of the fancy graphical features of the OS.

Now that you're on mobile - reboot your computer. When you get to the login screen, if you're on Ubuntu 17.10 or newer, you want to enter a TTY session by pressing Ctrl+Alt+F3. If you're on an older version of Ubuntu, you can use Ctrl+Alt+F1. If you're on RHEL, you're going to want Ctrl+Alt+F2. If you're on Centos, it's another F-key. Basically, all of the F-keys map to TTY sessions (you can read about them more here: https://askubuntu.com/questions/66195/what-is-a-tty-and-how-do-i-access-a-tty). You'll know you've entered the TTY session when your screen goes dark, and you're greeted by a login prompt. You can login here. On Ubuntu, it looks like this: Now that we're in a TTY, it acts just like a terminal. The first thing that we're going to want to do is kill the X-server. In the newer versions of Ubuntu, this can be done by running

sudo service lightdm stop

If this doesn't work - it's possible that you have a different display manager. You can try killing GDM with:

sudo service gdm stop

Most modern distributions of Ubuntu use one of these two X-server managers. If neither of these commands works, then it's time to find which X-server/window manager your distribution is using and kill that. I won't go into how to do this, but I suspect that if you're not using Ubuntu, you have at least a bit of Google-Fu.

Now that we're here, we can change to the directory of the downloaded Runfile. We can then execute the file:

sudo ./NVIDIA-Linux-<system>-<version>.run

If you haven't killed your X-server properly, you're going to get the following error: Make sure that you're in a TTY, and that you have killed the X-server which is running your GUI. I often get the following error: I've never actually had an issue with this - it's safe to continue the installation. If you have DKMS, you'll be gifted the following screen: It's up to you, but I like to put "Yes" for this. This means that if I rebuild my kernel, I probably won't have to fix the NVIDIA drivers. The next window prompts:

I like saying yes... But if you're never going to use them, why bother? After pressing Yes or No here, the screen may freeze for a moment, but then it will enter the installation phase. This phase usually take a minute or two, and then you'll have a working install of the NVIDIA drivers. The next window is: I put yes here. And you probably should as well. When you press "Yes", your system will now have brand new NVIDIA drivers. Reboot your machine, and everything should be set up properly!

A sometimes possible error (Login Loop)

Sometimes, when you install the NVIDIA drivers and restart your computer, you won't be able to login with the GUI. It will look like it's logging in, but it doesn't, and you get returned to the login screen. This is a problem with the X-Configuration, and it's relatively easy to fix. Just open a TTY session, and reinstall the NVIDIA drivers. This has always fixed it for me!

Validating your Driver Installation

You can validate that everything is set up properly by running nvidia-smi, which should, if the drivers are working, give you a screen similar to the following: The version is clearly specified, and your GPUs should show up. If your GPUs don't show up, then there's an issue with the GPU connecting to the program. If you get the warning "nvidia-smi not found", you can run the command:

modinfo nvidia

Which should give something similar to: If neither of these commands work, then the drivers have not installed correctly. Try reinstalling the drivers using the above steps. If it's still not working after that, it's time to get out the phone, and dial NVIDIA customer support at 1-(800) 797-6530.

2. Obtain the latest CUDA version

The first thing that we have to do is obtain the latest version of CUDA from *gasp *the internet***. **Shocking. I know. You're going to want to go here: https://developer.nvidia.com/cuda-downloads

**Note: (As of 6/14/2019) Tensorflow does not support the latest version of CUDA. You can get version 10.0, which is supported from here: **https://developer.nvidia.com/cuda-10.0-download-archive

Once you're on that page, you need to obtain the "runfile"for your distribution. On my computer, I fill out the form like so: If you're using a different distribution, you want to follow the instructions for your distribution. There is always a "Runfile (local)" option, and that is the one that we want. It's always tempting to download the .deb files (if you're on Ubuntu), but this will always bring you pain. If you're on a non-LTS version of Ubuntu, your distribution may not show up here. That's ok. You want to pick the version which is closest to yours. For example, if you're on 17.10, the 18.04 version should work fine.

We then want to click the download link here: This will download the latest version of CUDA. If you're on a remote machine, you can once again right click the download link, and click "save link address". Then, it's as simple as using wget to download the file:

wget <copied link>

(In my case, this looked like: wget https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linux )

Now that we have the latest version of CUDA downloaded, open a normal terminal. This time, we don't have to bother with the TTY session, and we can just use a normal terminal. Change directory to the location where you've downloaded the Runfile, and run the command:

sudo sh <downloaded file> (Once again, to me it looked like: sudo sh cuda_10.0.130_410.48_linux.run)

If you get an error about "This file is not executable", all you have to do is run the command chmod +x <downloaded file> and that should fix the issues that you're having. At this point, it really is as simple as following the command-line prompts. The first screen you get is this: Scroll down using the Down Arrow / Space Key and accept the TOS by typing "accept" at the end and pressing enter: The next prompt asks us if we want to install the graphics driver. We've already done that, so it's the right choice to say "No": The next prompt asks if we want to install the CUDA toolkit. The answer is probably yes, if you've made it this far in the tutorial: Putting the CUDA toolkit in the default location is a good choice, just hit enter to continue past this prompt with the default (unless you really want to put it somewhere else, and you know what you're doing): The next step is really cool (Oh, and you want to say yes): Because it's possible to keep multiple versions of CUDA on your computer, you can have a "system" CUDA version by linking against /usr/local/cuda, and keeping that symbolic link up to date. When the next version of CUDA comes out, you can update the symbolic link, but keep the old version around for compatibility. I always say yes to this - it makes life easier in the future.

The next step is this: If you want to install the samples, go for it. I usually don't - but if it's your first time using CUDA, you might want them to start playing around with the toolkit.

Finishing this step will install the CUDA toolkit in the chosen directory. Often, you might get the following warning: Don't worry about this. We've already installed the driver, so we're good. **We're not quite done yet though! **To actually use the toolkit, we want to do two things:

Add the new installation to your PATH. Open your ~/.bashrc file (or other system profile configuration file) and add the command:

export PATH=$PATH:/usr/local/cuda/bin

Add the new installation to your LD_LIBRARY_PATH. Open your ~/.bashrc file again, and add:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64

Both of the above steps you'll only ever have to do once. The first allows you to run the CUDA binaries such as nvcc and nvidida-smi without the full path. The second, allows you to link against the CUDA runtime libraries. If you don't understand either of those previous sentences, that's ok.

Restart your computer, and you'll finally have a working installation of the CUDA libraries and NVIDIA drivers. Running

nvcc --version

Will validate your installation. If you get errors here, this means that there was a problem installing the drivers or the CUDA libraries. The error itself will be informative. Googling the error is probably the best way to go from here, because there's a lot of things that could happen, and there's a lot of smart people on the internet who have likely already solved the problem.

A (possible) issue:

Command 'nvcc' not found : This means that you haven't installed the CUDA toolkit right. The most likely issue is that you didn't export your PATH variable correctly. Make sure you do this, and then restart your computer (or run source ~/.bashrc). Also make sure that /usr/local/cuda/bin/nvcc exists. If it doesn't, this is a much deeper problem.

3. You're Done!

Congrats! You now have a brand new installation of CUDA and the associated NVIDIA drivers! You can compile CUDA code. You can run pip install tensorflow-gpu. You can do what you will with this newfound parallel computation power!

For some more resources on what to do with CUDA, check out NVIDIA's website:https://developer.nvidia.com/cuda-toolkit

3.5 [Bonus] Installing cuDNN

If you're like me, and you want to train deep neural networks with your GPU, you're going to need cuDNN, NVIDIA's DNN library. Luckily, it's much easier to install than the other counterparts.

You can download the installation here: https://developer.nvidia.com/cudnn. Unfortunately, you're going to have to register as an NVIDIA developer to download this software. Once you do that, you can download the software. Make sure you get the cuDNN Library for Linux. Using the .deb files is, once again, a landmine of danger.

This will download a ".tgz" file: cudnn-<version>-linux.tgz. We can unzip this file using the command: tar -zxvf cudnn-<version>-linux.tgz. This will extract to a "cuda" folder: The CUDA folder mirrors the installation at /usr/local/cuda or wherever you installed CUDA in the previous section. Basically, we're just going to have to move all of the files from this folder to the right places in the CUDA directory: The following commands can do this:

sudo mv cuda/include/* /usr/local/cuda/include/

sudo mv cuda/lib64/* /usr/local/cuda/lib64/

sudo mv cuda/NVIDIA_SLA_cuDNN_Support.txt /usr/local/cuda/

Though really the third one is just for posterity's sake. There's no real reason to do that. Congrats! You have now installed cuDNN, and can deep-learn to your heart's content.

On Timely Feedback During Test Taking

2018-10-26T00:00:00Z

We've all been there. You go in, take a test, and then wait for the results. And wait. And wait. And eventually, a month or two later when you've completely forgotten about the test, and you've started to move on with your life, the results come back. We've all had professors like this.... But is it a good thing? This is certainly a common phenomenon for many college and high school students, and for somebody like me, who has a terrible case of test anxiety, it turns my life into something of a living hell. So I got thinking. Is there a reason that people might do this beyond the obvious case of "I don't want to grade 40 papers tonight, I'll do it tomorrow"? Is there perhaps some benefit to delayed feedback that we, as test-anxious students might be missing?

So I started, of course, by googling the problem at hand. "Is timely feedback necessary for students?" And the answers were discouraging for our tardy professors. One site remarked: Research has shown that the sooner students receive feedback after submitting work the more effective it is for their learning.

Ok, I thought. What research? They provided a link to [1], a paper which I was able to find, and read. The only mention in this paper, however, of the necessity of timely feedback was a reference to another paper, [2]. So, like any good researcher would do, I found that paper, and read that one as well, but they didn't really have a lot to say about the problem of instant feedback. In fact, they mentioned the following:

For ... feedback to be effective it needs to be provided early in the learning process (Brown & Knight 1994) and give guidance for improving performance (William & Black 1996, p. 543).

Ok. So two more references. But one of the interesting things to note about this statement is that it came with respect to "Web-Based" courses, and the "Test and rapidly retake" methodology. So really, there's nothing here about how instant feedback really helps students, just a few claims that it needs to be provided early in the learning process, and a few references.

So, given the breadcrumbs that we have, let's follow them - starting with Brown _et al. _[3]. This work has a more promising title: "*Assessing learners in higher education" *but it's longer. A lot longer. Actually it's a book. Not only is a book, the google books preview had a promising chapter title: "Chapter 8: Speedier Assessment."

Unfortunately, I wanted to write this blog post tonight, and it's in UCB's NRLF storage facility. Which means I probably couldn't get my hands on it until Monday. Thus, we'll have to do without.

Let's turn to the William & Black article [4], "Meanings and Consequences: A Basis for Distinguishing Formative and Summative Functions of Assessment?" With a title like that, how could it not be delightful? It turns out there's nothing here. Well, there is, but it's not that helpful - all it says is that feedback should be useful for improving performance. Nothing at all about timeliness.

Thus, we return once again to the original google search. There's plenty of opinion out there:

Feedback should be within 24 to 48 hours. This idea seems overwhelming for a teacher who sees 150 to 200 students in a day. But if students wait too long for feedback, they risk losing the context for the valued learning of the work. Strive to attain this response time as a goal, not a policy. With diligence, successes will happen when students need them most.

(That one is from: https://www.edutopia.org/blog/timely-feedback-now-or-never-john-mccarthy)

In most cases, the sooner I get feedback, the better. I don't want to wait for hours or days to find out whether my students were attentive and whether they learned, or which part of my written story works and which part doesn't. I say "in most cases" to allow for situations like playing a piano piece in a recital. I don't want my teacher or the audience barking out feedback as I perform. That's why it is more precise to say that good feedback is "timely" rather than "immediate."

(This one is from: http://www.ascd.org/publications/educational-leadership/sept12/vol70/num01/Seven-Keys-to-Effective-Feedback.aspx)

But really, it's difficult to find anything that's not based on opinion. Eventually, I found something:

Appropriate and timely feedback was considered to be the weakest area for more than 40% of Business institutions, with feedback noted as being provided too late to be useful. This was only a minor point across the Art & Design institutions, because of their tradition of oral feedback through tutorials (QAA, 2000).

This paper [4] referenced [5], the UK quality assessment in education. But it was our first real set of results. In business institutions in the UK in 2000, it was a problem. However in Art programs at the same time, it was less of an issue *but only because other feedback was provided earlier. *While it's too early to quit, I think we may finally be on the right track.

And then finally, I found something. "Timing Matters: The Impact of Immediate and Delayed Feedback on Artificial Language Learning" [6]. In this experiment, they taught two grammars, however one group received immediate feedback when they answered questions, while another received a 1s delay. Unfortunately, there's an issue with this. When you get test results back, it's usually much longer than a single second of delay - and nothing is immediate (barring modern web results). Further, it seems like theres another claim to consider:

However, feedback that's too immediate may cause students to rely on teachers for answers rather than persevering and figuring out problems on their own (Bangert-Drowns, Kulik, Kulik, & Morgan, 1991).

Hmm. Perhaps some counter-evidence? In their paper "The Instructional Effect of Feedback in Test-Like Events" [7] they reference another meta-review [8]. Finally - gold. In this meta-review of 53 studies, they split the set of studies into three varieties:

Applied studies measured learning on criterion examinations made up of items different from those used as learning material.

Experiments on Acquisition of Test Content used criterion measures made up of items identical to those used as learning material.

List-Learning Experiments used number of trials to acquisition or number of errors made during those trials as a criterion of learning.

In applied studies, immediate feedback was usually superior to delayed feedback (average effect size = 0.28).
In experiments on acquisition of test content, however, immediate feedback was almost always inferior to delayed feedback (average effect size = -0.36)
In list-learning experiments, effects of immediate feedback were moderately positive (average effect size = 0.34) but highly variable.

In more detail, for Applied Studies:

Nine of the 11 studies of this type found that students achieved more in classrooms where they received immediate rather than delayed feedback. In 4 of these 9 studies, the difference in achievement with immediate versus delayed quiz feedback was reported to be significant. In 2 of the 11 classroom evaluations, examination performance was better when students were taught with delayed feedback, but the gain associated with delayed feedback was not statistically significant.

Yet, when we are working with learning content, in Acquisition we find that:

The 14 studies using this type of measure produced fairly consistent results. In 13 studies, students who received immediate feedback performed less well than did those who received delayed feedback, and in 7 of these studies the result was reported to be statistically significant. In 1 study, students performed better with immediate feedback, and the result was reported to be statistically significant.

Hunh. So maybe it is possible that delayed feedback helps some in certain scenarios. In addition, there's some more surprising results:

Results on follow-up tests were consistent with results on the original measures of learning. The average effect size in the eight studies that used a follow-up measure was -0.44. That means that in the average study retention scores of those receiving immediate feedback were 0.44 standard deviations lower than the retention scores of those receiving delayed feedback.

Yet, there's something to note: **These were measures administered between 5 and 7 days after the original criterion measures were administered. ** 5-7 days seems to be helpful. Yet, what about longer than that? What about an entire semester?

It is from here that I wasn't able to find too many studies testing these effects. The closest I got was a figure from [7] where the columns go: Criterion, Number of Studies, Mean Effect, Standard Deviation of Effect What this says is that the effect of the four studies that returned criterion in greater than a week showed decreasing mean effect of feedback on performance. This is keeping with the common opinion that the longer you keep the feedback, the less useful it becomes.

Conclusion

So, after all this, what do those professors have to say for themselves? If you haven't gotten feedback all semester, you're probably in the scientific right - feedback is important, and every study makes that clear. But if you're upset about waiting a few hours, or a few days, science says be patient, the professor might not be lazy after all! Just encouraging you to get higher scores on your next exam :)

References

[1] Irons, Alastair. Enhancing learning through formative assessment and feedback. Routledge, 2007.APA

[2] Wang, Kua Hua, et al. "Learning styles and formative assessment strategy: enhancing student achievement in Web‐based learning." Journal of Computer Assisted Learning 22.3 (2006): 207-217.

[3] Brown, Sally, and Peter Knight. Assessing learners in higher education. Routledge, 2012.

[4] Melanie R. Weaver (2006) Do students value feedback? Student perceptions of tutors' written responses, Assessment & Evaluation in Higher Education, 31:3, 379-394, DOI: 10.1080/02602930500353061

[5] Quality Assurance Agency (QAA) (2000) Code of practice for the assurance of academic quality and standards of higher education. Section 6: Assessment of students. Available online at: http:// www.qaa.ac.uk/public/cop/ copaosfinal/COP_AOS.pdf (accessed June 2003).

[6] Opitz, Bertram, Nicola K. Ferdinand, and Axel Mecklinger. "Timing matters: the impact of immediate and delayed feedback on artificial language learning." Frontiers in human neuroscience 5 (2011): 8.

[7] Bangert-Drowns, Robert L., et al. "The instructional effect of feedback in test-like events." Review of educational research61.2 (1991): 213-238.

[8] Kulik, James A., and Chen-Lin C. Kulik. "Timing of feedback and verbal learning." Review of educational research 58.1 (1988): 79-97.

Why do today, what you can do tomorrow?

2018-08-31T00:00:00Z

So, I've learned something about writing regular blog posts. I suck at it. So much so that I've gone three weeks without something notable to talk about. Not that I've been lacking for ideas of course - I just have thought to myself every day that "Oh, I can just do it tomorrow!" Well, that turns out to be the wrong mentality, and leads to an endless amount of procrastination.

So this "Procrastination" thing... It's something that happens to everyone I think. Well, at least there's enough memes on the internet to suggest that it's a serious problem...

Yup. That's one.

Oh look! Another one!

From what I know about getting work done, everybody shares just a bit of being lazy. But what is it that brings on this procrastination? Why is it that we take one look at that assignment that's due - or that large research paper that we should be writing and think to ourselves "Eh, I'll do it tomorrow"?

A quick google search brings up an article from Psychology Today, listing the top five reasons that people procrastinate:

The absence of structure
Unpleasant tasks
Timing
Anxiety
Self-Confidence

Each one of these truly insightful bullet points is backed up by significant research - which talks about how (and why) it's in our brains to say "Let's just wait a bit longer." And while these are good points - I think that there's something missing. Many of my projects do have structure, deadlines, and direction. Many I'm excited about, and want to work on! They all have immediate rewards, aren't particularly difficult, and I sure as heck don't have anxiety or self-confidence issues about doing my homework. So what is it? More google is necessary.

And I've found it. In the Forbes article "Why do we procrastinate" it says:

Laziness

Yep. That's it. I'm lazy. But wait - then why do I procrastinate by, say, cleaning my bedroom, or learning about things on the internet? I wouldn't call that (precisely) the behavior of a lazy individual...

I think that for me, the root of my procrastination has to do with what I'm feeling like working on at any given time period. Some days, I just feel like it's exciting to work on one project, and less exciting to work on others. My interests drift from one project to another. My interests aren't constant. Thus, I believe that my procrastination is rooted in my fleeting ability to be interested in whatever I'm working on. Hence - why I always have so many projects I'm working on.

While having a large number of projects can be fun, and certainly has the ability to be engaging ("Oh, don't want to work on this today? Let's work on that instead!" my brain might say), it can sometimes be much more of a curse than a blessing. With so many projects, you're often left juggling too many tasks, or too many ideas.

Ian Somerhalder, the writer of Lost (and the Vampire Diaries) said once:

I kind of think too much, I try to do too many things at once.

That's the risk that you run when working on so many projects - trying to keep that many irons in the fire so that you're ready to strike with the right one when it gets hot (Yes, I think that metaphor was a bit of an over-extension as well).

So how do I deal with this added stress of having too many projects - so that I can find something to work on when I don't want to do anything? I use a todo list. Well, it's a bit more than that. What I do is I try to reduce my cognitive load - the things that I'm carrying around in my head at any given time to the things at hand. I keep notes (hundreds of sticky notes) on every thought that I have, so the paper can remember for me what I certainly wouldn't be able to juggle on my own. For me, my calendar and numerous trello boards filled with cards, my todoist lists and sticky notes, my pages of sketches and handwritten notes, and even some random text files with ideas serve to be a place where I can put my memory when it's not in use.

And for the most part, it works.

It's been a long time since I've felt stress about having too many projects going. I can't imagine a world in which I don't have something that I could be working on that I enjoy. These days, I find that often I have to force myself away from a project that I'm excited about to work on something that is less exciting (because it's due the next day, or is more academically interesting). I find myself waking up earlier, more excited for the day because there's something that I've always wanted to work on at the top of my todo list for the day. That's planning for you. Always have something fun to do.

Some people might, and probably will, say that this is too optimistic a view of how you can live your life. These people might say "You can't possibly always be excited about doing things" or "Isn't there anything that you aren't excited to do! Some tasks are unpleasant." And they're right. There are unpleasant tasks. There are tasks that make me anxious, where I don't have self-confidence, where there might be an absence of structure, or time might prevent me from doing other things. These things happen. But if you find these tasks happen one day out of 10, if you can devote just an hour each day to doing these things, and keep on keeping on, then you'll be fine. It all just works itself out (with a bit of work on your part). And you can get back to the things that you enjoy about life.

So then, we must ask the question: Why haven't I written a blog post in three weeks? And why is it that we now get some crappy writing about procrastination, and keeping a todo list?

Well, it's my blog. I can write whatever sappy heartfelt ode to todo lists that I like. And I finally had something that I wanted to say - today, writing a blog post seemed like fun, it didn't feel like a chore - it felt like something that I wanted to do. So I did it. And why not?