Free Parking in NYC can be found if you’re willing to navigate the world of “Alternate Side Parking” (ASP). A couple times per week, parkers need to accommodate street-sweeping trucks by moving their cars to the “alternate side” of the street. Ask any NYC street-parker about this and they’ll bore you with details of their little weekly routine.

Most of this free parking is in residential neighborhoods, but surprisingly, you can find it just about anywhere. If you know where to look, you’ll find spots in Midtown, SoHo, Chinatown, FiDi, Union Square, Flushing etc.

And when I say you’ll find these spots anywhere - technically true, but the spots you find will be occupied, as free parking is at nearly 100% utilization around the clock, especially in busy and/or commercial areas.

However, if you’re riding a Vespa or motorcycle, you don’t have this problem — you can always find space backed-in between parked cars (this scenario is the actual motivation for this post/app).

On two wheels, you can park anywhere in NYC for free at any time. Given this, I continue to be surprised at the low numbers of Vespas/motorcycles in NYC. Though, car drivers are insane and most people value their lives so perhaps that’s a reason.

Anyways, where are these spots? There isn’t a free parking map provided by the city. The NYC DOT provides this map, which has some fidgets and widgets that don’t seem to solve our problem.

So let’s make a map: parkgratis.com

Building the Map

All the code for this map is available here, and is separated into two areas - app and data_processing.

app builds the website and data_processing does … data processing.

I’ll spare you the details — feed the repo into your favorite LLM if you want to dig in — rather I’ll touch on a few points I think are worth mentioning

Data

NYC Open Data provides this dataset.

This file contains all Parking Regulation Locations and Signs in NYC. There are 440,363 rows, corresponding to 440,363 parking signs.

Here is an abridged sample:

This snippet only shows a handful of columns, but the full set has 25 columns:

"order_number","record_type","borough","on_street","from_street","to_street","side_of_street","sign_code","sign_description","distance_from_intersection","sign_x_coord","sign_y_coord"

Alternate Side Parking (ASP)

Unfortunately, none of the columns mention anything about “Free Parking”, because the truth is a bit more complicated.

Free parking sometimes exists on streets that have ASP

Some streets (or sections of streets) can have multiple overlapping signs; so a street with ASP doesn’t guarantee free parking.

Looking at the sample data in the sign_description column, we can equate ASP with the text “Broom Symbol” (ie street sweeping)

When is ASP free?

My heuristic, which seems right 95% of the time, is that ASP zones with:

• 90 minute ranges are free
• Shorter ranges (30 minutes) tend to come with additional restrictions (ie parking meters, special permits, etc).
• Longer ranges (4 hours) are hard to predict. Sometimes free, sometimes not.
• Overnight restrictions are also hard to predict. Sometimes free, sometimes not.

I’ve found this approach to be pretty reliable in practice (but suggestions welcome).

Data Processing

We have 440,363 rows that we need to parse, looking for “BROOM SYMBOL” with a 90 minute range.

Regex handles most of the heavy lifting here. You can see the details here.

For our output file, we don’t need all 25 columns of data - we just mapping these locations eventually, so we just bring along:

'borough',
'on_street',
'from_street',
'to_street',
'sign_description',
'sign_x_coord_lat',
'sign_y_coord_lon'

Dupes

On any given city block, there can be multiple signs saying the same thing at different locations on that block. For performance and UI reasons, we only want to display one of these signs per block.

To do this, we must check for duplicates:

# outside loop
seen_keys = Set.new

# inside loop (of 441k rows)
key = [
  borough,
  row['on_street'].to_s.strip,
  row['from_street'].to_s.strip,
  row['to_street'].to_s.strip,
  desc.to_s.strip
]

next if seen_keys.include?(key)

seen_keys << key

On any given block, all signs will have the same on_street, from_street, to_street, and borough. And we need to check the sign description desc to make sure it’s actually a duplicate.

To do this in Ruby, a Set datatype is ideal for identifying duplicates, we just fill it like an array, and checking for presence of data is very fast.

Coordinates

Eventually we’ll plot these signs on a map, and if we look back to the sample data, we’ll see that we have sign_x_coord and sign_y_coord.

Only thing is they don’t appear to be latitude and longitude. Rather they are EPSG:2263 coordinates (EPSG is the system, and 2263 corresponds to the NYC region).

EPSG is an alternate system meant for hyper-local location targeting - which in this case - signs on a city block - seems to be a valid use case.

We need to convert these coordinates to lat/long to feed into Google maps - here’s our converter.

App

The processed data is exported to a CSV file that we load into our frontend. It’s all fairly straightforward HTML, Javascript, Google Maps Apis, with most of the logic living here.

Frontend Performance

We started with 440,363 rows in our original dataset, and after filtering for 90 minute ASP and removing dupes, we have 57,587 rows in our CSV.

If we try to add 57,587 markers onto a Google Map, our browser will explode.

Instead, we have a couple of performance-related tricks.

1. Segment all the markers by borough, and default to showing Manhattan markers. You can still enable all 5 boroughs using the checkboxes, but at least on initial load, we reduce loading by 89% (only 11% of signs are in Manhattan).
2. Use heat maps when zoomed out. We determined a zoom threshold where showing individual location markers would render quickly (const MARKER_ZOOM_THRESHOLD = 16;), and when zoomed out farther than that, we’ll display a heat map of the markers, rather than the markers themselves. Heat maps are performant and useful for large chunks of location data. It’s still important to have the individual markers at an appropriate zoom level, so that we can see information regarding that sign, and even pop into a street view if need be.

The end

Ride safe and free park.

Ruby allows anyone … yes you … to customize things to your own liking. Update existing methods, add new ones, etc.

One way to accomplish this is through this concept of Open Classes. In our own code, we can open any class that has already been defined (either from the Core Library or our own code), and just put stuff in.

class String
  def disguise
	  self[1...-1] = "*" * (self.size - 2)
	  self
  end
end

"john smith".disguise
=> "j********h"

"43829".disguise
=> "4***9"

This #disguise method hides all characters but the first and the last.

This method is fun, but isn’t really the focus - the point is we can add methods to all classes, even those from the Ruby core library (class String in this case).

#All?

If you’ve used Ruby, you’ve probably used #all?.

#all? is defined in the Enumberable module, which is included in a few Ruby classes, and for Array, we can use it like:

[2,3,4].all?(&:positive?)
=> true

[2,3,4].all? do |number|
	number > 2
end
=> false

But what if we want to know whether most of the elements of an array satisfy a certain condition, not all.

Let’s say we wanted to check if all but one of the values of an array of numbers are positive. We could do something like:

arr = [2, 4, 1, -3, 9]
arr.count(&:positive?) >= (arr.size - 1)
=> true

This is okay, but if we need this behavior often, let’s create a method to handle it.

#all_but?

Ideally, we’ll be able to use this new method like:

arr = [2, 4, 1, -3, 9]
arr.all_but?(1) do |val|
	val.positive?
end
=> true

or with shorthand notation:

arr = [2, 4, 1, -3, 9]
arr.all_but?(1, &:positive?)
=> true

Open the Class

First step is to open the Array class.

class Array
  def all_but?(number = 0)
	  puts self
	  puts number
	end
end

[1,2,3,4].all_but?(2)
=> [1,2,3,4]
=> 2


[8,9,10].all_but?
=> [8,9,10]
=> 0 # default argument

Simple as that - now we have #all_but? available for our arrays.

Iterate

Now that we see that we have access to self inside our method, we iterate through it, to implement some sort of count.

class Array
  def all_but?(number = 0)
	  count = 0
	  # self is implied (ie self.each)
	  each { |item| count += 1 if [..some logic..] }
	  count
  end
end

Yield

We’ll use yield to grab any code that was passed into the block.

arr = [2, 4, 1, -3, 9]
arr.all_but?(1) do |val|
	val.positive?
end

In this example, we’ll access val.positive? via the yield method.

class Array
  def all_but?(number = 0)
    count = 0
	  each { |item| count += 1 if yield(item) }
    count
  end
end

We’re testing each element of our array against the code we passed in the block.

Temporarily, we’re just returning the count, so with this implementation, we should expect to return 4:

arr = [2, 4, 1, -3, 9]
arr.all_but?(1) do |val|
	val.positive?
end
=> 4

Semi-final Touches

Instead of returning the count of elements that pass our test, let’s actually check the computed count against our target count.

This method is meant to check if all but a certain number of elements pass the test. So essentially, we’re excepting some number of elements from our test.

And we pass this exception count as an argument to #all_but.

class Array
	def all_but?(number = 0)
		count = 0
		each { |item| count += 1 if yield(item) }
		count >= length - number
	end
end

That’s it!*

Let’s run a couple of tests:

arr = [2, 4, -1, -3, 9]
arr.all_but?(1) do |val|
	val.positive?
end
=> false

arr.all_but?(2) do |val|
	val.positive?
end
=> true

* Now let’s consider an edge case.

Blockless

What if someone doesn’t pass in a block?

arr = [2, 4, 1, -3, 9]
arr.all_but?(1)

Currently, this will raise the exception LocalJumpError: no block given (yield). We can’t call yield without a block.

Fortunately, Ruby gives us a handy block_given? method.

class Array
  def all_but?(number = 0)
    count = 0
    if !block_given?
			# do something?
    else
      each { |item| count += 1 if yield(item) }
    end
    count >= length - number
  end
end

Truthiness

The good news is that virtually all Ruby objects are truthy or falsey; meaning, they resolve to true or false.

All objects in Ruby except for nil and false evaluate to true true. One way to test this is using the double bang.

!!3
=> true

!!"string"
=> true

!!""
=> true

!!true
=> true

!!nil
=> false

!!false
=> false

The double bang works by negating the object twice.

!(!3) becomes !(false) which becomes true.

So we don’t actually require code coming in from the block to test each item in the array, since each object can be treated as its own test.

For example, #all? works in this same way:

[true, 2, "hello"].all?
=> true

[true, 2, "hello", nil].all?
=> false

Edge Case Continued

Let’s replace # do something :

class Array
  def all_but?(number = 0)
    count = 0
    if !block_given?
      each { |item| count += 1 if item }
    else
      each { |item| count += 1 if yield(item) }
    end
    count >= length - number
  end
end

Since each item of the array can evaluate to true or false, we can simply ask if [item].

In action:

[false, 2, "hello"].all_but?(1)
=> true

[true, 2, "hello", false, nil].all_but?(1)
=> false

Shorthand Notation

We’re basically done at this point, but just wanted to confirm that our method works with shorthand notation; that is; when the block is passed as the last argument of the method.

To accomplish this, we need to preface our block code with an &.

arr = [2, 4, 1, -3, 9]
arr.all_but?(1, &:positive?)
=> true

The end

class Array
  def all_but?(number = 0)
    count = 0
    if !block_given?
      each { |item| count += 1 if item }
    else
      each { |item| count += 1 if yield(item) }
    end
    count >= length - number
  end
end

Ruby is an Object Oriented Programming language, so most everything is an object (i.e. an instance of some class), and we can call methods on our objects.

3.class
=> Integer

"string".length
=> 6

[1,2,3].include? 1
=> true

Operators

In Ruby, there are a couple dozen operators: +, -, *, /, %, **, =, <, etc.

Some rocket science:

2 + 2
=> 4

Operators and OOP?

These operators … don’t really look like methods being called on objects?

Well, turns out 2 + 2 is a shorthand version of 2.+(2) where + is just a method of the Integer class.

2.+(2)
=> 4

2.+ 2 # parens optional
=> 4

Native classes

Not all operators are available everywhere - each class specifies which operators to support.

For Integer, there are many operators, as we tend to like to operate on integers.

A quick glance via Integer.instance_methods.sort:

[:!, :!=, :!~, :%, :&, :*, :**, :+, :-, :/, :<, :<<, :<=, :<=>, :==, :===, :=~, :>, :>=, :>>, :[], :^, ...]

There are fewer available for a class like String; and this makes intuitive sense, as it isn’t clear what a - would do for an instance of the String class.

String.instance_methods.sort

=> [:!, :!=, :!~, :%, :*, :<, :<<, :<=, :<=>, :==, :===, :=~, :>, :>=, ...]

But one common use of + for strings is concatenation:

"my string" + " has been concatenated"
=> "my string has been concatenated"

Custom Classes

We can define operators on our own custom classes if we’d like.

class Cat
	attr_reader :lives_remaining

	def initialize
		@lives_remaining = 9
	end

	def -(lives)
		@lives_remaining -= lives
	end
end

cat = Cat.new
cat.lives_remaining
=> 9
cat - 3
cat.lives_remaining
=> 6

Why does this matter?

Since operators are just methods (usually displayed in shorthand), we can do method-y things!

One example is the safe navigation operator &.

Sometimes, if we’re not sure whether an object is nil, we can chain & before the method call, so we don’t get an undefined method for nil:NilClass.

It’s quite common to see stuff like my_object&.my_method.

Using the Safe Navigator for Operators

Let’s say we have a method that calculates the tax on a transaction.

def tax(amount)
	amount * 0.07
end

tax 100
=> 7

But perhaps in practice, we don’t know the values that will find their way into this method. Occasionally, the argument could be nil.

Currently, our app would blow up:

amount = nil
tax(amount)
`undefined method `*' for nil:NilClass (NoMethodError)`

Let’s redefine our method using the safe navigation operator:

def tax(amount)
	amount&.* 0.07
end

tax(nil)
=> nil

Another possible solution would be to protect areas of code with guard clauses (ie return if amount.nil?), but the save navigation operator is nice and tidy, and in some circumstances, might be enough.

That’s all.

I made a website with user accounts, and put it online. Within a couple of days, the bots managed to track it down.

Every 20 minutes or so, an account would be created with a username that looked like mBxzKObvCDiWhgt

Bots that create accounts systemically, every 20 minutes, forever, is not good.

1. They take up server resources and slow down the site for actual humans.
2. They also mess up the product. When actual humans try to find other users, they shouldn’t have to wade through a mess of bot accounts.

How they get in

They arrive on the Sign Up page, and submit a form. The form was designed to be simple and quick, so there’s minimal friction for new (human) users.

The flip side is that the form is so simple that robots can figure it out. Only the regular stuff is required: username, password, email. Nothing fancy in the way of Captchas.

Captchas

Captcha is short for Completely Automated Public Turing test to tell Computers and Humans Apart. I just double-checked that was correct, because it seems it should probably be CAPTTTTCAHA.

Anyways, what’s important is what they do — attempt to distinguish between humans and robots. Unfortunately this is usually to the detriment of the user experience.

Before you can post something, you have to find a bridge in 9 small, grainy images.

Apparently robots aren’t good at this practice, and turns out, neither am I.

Honeypots

Honeypots are like reverse Captchas. Instead of asking humans to do the work to prove they aren’t a robot, let’s trick the robot into doing some work.

Once we see a user with his (mechanical) finger in the honeypot, we can safely assume we’ve identified a robot.

This works by adding an additional field to a form, that’s hidden from view. A human won’t see this field, and won’t fill it out. A robot doesn’t have eyes, so they just fill out everything in a form that seems important.

When our application receives the form, if the honeypot field is filled out, we have ourselves a robot. In our case, let’s throw out this data, and not sign up mBxzKObvCDiWhgt.

Implementation using Rails

There’s a Rails gem called invisible_captcha that does exactly what we just described, but it didn’t seem to work for the bots we are dealing with at Bounce, so let’s build a custom honeypot.

Sign Up Form

Inside our form, we include this:

<div class="hp">
  <input type="checkbox" value="1" name="hp" id="hp" autocomplete="off" tabindex="-1">
  <label for="hp">Huemans should NOT clik this.</label>
</div>

And the .hp class has these attributes:

.hp{
  display:none;
  position:absolute;
  overflow:hidden;
  width:1px;
  height:1px;
  opacity:0;
}

It’s probably clear from this CSS that this checkbox is hidden many times over.

For the checkbox itself, we set autocomplete="off" so your browser doesn’t get involved with any sort of autocompletion.

We have tabindex="-1" which means that you can’t tab your way into this field.

We also have an invisible label for this invisible checkbox that says “Huemans should NOT clik this.”. We want this label in the rare chance that the checkbox is visible — maybe a user doesn’t load CSS or they have some weird browser add-on that displays all form fields. Or, if the user is using a screenreader — the label will indicate what’s going on.

So why do we spell it weird? This isn’t an exact science, but there’s a chance that bots have been programmed to actually look out for honeypots, so we don’t want our warning message to be easily parsed by a robot.

Registrations Controller

class Users::RegistrationsController < Devise::RegistrationsController

  before_action :find_bot, only: :create

  private

  def find_bot
    return unless params[:hp] == '1'
    head :ok
  end
end

We add a before_action to the create action of our RegistrationsController that checks to see if the checkbox from our form was checked.

The checkbox had a value set of 1; so when checked, we can can expect our params[:hp] to equal 1.

In our find_bot method, when the checkbox is checked (params[:hp]==1), we return head :ok.

This will halt the request cycle, meaning that the actual create action never runs. Additionally, by returning head :ok, we’re returning the status code of 200 to the client, which the robot will interpret as a success.

Conclusion

This custom honeypot seems to work ok for now. We’ve routed our robots into a false sense of success, without cluttering our database with fake accounts. If they become sentient enough to evade this honeypot, I’ll update this post.

Bounce is a platform where people pay to chat with other people. Everything happens in real-time, and is billed on a per-minute basis.

Seems fairly straightforward. We’ll create a Payment Intent (which is Stripe’s newer way of saying charge) after each conversation takes place.

Using Stripe Connect, we can even specify transfer_data — this allows us to transfer money directly from student to teacher.

First Complication

Bounce’s network is global. A teacher in Mexico can help a student in Japan. Luckily, Stripe handles most of the heavy-lifting here.

However, Stripe Connect transactions have varying requirements depending on the location of the teacher. This is probably due to some combination of local regulations, fraud protection, and other things we don’t need to worry about.

So, before we create a Payment Intent, we need to know where the teacher is located — or more specifically, where her bank account is located.

Stripe has this information — and instead of making an API call to Stripe before creating every payment intent (to find the user’s location), we’ll save the location in our database.

Location Differences

From Stripe’s docs:

Since cross-border transfers are not supported in countries outside your region, some accounts will need to use the card payments capability and act as the settlement merchant to process payments.

So, when the teacher’s country differs from the platform country (this is our location, USA), the platform can no longer act as a middle man in the transaction — the teacher becomes the settlement merchant.

This distinction may seem confusing and boring, so we won’t linger here, other than to say that we need to specify something about card payments later on, when the teacher’s country differs from ours.

Payment Intents

Here’s what a generic Payment Intent might look like in Ruby:

pay_int = Stripe::PaymentIntent.create({
			amount: amount,
			customer: student.stripe_student_id,
			currency: 'usd',
			payment_method: customer_payment_method,
			off_session: true,
			confirm: true,
			transfer_data:{
				destination: teacher.stripe_teacher_id,
				amount: transfer_amount
			}
		})

The first few parameters (amount, customer, currency, payment_method) are fairly straightforward.

off_session: true means the student isn’t present in the checkout flow, which is half true for us — technically he might be present, but his saved card is automatically charged when a session ends.

confirm: true means it’s go time, we’re ready to charge.

transfer_data is the crucial parameter — this is how the platform operates, by transferring money from one person to another (minus the platform fee).

Cross border transfer

If a Mexican teacher helps a Japanese student, we need to use the card payments capability. Stripe Connect will automatically enable this for international accounts in the admin (for Express accounts, at least).

As mentioned in the docs, card payments allows teachers to receive funds directly from students, without the platform acting as an intermediary.

If you’re wondering why we don’t treat domestic transfers this way — the reason is that the account setup for a teacher is significantly more of a pain when they need to act as a settlement merchant. If the platform acts as the settlement merchant, things are easier for everyone involved.

Card Payments are already enabled in the admin (for international teachers), but we also have to specify this directly in the Payment Intent. To do this, we include an additional parameter on_behalf_of — this specifies that the teacher will be the settlement merchant for this transaction.

However, we don’t want to have one Stripe::PaymentIntent.create flow for US teachers and another Stripe::PaymentIntent.create for International teachers, because we’d be repeating ourselves and risking a DRY violation.

Ruby’s tap

tap is a ruby method that allows us to … tap into attributes of an object or the keys of a hash.

For example, when creating a new user, we could do something like this:

user = User.new.tap do |u|
  u.username = "horat22"
  u.name = "Horatio"
  u.location = "Earth"
  u.save
end

This mini-bind that we’re in with Stripe presents a great opportunity for tap, since we can conditionally decide whether to add a particular attribute.

And we only want to include the on_behalf_of sometimes, in the case of the cross-border payment mentioned above.

obj = {
	amount: amount,
	customer: student.stripe_student_id,
	currency: 'usd',
	payment_method: customer_payment_method,
	off_session: true,
	confirm: true,
	transfer_data:{
	  destination: teacher.stripe_teacher_id,
	  amount: transfer_amount
	}
}

obj.tap do |hash|
	hash[:on_behalf_of] = teacher.stripe_teacher_id if not_us(teacher.country)
end

pi = Stripe::PaymentIntent.create(obj)

We have a helper method not_us

  def not_us(country)
    country != 'US' && !country.nil?
  end

which returns true if the teacher’s bank account is not located in the USA.

To recap - we first tap into our obj which contains the Payment Intent data. Then we decide whether we want to add :on_behalf_of. Then, we pass obj into Stripe::PaymentIntent.create.

Ruby’s tap has allowed us to maintain a single payment flow, and keep our code DRY.

Conclusion

That’s all for now.

Back in the days before everything auto-saved, we actually had to remember to save things.

It wasn’t all that painful, aside from the occasional unsaved app crashing fiasco. It gave a moment to pause, reflect, and mark a point in progress.

Autosave has eliminated these pause breaks for nearly all computer-related work, though one exception comes to mind: programming.

Version control forces us to pause every now and again, and deliberately mark a save point for our work. Whether or not this pause break is a peaceful moment of reflection is another question.

These pause breaks look something like:

git add -A
git c -m 'my commit message'
git push

This is fine, but it can be a bit tedious - stage, commit, push each time.

The temporary pause is refreshing, though perhaps we can reduce the number of steps required.

Let’s try to make our pause-break resemble a single command S.

git save

Here’s how:

git config --global alias.save '!f() { message=${1:-"default message"}; git add -A && git commit -m "$message" && git push; }; f'

This works right out of the box. Typing git save will stage all files, create a commit with “default message” (or whatever we replace “default message” with), and then push the files.

If we want to specify the commit message, we can say git save 'my custom message' and we’re on our way.

How

Create the git alias save

git config --global alias.save

Create a shell function and then call it.

'!f() { ... }; f'

Create a default message, or take one is as the first argument.

message=${1:-"default message"};

Stage, commit, and then push.

git add -A && git commit -m "$message" && git push; 

And altogether again:

git config --global alias.save '!f() { message=${1:-"default message"}; git add -A && git commit -m "$message" && git push; }; f'

Our app is a web chat - so a variety of data needs to be broadcasted directly to a user in real time.

This is handled via WebSockets - instant sending and receiving of data. Rails implementation of WS is called Action Cable.

Incoming Requests

For this chat application, users can request chats with pros.

So a pro will receive an incoming request on her dashboard.

Before request:

After request:

As mentioned above, we need to send this data directly to the pro, without the pro needing to refresh her page.

Let’s look in the Controller that handles chat sessions.

ChatSessions Controller

When a user requests a chat, we route to the create action of the ChatSessionsController

# ChatSessionsController

def create
	...
	# append new student to pro buddy list
	SessionChannel.broadcast_to(
	    @pro,
	    action: 'create',
	    new_student: new_student(@chat_session),
	    ... more stuff ...
	  )
	...
end

private

def new_student(session)
	ApplicationController.renderer.render(partial: 'shared/list_user_pro_dash', locals: { session: session, session_status: 'active' })
end

The SessionChannel is the Action Cable channel where we handle chat sessions throughout their lifecycle (request, accept, cancel, end).

In this case, we are broadcasting an HTML partial (list_user_pro_dash.html.erb) directly to the @pro’s dashboard.

This partial generates the HTML we see in the image above — a link with the username of the incoming request.

Receiving the Partial

The Javascript component of the SessionChannel lives in session_channel.js, and we receive the data that was broadcasted from the controller.

#session_channel.js

import consumer from "./consumer"

consumer.subscriptions.create("SessionChannel", {
  connected() {},
  disconnected() {},
  received(data) {
	  if (data.action == 'create'){
	  // append data.new_student to correct location in DOM
	 }
})

Appending data.new_student

data.new_student is a string that contains the HTML corresponding to this requested chat. Just a minute ago, this was rendered via ApplicationController.renderer in our ChatSessionsController.

To append it to the pro’s dashboard, we could do something like this:

const new_student = data.new_student

// active sessions live in #active element
const active = document.getElementById('active')
active.append(new_student)

However, this won’t work because new_student is a string (and not a Node element), so append will treat it like a string.

In other words, we will see HTML, instead of experiencing HTML.

In other words, actually, in images, our view will look something like:

instead of what we want:

Strings or Nodes

So at this point, we either need to turn our HTML string into a Node object (because append can also append nodes) or somehow ask Javascript to interpret our string as HTML.

The latter option probably would have been the easiest. All we would have to do is something like this:

const new_student = data.new_student

// active sessions live in #active element
const active = document.getElementById('active')
active.insertAdjacentHTML('beforeend', new_student)

insertAdjacentHTML will parse a string as HTML.

However, I decided to stick with append to see if we could get it to work.

To do this, we do a little hack: we create an element, then fill it with our HTML string. At this point, our string magically becomes a node object. And we can extract it from our temporary element (temp_obj) via firstChild.

// create temp Node object
const temp_obj = document.createElement('div')
temp_obj.innerHTML = data.new_student
const new_student = temp_obj.firstChild

// append to list
const active = document.getElementById('active')
active.append(new_student)

At this point we have a working solution. We turn a string of HTML from our Controller into a node, and then append it to the correct element.

The Application Breaks

Initially, the partial we append (new_student) looked something like this:

<li>Stuff in here</li>

Then, we make a seemingly innocuous change to the partial.

<% if something? %>
	<li>Stuff in here</li>
<% else %>
	<li>Different stuff in here</li>
<% end %>

Now the application is broken.

Huh?

It seemed bizarre that the addition of a conditional in a partial on the backend would break the broadcasting elsewhere in the site.

It turns out that the broadcast wasn’t broken, but our Javascript was unable to deal with the new HTML string from our partial.

What’s new in this partial?

It would seem that the partial should render a virtually identical result — a simple <li> element containing some info.

However, there is one difference. As is standard in code syntax, we indent the nested line inside of a conditional.

And because of this, a whitespace is created before our <li> element.

FirstChild

Now back to our Javascript. Our data.new_student now contains a single whitespace, and then an <li> element.

// temporary object
const temp_obj = document.createElement('div')
temp_obj.innerHTML = data.new_student
const new_student = temp_obj.firstChild

const active = document.getElementById('active')
active.append(new_student)

When we temporarily created the element and then grab the firstChild, we are actually grabbing the whitespace instead of the <li>.

So, we are appending something, but we are appending whitespace.

Fixes

At this point we have a couple of fixes.

The easiest would be to use firstElementChild instead of firstChild, as that will grab the <li> and ignore text(whitespace).

Alternatively, we can be more specific than firstElementChild and instead look directly for an <li> element.

// temporary element
const wrapper = document.createElement('div')
wrapper.innerHTML = data.new_student
const new_student = wrapper.querySelector("li")

const active = document.getElementById('active')
active.appendChild(new_student)

The End

Be careful when using firstChild as whitespace could potentially find its way to the front of the child you were initially targeting.

Let’s say we’re running this simple little Ruby program:

i = 0
loop do
	exit if i == 5
	puts 'Doing something'
	i += 1
	sleep 10
end

This will print Doing something every 10 seconds, five times, and then exit.

A different, more relatable example of this might be if we we want to run something continuously until a given time.

In that case, it might look like:

y = 2030; mo = 2; d = 28; h = 17; mi = 0;
cutoff = Time.new(y, mo, d, h, mi)
loop do
	exit if Time.now > cutoff
	puts 'Doing something'
	sleep 10
end

In this case, we’ll print Doing something every 10 seconds until February 28, 2030 at 5PM. Here’s a link to a working program if you’d like to run this until 2030.

Manual Exit

In these examples, the programs would automatically exit at some point via an exit statement.

Let’s say we want to exit earlier; we can manually force an exit by running CTRL C in our Terminal.

Exit Cleanup

When our program exits, either automatically or manually, we might want to do some clean up.

Let’s say our program is a stock trading bot. When we exit, we want to make sure all the positions are closed.

Luckily, we can use Ruby’s at_exit block to handle this.

at_exit do
	# close all positions
end

This will run, as we’d expect, when the program ends. We just need to make sure this is defined before exit is called, or it won’t be found.

Exceptions

When we exit the program early via the CTRL C method, everything seems to operate normally, but we also notice that we receive a messy error message in our Terminal.

This is because CTRL C triggers a SignalException and the program terminates. When exceptions are unhandled, our Terminal spews out some stuff that’s supposed to help us out.

If we want to exit the program more gracefully during a CTRL C, we can rescue ourselves from this SignalException using a begin - rescue block.

Begin / Rescue Detour

The boilerplate begin - rescue block looks something like this:

begin
	# do something bad
	1/0
rescue
	'something bad happened'
end

If we want to know what the bad thing was, we can pass the exception object into the rescue.

begin
	# do something naughty
	1/0
rescue => e
	puts e.message
end

This will print:

divided by 0

In addition to message, there are a variety of instance methods available to call on our exception object (full_message, backtrace, etc) to give us whatever info we might want.

One other thing worth mentioning — method declarations are implicit begin blocks, so we can do something like this:

def my_method
	1 / 0
rescue
	puts 'Something bad'
end

(And so are Class and Module declarations.)

Why do we keep using 1/0 as an example? It’s a quick way to trigger an Exception, as programs (just like calculators) cannot divide numbers by zero.

One Large Caveat

If we don’t specify the Exception type on our rescue it will default to a StandardError.

So

begin
	# do something bad
	1/0
rescue
	'something bad happened'
end

is identical to

begin
	# do something bad
	1/0
rescue StandardError
	'something bad happened'
end

StandardError is an umbrella Error class, that contains many of the standard exceptions, and as we’ve seen above, ZeroDivisionError is one of these. There are a dozen or so of these children - link here.

However, the exception created by a CTRL C, SignalException, is NOT a child of StandardError, so it will not be handled by the default rescue. We have to specifically handle this scenario.

begin
	# Code here
rescue SignalException
	puts "CTRL C"
rescue
	puts "StandardError"
end

Rescuing SignalException

Now that we’re able to rescue SignalException, our terminal won’t freak out when we CTRL C.

However, we need to be careful about how we exit at this point.

Since we’re rescuing from this exception, the program will not automatically exit anymore. We need to explicitly tell it to exit.

at_exit do
	# clean up program
end

begin
	# Code here
rescue SignalException
	puts "Program terminated via CTRL C"
	exit
rescue
	puts "StandardError"
end

You might be wondering if adding that explicit exit statement was necessary. The answer is — sort of.

The Third Exit

So far, we’ve seen exits from CTRL C and exit statements in our code. Another way a program will exit is when it simply reaches the end of the program.

Back to the question above - we might get away with not calling exit in our SignalException if the program will exit on its own immediately after.

When the program exits on its own, at_exit is still called, and everything goes as planned.

But it pays to be explicit in this case, because there’s a chance there could be something that happens after the begin - rescue block, but before the program naturally concludes and exits.

For example:

at_exit do
	puts "in at_exit"
end

begin
	sleep 10
rescue SignalException
	puts "Program terminated via CTRL C"
	# no exit call here
end

sleep 100

This program will initially sleep for 10 seconds. If we call CTRL C during this 10 seconds, we will be rescued, but we have removed the exit call.

Now the program won’t exit right away. at_exit won’t run for 100 seconds.

If we add back the exit call in our SignalException, we can guarantee that we immediately exit when CTRL C is called.

at_exit do
	puts "in at_exit"
end

begin
	sleep 10
rescue SignalException
	puts "Program terminated via CTRL C"
	exit
end

One Last Rescue

Everything should be working nicely now. For clarity, we’ll add one last rescue statement.

at_exit do
	puts "in at_exit"
end

begin
	sleep 10
	exit
rescue SignalException
	puts "Program terminated via CTRL C"
	exit
rescue SystemExit
	puts "Program terminated via exit command"
	exit
rescue => e
	puts "StandardError"
	puts e.message
end

This step isn’t completely necessary, but I think it helps understand what’s happening. We’ve added a SystemExit rescue, which captures any exit statement in our begin block. As you may have expected, SystemExit is not a child of StandardError, so we have to explicitly declare it.

This is helpful because we receive information about how the program exited. Every time the program exits, we’ll know whether it came from an exit call or from a CTRL C.

And again, since we’re capturing this in a rescue block, we have to explicitly exit for the same reasons as mentioned above.

The end

Our program can exit via (1) an exit statement in the begin block, (2) a CTRL C user input, or (3) by reaching the end of the program.

(1) is captured by a SystemExit and explicitly exited. (2) is captured by a SignalException and explicitly exited.

(1),(2), and (3) find their way to the at_exit block, allowing us to clean things up.

Let’s say we want to add something to a webpage after the initial load. JavaScript gives us a variety of tools. Perhaps you’ve used some of them, like append, appendChild, insertAdjacentHTML, or innerHTML.

The difficult thing about appending and inserting things with JavaScript isn’t so much about the tools it offers, but which one to use, when to use them, and understanding how each one works.

Let’s try to clear things up.

Quick Context

Your browser converts the HTML tags inside your HTML file into a bunch of objects that can be manipulated with JavaScript. These objects construct a Document Object Model (DOM) tree. This tree is a series of objects that are structured as parent-child relationships.

In DOM parlance, these objects are called nodes, or more specifically, HTML elements.

<!-- I'm the parent element -->
<div>
  <!-- I'm a child element -->
  <span>Hello</span>
</div>

In this example, the HTML span element is the child of the div element, which is the parent.

And I know that some of these terms are weird and possibly confusing. We say “node”, but other times we may say “element” or “object” instead. And, in some cases, they refer to the same thing, just depending on how specific we want to be .

For example, an “element” is a specific type of “node”, just like an apple is a specific type of fruit.

We can organize these terms from most general, to most specific: Object → Node → Element → HTML Element

Understanding these DOM items is important, as we’ll interact with them to add and append things with JavaScript after an initial page load. In fact, let’s start working on that.

Setup

These append and insert methods mostly follow this pattern:

Element.append_method_choice(stuff_to_append)

Again, an element is merely an object in the DOM Tree that represents some HTML. Earlier, we had mentioned that the purpose of the DOM tree is to give us a convenient way to interact with HTML using JavaScript.

So, how do we use JavaScript to grab an HTML element?

Querying the DOM

Let’s say we have the following tiny bit of HTML:

<div id="example" class="group">
  Hello World
</div>

There are a few common ways to query the DOM:

// Query a specific selector (could be class, ID, element type, or attribute):
const my_element1 = document.querySelector('#example')

// Query an element by its ID:
const my_element2 = document.getElementbyId('example')

// Query an element by its class:
const my_element3 = document.getElementbyClass('group')[0]

In this example, all three lines query the same thing, but look for it in different ways. One looks at any of the item’s CSS selectors; one looks at the item’s ID; and one looks at the item’s class.

Note that the getElementbyClass method returns an array. That’s because it’s capable of matching multiple elements in the DOM and storing those matches in an array makes sure all of them are accounted for.

What we can append and insert

// Append Something
const my_element1 = document.querySelector('#example')
my_element1.append(something)

In this example, something is a parameter that represents stuff we want to tack on to the end of (i.e. append to) the matched element.

We can’t just append any old thing to any old object. The append method only allows us to append either a node or plain text to an element in the DOM. But some other methods can append HTML to DOM elements as well.

1. Nodes are either created with document.createElement() in JavaScript, or they are selected with one of the query methods we looked at in the last section.

2. Plain text is, well, text. It’s plain text in that it does not carry any HTML tags or formatting with it. (e.g. Hello).

3. HTML is also text but, unlike plain text, it does indeed get parsed as markup when it’s added to the DOM (e.g. <div>Hello</div>).

It might help to map out exactly which parameters are supported by which methods:

How to choose which method to use

Well, it really depends on what you’re looking to append, not to mention certain browser quirks to work around.

• If you have existing HTML that gets sent to your JavaScript, it's probably easiest to work with methods that support HTML.
• If you're building some new HTML in JavasScript, creating a node with heavy markup can be cumbersome, whereas HTML is less verbose.
• If you want to attach event listeners right away, you'll want to work with nodes because we call addEventListener on nodes, not HTML.
• If all you need is text, any method supporting plain text parameters is fine.
• If your HTML is potentially untrustworthy (i.e. it comes from user input, say a comment on a blog post), then you’ll want to be careful when using HTML, unless it has been sanitized (i.e. the harmful code has been removed).
• If you need to support Internet Explorer, then using append is out of the question.

Example

Let’s say we have a chat application, and we want to append a user, Dale, to a buddy list when they log in.

<!-- HTML Buddy List -->
<ul id="buddies">
  <li><a>Alex</a></li>
  <li><a>Barry</a></li>
  <li><a>Clive</a></li>
  <!-- Append next user here -->
</ul>

Here’s how we’d accomplish this using each of the methods above.

append

We need to create a node object that translates to <li><a>Dale</a></li>.

const new_buddy = document.createElement('li')
const new_link = document.createElement('a')

const buddy_name = "Dale"

new_link.append(buddy_name) // Text param
new_buddy.append(new_link) // Node param

const list = document.querySelector('#buddies')
list.append(new_buddy) // Node param

Our final append places the new user at the end of the buddy list, just before the closing </ul> tag. If we’d prefer to place the user at the front of the list, we could use the prepend method instead.

You may have noticed that we were also able to use append to fill our <a> tag with text like this:

const buddy_name = "Dale"
new_link.append(buddy_name) // Text param

This highlights the versatility of append.

And just to call it out once more, append is unsupported in Internet Explorer.

appendChild

appendChild is another JavaScript method we have for appending stuff to DOM elements. It’s a little limited in that it only works with node objects, so we we’ll need some help from textContent (or innerText) for our plain text needs.

const new_buddy = document.createElement('li')
const new_link = document.createElement('a')

const buddy_name = "Dale"

new_link.textContent = buddy_name
new_buddy.appendChild(new_link) // Node param

const list = document.querySelector('#buddies')
list.appendChild(new_buddy) // Node param

Note that appendChild, unlike append, is supported in Internet Explorer.

Before moving on, let’s consider a similar example, but with heavier markup.

Let’s say the HTML we wanted to append didn’t look like <li><a>Dale</a></li>, but rather:

<li class="abc" data-tooltip="Click for Dale">
  <a id="user_123" class="def" data-user="dale">
    <img src="images/dale.jpg" alt="Profile Picture"/>
    <span>Dale</span>
  </a>
</li>

Our JavaScript would look something like:

const buddy_name = "Dale"

const new_buddy = document.createElement('li')
new_buddy.className = ('abc')
new_buddy.setAttribute('data-tooltip', `Click for ${buddy_name}`)

const new_link = document.createElement('a')
new_link.id = 'user_123'
new_link.className = ('def')
new_link.setAttribute('data-user', buddy_name)

const new_profile_img = document.createElement('img')
new_profile_img.src = 'images/dale.jpg'
new_profile_img.alt = 'Profile Picture'

const new_buddy_span = document.createElement('span')
new_buddy_span.textContent = buddy_name

new_link.appendChild(new_profile_img) // Node param
new_link.appendChild(new_buddy_span) // Node param
new_buddy.appendChild(new_link) // Node param

const list = document.querySelector('#buddies')
list.appendChild(new_buddy) // Node param

There’s no need to follow all of above JavaScript – the point is that creating large amounts of HTML in JavaScript can become quite cumbersome. And there’s no getting around this if we use append or appendChild.

In this heavy markup scenario, it might be nice to just write our HTML as a string, rather than using a bunch of JavaScript methods…

insertAdjacentHTML

insertAdjacentHTML is is like append in that it’s also capable of adding stuff to DOM elements. One difference, though, is that insertAdjacentHTML inserts that stuff at a specific position relative to the matched element.

And it just so happens to work with HTML. That means we can insert actual HTML to a DOM element, and pinpoint exactly where we want it with four different positions:

<!-- beforebegin -->
<div id="example" class="group">
  <!-- afterbegin -->
  Hello World
  <!-- beforeend -->
</div>
<!-- afterend -->

So, we can sorta replicate the same idea of “appending” our HTML by inserting it at the beforeend position of the #buddies selector:

const buddy_name = "Dale"

const new_buddy = `<li><a>${buddy_name}</a></li>`
const list = document.querySelector('#buddies')
list.insertAdjacentHTML('beforeend', new_buddy)

Remember the security concerns we mentioned earlier. We never want to insert HTML that’s been submitted by an end user, as we’d open ourselves up to cross-site scripting vulnerabilities.

innerHTML

innerHTML is another method for inserting stuff. That said, it’s not recommended for inserting, as we’ll see.

Here’s our query and the HTML we want to insert:

const buddy_name = "Dale"
const new_buddy = `<li><a>${buddy_name}</a></li>`
const list = document.querySelector('#buddies')
list.innerHTML += new_buddy

Initially, this seems to work. Our updated buddy list looks like this in the DOM:

<ul id="buddies">
  <li><a>Alex</a></li>
  <li><a>Barry</a></li>
  <li><a>Clive</a></li>
  <li><a>Dale</a></li>
</ul>

That’s what we want! But there’s a constraint with using innerHTML that prevents us from using event listeners on any element inside of #buddies because of the nature of += in list.innerHTML += new_buddy.

You see, A += B behaves the same as A = A + B. In this case, A is our existing HTML and B is what we’re inserting to it. The problem is that this results in a copy of the existing HTML with the additional inserted HTML. And event listeners are unable to listen to copies. That means if we want to listen for a click event on any of the <a> tags in the buddy list, we’re going to lose that ability with innerHTML.

So, just a word of caution there.

Demo

Here’s a demo that pulls together all of the methods we’ve covered. Clicking the button of each method inserts “Dale” as an item in the buddies list.

Recap

Here’s a general overview of where we stand when we’re appending and inserting stuff into the DOM. Consider it a cheatsheet for when you need help figuring out which method to use.

If I had to condense all of that into a few recommendations:

• Using innerHTML for appending is not recommended as it removes event listeners.

• append works well if you like the flexibility of working with node elements or plain text, and don’t need to support Internet Explorer.

• appendChild works well if you like (or need) to work with node elements, and want full browser coverage.

• insertAdjacentHTML is nice if you need to generate HTML, and want to more specific control over where it is placed in the DOM.

I’m making a web app using Rails, and the gem Faker allows us to work with dummy user data before actual non-dummies Log On.

I was using the Harry Potter flavor of Faker’s dummy data, so my users had names like Fluffle BippleBap and emails like fbibblebap@gmail.com.

When a user creates an account, she gets a welcome email. Rails ActionMailer makes this nice and easy.

Since we’re in Rails Land, of course we’re using Devise - this allows us to create and manage user accounts with ease.

Controllers and Models

Since we’ve outsourced our account creation to Devise, we don’t have immediate access to a create action in the users_controller - Devise has it’s own controllers.

But we want to send a user an email after her account is created - seems like this could live happily in the create action that we don’t have.

So we seem to have at least 2 options.

1. We can extend the Devise registrations_controller and add an after_action to the create action that sends a welcome email. I probably should have done this.
2. We can add an after_create callback in the User model that sends the email. Needless to say, this is what I did.

Now this toxic seed was planted weeks before it grew into a full banning from SendGrid, thanks in part to another handy Rails gem called Letter Opener.

Instead of sending an actual email to fbipplebap@gmail.com, Letter Opener just shows us a pretend HTML email in our browser.

The Main Event

So we’re using Letter Opener in our development environment, but not for our staging environment, because in staging, we want to see that emails actually get delivered.

Now you can probably see where this is headed. Here are the magical 10 seconds that led to my banning:

On my Heroku dashboard for the staging site, I added the SendGrid add-on. (3 seconds)

Then, I seeded the staging database: rails db:seed . (3 seconds)

Then, 100 Harry Potter dummies were created. The after_create callback mentioned above is fired 100 times, and 100 emails are sent to Harry Potters. And 100 emails bounce. (4 seconds)

I set up my computer to fail. And SendGrid’s computer is smart enough to notice my 100% bounce rate, and promptly banned me.

A Better Approach

Instead of the after_create callback on the model, I could have extended the Devise controller, and used an after_action .

The difference between these two approaches may seem minor, but the difference between being banned and not being banned is major.

The after_create method will fire any time a record is created for the corresponding model - i.e. User.create This could happen from the console, the seed file, or from inside a controller somewhere. My self-inflicted damage happened in the seed file.

If I had used an after_action in a controller, I would have been safe. This is because the controller action is only called from a specific route - when the user actually interacts with the UI. The controller action is unrelated to me calling User.create somewhere else in my code.

Another Approach

Use real emails! This was actually my first fix. I created an alias email in the Google Apps admin (t@bouncerip.com) that gets forwarded and skips the inbox. And in the seed file, just used something that looked like this:

100.times do |i|
	u = User.new
	u.email = "t+#{i}@bouncerip.com"
	u.save
end

Post Morten

SendGrid, please unban me!