Ever crash after a long night of coding only to wake up realizing you didn’t turn off your EC2 instances and now you owe AWS $10? This is surprisingly easy to avoid in 2021.

This guide will show you how to automatically turn off your instance after a period of inactivity.

1. Find your EC2 Instance and click “+” in the Alarm Status column

Our first step is to find the EC2 instance we want to automatically turn off.

Clicking the “+” icon in the Alarm Status column will open up the page to manage CloudWatch Alarms (the feature we’ll be using)

You should now be on Manage CloudWatch Alarms page.

2. Set the Alarm Action to Stop your Instance

When our EC2 instance becomes inactive, the behavior we want to happen is that our instance will be stopped. To specify the action to take, toggle the Alarm Action section to be on, then select Stop.

3. Set the Alarm Thresholds as Needed

Like a lot of AWS in my opinion, this section is powerful, but not the most intuitive.

Below are the settings I recommend for an inactivity filter. Essentially, when the maximum CPU usage hasn’t been above 2% for an hour, turn off the instance. I also gave my alarm a more readable name.

4. Click Create and get more Peace of Mind

It really is that easy. Whenever I create an EC2 instance, I’m now going to be doing these steps each time to ensure I don’t end up with a big bill at the end.

5. Optional: Add an Email Alert when your instance shuts down

If you’d like to receive an email for extra peace of mind when your instance shuts down, there is an Alarm Notification section where it asks you to type in the name of a new SNS topic to create an email notification.

I actually couldn’t get this to create a new, usable SNS topic / notification. Feel free to comment about your own experience below.

If you do want to set this up, I found that after I created my alarm, I could go back to the EC2 instance overview and click on the Alarm Status cell for my EC2 instance, then go to the full edit mode.

Once on the Edit page, we’ll want to scroll down and click Next to go to the Step 2: Configure Actions section.

Click Add Notification.

Lastly, we’re going to create a new topic by giving it a topic name and specifying your email address.

Click Create topic and click through the confirmation email you’ll receive.

You’re now all set to get email alerts the next time your instance shuts down.

Additional Reading

This is truly just the tip of the iceberg — For more information, check out the resources below:

AWS Guide on CloudWatch Alarm Creation for EC2 instances

AWS Overview on their Instance Scheduler — a way to automatically turn on and off your EC2 instance at set intervals

Have any other tips to more efficiently use your EC2 instances? Leave them in the comments below!

Prop 16 is on the ballot and it’s confusing as heck. Ensuring racial equality shouldn’t be this twisted. Here’s my breakdown.

In a few weeks, California will be voting on Prop 16, a repeal of Prop 209 which banned discrimination or preferences “on the basis of race, sex, color, ethnicity, or national origin in the operation of public employment, public education, or public contracting.”

Since Prop 209 passed in 1996, California has been 1 of 9 states that ban “affirmative action”, a reference to a 1961 executive order from JFK ordering federal contractors to “take affirmative action to ensure that applicants are employed and that employees are treated during employment without regard to their race, creed, color, or national origin”

Well, this is easy, I’m against discrimination so why would I vote “yes”?

Like most things in the world, nothing is ever as easy as it seems. This is no exception.

1996, when it all started

Since we’re considering repealing Prop 209, let’s dive into how we got Prop 209 passed in the first place.

Prior to Prop 209, California set goals for how many contracts they awarded to women-owned and minority-owned businesses, as well as evaluated race as a contributing, but not an exclusive factor to university admissions.

Quota systems were not part of the original nor current proposition, they have been banned by the supreme court since 1978, and won’t be making a comeback regardless of the outcome of this vote.

Prop 209 was passed in 1996 with the support of 54.5% of that year’s voters, nearly three-quarters of whom were white and voted for it 63% to 37%.

Understanding both sides

CalMatters.org is an excellent resource to understand both sides of a given ballot measure. They have summarized the arguments for Prop 16 below:

Supporters of Prop 16 say:

Structural racism exists and to preach a color-blind philosophy is to be blind to the impacts of racism. Instead, for example, principals should be able to specifically seek to employ qualified Latino teachers in a school where most teachers are white but most students are Latino. Public universities should be able to consider a student’s race as one of numerous admissions factors, including grades and school work.

Opponents of Prop 16 say:

Allowing schools and government offices to make decisions based on race, ethnicity or sex is its own kind of prejudice. Equal rights mean everyone is treated equally. The claim that America is systemically racist is a false narrative that “fuels racial paranoia, division and hatred.” The state already has made strides in diversity. And it’s legal now to give preference to students who really need it — those who grew up in low-income families. As for who gets into the public universities, the fault lies with inadequate K-12 schooling.

Prop 16 boils down to a vote on how you recognize and want to solve systemic discrimination based on race and gender.

Systemic racism and discrimination are not comfortable, easy, nor straightforward topics.

Systemic racism is a form of racism that is embedded as a normal practice within society or an organization. I would hope we can agree that throughout America’s history, there have consistently been laws, policies, and regulations meant to keep groups of people at a disadvantage, whether that’s women, African Americans, Latinos, Native Americans, Asians, or any other minority group. To varying degrees throughout history, every non-white group has had to overcome challenges that were a direct result of white men wanting to stay in power.

Tying this back to Prop 16, the naive argument to vote no usually goes something like these comments pulled from an educational video on Prop 16.

You should be hired on your qualifications not the color of your skin

I’m voting NO. I say to the best qualified individual. It motivates all people to try harder to get what they want rather given a hand me down or pass for race.

The only systemic racism taking place is that towards white people.

If you don’t recognize that systemic discrimination has played a large role in America’s history and that the ramifications are still being felt today, then each of these arguments makes perfect sense.

It is indeed much simpler and easier to imagine the world being fair to everyone and that everyone has an equal opportunity to succeed in America, it certainly feels American to say that.

Is Affirmative Action the right response to Systemic Discrimination?

Frankly, to me, I find the naive arguments of “Vote No because everyone has an equal opportunity, they just need to work harder” relatively easy to counter by pointing out simple, indisputable facts about how groups have been systemically discriminated against.

It does get more nuanced once you do recognize that systemic racism exists and we look to the possible solutions to reconcile those imbalances.

To me, this issue has two main options:

Option 1: Do not leverage Affirmative Action and trust that those in power today will, on their own, introduce policies and initiatives that correct the historical discrimination women and minority groups have faced — and within a timeline competitive with other options seeking to right these discriminations.

Option 2: Allow the possibility of Affirmative Action to level the playing field by considering more minority contenders for university admissions and government contracts. — Affirmative action isn’t a slam dunk, repercussion-free option. By definition, it would raise the bar for white men to compete for opportunities and it’s a bit of a “fighting fire with fire” approach that can feel uncomfortable to some.

To me, Affirmative Action is not an amazing choice, but it is better than the alternatives. With Affirmative Action, it in no way addresses the heart of the issues, say, quality K-12 education that gives everyone an equal opportunity to compete during and after their high school years, or the myriad other causes of systemic discrimination.

You don’t want discrimination to be legal, do you?

Like many things in our political system today, there’s a game to play when introducing laws and policies. Efforts to codify the intent and ensure things at the surface-level seem great have been used over and over again, it’s practically in the playbook.

In this case, it manifests itself as “You don’t want discrimination to be legal, do you?” - Then vote “No”.

And sure enough, it seems successful.

Watching a focus group with Black voters from Los Angeles, they all said no we won’t vote for this as it was read to them,” said Eva Patterson, who co-chairs the Yes on 16 campaign. “Then we explained that it was in favor of affirmative action and equal opportunity, and they all said, ‘Of course we’ll vote for this.’
(ABC News)

The team introducing Prop 209 back in 1996 expertly knew how to word this to kill affirmative action in California and ensure that efforts to bring it back would be extremely difficult. They seem to be counting on an uneducated voter base.

Who’s voting “Yes” on Prop 16?

• Governor Gavin Newsom
• Mayors of Los Angeles, San Francisco, and San Diego
• Many Democratic senators, representatives, and other legislators
• ACLU, Anti-Defamation League, California Hispanic Chambers of Commerce, California Teachers Association, many other groups
• Kaiser Permanente and PG&E

Who’s voting “No” on Prop 16?

• 2 Republican State Senators
• Ward Connerly, the person who introduced Prop 209 in 1996.
• Organizations helping Asian students get into California’s premier schools, like IvyMax.

Where can I get just the facts, no opinions?

BallotPedia

CalMatters.org

It took me a really long time to unpack what was going on with this proposition and to get to the heart of the issues we’ll be voting on.

My goal in writing this is to provide a breakdown of the issue as I’ve learned about it in hopes that you’ll be able to form a more educated and precise opinion on the matters at hand.

Do your own research. I don’t have all the answers myself. I’d love to understand your perspective.

Since joining Scale AI as a Customer Engineer, I’ve been diving deep in to all things self-driving and trying to learn as much as I can about AI, Computer Vision, and anything remotely related to those things.

I recently signed up for Udacity’s Self-Driving Car Nanodegree program and found the first project around Lane Finding with Computer Vision techniques to be very engaging.

The Problem:

The goal is to take videos of highway driving and annotate the vehicle’s lane markings by creating an image pipeline to identify and ultimately draw lane markings.

In the end, our output will look like this:

My Approach:

We’ll be going into the details below, but at the highest level, the approach I developed was:

1. Extract only the white and yellow-related pixels from a given video frame
2. Convert image to grayscale and apply a slight gaussian blur to remove noise
3. Calculate edges using a canny edge detector
4. Apply a regional specific mask to focus on the area on the front of the car only
5. Run a hough transform over the remaining edges to identify straight lines
6. Split lines into “left” and “right” by their slope, calculate median slope and y-intercept for all “left” and “right” lines
7. Calculate line intersection point and x-intercept points
8. Keep track of the last few frames line intersection points and x-intercept points as well for smoothing from frame to frame
9. Draw smoothed intersection and x-intercept points on the frame
10. Grab some popcorn and enjoy the smooth as butter lane annotations

I then end this blog with some reflection and thoughts about enhancements.

1. Extracting the White and Yellow Pixels

To start off, I want to explain why this step is a game-changer.

If you were to jump straight into detecting edges and calculating lines without first extracting the color hues you care about, your resulting edges would look something like what’s below for this image:

Without color isolation, if you look closely, the right, long line isn’t even on the actual lane marker, it’s the area where it goes from shadows to the sun.

With color isolation, the yellow and white line markings come through clearly in the image.

Enter the HSV Color Space

We usually think about colors in terms of Red, Green, and Blue or RGB values, where different ratios of Red, Green, and Blue make up the color a pixel has.

With RGB, a pixel gets its color from how bright or intense each of the three colors are, but that becomes an issue when you instead want to find colors of varying brightness.

HSV flips RGB on its head and separates a pixel’s color into Hue, Saturation, and Value. All the “Yellow”-ish pixels would have about the same hue value regardless of it was a dark yellow or really bright yellow, which is exactly what we want since roads and lighting conditions change a lot!

Extracting the Colors:

I had to do some playing around with an online tool myself to figure out how variations of yellow and white were represented, and then applied it to my image by creating a mask of where the colors I want are and removing everything else.

# Convert image to HSV colorspace
    hsv_image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)

# Define range of colors in HSV
    lower_yellow = np.array([30,80,80])
    upper_yellow = np.array([255,255,255])
    lower_white = np.array([0,0,200])
    upper_white = np.array([255,20,255])

# Threshold the HSV image to get only blue colors
    yellow_mask = cv2.inRange(hsv_image, lower_yellow, upper_yellow)
    white_mask = cv2.inRange(hsv_image, lower_white, upper_white)
    mask = cv2.bitwise_or(yellow_mask, white_mask)

# Bitwise-AND mask the original image
    color_isolated = cv2.bitwise_and(image, image, mask=mask)

A quick tactical note, you need to convert HSV values into a scale of 0 to 255 instead of say, 0–360 degrees for a Hue or “70%” Saturation.

2. Converting Image to Grayscale and Blurring

Blurring is a common technique to remove “noise” from particular pixels, this helps lines and edge transitions be more smooth instead of jagged.

# Convert to Grayscale
    gray = cv2.cvtColor(color_isolated,cv2.COLOR_RGB2GRAY)

# Define a kernel size and apply Gaussian smoothing
    kernel_size = 5
    blur_gray = cv2.GaussianBlur(gray,(kernel_size, kernel_size),0)

3. Edge Detection

We want to extract all the edges around the colored regions to extract the specific lines in our image, we’ll want to use that in a few steps to approximate the lane lines.

I used the Canny Edge Detector algorithm with some reasonable detection thresholds.

# Define our parameters for Canny and apply
    low_threshold = 50
    high_threshold = 150
    edges = cv2.Canny(blur_gray, low_threshold, high_threshold)

4. Regional Masking

You may notice there’s a bunch of noise hanging out on the far left and right of the image.

Given the fixed camera position on the vehicle, it is safe to assume that the lanes in front of the vehicle that we care about will always be roughly in the same parts of the image.

We can remove other potential lines and noise in our image by applying a regional mask that allows only pixels in the region to exist, making all others black.

I came up with this region to limit my lane searching to:

We use OpenCV’s fillPoly method to create a regional mask and apply it to our image.

# Next we'll create a masked edges image using cv2.fillPoly()
    mask = np.zeros_like(edges)   
    ignore_mask_color = 255

# Define a four sided polygon to mask
    imshape = image.shape
    vertices = np.array([
        [
            (0,imshape[0]), # bottom left
            (imshape[1] * .35, imshape[0] * .6),  # top left 
            (imshape[1] * .65, imshape[0] * .6), # top right
            (imshape[1],imshape[0]) # bottom right
        ]
    ], dtype=np.int32)
    
# Do the Masking
    cv2.fillPoly(mask, vertices, ignore_mask_color)
    masked_edges = cv2.bitwise_and(edges, mask)

We now have an image looking like this:

5. Get lines with a Hough Transform

A Hough transform is one of the cooler and more mind-blowing things in Computer Vision for me.

I won’t be able to explain it fully here, but it essentially maps every pixel to a linear equation where the 2d axes go from “x” and “y” to “m” and “b” representing the parts of the equation for a line, y = mx * b , and then we put everything in polar coordinates instead of cartesian coordinates to avoid dividing by 0 errors, lots of fun!

The important part is that we get a set of points, x1, y1, x2, y2 representing the start and endpoints of lines it found in our image.

# Define the Hough transform parameters
    rho = 2 # distance resolution in pixels
    theta = np.pi/180 # angular resolution in radians
    threshold = 30 # minimum number of votes 
    min_line_length = 20 # minimum number of pixels making up a line
    max_line_gap = 1 # maximum gap in pixels between lines

# Run Hough on edge detected image
    # Output "lines" is an array containing endpoints of lines
    lines = cv2.HoughLinesP(
        masked_edges, rho, theta, threshold, 
        np.array([]), min_line_length, max_line_gap
    )

6. Identify our Lane Equations

Our Hough transform gave us a set of line endpoints for any lines that it found in our last image above. Hopefully, most of these lines are going to be representing the long edges of the lane markings. There can be 5, 10, or even 20+ lines returned sometimes depending on the level of noise and lane markings captured.

Step 1: Break out the middle school math and calculate the m and b parts for every line with our handy rise/run formula.

# Iterate over the output "lines" to calculate m's and b's 
    mxb = np.array([[0,0]])
    for line in lines:
        for x1,y1,x2,y2 in line:
            m = (y2-y1) / (x2-x1)
            b = y1 + -1 * m * x1   
            mxb = np.vstack((mxb, [m, b]))

Step 2: Separate line equations into the left and right lanes and calculate the median slope and median y-intercept.

I chose to use the median to reduce outliers, outliers are actually quite possible if we happened to draw a line for another part of the road or scene that doesn’t relate to a line marker directly.

An important realization is that the left lane and right lane markers will inherently have opposite slopes, such that one is positive and one is negative.

We’ll do some fancy array indexing to calculate our median slope and intercept for the left and right lines of our slope.

median_right_m = np.median(mxb[mxb[:,0] > 0,0])
median_left_m = np.median(mxb[mxb[:,0] < 0,0])

median_right_b = np.median(mxb[mxb[:,0] > 0,1])
median_left_b = np.median(mxb[mxb[:,0] < 0,1])

Awesome, we have our linear equations for our lines!

7. Calculate Line Intersection Point and X-Intercept Points

Now that we have our two lines, let’s figure out where they intersect with each other, as well as the bottom of the image. This will be useful when we want to draw them on our original image.

For a refresher, here’s some equations from Wikipedia given ax+c = bx+d.

# Calculate the Intersect point of our two lines
    x_intersect = (median_left_b - median_right_b) / (median_right_m - median_left_m)
    y_intersect = median_right_m * (median_left_b - median_right_b) / (median_right_m - median_left_m) + median_right_b

# Calculate the X-Intercept Points
    # x = (y - b) / m
    left_bottom = (imshape[0] - median_left_b) / median_left_m
    right_bottom = (imshape[0] - median_right_b) / median_right_m

8. Tracking Line History for Smoothing

If we stopped here, we would have lines drawn on our images, but when you put these annotations into a video, you’ll see a lot of jumpiness as slight differences in the line equations and how the edge detection algorithms highlight pixels cause minor variations in our equation variables.

The solution is to sacrifice a small amount of responsiveness for a smoothed transition from the previously seen line equations.

When used in a video setting, my algorithm will actively take in as input, utilize, and return a history of previously seen line equations. We’ll see what that looks like further below.

# Create a History array for smoothing
    num_frames_to_median = 19
    new_history = [left_bottom, right_bottom, x_intersect, y_intersect]

    if (history.shape[0] == 1): # First time, create larger array
        history = new_history
        for i in range(num_frames_to_median):
            history = np.vstack((history, new_history))

    elif (not(np.isnan(new_history).any())): 
        history[:-1,:] = history[1:]
        history[-1, :] = new_history

# Calculate the smoothed line points
    left_bottom_median = np.median(history[:,0])
    right_bottom_median = np.median(history[:,1])
    x_intersect_median = np.median(history[:,2])
    y_intersect_median = np.median(history[:,3])

It turns out lanes move relatively slowly from one video frame to the next so we can have a relatively high count of how many frames we’re taking the median of to calculate the current line.

After some experimentation, I chose 19 frames to take the median of, which at 25 frames per second, amounts to a theoretical delay of (19 / 2) / 25, or 0.38 seconds. When doing this, you balance smoothness with delay.

9. Drawing Lines on Video Frames

Step 1: Once we have our final line points, we are able to draw lines on a single image.

# Create a blank image to draw lines on
line_image = np.copy(image) * 0

# Create our Lines
cv2.line(
    line_image,
    (np.int_(left_bottom_median), imshape[0]),  
    (np.int_(x_intersect_median), np.int_(y_intersect_median)),   
    (255,0,0),10
)

cv2.line(
    line_image,
    (np.int_(right_bottom_median), imshape[0]),
    (np.int_(x_intersect_median), np.int_(y_intersect_median)),
    (0,0,255),10
)

# Draw the lines on the image
    lane_edges = cv2.addWeighted(image, 0.8, line_image, 1, 0)

Step 2: Process an entire Video

history = np.array([[0, 0, 0, 0]]) # Initialize History
images_list = []

# Read in Base Video Clip
base_clip = VideoFileClip(input_filename)

# Process each video frame, note how "history" is passed around
for frame in base_clip.iter_frames():
        [image, history] = draw_lanes_on_img(frame, history)
        images_list.append(image)

# Create a new ImageSequenceClip, in other words, our video!
clip = ImageSequenceClip(images_list, fps=base_clip.fps)

# Save our video
%time clip.write_videofile(output_filename, audio=False)

10. Enjoy some smooth Lane Annotations

Below are some example videos processed with the above code:

Reflections

What could be done still:

I had three ideas I didn’t yet implement in the above code.

Creating a Custom Kernel for Edge Detection

In Step 2 and 3, we applied a blur to our image to remove noise and then found edges. While these are things you can do to convolve an image, there’s all kinds of different kernels you can build to highlight, lessen, or otherwise modify a pixel’s value based on the pixels around it.

In theory, since lanes have somewhat fixed angles, we can build a custom kernel that highlights pixels where the pixels above and below it form an angled line.

A Sobel filter is another kind of edge detector useful for finding, very specifically, straight lines in images. We could borrow similar concepts from how Sobel filters work to create an edge detector for angled lines at the approximate angles lane line follow.

Filtering Hough Transform Lines

While most lines coming back from our Hough transform are hopefully related to the lane markers, we can engineer out any obvious lines where that is not the case.

Directly in the Hough transform, or in the processing of the lines, we should be able to filter out, say, any horizontal lines as we know those could not be related to a lane marker.

Leverage Parallel Lines

Limitations:

No Lane Switching
This algorithm assumes you are already in a lane, and will stay in that lane. When you are in the middle of switching lanes, your line angles become near vertical and I think this algorithm would struggle with that given how I am separating lanes into “left” and “right”. A better approach would be to draw and track ALL lanes seen and then compute the car’s position relative to the lanes.

Lane Markers must be Present and Explicit
Typically, lane markers on highways are pretty explicit. If a lane marker was simply not painted, covered over by something, or if a highway exit showed up, this algorithm wouldn’t know what to do.

All in All

I had so much fun working on this project. There are so many much better ways to solve this in 2020, but getting back to basics and seeing how far you can take classic computer vision techniques was a blast!

As always, feel free to comment with your thoughts, connect with me on LinkedIn, or otherwise email me at shaun.t.vanweelden [at] gmail.com

Lane Finding with Computer Vision Techniques was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

I don’t know about your team, but at least with the team I’m on, we have a lot of crazy, silly ideas that are very fun to discuss, but likely shouldn’t ever happen to preserve the sanity of those around us.

After seeing our sales team celebrating closed deals with a smash of the carnival hammer game, and our CSMs celebrating closed renewals with a gong-type thing, I’ll admit we were feeling a bit left out.

We started brainstorming what we could do, and our conversation about “how everyone looks cool dancing in strobe lights no matter what” turned into “We should have a fog machine and strobe light for a few seconds every time we close a conversation”

Alas, the smoke machine wasn’t meant to be.

Despite not having a smoke machine, I REALLY wanted to do something now that I was brainstorming.

I arrived at Christmas Lights blinking on and off every time we closed a conversation which seemed both subtle enough to have in the office and timely for the season.

System Overview

At a high-level, we need a few things to happen to make this work:

1. A smart plug that can be reachable with an API endpoint
2. A support tool that can fire a Webhook when a conversation is closed (most have this, more below)
3. Some kind of server or service that can receive the webhook from the support tool and hit the smart plug’s API to toggle the state

Smart Plug Selection

I do not have a ton of experience in the “Internet of Things” or “Smart Home” space so this was my first real introduction.

All Smart Plugs will have a mobile app that is paired with them, and then you control your Smart Plug primarily with that, or you hook it up with Alexa or something like IFTTT (If This Then That — a very consumer-ized version of Zapier for better and worse).

Since no one has built a Support Tool -> Smart Plug service before, there isn’t anything out of the box for us to use and we’ll need our own way to connect to it.

You must buy a Smart Plug that you can toggle on and off with your code or service of choice. Determine how you’ll turn it off with your code or service BEFORE deciding which plug to get.

For myself, I already had a Node.js server up and connected to our Support Tool for some other operational things. The first thing I googled was “smart plug node sdk” and I found the link for https://www.npmjs.com/package/tplink-cloud-api

This one seemed (and was) perfect because I’ll be able to use my smart app credentials to authenticate and mimic the requests I’d make from the app on my phone.

In conjunction with this Node.js library, I bought this Smart Plug on Amazon:

https://www.amazon.com/Smart-outlet-Google-Simple-Required/dp/B07TXM4MT3

Support Tool Webhook

A webhook is a way for a service such as Help Scout or Zendesk to let you know that something just happened on their platform. When an event occurs within the platform, it triggers a call to be made to a URL you specify, usually pointing to an API endpoint you’ve designed. The call it makes to your URL contains the data you need to take action.

In our case, we want a call to be triggered when a conversation is closed.

Now that we have a webhook set up, our server will get a ping every time a conversation is closed.

Putting it Together

With a Smart Plug that you can toggle on and off with code, and a way to get notified when a customer conversation is closed, we just need to put these two pieces together and we’ll be set.

Since I was using Node, I built out a helper class like this following the Node SDK for this Smart Plug.

Next, add a Help Scout Webhook function like this:

My Node.js app simply uses Express.js to handle the routing and all of that.

Depending on the plug you buy and your preference for hosting code, your mileage may vary.

Learnings so far:

1. Easy to work with APIs for “Smart Home” devices seem to be really hard to find. It makes sense given how consumer-driven and non-technical the market is, but when you want to be technical… it was hard.
2. Similarly, I was really hoping to abstract this project out so it wouldn’t be as technical. I had explored services like AWS Lambda, Zapier, and IFTTT to see if I could get to a more “Plug n Play” model for other support leaders wanting to do this. I haven’t had a lot of luck so far, but if you have ideas, I’d love to hear them!
3. It’s not the most robust lighting system ever — If it gets two “close” requests at about the same time, it will get off it’s toggle track and the lights will stay on indefinitely. Having the mobile app handy to turn them off when that happens is nice.. :)

Most people don’t realize this is something you can do by yourself in about 30 seconds with no tools, apps, or money needed.

There’s only 4 steps needed to download public or private videos to your computer.

1. With Google Chrome, use the actual Instagram website to get to the page where the image or video you want to download is.
   If the page is private, you will need to log in to your Instagram account first.

2. Click on the Image or Video you want to download to make it “full screen”

3. Right-click directly on the Image or Video, then click “Inspect”

This crazy looking HTML should pop up:

4. Get the URL to your Image or Video

You’ll see an HTML bit starting with “<div …” highlighted on the right.

For Images:

There’s a line right above the line that’s highlighted also starting with “<div …”.

Click the “▶” next to the “<div …” that’s above what’s highlighted.

You’ll see a bunch of URL-looking things, the one we want to grab is the bottom one. It starts with src=”long link”.

Copy that URL and paste it into a new tab. You can do whatever you want with the image now.

For Videos:

There are two lines right above the line that’s highlighted also starting with “<div …”.

Click the “▶” next to the “<div …” that’s two lines above what’s highlighted.

Keep clicking the “▶” button until you can not any more and instead see URLs.

Find where it says type=”video/mp4" src = “ … “ — Copy the URL for src and paste in a new tab.

Click the three dots and “Download”, you’re done!

How to Normalize Websites in Salesforce and Excel

I was recently tasked with seeing how many customers on a list we had were shared with lists others were providing us.

No problem I thought, I’ll just export the report in Salesforce, do a quick VLOOKUP or COUNTIF based on the Account Website in Excel, and Voilà, I’ll have my list!

I exported my report and realized it wasn’t going to be quite as easy… we had all types of values in our “Website” field and the lists to compare to also had different domain formats.

I needed a way to normalize everything together:

In this post, I’ll share how I made:

1. A Normalized Website Formula Field in Salesforce

2. The Excel version of that Formula field so you can normalize other lists

3. A Process Builder flow so the Website field is always normalized, and you don’t need to create a second Website field.

The Recipe

First things first, I made a list of what had to be normalized:

1. Remove https://
2. Remove http://
3. Remove anything after the first remaining “/” (including the slash itself)
4. Remove “www.”
5. Make it all lowercase

The one thing you could argue should also be removed is any additional subdomains. “blog.google.com” should in theory be “google.com”.

This specific case becomes quite challenging considering domains like “blog.gsuite.google.co.uk” to the point where I find it’s not worth the complexity.

Luckily for us, the probability that website fields include sub-domains is generally very low and can generally be dealt with manually or in the acceptable margin of error.

Salesforce Formula for Normalized Website

SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(
  LEFT(
    LOWER(Website),
    IF(FIND("/", Website, FIND(".", Website)) = 0,    
      LEN(Website),
      FIND("/", Website, FIND(".", Website)) - 1
    )
  ),
"https://", ""), "http://", ""), "www.", "")

If you copy and paste this into a new text-based formula field on the Account, you’ll have a lovely normalized website field and be all set.

Break it down

If you find longer formulas interesting and fun like me, here’s a breakdown of what’s happening starting from the middle outwards.

We start off with LOWER to ensure casing never gets in our way of comparing websites.

Next, we need a way to see if there’s a “/” or things after a “/” at the end of our lower-case domain to remove. We’ll use LEFT to get everything to the left of our potential “/”. LEFT takes in two arguments:
1. Text to potentially cut off
2. The Nth letter in the text where it should cut things off

To get the Nth letter in our website for LEFT, we need to see IF there’s a “/” somewhere in the domain. To FIND if there is a “/”,
1. We pass in our search text, “/”
2. We pass in the text to look at, Website
3. We pass in the first place in that text we want to look for our slash. In this case, we want to look after the first “.” we see because we need to avoid finding a “/” in “http://” and we can count on there always being a “.” before a final slash because of the “.com” or similar part.

IF there is not a “/” in Website (as indicated by FIND returning 0), return the entire Website by getting the length of the Website field.
IF there was a “/”, only return the first N letters of the Website field before the “/”

Now with our LOWER-cased text and everything to the LEFT of a potential “/” at the end of our website, SUBSTITUTE out the “www”, “http”, or “https” part of the domain with an empty string.

🎉 You made it!

Excel Formula for Normalized Website

Luckily for us, Salesforce and Excel formulas are quite similar. We can use essentially the exact same logic in Excel as Salesforce.

=SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(
  LEFT(
    LOWER(A2),
    IF(ISERROR(FIND("/", A2, FIND(".", A2))), 
      LEN(A2),
      FIND("/", A2, FIND(".", A2)) - 1
    )
  ),
"https://", ""), "http://", ""), "www.", "")

If you’re paying attention, the only change here is that we need to swap out our IF statement to use ISERROR instead of “= 0” because if text can’t be found in Excel, it returns an error instead of just 0.

Normalized Website Process Builder

If you too like to avoid making unnecessary fields but don’t want to give up this normalization efficiency, you can get a bit creative in Process Builder.

Goal:

On any changes to an Account’s Website field, re-normalize the value in the Website field. This works well when doing data imports from different vendors.

Create a New Process

Our Process is for the Account object

Define our Criteria

If our Website field changed, let’s go update it!

Define our Action

Have the Website be set to our formula above with one small twist, we need to explicitly tell it what object to pull our fields from (Account).

SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(
  LEFT(
    LOWER([Account].Website),
    IF(FIND("/", [Account].Website, FIND(".", [Account].Website)) = 0,    
      LEN([Account].Website),
      FIND("/", [Account].Website, FIND(".", [Account].Website)) - 1
    )
  ),
"https://", ""), "http://", ""), "www.", "")

Click Use this Formula and Save

Turn it On

Click Activate, and maybe do a quick data load to set all existing websites with their normalized version, and you’ll be all set!

Oh, The Places You’ll Go

Hopefully this saves you a bit of time, gives you some inspiration, and helps you get back to making the most of your data instead of fighting domain variances.

If you have ideas, I’d love to hear them! Please leave a comment or find me on LinkedIn.

Setting up GoDaddy and Heroku was a particularly frustrating experience and I thought I’d try and help you avoid the same pitfalls.

As it turns out, it’s impossible to perfectly get Heroku and GoDaddy to play together well, but below is as close as you can get.

Why connecting GoDaddy and Heroku is challenging:

1. Most platforms give you a CNAME record to point your www subdomain to, and then some IP Addresses for your A records to point to. Heroku only provides CNAME records and ANAME or ALIAS records GoDaddy does not have.
2. You can add Custom Domains on the Heroku site, but the domain format isn’t clear, should I add the www subdomain and the “naked” domain, one, or the other?
3. When you do add a Custom Domain in Heroku, the domain to point to is cut off on the Heroku website.
4. SSL requires you to upgrade to a Hobby Plan in Heroku, and then https needs to still be managed somewhat carefully on the DNS side.

Initial Goals:

Going to:
http://yoursite.com
https://yoursite.com
http://www.yoursite.com
https://www.yoursite.com

All work and end up, without scary warnings, at https://www.yoursite.com

GoDaddy does NOT Support Heroku

Getting “https://yoursite.com” to go to your website without a scary warning is not something that is possible today when using GoDaddy.

Heroku documents why over here:
https://help.heroku.com/NH44MODG/my-root-domain-isn-t-working-what-s-wrong

Thank you @Aidlab for providing the link and context.

Final Solution Preview:

In Heroku:
1. The ONE domain you need to set up is the WWW version, www.your-site.com, do not set up your-site.com
2. Pay $7 a month to upgrade to the hobby plan because your site will load faster and you can have https://

In GoDaddy:
1. Add a CNAME record for “www” pointing to the site Heroku provides
2. Set up Domain Forwarding to point to the “https” version of your “www” domain.

Step by Step Guide:

In Heroku:

Step 1: Add your Custom Domain
Use the Heroku CLI to add your domain instead of doing it from the web application. Ensure you add the “www” version instead of just “yourdomain.com”

heroku domains:add www.yourdomain.com

Step 2: Retrieve the DNS Target
If it’s not already there, you can always type the following to get a list of your domains

heroku domains

Step 3: Double check SSL is enabled
If you’re like me and weren’t on a paid project when setting up your custom domain, you need to log in to Heroku and go to your Application Settings page and enable SSL.

In GoDaddy:

Step 0: Capture Original State
Before making changes, always take a screenshot or note of what settings looked like

Step 1: Remove Extraneous A Records
Remove any A records pointing to other IP Addresses from past projects or the GoDaddy defaults.

Step 2: Add a CNAME Record
Your CNAME record will have
Name = www
Value = Heroku domain from Step 2 above

Step 3: Set up Forwarding
Heroku mentions setting up ANAME or ALIAS records, GoDaddy does not have that. Instead, that is done through Forwarding. I personally spent way too long here trying to figure that one out…

You want to forward with the “https://” protocol to your “www” sub-domain.

In Conclusion:

Hopefully this saved you a few clicks and minutes. Like all domain stuff, there can be some nerve-wracking minutes while things propagate, but hopefully this will allow things to make much more sense!

Traditionally, there have been 3 main models for doing customer support for a SaaS-based business; Email, Phone, and Chat.

When I get asked how Engagio supports customers, I have always struggled to put it in one of those buckets.

At Engagio, we, like many SaaS-based startups turned to tools like Intercom to offer our customers an easy way to get in touch with us, ask questions, and get support.

The premise of Intercom (and other similar tools) is pretty simple, customers click the chat bubble, they open it up, and type their question. We get back to them, and everyone is happy.

So it seems like we have a chat experience going on, heck, we can even add emojis.

One issue. There’s a little line below the friendly faces.

The Team typically replies in a few hours.

Clearly, we’re not about to have a full chat agent experience, and I probably shouldn’t be expecting a reply in seconds.

So it’s not Chat

I think it’s fair to say, using support tools like Intercom is not typically a full blown chat experience with wait times measured in the seconds. At the very least, it does not have to be that model of support.

Many SaaS support teams, including Engagio’s, simply don’t have the bandwidth to fully support a “chat” experience all day every day, but we want customers to have a seamless experience with us and we want to be able to communicate quickly back and forth when we are able to.

When I reply late at night, you’re not in the app, and you get an email copy of my response that you reply to the next day; we’re now very much emailing about your question more than anything else.

While some customers may have a few hour wait period throughout the week, if you send a question over to us at 2pm on Friday when it’s more quiet, it’s very possible we’ll reply 18 seconds later with an answer and can help you in real-time get things figured out.

So it’s not Email either

Going back and forth in a matter of seconds with a user experience modeled after iMessage to the point where you have the “___ is currently typing …” appear alongside buttons for GIFs doesn’t really feel like email either.

From both an agent and a customer perspective, it can very much operate like you’re chatting with a friend. Something tells me that this experience has been a consistent part of Intercom’s Success.

So if it’s not Chat, and it’s not Email, what is it?

It took me a long time to see it, longer than I’d like to admit, but it’s like texting or messaging.

When you text a friend, you know they can get back to you in 15 seconds, but more than likely, there’s going to be a few minute to a few hour delay between responses because they have their own things they are working on and so do you.

It’s mutually understood that you’ll both do your best to get back to each other as quickly as you can, and the conversation can go on in an ad-hoc way as you both get more context and get things figured out.

After using Intercom for over two years as we grew our support team with the coincidental “messaging” approach, I’ve been reasonably satisfied with it’s scalability.

As you can imagine, handling 50 different text conversations with friends in a given day would be error-prone and hard to stay on top of, things would get lost, there’s a lot of cherry picking the easy stuff. Setting up workflows, teams, and multiple inboxes to handle the increased load is critical, and this is where I feel the tools to start differentiate themselves quite a bit.

Is the “Messaging” Support Model here to stay?

I honestly think so, especially in the software and startup space. I think it strikes a good balance between timeliness, experience and accessibility, and scalability.

As society becomes more text and messenger driven, I feel we are already accustomed to the experience and willing to take it into our work-life as well.

What are your thoughts, do you have a better name for this support model? Does it deserve it’s own channel?

Leave a comment or get in touch!

If you’re a Salesforce Admin, there’s a good chance you’ve checked out the “Bulk Data Load Jobs” admin page to monitor the status of a big batch update or debug why an integration push didn’t go as expected.

This page is unique, it’s the ONLY place in Salesforce to see:
➤ What Object was Updated
➤ # of Records Processed
➤ # of Records Failed
➤ Job Start Time

For a lot of debugging, you’ll want to count the total number of records updated in a given time window, or figure out how many records a specific user submitted for updates.

This is all super helpful data, but there’s a catch.

The table view has two main issues,
1) It only shows ~50 records per page in Lightning and 110 per page in Classic
2) Clicking “View more records” on the page only adds 10 records, 10!

There is no way to export this data, no way to query it with SOQL or Apex, and it’s a total pain in the ass to page through it manually.

Lucky for you, I was determined to get the full set of our batch jobs into Excel.

My Attempts to Solve This

Write a SOQL query

My end goal was to filter this data down by the parameters in the table, and then export it. A SOQL query seemed perfect. I found the AsyncApexJob object which seemed to roughly mirror what the table was showing, but it was missing a few critical updates, namely the User who submitted the request and # of Records updated.

Write JavaScript to click the “View More” button programmatically

Trying to view the last 3,000 batch jobs by slowly paging through wasn’t going to happen manually.

I know how to write some code though, I thought, let me try to click the “View More” button with JavaScript every few seconds. I’ll let it sit for an hour and come back to the full table view, it sounded great.

It didn’t go very well…

1. In Lightning Mode, the table is actually loaded as an iframe on the page holding a VisualForce component, trying to click on things programmatically in an iframe that reloads itself every time it’s clicked was proving harder than imagined.
2. In Classic, the page refreshed itself every time the “View More” or “Next Page” button was clicked so looping through with JavaScript wasn’t working out well. This ultimately proved to be key to the solution.

The Break Through

When testing the JavaScript method in Salesforce Classic, I realized the URL had pagination parameters that would appear in the query string only after clicking the “View More” and “Next Page” links.

If you were to click those a few times, you’d get an ugly link like below, but it has two very important components:

https://na1.salesforce.com/750?j_id0%3Aj_id6%3Arowsperpage=140&j_id0%3Aj_id6%3Alsr=280&retURL=%2Fui%2Fsetup%2FSetup%3Fsetupid%3DJobs&setupid=AsyncApiJobStatus

Rows per Page
The first parameter ending in “rowsperpage=140” is how many rows that page should return.

Page Number
The second highlighted parameter ending in “lsr=280” is how many records it should skip, or what page it’s on.

In my example, I’m returning 140 batch jobs per page, and I’m on page 3 because I’m skipping the first two pages (140 * 2 = 280 records)

Let’s Optimize

After some testing, you can return a maximum of 1,000 records per page. This means you can click on the below URLs to get the full set of the data and follow the pattern:

First Page:
https://na1.salesforce.com/750?j_id0%3Aj_id6%3Arowsperpage=1000

Second Page:
https://na1.salesforce.com/750?j_id0%3Aj_id6%3Arowsperpage=1000&j_id0%3Aj_id6%3Alsr=1000

Third Page:
https://na1.salesforce.com/750?j_id0%3Aj_id6%3Arowsperpage=1000&j_id0%3Aj_id6%3Alsr=2000

This MUST be done in Salesforce Classic because Lightning doesn’t let you pass in the pagination parameters.

Getting into Excel

Alright, so this part isn’t quite as sexy, but it’s surprisingly effective.

Salesforce is rendering the table, as well, an actual HTML table. This means when we copy it, it can be nicely pasted into Excel.

Doing this, I was able to make a pivot table of record updates submitted by user, by object for the last 5,000 batch jobs in about 5 minutes, it was great!

Do you have a better way to do this?

I’d love to hear it, while the approach above seems a lot better than the alternatives, it’s still not as sexy as I’d like it to be. If there are some cool SOQL queries you can run to avoid all this completely, I’d love to highlight that here.

Leave a comment or email me at shaun.t.vanweelden [at] gmail.com

Helpful Links:

Monitoring Bulk Data Load Jobs Help Article

Salesforce Reference on AsyncApexJob object

One of the earliest and most consistent pain points for us was managing job titles. We have more than a handful of “title” related fields in our Salesforce instance, some examples would be

• The default “Title” field
• Title derived from Outsourced Work (UpWork, etc.)
• 2 LinkedIn Title fields from two different apps
• 4–5 Data Vendors fields (Clearbit, Leadspace, Reachforce, etc.)

The Problem

Most people and applications only look at and use the default “Title” field.

Not touching the Title field when augmenting data seems like a waste of potentially viable data. This weekend, I found over 15% of our leads had a job title in some field, but nothing in the default Title field everyone uses in their reports and our app leverages for segmentation. That sucks.

On the other hand, I trust some of these sources significantly more than others. If an SDR goes through and populates a bunch of job titles manually, but I overwrite their changes in the next weekly import from a 3rd party, I will have a frustrated team that doesn’t trust the data and doesn’t want to make updates.

The last problem today is that many platforms and integrations do update the default title field with their debatably trustworthy information. In a perfect world, I’d trust data we manually updated on a lead or contact or paid someone to look up on LinkedIn over a tradeshow upload that populates the title field. Forget about other fields, even the title field itself isn’t always trustworthy.

What we do Today

To partially solve this, we have came up with a “Best Title” formula field in Salesforce that is simply a rank ordered precedence of title fields. If the top trusted field is blank, go to the next. Our key marketing programs and Salesforce flows use this Best Title field for segmentation.

This solves the second part of the problem above, trusting the data, but leaves the first part of the problem woefully unsolved. Other applications have no idea we have a custom title field, and for better or worse, the field is read-only as it’s a formula field. It’s not even on our page layouts currently, it’s just used by those “in the know” behind the scenes.

A Potentially Better Way

To solve this problem well, you seemingly would need to be able to establish a “title source” precedence and then capture any update to a title-related field and see if the default “Title” field should be updated or not based on that precedence.

I think someone (including myself) could build exactly this in Salesforce today.

We could create a trigger to intercept any lead or contact update that contains a title field we’re tracking. Instead of simply letting the title field update go through, we look at what field updates have been made in the past and see if the update that’s being pushed through should take precedence on the title field. If so, update the “title” field, if not, leave the “title” field alone and either drop the update or let it update the 3rd party title field.

An Example —

Our field precedence may look something like this:
1. UpWork / manual field
2. Title field
3. DiscoverOrg field
4. Clearbit field
5. Leadspace field

DiscoverOrg creates a new lead and populates their DiscoverOrg Job Title field with the title it came up with. The trigger would then note that the DiscoverOrg title was populated with the initial value, and then it would populate the default “Title” field with the DiscoverOrg Value.

2 Months later, this lead is at a trade show and our marketing team does a list upload and updates this person’s “title” field to be what they found at the trade show. Because the update was to the default “Title” field, we know that takes higher precedence than the DiscoverOrg field and so we populate the “Title” field with the tradeshow value.

The next month, we get Clearbit to augment job titles. They populate their job title field with the value they came up with. Since we know this lead has had it’s “title” populated by the default “title” field in the past from our behind the scenes tracking, we leave the default “title” field alone.

Quick notes on Logistics as they are in my head now
The behind the scenes historical tracking would be done in a hidden textarea field on the lead/contact that is populated and managed by the trigger.

An app with a VisualForce component to set the field precedence would be helpful so no one needs to write code.

If someone manually updated the “title” field, that should always take the highest precedence. An easy, not foolproof, way to determine this case vs. a list upload using the default “title” field is by looking at how many records are included in the change. If it’s just 1, it’s likely a manual change from someone. If more than 1, it’s a bulk import/update.

Do you have this problem too?

If so, I’d love to hear if the above solution resonates with you and if you’d be interested in such a Salesforce application? See any holes, let me know!

Email: shaun.t.vanweelden [at] gmail.com
LinkedIn: Shaun VanWeelden