Tag: testing

How I’m testing seed data generation

August 6, 2020

Technical

Ruby, Ruby on Rails, sample data, testing
When I create a new Rails project I like to have a robust seeds that can be used to quickly bootstrap development, testing and staging environments to interact with the application. I think this is critical for development speed.

If a developer creates a feature to, for example, connect two records together, you just want them to fire up the application and connect two records to see it work. You don’t want them spending time creating the records because that’s a waste of time, but also, because all developers end up having different subsets of testing data and generally ignoring everything that’s not included in them. It’s better to grow a set of testing data that’s good and complete.

One of the problem I run into is that generating testing data, or sample data, doesn’t happen often enough and changes in the application often break it. Because of that, I wrote this simple test:
```
RSpec.describe "rake db:seed" do
  it "runs" do
    Imok::Application.load_tasks if Rake::Task.tasks.none? { |t| t.name == "db:seed" }
    ENV["verbose"] = "false"
    Rake::Task["db:seed"].invoke
  end
end
```
It doesn’t have any assertions, but with just a few lines of code it probably covers 90% of seed data creation problems, that generally result in a crash. I would advice against having assertions here, as they may cost more time than the time they’ll save because sample data evolves a lot and it’s not production critical.
Data driven tests

March 22, 2011

Technical

data driven tests, test, testing

I’m not sure if anybody uses the terminology “data driven test” but if you explain what it is, experienced people will tel you that they are bad. Data driven tests are tests with the same code repeating over many different pieces of data.

Let’s show an example. For my startup project Keep on Posting, I have a method that turns a blog url into a feed url. That method is critical for my application and there are many things that can go wrong, so I test it by querying a sample of real blogs. The test would be something like this (this is in Ruby):

[sourcecode lang=”ruby”]
class BlogToolsTest
BLOGS_AND_FEES =>
"http://blog.sandrafernandez.eu" => "http://blog.sandrafernandez.eu/feed/",
"http://www.lejanooriente.com" => "http://www.lejanooriente.com/feed/",
"http://pupeno.com" => "http://pupeno.com/feed/",
"http://www.avc.com/a_vc" => "http://feeds.feedburner.com/avc",
}

def test_blog_to_feed_url
BLOGS_AND_FEEDS.each_pair do |blog_url, feed_url|
assert_true feed_url == BlogTools.blog_to_feed(blog_url)
end
end
end
[/sourcecode]

Note: I’m using assert_true instead of assert_equal to make a point; these kind of tests tend to user assert_true.

The problem with that is that eventually it’ll fail and it’ll say something like:

[sourcecode]
false is not true
[/sourcecode]

Oh! so useful. Let’s see at least where the error is happening… and obviously it’ll point to this line:

[sourcecode lang=”ruby”]
assert_true feed_url == BlogTools.blog_to_feed(blog_url)
[/sourcecode]

which is almost as useless as the failure message. That’s the problem with data drive tests. You might be tempted to do this an re-run the tests:

[sourcecode lang=”ruby”]
def test_blog_to_feed_url
BLOGS_AND_FEEDS.each_pair do |blog_url, feed_url|
puts blog_url
puts feed_url
assert_true feed_url == BlogTools.blog_to_feed(blog_url)
end
end
[/sourcecode]

but if your tests take hours to run, like the ones I often found while working at Google, then you are wasting time. Writing good error messages ahead of time help:

[sourcecode lang=”ruby”]
def test_blog_to_feed_url
BLOGS_AND_FEEDS.each_pair do |blog_url, feed_url|
assert_true feed_url == BlogTools.blog_to_feed(blog_url), "#{blog_url} should have returned the feed #{feed_url}"
end
end
[/sourcecode]

and if half your cases fail and the whole suit takes an hour to run and you have 1000 data sets you’ll spend hours staring at your monitor fixing one test every now and then, because as soon as one case fails, the execution of the tests is halted. If you are coding in a language like Java, that’s as far as you can take it.

With Ruby you can push the boundaries and write it this way (thanks to executable class bodies):

[sourcecode lang=”ruby”]
class BlogToolsTest
BLOGS_AND_FEES =>
"http://blog.sandrafernandez.eu" => "http://blog.sandrafernandez.eu/feed/",
"http://www.lejanooriente.com" => "http://www.lejanooriente.com/feed/",
"http://pupeno.com" => "http://pupeno.com/feed/",
"http://www.avc.com/a_vc" => "http://feeds.feedburner.com/avc",
}

BLOGS_AND_FEEDS.each_pair do |blog_url, feed_url|
define_method "test_#{blog_url}_#{feed_url}" do
assert_true feed_url == BlogTools.blog_to_feed(blog_url), "#{blog_url} should have returned the feed #{feed_url}"
end
end
end
[/sourcecode]

That will generate one method per item of data, even if one fails, the rest will be executed as they are separate isolated tests. They will also be executed in a potential random order so you don’t have tests depending on tests and even if you don’t get a nice error message, you’ll know which piece of data is the problematic through the method name.

Note: that actually doesn’t work because blog_url and feed_url have characters that are not valid method names, they should be replaced, but I wanted to keep the example concise.

Since I’m using shoulda, my resulting code looks like this:

[sourcecode lang=”ruby”]
class BlogToolsTest
BLOGS_AND_FEES =>
"http://blog.sandrafernandez.eu" => "http://blog.sandrafernandez.eu/feed/",
"http://www.lejanooriente.com" => "http://www.lejanooriente.com/feed/",
"http://pupeno.com" => "http://pupeno.com/feed/",
"http://www.avc.com/a_vc" => "http://feeds.feedburner.com/avc",
}

BLOGS_AND_FEEDS.each_pair do |blog_url, feed_url|
should "turn blog #{blog_url} into feed #{feed_url}" do
assert_equal feed_url, BlogTools.blog_to_feed(blog_url), "#{blog_url} did not resolve to the feed #{feed_url}"
end
end
end
[/sourcecode]

and running them in RubyMine looks like this:

Rails come with some awesome assertion methods for writing tests:

assert_difference("User.count", +1) do
  create_a_user
end

That asserts that the count of user was incremented by one. The plus sign is not needed, that’s just an integer, I add it to make things clear. You can mix several of this expressions into one assert_difference:

assert_difference(["User.count", "Profile.count"], +1) do
  create_a_user
end

That works as expected, it asserts that both users and profiles were incremented by one. The problem I have is that I often found myself doing this:

assert_difference "User.count", +1 do
  assert_difference "Admin.count", 0 do
    assert_difference "Message.count", +3 do  # We send three welcome messages to each user, like Gmail.
      create_a_user
    end
  end
end

That looks ugly. Let’s try something different:

assert_difference("User.count" => +1, "Admin.count" => 0, "Message.count" => +3) do
  create_a_user
end

Well, that looks nicer, and straightforward, so I implemented it (starting from Rails 3 assert_difference):

def assert_difference(expressions, difference = 1, message = nil, &block)
  b = block.send(:binding)
  if !expressions.is_a? Hash
    exps = Array.wrap(expressions)
    expressions = {}
    exps.each { |e| expressions[e] = difference }
  end

  before = {}
  expressions.each {|exp, _| before[exp] = eval(exp, b)}

  yield

  expressions.each do |exp, diff|
    error = "#{exp.inspect} didn't change by #{diff}"
    error = "#{message}.\n#{error}" if message
    assert_equal(before[exp] + diff, eval(exp, b), error)
  end
end

Do you like it? If you do, let me know and I might turn this into a patch for Rails 3 (and then let them now, otherwise they’ll ignore it).

Update: this is now a gem.

Tag: testing

How I’m testing seed data generation

Data driven tests

Better assert difference?