Implement Teardown (added back DB cache)

[nikhilmat]

This is a continuation of #479 and implements a migration specific DB cache that was causing memory issues observed in #526

There are two important differences added in separate commits:

• fac1ba7 - Throttler teardown
  • We noticed in our own monitoring that there were still go routines that were hanging around in the throttler. This commit makes sure to close those out and implements the Teardown method in the same fashion
• ec6ceff - Pass in a migrationContext UUID for a migration specific connections
  • This implements the migration specific database connection cache (a migration UUID was added to the migrationContext), which should hopefully fix the memory leak. Things look stable from a migration we ran as a test from our end, but we never saw the memory get as high as a gig like you had mentioned - would be interested to hear what load you are putting it under to reproduce that scenario.
• e82d563 - Close the binlog syncer
  • I noticed in my testing that the binlog syncer was not stopping, despite the comments indicating that the library itself should do this. Explicitly adding back the close fixed this issue for me, curious to hear the context behind taking it out.

Let me know your thoughts on this approach! Happy to revise what's needed.

Thanks again :)

[nikhilmat]

@shlomi-noach I noticed from my testing that this was required to stop the binlog syncer from running indefinitely. i'm not sure if this comment is specific to running this code as a process that exits, but it seems this is necessary from my end. would appreciate any context you have on this, and if this change looks right to you

[nikhilmat]

I do get the following error in the log:

2017/12/18 11:51:15 binlogsyncer.go:126: [info] syncer is closing...
2017/12/18 11:51:15 binlogsyncer.go:554: [error] connection was bad
2017/12/18 11:51:15 binlogstreamer.go:47: [error] close sync with err: Sync was closed
2017/12/18 11:51:15 binlogsyncer.go:141: [info] syncer is closed

But it seems to not cause any issues.

If I don't have this explicit close, the binlogsyncer lingers on after the migration has finished:

2017-12-18 11:53:40 INFO Done migrating `zenpayroll_development`.`companies`
2017-12-18 11:53:40 INFO Removing socket file: /gh-ost/gh-ost.test.sock
2017-12-18 11:53:40 INFO Tearing down inspector
2017-12-18 11:53:40 INFO Tearing down applier
2017-12-18 11:53:40 INFO Tearing down streamer
2017/12/18 11:55:00 binlogsyncer.go:632: [info] rotate to (mysql-bin-changelog.001359, 4)
2017/12/18 11:55:00 binlogsyncer.go:632: [info] rotate to (mysql-bin-changelog.001359, 4)
2017/12/18 11:55:00 binlogsyncer.go:632: [info] rotate to (mysql-bin-changelog.001359, 4)
2017/12/18 12:00:00 binlogsyncer.go:632: [info] rotate to (mysql-bin-changelog.001360, 4)
2017/12/18 12:00:00 binlogsyncer.go:632: [info] rotate to (mysql-bin-changelog.001360, 4)

[shlomi-noach]

Thank you! To avoid repeated work I changed the base branch from github:master to github:teardown-contrib, so that it is easier to track the changes.

[shlomi-noach]

I'll merge this and send for tests

[shlomi-noach]

I will just need to test it in production and report back