Using Git like CVS

Tonight I moved the http://rainbowcinemas.ca sources from phpeclipse and CVS to PDT and Git.

Below are some gotchas and tips for initial repo creation, how to keep the central remote copy up to date, and how to work around complaints about updating master directly from remote. I'm sure there's a better way to do this w/o the need for the workaround, but this is what I found worked.

Initial setup To crawl a directory and create a git project for each subfolder: for d in $(find . -maxdepth 1 -type d -not -name "." | \ egrep -v ".ssh|logs|OLD|download|upload"); do cd $d; \ git init; git add .; git commit -m "initial commit" .; \ cd ..; \ done See also Setting up a shared repository.
Create local clone via commandline git clone ssh://servername/path/to/.git folderToCreateLocally	Create local clone w/ eGit Once your repo is created, you can clone a copy from the remote server onto your local box, and import it into Eclipse (with eGit installed) using File > Import > Git > Projects from Git > Clone... .
Commit local changes via commandline As outlined before, you can git pull, git checkout, git commit, and finally git push your changes. If you encounter an error trying to commit changes back to the repo, see the section below, "Allow a ref update to the currently checked out branch of a non-bare repository".	Commit local changes w/ eGit With eGit, you can pull, push, checkout, commit, merge, etc. using the context menu on a Git project or with the Synchronize view. I don't recommend using any of the change sets / models except the Git Change Set, since the others will tend to show more than is actually needed (like local changes which Git doesn't track).
Allow a ref update to the currently checked out branch of a non-bare repository Update the ~/.gitconfig file on the remote server to look something like this: [user] name = Your Name email = [email protected] [color] branch = auto diff = auto interactive = auto status = auto [core] editor = vim [merge] tool = vimdiff [receive] denyCurrentBranch = warn Retrieve changes into remote repo Because I'm using the remote server to both host http-accessible files AND host the git repo, it's important that changes checked into the git repo be then checked out into the local filesystem so that the local workspace is in synch with the repo's metadata. To pull changes, I use git status (to review changes), git reset HEAD <filename> (to reset specified file, or omit filename to reset all changes) and finally git checkout to retrieve the changed file from the repo into the working directory. Access the server w/o a password prompt To skip being prompted for a password when you connect over ssh, scp, or fish, add your public SSH key to the ~/.ssh/authorized_keys file on the remote server. Access the server via an alias Instead of having to reference the server as username@server when connecting, you can add an entry to your ~/.ssh/config file that looks like this: Host shortName Hostname fully.qualified.domain.name.or.IP.address User yourUsername Port 22

Feeding The Right Sources To Your Builds

I was recently asked the best approach for how to control the input to one's builds, and while there is no single solution to this that fits all projects, here are two ways you might want to go.

Freeze Then Branch

The current approach for JBoss Tools development involves continuous nightly builds from trunk until such time as the project is deemed frozen in prep for an upcoming milestone or release candidate. At that time, the trunk is copied to branches/JBossTools-3.1.0.M4, and very little is done to that branch - only urgent tweaks and fixes. A new Hudson job is cloned from the current one, and adjusted to point at the branched sources instead of trunk. The maps are also updated to point at the new source location in SVN.

This allows more nightly builds towards an upcoming stable development milestone, while new development can continue in parallel in trunk. When the milestone is released, the unneeded job is disabled or removed.

The only issue with this approach is that all plugins, built from the head of a branch (or trunk) are re-versioned w/ the latest timestamp every time they're compiled. Thus upgrading from one 80M update to the next requires another 80M download. To mitigate this, milestones are released only once every month or two.

Tag And Release

The current approach for numerous Eclipse.org projects, such as GEF, is to develop continuously in HEAD and when a number of changes are ready to be released, a tool such as the org.eclipse.releng.tools plugin can be used to, in a single operation, both tag the sources and release those tags into the map file.

This permits a granular approach to plugin versioning wherein only the plugins which have actually changed are renumbered, and thus incremental updates between releases is possible, and if only a single plugin changes from week to week, only that plugin will be upgraded.

This approach also allows your adopters to get more stable, weekly updates rather than hourly or ad hoc nightlies which may include incomplete changes. Sure, you can only watch for good builds in Hudson, but a more predictable schedule makes inter-build communication easier.

The only issue with this approach is that it introduces extra overhead, unless the tag&release process can be automated and run on a (weekly?) schedule. For CVS sources, there is the org.eclipse.releng.tools.tagandrelease crontab script available; for SVN, no such script (yet) exists. Want to help? See bug 264713.

Hybrid Approach

With the Athena Common Build, you can set up a Hudson job to run from tagged maps, but also override those tags on demand to force a build from head (or trunk) instead of from the specific repo tags.

To do so, pass in the following flags to EXTRAFLAGS via your build's job configuration / parameters. To build from the workspace's sources instead of fetching anything new from the repo (bug 252774):

-localSourceCheckoutDir /opt/users/hudsonbuild/.hudson/jobs/${JOB_NAME}/workspace/

To fetch from the repo, but ignore tags (bug 251926):

-forceContextQualifier -fetchTag HEAD

Or, like many projects on build.eclipse.org have done, set up two jobs: one for nightly from trunk, and one for weekly integration from maps. Then instead of doing CI builds when you remember to, they will run automatically when the repo changes so you'll have immediate feedback when something has broken. Add email, IRC, or Twitter notification to the job, and everyone else will know too.

Dash Athena News, Oct 2009

Have you heard the latest? Here's what's been going on in the Dash Athena project lately.

Infrastructure Changes

• There are now 43 Athena jobs on build.eclipse.org! Of those, 30 are green, 1 is yellow, and 6 have not yet been enabled. These jobs represent 29 different projects' builds! 6 of them use SVN sources instead of CVS.


• bug 257074 comment 12 build.eclipse.org now has SVN 1.6.5; if your map files no longer work (your build complains it can't resolve plugin or features sources) then read this.

New Features

• bug 291446 Provide hook for extra actions after fetching code from repo and before compiling it (e.g. code generation, parser generators, etc.)


• bug 275529 Athena is now a full Project rather than a Component! Now if we could just get someone to design a logo... Do I need to offer up prizes? I will, if you comment on bug 272723 with some design ideas (or prize ideas).

Better Documentation

• Tips for Building on Windows - community contributed! Thanks to Nicolas Bros!


• New category created for User-contributed build stories: Category:Athena Common Build Users


• Athena bugs moved from product=Dash&component=Common+Builder to product=Dash+Athena. Here's what's changed this week.

See also Older New & Noteworthy.

Git 'Er Done

The discussions in bug 257706: Host a git repository on Eclipse Foundation servers, support git as the repository of Eclipse projects rages on, wind blowing in both directions.

Let's look at the objections to Git @ Eclipse:

Implementing a common build infrastructure would also be complicated by additional code repositories as well.

Not true; the Athena system already supports CVS and SVN, plus a "build from local sources" mode which works w/ a cvs/svn tree dump, a workspace w/ checked out projects, or (TBD, we haven't tested this yet) with a git repo. And we have an open bug to make repo tree structure irrelevant to the local checkout mode. Party on. There is even a Git plugin for Hudson so you can use Hudson to watch your repo for changes, like it does with CVS and SVN.

Can't use unapproved or non-EPL code at Eclipse.org

Not true; from discussions w/ [email protected], I've been told at least twice that as long as you're not SHIPPING code that falls under a non-EPL or non-approved-CQ you're entirely fine to USE that code as server-based infrastructure. Rock on.

Cannot include tooling in a release train or host its project at Eclipse.org

Not true; since eGit is EPL and jGit is BSD, I don't see a problem with distributing the tooling that would connect to a Git repo hosted at Eclipse.org. We worked around the license woes for SVN tooling support. We can do it again. (Of course IANAL, TINLA.)

Conclusion:

No legal concerns with use of Git as hosted server infrastructure. Dash Athena (Common Builder) will support Git. Tooling is safe for inclusion in release trains (either fully like CVS is or partially like SVN is).

Only remaining issue is therefore to get it installed and allocate resources to support/manage it. With all the erosion going on lately, this need should not be trivialized.

Before the thaw this spring, this tree was on top of the bluff. With nothing to support it, it was dropped like an unchampioned feature request.

However, in the spirit of open source, several people on the above bug have offered to help w/ setup, testing, support, etc. So the burden here will be shared, like many things at Eclipse (eg., Babel, Hudson). Erosion continues, but we can all help to shore up the loss.

As most people prolly already know, Sourceforge supports the whole spectrum of VCS and DVCS options. If we don't want people to host projects there, Eclipse has to at least offer something from the DVCS world to encourage participation here. Keep the barrier to entry high, and people will go elsewhere. Lower the barrier, and people will come here to party instead.

With everyone feeling the economic- and time-pinch these days, can we really afford to discourage contributions at Eclipse simply because, as the silverbacks say, "why, back in my day, we only had CVS, vi, and notepad, and dangit, that was good enough!" ?

After all, the new world is inevitable.

Synch View Magic

Earlier this week, I opened bug 228701 asking for Mylyn filtering on the Synch view. (Please vote if you agree.) This afternoon in #eclipse-dev, Eugene pinged to ask why I don't simply use the Change Sets presentation in the Synch view, and Boris chimed in to tell me about bug 143419, "Strange tree rendering in synchronize view". Strange indeed.

I agreed I'd document my experiences with this view, but because Bugzilla is undergoing maintenance at the moment (thanks to our tireless webmasters!) and I want to share it while it's fresh, here's my account of how to make files vanish from the Synch view, though Mylyn continues to track them.

DO try this at home, as magic tricks in Eclipse are rare (e4 demo at EclipseCON with spinning views & scriptable UI elements notwithstanding), and this one's pretty cool.

Steps to reproduce

1. Install the latest Eclipse 3.4 I build (I've got eclipse-SDK-I20080422-0800-linux-gtk.tar.gz.)
2. Install Mylyn from its update site (mylyn-2.3.2-e3.4.zip).

   DO NOT unpack it first, just give the URL to Eclipse's new p2 / Install Manager or download the zip and hand it that. If you unpack it first, p2 doesn't seem to know how to find the features therein. But I digress...

   

3. Check out some CVS project into your workspace. I used :ext:[email protected]:/cvsroot/modeling/org.eclipse.emf/org.eclipse.emf.releng, from HEAD.
4. Open Mylyn's Task List view (Shift-Alt-Q, Q). Create a new Local Task or open an existing Bugzilla task. Enable the task.
5. Unfilter the Package Explorer view.

   This is one of my *very* few complaints about Mylyn -- I'm starting a new task w/ no context, yet it insists on filtering my view to show me my non-existent context, so I can't start selecting files on which to work until I unfilter the view. But I digress again...

6. Open the Synchronize View (Shift-Alt-Q, Y).
7. Configure the view as shown below:

   

8. In the Synchronize view, select the Change Sets presentation (the toolbar icon that looks like a file tree).
9. Create a new Synchronization. Type CVS, Resources to synch: Workspace. Hit Finish.
10. Open some files in the Package Explorer view, and edit them. Watch as they appear in the Synch view as items under your Change Set (ie., as changes for your Local or Bugzilla Task)

    It would be nice if the Change Set title included the bug number, or could be configured the same way that commit comments can be. Not crucial, just a handy enhancement. For now, the task's title is sufficient.

11. Pick some file in the CVS-backed project and copy it within the same folder. Notice how it appears in the Synch view as an outgoing addition.

    

    
    
    
12. Here's where the magic trick starts. Made sure your audience is still awake.
    
    
13. Disable your Task.
14. Copy some file (could be the same one as before, or a new one -- doesn't matter as long as it's in a CVS-backed project. The new file will appear in the Synchronize view under a new <Unassigned> Change Set.

    

15. Re-enable your Task.
16. In the synch view, select the new file from under the <Unassigned> Change Set.
17. As you select the file, it will be added to Mylyn's context for that task, and appear in the filtered Package Explorer. Note however that it didn't change from the <Unassigned> Change Set to your Task's change set. So begins the magic.

    

18. In the Synchronize view, select the Workspace presentation (the toolbar icon that looks like a file tree).
19. When you expand the tree, you'll see that your new file does appear as you'd expect. This tree should mirror what's in the Package Explorer (as in previous screenshot).

    

20. Switch back to the Change Sets presentation, and get ready for your exit with flourish.

    

21. Lo, that new file has vanished!

Image from Happy Tree Friends - I Get a Trick Out of You.
Courtesy of Mondo Media / YouTube. Used without permission.

Personal Changelog

To kick off a new blog series on change and change accessories, I'd like to start with a bit of self-promotion about a great bit of system integration that would not have been possible without the Magic of Co-Op Students^TM. Anyone who's been to the Eclipse Modeling Project's websites has probably seen how we generate release notes per build/release and per-bug deltas & patches. And as I've shamelessly promoted this a number of times in numerous bugzillae, it seems fitting to use a song we've most likely all heard before...

Reach out and touch change

Your own personal Changelog
Something that's accountable
Something able
Your own personal Changelog
Something for forensics
Something that's sick

Hist'ry unknown?
But you're not alone
Commits you own
Bugs that you have pwned
Load your web retriever
It'll make you a believer, oh

Search CVS
Put it the test
Well, it's not Jazz
But it's still real snazz
It will deliver
You know truth's a forgiver, oh

Reach out and track change [x2]

Your own personal Changelog
Something to show off your skillz
Your tricks and your spills
Your own personal Changelog
Something to show you're 1337
(And) don't miss a beat

Reach out, audit change

With hist'ry known
Your devs will come home
Flesh and bone
See what they have pwned
Load your web receiver
It'll make you a believer, oh
It will deliver
The truth's a forgiver, oh

Reach out and watch change

Your own personal Changelog

Reach out and touch change [x5]

Marilyn Manson (Depeche Mode cover) - Personal Jesus

Beyond CVS, Subversion & Cirrus

Some news about the Beyond CVS project:

1. With the release of version 0.7.0, Chris recently added support for doing SVN compares.
2. Because of this, "Beyond CVS" is now a bit of a misnomer, so we're thinking of refactoring the project to better represent its scope and use. As part of this, we're wondering if the WinSCP and Putty support should be moved to another project or just a secondary feature/plugin. If you use this project, please let us know your thoughts.
3. Beyond Compare 3 Beta (aka Cirrus) is almost ready for its public beta release. It's got some cool new features like native Linux support (including -flags instead of /flags), three-way compares/merges, better syntax colouration, the ability to copy entire blocks of changes instead of just single lines, and the ability to edit files right in the Text Compare view.

For anyone who's in the private beta group, here's a quick patch in order to be able to use Beyond CVS with Cirrus on Linux. (Windows users need no such patch as the old BC2 commandline syntax is still supported in BC3.)

$ diff -u Cirrus.sh Cirrus.sh.new
--- Cirrus.sh   2007-11-01 22:08:42.000000000 -0400
+++ Cirrus.sh.new       2007-11-01 22:25:39.000000000 -0400
@@ -22,7 +22,12 @@

 #execute the program.
 chmod u+x $EXEC
-$EXEC $* >& /dev/null
+while [[ $# -gt 0 ]]; do
+  if [[ $1 != "/rightreadonly" ]]; then
+    args=$args" "$1; shift 1;
+  fi
+done
+$EXEC $args >& /dev/null
 #$EXEC $* >& $BASE/Cirrus.log

Stay tuned for news of the public beta!

Compare Different

In my role as the weekend webmaster for Rainbow Cinemas, I have on occasion the need to compare local and remote (ftp or ssh) files between my staging server and the production website; I also have to compare files between my local Eclipse projects and my local CVS repo, to see why and when changes were made.

So, what do I use? Well, I run KDE on my MEPIS box, so I've set up Konqueror profiles for the various admin tasks I need to do. For the web, I split a Konq window into four panes and can drag/drop files between local and remote, and also compare local and remote versions of the site.

When diffing files, I use Beyond Compare 2 (or Cirrus, their new beta), and have quite enjoyed it since the dark ages (circa Eclipse 3.0) when I first joined the IBM Toronto Lab (which isn't in Toronto anymore, but I digress). It's my not-so-secret weapon for releng work, web work, java work, ... really, everything. BC supports comparing/synching from local to a remote FTP site, but in most cases I find it slow as it has to download all the remote files, compare them to the local ones, and update the view. It's handy for single file compares or small folder compares, but doing entire trees requires a lot of patience as it builds the diff tree. So, more often than not, I just use Konqueror for swapping files back and forth, since its ftp and fish ioslaves rock. (This is perhaps the biggest reason I won't change to another wm like gnome or xfce.)

I can be pretty jaded sometimes when it comes to trying new software, especially if I've grown happy and complacent using a set of tools that I find work well. Switching to something new doesn't necessarily mean finding something better; often it just means time wasted and functionality lost.

While it's cool to use a mix of tools, sometimes it's nice when you can do everything in a single IDE. So, to that end, this past weekend I started playing with DSDP's Target Management (TM) offering. Very cool. It doesn't do folder compares yet, though there's been some talk about how that could be implemented either at the Eclipse Compare level or at the DSDP-TM level. I like being able to finally browse remote files within an Eclipse view (rather than in Konq or Konsole), but the absence of folder-to-folder comparison makes TM a bit of a non-starter for me, so far.

So, to that end, I opened bug 203001 to see if folder compares were possible, and now I'm realizing that it's both not a trivial thing to implement and also introduces some philosophical questions about "derivative work", "ethical competetion", "changing your business model" and "reinventing the wheel".

If there's a commercial piece of software, like Beyond Compare, with a superior (IMHO) diff engine, should they be persuaded to contribute to Eclipse? The current version of BC can be launched from inside Eclipse using Beyond CVS, and it does the job when Eclipse's internal compare isn't enough. But it's not a plugin, nor does it integrate into the IDE. It also requires WINE, but this is being addressed in Cirrus (BC3), with a linux-friendly qt version.

But should Scooter provide their code to Eclipse, and potentially lose customers in order to make Eclipse's Compare better?

If they did, they could grow their market share and get some good PR, but they wouldn't sell licenses. They could instead switch to a donation/swag model, I suppose.

Or they could sell a standalone RCP app version with just their compare stuff in it, so people could choose between the free integrated-in-Eclipse version and the paid standalone version. Problem is, integrated in Eclipse is much more useful IMHO so I'd think that would be a non-starter too.

What would you do, if you were were Scooter?

Conversely, if you were a committer for Eclipse Compare or DSDP-TM, how much functionality in BC would you attempt to recreate, given that you're reinventing the wheel, and that by giving it away for free you're taking business from a small company who's been in the biz of compare for years. I mean, sure, it would be great if all the functionality of BC was in Eclipse so that IBM could build on it and sell product based on it, but personally I'd feel a very dirty if by opening a feature request against DSDP-TM / Eclipse Compare I was helping IBM make money at the expense of all the good work Scooter has done over the years to make my life (and my job at IBM) so much easier.

Me, I'd just like to see a way to hook in different compare engines (eg., via a plugin like Beyond CVS), but is doing the work to allow what's in Eclipse Compare to be replaced by something else in Eclipse's best interest?

Yes, there'll always be competition in the marketplace, but how does a community like Eclipse compete ethically in a situation like this? And if open source is about enabling business to succeed faster/better/smarter/whatever (among other motivations), how do small companies survive if their product gets replaced by a free open source alternative? Does anyone have experience with their product being "Eclipsed" by something free?

Blogged with Flock

Beyond CVS for Linux

Solved a long-standing annoyance this week - using the BeyondCVS Eclipse plugin on Linux.

One day, if I have time I'll try to turn this into a fragment so that it'll work without outside effort, but for now, here's the HOWTO:

0. Install WINE using yum, apt-get, etc. as appropriate for your linux distro. (Chances are this is already installed.)

1. Install beyond compare, from Scooter Software. I installed it using wine into /opt/beyondcompare2:

 $ wine beycomp.exe 

2. Create a script (eg., ~/bc2.sh) to handle passing commandline args. Your script might require a different drive letter prefix, depending on your wine install - I've had to use both s: and z:. chmod your script so it's executable (chmod +x ~/bc2.sh).

#!/bin/sh

# collect commandline params and turn them into wine-friendly paths
opts="";
while [ "$#" -gt 0 ]; do
  opts=$opts" s:"$1; shift 1;
done

/usr/bin/wine /opt/beyondcompare2/BC2.exe $opts

3. Test your script by running it in a console window.

$ ~/bc2.sh

4. Install BeyondCVS plugin and feature into your Eclipse 3.2+ install (or using a .link file).

5. Start Eclipse. Open up Help > Window > Preferences... > External Tools Launcher > Beyond Compare. In the field Path To Beyond Compare, enter the path to your shell script, eg., /home/nickb/bc2.sh. Leave the Show Debugging Info? checkbox enabled.

6. Select two files or folders in your Package Explorer or Navigator. Beyond Compare should launch under WINE. If it doesn't, open a console and run your bc2.sh script, passing the same two parameters shown in the debug info popup in Eclipse. As noted above, you might have to change the drive letter from s: to z: or something else so that WINE can find the files you passed it. If all worked, you can remove the Show Debugging Info? checkbox in the Preferences panel.

Happy comparing!

divbyzero is back - now with CVS!

I've still got a lot of code to post, but the Eclipse External Tools Launcher 0.2.2 is back. I've also posted the code into CVS, so you can browse it there if you want or extract it directly into an Eclipse project using this location: :pserver:[email protected]:/cvsroot/divbyzero

Damn scriptkiddies

It's funny. You leave your password the same thing - stupidly - for 3 years, and nothing happens. Then one day you decide to get smart, update it to something longer and safer, and boom, you get hacked a couple weeks later and turned into a PayPal phishing scam site. So, I'm blowing away divbyzero.com's blog and moving it here, and will be redoing that site entirely in the coming months. If anyone knows where I can post to a public CVS so that I can provide PHP script source w/o the danger of some scriptkiddy using it that source viewer script to do something malicious, please let me know.