Category Archives: Programming

Gogs – Self-Hosted Source Code Repository Review

Written by William Roush on June 26, 2015 at 5:14 pm

A review of Gogs – Go Git Service, a new self-hosted website for managing your Git repositories.

 

Gogs is a self-hosted code repository and collaboration platform for the Git distributed version control system. Gogs follows a lot of GitHub’s workflow and design decisions, all built on top of the Go language. The result is a familiar — and fast experience.

Installation

Gogs has repositories for various popular platforms including Debian and Redhat based systems. Installation is pretty easy, with a simple INI file for modifying it’s configuration (I haven’t seen one of those in new software in forever).

I’ve installed mine through the Puppet module Siteminds-gogs which leveraged the Debian 7 packages from Gogs’ own repository, though it has some bugs on it’s configuration templates and could use some love, it’s most of the way there for being version 0.0.3!

First Impression

One word: Fast.

It’s really fast, the website is extremely responsive and push/pulls are faster than anything I’ve experienced using cloud based solutions — I honestly thought Git was just slow at it.

Gogs - Chadev Files

The second thing you’ll notice is it’s pretty much GitHub, organizations work the same, repositories work the same, forking is the same, management is the same. You’ll feel right at home coming from GitHub. It is however currently missing pull request support but that is in the works.

Collaboration

Again — if you’re a GitHub user this will all feel familiar. You can have private repositories like normal, you can add users to these repositories. Additionally you have organizations, organizations can have repositories and teams. Teams can be granted access on all projects under an organization, or you can add users individually to projects under them. The only thing I could see that would be helpful here is the ability to create and reuse teams.

User Experience

Gogs - Activity

Activity page, some private repository information redacted.

We’re missing some key pieces to make Gogs a replacement to something like GitHub, pull requests are one of them, gists are another. Of course Gogs is young and is receiving a lot of help so I expect these to get finished soon.

Other than that, everything is clean and easy to use. There is little functionality I see missing from my day-to-day usage, and a few extra freebies such as 3rd party issue support that I need to investigate (for those of us that don’t like integrated issues in our DVCS hosting platform).

Stability

Gogs - New Repo

Handful of 500 errors aside from doing dumb things, LDAP integration went really screwy when I set a filter incorrectly. So far I’ve moved all of my development to it, and minus the LDAP issue and cloning empty repositories I haven’t seen a 500 error. Solid!

Though I’m sure there are issues here and there, for the age of the project I’m impressed.

The Future

Gogs - Chadev Settings

Gogs is looking to add pull requests relatively soon, I’m personally rooting for Mercurial support (but don’t hold your breath). Their Trello board is available online showing what they’re currently working on and their Github is thriving with pull requests, issues, and discussions. I’m excited for this to continue to mature.

Overall

My experience with Gogs has been so positive that I’ll likely be moving all of my Mercurial repos over to it. RhodeCode was nice but the pricing structure for 10+ users has really killed it (more on that in the RhodeCode review) with Kallithea being the only one I’m seriously looking at anymore (but don’t have high hopes being as the forking point for RhodeCode was when it was still pretty slow, and Gog’s speed has spoiled me).

The main worry I have about Gogs is a somewhat slow development process. Currently the project is lead by Unknwon over on GitHub, and they’re currently very busy — leading to a nearly complete halt on the project, some forks have emerged but this project has thrived under Unknwon and their dedication to this and I’d love to see it continue under them.

Introduction – Self-Hosted Source Code Repository Review

Written by William Roush on April 19, 2015 at 9:37 pm

There are many self-hosted source code repositories out there, but which one is right for you?

Looking For the Right Self-Hosted Solution

You may have many reasons to implement a self-hosted solution, and depending  on what you’re looking for you may have pretty limited options (Git + Mercurial support sure narrows it down to two for me for some solutions). We’re going to review quite a few systems that revolve around the two popular DVCS systems, Mercurial and Git (I may add other version control systems if people are interested enough and the revision control systems are non-proprietary or I can get my hands on a trial), check back or subscribe for updates!

Why Self-hosted?

Self-hosting comes with a range of benefits, including but not limited to:

  • Faster response times due to being on a local network with less resource contention.
  • Security of source code behind a firewall on a private network.
  • Tighter integration with your internal authentication systems (ex: LDAP, or the Atlassian stack).
  • Typically cheaper than a cloud solution.

What I Look for in a Self-Hosted Solution

I work on a variety of projects, from open-source to closed-source and proprietary, code for myself, code for companies, code for the community. I need a place to store all of this code in a safe manner and backup to some cloud solution (and RhodeCode has been good for this in the past), I want a place I can delegate access to companies that want access to their repositories, or a consultant they have come in and make a minor change or wants to review something I did. All while staying cost effective. Hosted solutions cannot work for me due to the high costs of onboarding 3rd parties (and what do I do when they no longer need access? I’m stuck paying a seat).

Some positives include fork management, organization control and LDAP integration.

Self-hosted Solutions

Product Source Available Cost Free Use? Language Review Available
RhodeCode Yes  ~$5/user/mo For <= 25 users Python Review
Kallithea Yes Free Yes Python Planned
GitLab Yes Free Yes Ruby Planned
Gogs Yes Free Yes Go Review
Stash Yes Tiered pricing No Java Planned
Github Enterprise No  $21/user/month No Ruby No

Cloud Based Solutions

Throughout reviews we’ll refer to some cloud based solutions, their features and their pricing, here is a list of some popular ones:

Product Price Free Public Repos? Free Private Repos?
Github ~$1/repo/month Yes No
BitBucket $1/user/month Yes Yes – Up to 5 users

What We’ll Be Evaluating

  • Product functionality (both the basics needed for source control management and beyond)
  • Repository systems supported
  • Integration with existing infrastructure
  • Ease of use
  • Ease of administration
  • Community resources
  • Pricing schemes

Missing a Solution?

Please leave a comment!

TeamCity vSphere (ESXi) Integration Explained

Written by William Roush on January 25, 2015 at 5:15 pm

I found the documentation of configuration and why certain things are set up a specific way a bit lacking on the new vSphere support from TeamCity, so here we do a dive into how everything works.

I’m going to assume you have a fairly good grasp on what TeamCity is and how to manage it, if you feel I’ve skipped anything and should go into better detail drop me a message either through the site’s contact page or a comment here.

TeamCity Cloud Integration

TeamCity’s cloud integration allows you to move your build agents from machines you may have online all day to a base image you clone out and spin up as required (and as many as required in whatever combination).

Benefits:

  • You’re not limited to 3 build agent configurations with the base TeamCity install, you’re limited to 3 active at any one time. Good for multi-platform environments.
  • Resources are only used when needed.
  • Every build can be a clean build (if you trash your build agents after a build).

Drawbacks:

  • Each virtual machine being a single build agent vs multiple build agents on a single machine with multiple cores may waste resources.
  • Build times will increase due to virtual machine build time.
  • You need to move to your build agents being stateless. (I’m dumb, Jody Shumaker corrects me in the comments)

Useful Repositories

JetBrains has published the plugins required to integrate with vSphere on Github here: https://github.com/JetBrains/teamcity-vmware-plugin/. I’m very thankful they’ve open sourced this because of the hangup mentioned later with the required resource pools.

Preparing Your Base VM

  1. Install the OS of your choice (so far Windows and *nix environments are supported out of the box, plugin needs updating if you want to support more).
  2. Install VMware tools.
    • This is used by TeamCity cloud to properly configure your build agent, and is required.
  3. Install all of your build tools
  4. Install Java for TeamCity build agent (if it isn’t part of your build tools).
  5. Install TeamCity build agent.
  6. Verify it shows up in TeamCity’s unauthorized agents list, check your agent parameters and compatible configurations.
  7. Shut down TeamCity build agent service on build agent virtual machine.
    • May want to remove the build agent from the unauthorized list at this point just to clean things up, but this is up to you.
  8. Remove name, serverURL and authorizationToken from conf/buildAgent.properties on the build agent.
    • This is to make your image generic, your cloud plugin and VMware tools will auto-populate these values for you, and in the event of you wanting to tweak the base virtual machine you don’t have to worry about it booting up as a valid build agent.
  9. Shut down the VM, and snapshot it (without a snapshot your virtual infrastructure will try to clone the entire VM, this will make spin up times for build agents extremely high.

Linux Caveats

In the most recent version of VMware on some Linux platforms when you go to install VMware tools you’ll be told to use open-vm-tools instead. At least on Debian 7 (and I haven’t tested other platforms) the vmware-rpctool binary ends up in /usr/bin instead of /usr/sbin like TeamCity expects. So we’ll just make a link for it:


$> ln /usr/bin/vmware-rpctool /usr/sbin/vmware-rpctool

I have an open issue with JetBrains on it to update their documentation.

Update: No longer an issue on the latest patch, quick turn-around from JetBrains!

Configuring Your Cloud

Go to Administration > Agent Cloud (under Server Administration) and add a new profile and give it a name.

I like to use a dedicated account for vSphere that has access only to specific folder in our environment, this includes network and datastores to prevent it from doing anything to the rest of the infrastructure.

  • Terminate instance idle time – This is the time a virtual machine that is spun up will wait without a task before being shut down, I set this fairly low (10 minutes).
  • Terminate instance (after first build completed) – This will trash a virtual machine after the build process is complete, I keep this on because I find this to be one of the major reason I’m doing this.
  • Cloud type (set to VMware vSphere)
  • vCenter SDK URL – You’ll set this to https://[vCenter FQDN]/sdk, it accepts self-signed certs provided by your vCenter box by default just fine.

Next we’ll need to configure some images, click “Add Image”, select a virtual machine from the pull-down list, pick a snapshot (you’re using snapshots, right?), select a folder that the clones will go into, select a resource pool (more on that later) and set the maximum number of virtual machines you want to be able to run at once.

If you don’t have resource pools because you’re on a version of vSphere that doesn’t support them, you’ll either have to wait till I get my fork done that removes this requirement and have JetBrains pull or, or download and remove the requirement yourself (I hear an older version of the plugin doesn’t require it but haven’t verified it).

Update: JetBrains beat me to it, issued a patch, works great now.

Verification And Validation

When you build a project with zero build agents installed, it’ll go into the queue. The TeamCity cloud plugin will spin up a virtual machine for you, and when that machine is registered it should show up with the name of the virtual machine. If this is what you got, success!

If not, feel free to post comments and I’ll see if I can help out.

Upsource EAP Preview – Repository Browser and Code Review From JetBrains

Written by William Roush on August 28, 2014 at 10:38 pm

This article is a bit out of date, JetBrains has made some changes that I need to review and update the relevant sections of this article.

Upsource is a new code browser and code review system by JetBrains, the guys behind popular development software like Resharper, IntelliJ IDEA and TeamCity.

Update: I’ve been talking with some of the developers, I’ve made notes around the article with things they’ve gotten back to me about.

Upsource Early Access Program

Upsource is currently available through it’s Early Access Program — Mind you everything you read here is about Upsource in it’s EAP state, I’ll be doing more reviews as the software evolves (JetBrains continuously adds awesome stuff to their software).

Requirements

Shame on me, didn’t check this at first. Threw it on a Linux machine with 2GB of memory, watched the entire system crash with out of memory exceptions. Oops.

The 8GB of RAM is a little steep, but it’s sitting on top of Java, so some memory hungry behavior is expected.

UpSource is a bit hungry resource wise.

Upsource is a bit hungry resource wise.

You may want to throw two CPU cores at it, startup was a bit slow, however once it was running the application was smooth.

Lets Get Started

So I unpack the zip file, install OpenJDK 7’s JRE, and fire up the services, Upsource is reported as online, but I’m met with a blank page while Java thrashes about. Very easy to set up though.

Project Management

Creating A Project

Creating a project is quite easy, give it a name, point it at your current repository sever, and away you go. No hang-ups here really.

Project Sprawl

At the current moment, for those of us working at places that have 40+ projects, it looks like there will just be a massive amount of horizontal sprawl going on, TeamCity had a similar issue until recently when project hierarchies were added, I’m sure this will end up in Upsource at some point.

This also includes the groups that are automatically created (“[project] Developers”, “[project] Observer”, and “[project] Project Admin”), these will be created for every repo you have, even though you’d likely have a ton of repos under one group schema.

Main Upsource UI

Main UpSource UI, showing projects and a news feed.

Main Upsource UI, showing projects and a news feed.

When logging in you view all available projects and your news feed. I really dig the news feed, showing all comments on your reviews and mentions for your user account. The main thing I don’t like is that getting to this page from a child project is annoying.

Project Status UI

Project overview UI is pretty straightforward.

Project overview UI is pretty straightforward.

The main project UI is pretty basic but gives you what you want, on the left side you see a handful of recent changesets with a nice visual tree. From here you can navigate to a commit to start a review, browse code in the repository or jump straight to analytics.

Analytics

This is one of those major extras I love with systems like Upsource and Crucible. This gives a very visual breakdown of who handles what commits, and who is handling what reviews.

Statistics for Redmine contributions pulled from Bitbucket

Statistics for Redmine contributions pulled from Bitbucket

The visualization is powerful, I’d like the ability to map multiple users together to a single user in Upsource (doable in Redmine, Crucible, etc.), please allow this setting to be global (a major problem that Crucible had the last time I used it).

 

Code Reviews

This is by far the biggest reason I was looking towards Upsource. Previously I’ve looked at using Crucible for our code review system, and like 90% of it, however there is a critical piece of functionality it lacks, and that is supporting a good UX workflow for post-commit review for multiple branches of code. Sadly Upsource also suffers from this problem (and straight up doesn’t support it, more on this later).

My Mercurial Workflow

Basic mercurial workflow I follow.

Basic mercurial workflow I follow.

The workflow process I follow is pretty straight forward, each bug or feature gets it’s own named branch on Mercurial, each branch is closed and merged onto a staging branch where things are tested, verified and reviewed, and once everything is approved onto default it goes to get tagged and released.

Where Upsource Fails To Support This Workflow

Update: This is planned to be fixed in the 1.0 release, I’ll see if the UI works well.

To create a review in Upsource you select a single commit and click “Create Review”, from here you can add more revisions using Upsource’s powerful search system.

There are a couple issues I have however, if I search for “branch:stage-1.1.0”, it will also apparently bring in any parent of the named branch “stage-1.1.0” which will include all previous commits, so I can’t easily just add all commits I see on the search page.

I can search for each invididual commit ID by searching for “id:[] OR id:[] OR id:[]” for every included changeset, however there is a problem with including changesets from different branches (more on that later).

I’d love for some powerful way to select all items between the default commit at the bottom of the workflow image above and the last staging commit, this would make the workflow a thousand times easier and would instantly have me sold.

Code Review UI

 

Inline review is pretty clean, single-line comments leaves a little to be desired.

Inline review is pretty clean, single-line comments leaves a little to be desired.

The UI is pretty clean, one feature I do miss from ReviewBoard is being able to make block comments. This allows you to choose a range of line numbers by clicking and dragging and putting a comment on them. This allows you to bring a developer’s attention to a block of code instead of an individual line.

Side-by-side diffs are available too.

Side-by-side diffs are available too.

Side-by-side diffs are nice, double clicking allows you to still add comments to the commit, this wasn’t immediately apparent and they should probably leverage the same edit icon on the inline code review diffs.

 

Additionally, I really did prefer the nice single file UI that Crucible has, I’ve worked with some changes that can span 10-15 files, and hundreds of lines of changes. When these are all crammed onto the same page your scroll bar can get really tiny (and some other ill effects in terms of browser performance can begin to happen). These aren’t often, but when refactoring old code bases it’s bound to happen.

Code Review Process

Unlike Crucible there isn’t a predefined process, which is unfortunate, every time you do a review you add individual users to the review and individual users as watchers.

Completed reviews show a green check box near the user's icon.

Completed reviews show a green check box near the user’s icon.

I do like that you complete reviews, which is better over ReviewBoard which only includes a “ship it” button, so no way to mark a review as completed but the code quality insufficient for production.

Reviewing Code From Multiple Branches

Update: This is planned to be fixed in the 1.0 release.

So when attempting to review code from multiple branches (say from my example of the resulting code from bug 1 and bug 2 to be pushed to live), you are greeted with this error:

No multi-branch review support.

No multi-branch review support.

So even if I was patient enough to select each individual changeset it won’t support it.

Repository Browser

Code Display

Code browser, limited language support.

Code browser, limited language support.

The code display is pretty straight forward, lets you browse through your repository, uses the same display as your inline diff UI, only thing I wish I had here was multiple language support (even if it was a framework that 3rd parties could write tools for).

File History

File history, what more is there to ask for?

File history, what more is there to ask for?

File history is pretty straight forward, I seriously can’t think of anything to ask for.

Code Annotation

Annotation works like blame in Git/Mercurial.

Annotation works like blame in Git/Mercurial.

Code annotation is pretty nice, shows you who and what changeset the last changes came from, much like blame in Git and Mercurial, but this UI is a bit nicer and will let you navigate straight to the related changeset.

Final Thoughts

Sadly Upsource doesn’t support the code review process I’d need to adopt it, however JetBrains is known to constantly improve their product line and I’m eager to see this product continue to evolve. If you don’t mind per-commit reviews (instead of batching reviews based on branching policies) then I’d recommend you keep an eye out for Upsource coming out.

Required Features Before I’d Use it

  • Support for batched reviews, based off the results of multiple branches coming together.

Tasty Features

  • Ability to define a more strict revival process (all reviews require approval from [x] people from team [a] and/or [y] people from team [b]), with the ability to automatically mark a review as complete with this objective is met.
  • Approval/rejection of a code review.
  • Single-file review process that isn’t just side-by-side.
  • A little more love for the side-by-side UI.

Bugs I Ran Into

  • Every once in awhile when refreshing a page, I was greeted with an authentication failure error, refreshing the page a few times would generally fix this.
  • Adding an empty repository will have pretty bad consequences and errors everywhere.

Curiosities

  • On every page you navigate around Upsource you are greeted “Loading Upsource” UI element while the page loads, as if they intended it to be a single page application but moved away from it. At least I don’t typically see this behavior in an application that has a bunch of individual pages. Update: The application is supposed to be a single page application! Found a bug that apparently is known but they’re gathering info on, will be providing info to JetBrains to fix.