Tag Archives: vSphere

TeamCity vSphere (ESXi) Integration Explained

Written by William Roush on January 25, 2015 at 5:15 pm

I found the documentation of configuration and why certain things are set up a specific way a bit lacking on the new vSphere support from TeamCity, so here we do a dive into how everything works.

I’m going to assume you have a fairly good grasp on what TeamCity is and how to manage it, if you feel I’ve skipped anything and should go into better detail drop me a message either through the site’s contact page or a comment here.

TeamCity Cloud Integration

TeamCity’s cloud integration allows you to move your build agents from machines you may have online all day to a base image you clone out and spin up as required (and as many as required in whatever combination).

Benefits:

  • You’re not limited to 3 build agent configurations with the base TeamCity install, you’re limited to 3 active at any one time. Good for multi-platform environments.
  • Resources are only used when needed.
  • Every build can be a clean build (if you trash your build agents after a build).

Drawbacks:

  • Each virtual machine being a single build agent vs multiple build agents on a single machine with multiple cores may waste resources.
  • Build times will increase due to virtual machine build time.
  • You need to move to your build agents being stateless. (I’m dumb, Jody Shumaker corrects me in the comments)

Useful Repositories

JetBrains has published the plugins required to integrate with vSphere on Github here: https://github.com/JetBrains/teamcity-vmware-plugin/. I’m very thankful they’ve open sourced this because of the hangup mentioned later with the required resource pools.

Preparing Your Base VM

  1. Install the OS of your choice (so far Windows and *nix environments are supported out of the box, plugin needs updating if you want to support more).
  2. Install VMware tools.
    • This is used by TeamCity cloud to properly configure your build agent, and is required.
  3. Install all of your build tools
  4. Install Java for TeamCity build agent (if it isn’t part of your build tools).
  5. Install TeamCity build agent.
  6. Verify it shows up in TeamCity’s unauthorized agents list, check your agent parameters and compatible configurations.
  7. Shut down TeamCity build agent service on build agent virtual machine.
    • May want to remove the build agent from the unauthorized list at this point just to clean things up, but this is up to you.
  8. Remove name, serverURL and authorizationToken from conf/buildAgent.properties on the build agent.
    • This is to make your image generic, your cloud plugin and VMware tools will auto-populate these values for you, and in the event of you wanting to tweak the base virtual machine you don’t have to worry about it booting up as a valid build agent.
  9. Shut down the VM, and snapshot it (without a snapshot your virtual infrastructure will try to clone the entire VM, this will make spin up times for build agents extremely high.

Linux Caveats

In the most recent version of VMware on some Linux platforms when you go to install VMware tools you’ll be told to use open-vm-tools instead. At least on Debian 7 (and I haven’t tested other platforms) the vmware-rpctool binary ends up in /usr/bin instead of /usr/sbin like TeamCity expects. So we’ll just make a link for it:


$> ln /usr/bin/vmware-rpctool /usr/sbin/vmware-rpctool

I have an open issue with JetBrains on it to update their documentation.

Update: No longer an issue on the latest patch, quick turn-around from JetBrains!

Configuring Your Cloud

Go to Administration > Agent Cloud (under Server Administration) and add a new profile and give it a name.

I like to use a dedicated account for vSphere that has access only to specific folder in our environment, this includes network and datastores to prevent it from doing anything to the rest of the infrastructure.

  • Terminate instance idle time – This is the time a virtual machine that is spun up will wait without a task before being shut down, I set this fairly low (10 minutes).
  • Terminate instance (after first build completed) – This will trash a virtual machine after the build process is complete, I keep this on because I find this to be one of the major reason I’m doing this.
  • Cloud type (set to VMware vSphere)
  • vCenter SDK URL – You’ll set this to https://[vCenter FQDN]/sdk, it accepts self-signed certs provided by your vCenter box by default just fine.

Next we’ll need to configure some images, click “Add Image”, select a virtual machine from the pull-down list, pick a snapshot (you’re using snapshots, right?), select a folder that the clones will go into, select a resource pool (more on that later) and set the maximum number of virtual machines you want to be able to run at once.

If you don’t have resource pools because you’re on a version of vSphere that doesn’t support them, you’ll either have to wait till I get my fork done that removes this requirement and have JetBrains pull or, or download and remove the requirement yourself (I hear an older version of the plugin doesn’t require it but haven’t verified it).

Update: JetBrains beat me to it, issued a patch, works great now.

Verification And Validation

When you build a project with zero build agents installed, it’ll go into the queue. The TeamCity cloud plugin will spin up a virtual machine for you, and when that machine is registered it should show up with the name of the virtual machine. If this is what you got, success!

If not, feel free to post comments and I’ll see if I can help out.

How VMware Can Make The Web Client Awesome

Written by William Roush on August 4, 2014 at 12:39 pm

Some pretty basic design principals that would make the web client on VMware awesome, including the ability to make it redundant and supported on free systems!

I was reading this article by Trevor Pott, which does a fairly good job dealing some major problems on VMware’s vSphere web client, and how absolutely terrible it is. However I have some major issues with this article, first of all is no real concrete suggestions on architecture changes (how do we handle the vCenter single point of failure? What about free clients? What about the Flash plugins?). Here I’m going to offer up some suggestions to reaffirm Trevor’s stance that VMware could and should do this better!

A True Single-Page Application

By far I figure one of the easiest ways to resolve all of our issues is a solid single-page application. This is the concept that the website you visit will load all the resources needed on your computer to run without refreshing the page. This is generally done using HTML5 and Javascript, common frameworks include AngularJS and Ember.JS. A giant flash application like the vSphere web client has now doesn’t really count.

How to Handle the API

Some suggestions on how to handle API calls to the hosts/vCenter:

  1. Transparent layer – Have the web server host a JSON based API that gets translated into the API calls to the host/vCenter box. This allows you to have very low overhead calls (as opposed to very noisy SOAP), and allow Javascript to do what it does best (talking in a native tongue instead of using Apache CXF for Javascript clients). This incurs minor overhead on the host running the web server to do the translations, this also effectively creates two web APIs you use (though arguably you wouldn’t support consumption of the JSON API).
  2. Reverse Proxy – This allows you to remove any difficulty with Javascript dealing with cross-port requests, but you’re going to be leveraging something like Apache CXF for the web services.
  3. Direct Communication – vCenter and VMware’s APIs already exist over HTTPS for web services, if you serve up the single-page application from the same domain/port in a hybrid host setup there will be no additional overhead!

In-Browser Remote Console

Now this is the one piece I will detail is pretty experimental, and by all means feel free to fall back to a Flash/Java console, but what I’d really like to see is a true in-browser console, look at solutions like Guacamole which runs a full VNC client in-browser. Of course there may be some barriers here (Guacamole requires server-side code, not sure how much overhead is acceptable on the Busybox management VM on vSphere).

The only feature I can’t think of reproducing in HTML5 is direct device access required for mounting ISOs/USB devices.

Addressing The Single Point of Failure

These thin API layers (or in one case non-existent) allows not only vCenter to support these single-page web applications, but also the individual hosts. Now it becomes safe to completely scrap the old vSphere desktop clients.

Browser Security

Trevor Pott does some hand-waving about security issues on browsers, and then goes in to complain that the problem really relies with Flash and Java Applets. I’d recommend dropping auto-sign on removing all need for plugins and leaving it at that.

The current desktop client embeds Java applets for some 3rd party tools, so to say it’s more secure is silly.

Speed

The old Windows client is imperceptible. Click and the info is there. Expanding a tree just completes in a time frame so short that a human can’t tell there was a delay.

Yeah, I’m not going to stand by this stance at all, the desktop client is a massively bloated slow piece of garbage. It eats a massive amount of memory, is prone to killing consoles and requiring you to play whack-a-mole in your process manager to kill the spawned processes and get it online again.

The web client is slower, but the desktop client isn’t some kind of idea of what we’d want to achieve, that was pretty bad to begin with.

Using PowerCLI it seems like most operations are pretty instant, so it just seems to be entirely overhead on the applications themselves, so a well-written single-page application could easily handle this and be lightning fast.

What’s ultimately the damning element of this is that Internet Explorer is the most common enterprise browser. In many environments, browsers that aren’t Internet Explorer are outright banned. 

This is more of a problem with your work environment than the web application itself. If you’re on IE11, things are pretty decent (Javascript is fast, support for modern things is pretty up to date). If you’re at a company that keeps you on IE8 and wont let you install Chrome, that is absolutely no fault of VMware’s.

 Other Options and Why I Think They’re Not Good Routes

Native Application

This is going back to the roots of the vSphere desktop client, which generally comes with the same problems (going to be Windows only). I highly doubt VMware will write some GTK+ Windows/Mac/Linux client. So far VMware has still been unwilling to patch a major problem with RVC, so I don’t think they’re giving attention to more “hip” languages like Python and Ruby.

Cross-Platform Application

The next option is planning on a cross-platform application, and I know what they’re going to do: what ever other vendor has done.

Java.

I don’t really think I need to say more, I have a love/hate relationship with Java, but most system admins have just the hate side. Mainly it comes down to writing cross-platform applications can be more costly in languages that aren’t like Java with a nice solid platform.

Mono is also an option, but I have a feeling VMware won’t jump on that boat this early.

Freebies

By far, one of the best parts of major infrastructure decisions: freebies. Additional features or supported platforms with reduced, little or no effort. This list is by no means exhaustive.

OSX/Linux Support

This has been a goal off and on for VMware, obviously fully HTML5 will get you 98% of functionality on OSX and Linux, with minor plugins needed for device management.

Mobile Support

Take that single-page web application, wrap it in a delivery method like PhoneGap, stylize it so that it fits better on the device (different CSS files for phone/tablet), and you’re going to have not just a small subset of features (like most current mobile apps available), but the ability to fully manage your VMware cluster from the ground up.

Overall

There is no reason that VMware should have shipped the web client in it’s current state, nor is it an example of why VMware shouldn’t dedicate resources to writing solid web-based management software, it misses most of the point while throwing all of it’s resources into a dying framework. A bit of design centered around delivering the things customers have been asking for could lead to a product that will put all competitors to shame, instead of turning them away from vSphere.

vSphere 5 Memory Management

Written by William Roush on November 12, 2013 at 7:14 pm

A very informative video describing how the VMware hypervisor vSphere (AKA: ESXi) handles your virtual machine’s memory. The video below shows how to dig into your virtual machine’s memory usage statistics, read esxtop, and discusses how memory compression and Transparent Page Sharing (TPS) works!