Bootstrapping Chef with Ruby-1.9.2-p290 on Ubuntu 10.04 LTS
I just got going with Chef and then realised that it was installing Ruby 1.8.7 on my server to bootstrap itself. How last decade is that?! I was shattered, I thought all the cool kids were using Chef. I did start to wonder what I was getting myself into.
Turns out it is not too difficult to change the bootstrapping behaviour. So I set up a bootstrap template to install Ruby 1.9.2-p290 instead. I had to do a little tweaking and then knife installed the chef-client without any real problems. Below is my bootstrap template. I put it in my chef-repo under .chef/bootstrap/install_ruby_192-p290.erb
This script updates apt with build-essentials and all the other goodies required to compile ruby successfully. It then pulls down 1.9.2-p290 from ruby-lang.org, unpacks it, compiles it and installs it. It follows that with updating ruby gems to an appropriate version for 1.9.2 and then installing the ohai and chef gems. Ruby will end up in /usr/local/bin rather than /usr/bin and so does ruby-gems and so you can see that the call to chef-client at the end of the script had to be hard coded rather than calling to chef_start since chef_start seems to assume the the gem executable is in /usr/bin.
bash -c '
if [ ! -f /usr/local/bin/chef-client ]; then
apt-get update
sudo apt-get install -y build-essential zlib1g zlib1g-dev openssl libssl-dev
cd /tmp
wget ftp://ftp.ruby-lang.org//pub/ruby/1.9/ruby-1.9.2-p290.tar.gz
tar xvf ruby-1.9.2-p290.tar.gz
cd ruby-1.9.2-p290
./configure
make
sudo make install
sudo gem update --system
gem update
gem install ohai --no-rdoc --no-ri --verbose
gem install chef --no-rdoc --no-ri --verbose <%= bootstrap_version_string %>
fi
mkdir -p /etc/chef
(
cat <<'EOP'
<%= validation_key %>
EOP
) > /tmp/validation.pem
awk NF /tmp/validation.pem > /etc/chef/validation.pem
rm /tmp/validation.pem
(
cat <<'EOP'
<%= config_content %>
EOP
) > /etc/chef/client.rb
(
cat <<'EOP'
<%= { "run_list" => @run_list }.to_json %>
EOP
) > /etc/chef/first-boot.json
/usr/local/bin/chef-client -j /etc/chef/first-boot.json'
Once you have that in place you can simple type:
$knife bootstrap NODE --distro ubuntu10.04-gems -x ubuntu --sudo --template-file .chef/bootstrap/install_ruby_192_p290.erb
and your node should be bootstrapped with ruby 1.9.2 instead of 1.8.7. Obviously replace NODE with your node hostname. I hope this helps someone else, it took quite a while of digging to work this out. I won't be surprised to find out in a week or so that there is some existing bog standard way of doing this and I just didn't know what it was, but until then, hopefully this can shortcut anyone else in a similar situation to me.
Whoops, forgot to background that 20gb scp
How pissed off do you get when you start a massive copy from one server to another via an ssh shell and forget to background it?! I was faced with this today. I had already copied 2gb when I realised my error and rather than stay at work over the weekend I had to do a little research to find out how to bail out of the terminal session without killing my scp process. Bash to the rescue:
$^z
control-z suspends the current process and gives you back your prompt,
$bgbg backgrounds your suspended process and lets you get back to the essential task of packing up so you can go get some beers.
OS X GCC Installer
After having a little bitch about the size of the Xcode download required just so I can compile binaries I saw a link to this little baby. I haven't tried it myself yet, since I already sucked up the anguish of sacrificing a fair whack of my download cap in exchange for the joys of Xcode on my mac, but it looks like an awesome option for those of us who are as little bandwidth challenged at times. It is essentially an installer for just the important parts of Xcode, i.e. GCC and other essential binaries for linking and making. I'll definitely give this a go next time.
RMagick / ImageMagick on OS X Lion
I have had a few interesting problems with this so I thought I would document how I went about fixing my ImageMagick problems after I upgraded to Lion.
First I had to uninstall the rmagick gem then uninstall ImageMagick, I use homebrew so I just did
$brew uninstall imagemagickNext I had to get the new Xcode version from the appstore, thankfully it is free but at three point something gigabytes it certainly put a dent in my download cap.
With Xcode installed I was able to re-install ImageMagic with the usual
$brew install imagemagicLovely, image magic installed and now
$gem install rmagickworks without complaint, everything is great, right? …wrong! Unfortunately ghostscript seems to have disappeared from my system, I can't say I ever used it intentionally so I'm not sure when I lost it. What I can say is that any ImageMagic font work is done using the ghostscript fonts, so anything like captcha generation fails badly with this new 'working' rmagick install as ImageMagick can't find the fonts where it expects them (in the Ghostscript directory).
The answer to this problem is to install ghostscript
$brew install ghostscriptUnfortunately this failed for me due to the SHA1 being wrong for the current ghostscript version - 9.02. Wow, what a pain in the butt. The fix for this turns out to be applying a patch to the ghostscript recipe
$brew edit ghostscriptThen add:
def patches { :p0 => 'http://ftp.netbsd.org/pub/NetBSD/packages/pkgsrc/print/ghostscript/patches/patch-ak' } end
Problem solved, ghostscript installs normally and now ImageMagic can do its thing with fonts, which means that rmagick can quit with its complaining and I can get back to development…awesome.
Rails 3 Error Display
Rails error management is helpful and automatic, and a total pain in the ass sometimes. The reason it can be annoying is that when a field has an error it wraps both it and it's label with separate divs with the class "field_with_errors". From an HTML layout point of view this approach is a long way from awesome.
If you are putting your labels above your fields the only simple way to manage this is to make your label a block level element. If you put a br after your label it makes the label jump up a line above the field when an error is present. So that approach is out. Well that is just great for people who want to make their labels block level elements but I find that it totally sucks. I prefer to put a red asterisk or red dot at the end of the label of a field that is mandatory. This is trivially easy on inline elements, just add some right padding and position the mark as a background image with no-repeat at the right.
Unfortunately it is all but impossible to do for a block level label. I found a way around this, hopefully it can help others. Not exactly rocket science, but I wrote a helper to generate the br element only if their is no error on row field. Here it is:
def br_if_no_errors(model, field) "<br>" unless model.errors.get(field).present? end
BDD or Bullshit?
The BDD debate goes on and on. Every new place I work people have a new and completely different take on what it is and whether it is good or bad. I have to admit that I have gotten to the point where if a codebase I am about to start working on already has a Cucumber test suite, I start to get nervous.
I am now at the point where, on a project using BDD and I wasn't there at inception, I expect half my time to be spent fixing the Cucumber end of the build. If I'm spending that much time fixing them they would want to be providing some pretty airtight certainty of correctness and a certainty that I couldn't get with well scoped specs, usually they aren't.
Replacing Dozer with SimpleObjectAssembler
Since I made the painful move to using DTOs in my Java applications (at least when any remoting is involved) I have been less than excited with the work involved in mapping domain objects to their DTO counterparts. There is a whole, lengthy conversation in why I abandoned trying to remote with domain objects (even when the domain is reasonably trivial) and went to DTOs but ostensibly it was driven by Hibernate and JPA and the enormous pain in the ass involved in detaching the object graph.
Until recently I have been using Dozer to do the heavy lifting in transforming from domain to DTO and back. It is heavily XML based and requires a fair bit of XML configuration for anything other than the most simple mappings. Since I have a long history with Spring (since long before annotations came along) I just accepted this along with the other billion lines of XML required for defining my applications. I guess I had just become immune to configuration fatigue, it all seemed quite normal to me. Even when a friend and colleague (Rob Monie) told me about a little open source project he was working on to simplify the mapping process without the need for XML (SimpleObjectAssembler) I initially ignored it and stuck with the devil I knew.
Recently, I just got sick of it all and decided to move to the Simple Object Assembler. My main motivation was not so much about Dozer as that I had implemented my mappings without tests to verify them and could no longer stand the pain of supporting that approach. I decided to properly test my transforms and decided it was a good opportunity to scrap Dozer and clean up.
SimpleObjectAssembler allows you to define field to be excluded inline when you do the mapping. There is no need to set up an XML descriptor for each type-to-type mapping. It does some pretty reasonable inferences where fields are named the same and allows you to register custom converters for complex type conversions. I like this approach as it splits out the conversion logic into classes that are highly specialised to the task. So long as you name them clearly it is simple to track down the conversion logic later and even easier to test it in isolation. The same cannot be said for Dozer.
I have converted Express 0.8-RC2 to SimpleObjectAssembler and aside from the increased testability I have achieved I am also getting a slight performance increase. This is likely due to the fact that it is easier to highly customize the mappings a service performs as you can rework the exclusions in-line rather than just putting up with an XML configured conversion that is close to what you want (the Dozer approach).
Install ruby-pg gem on snow leopard
I can't hunt the web for how to do this again:
export PATH=/Library/PostgreSQL/9.0/bin/:${PATH} env ARCHFLAGS="-arch x86_64" gem install ruby-pg
Automatic Getter And Setter Testing with Hamcrest
Getter and setter testing is a pain in the behind. This kind of low value testing is time consuming and really hard to justify, but every now and then it turns up a variable shadowing bug or some problem in assignment that would be brutally hard to work out based on user defect reports. The other down side of not doing it is that it really hits your test coverage metrics hard and causes no end of pointless depression for teams trying to keep high quality code metrics.
I am a strong exponent of TDD but, personally, I rarely drive out domain getters and setters with tests. I'm certain that I am committing some type of mortal sin by not doing so, but I usually have a fair idea about the data required from domain modelling discussions and I like to get out a bare bones implementation of the domain quickly to facilitate initial stories.
Over a number of years I have used a getter/setter invoking strategy that is fairly horrible but functional. Recently I had cause to go back and have a look at it and decided to rewrite it using Mockito, Hamcrest matchers, and some varargs to make the whole thing a lot more useable.
Two matchers are exposed, one that will invoke a getter and setter for all member variables of a class and will exclude zero or more fields that the user provides
@Test public void shouldSetAndGetProperties() { assertThat(user, hasValidSettersAndGettersExcluding("fieldX", "fieldY")); }
The other matcher will invoke a getter and setter only for the fields provided
@Test public void shouldSetAndGetProperties() { assertThat(user, hasValidSettersAndGettersFor("fieldA", "fieldB", "fieldC)); }
The heavy lifting is done by a GetterSetterInovker class:
package com.express.testutils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import java.lang.reflect.Field; import java.lang.reflect.Method; import java.lang.reflect.Modifier; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; public class SetterGetterInvoker<T> { private static final Log LOG = LogFactory.getLog(SetterGetterInvoker.class); private T testTarget; private Class testClass; private ValueFactory valueFactory; private Map<String, Method> methodMap = new HashMap<String, Method>(); public SetterGetterInvoker(T testTarget) { this.testTarget = testTarget; this.testClass = testTarget.getClass(); this.valueFactory = new ValueFactory(); for (Method method : testTarget.getClass().getMethods()) { methodMap.put(method.getName(), method); } } public boolean invokeSettersAndGettersFor(String[] fields) { return invokeSettersAndGetters(fields); } public boolean invokeSettersAndGettersExcluding(String[] excludedFields) { Field[] classFields = testClass.getDeclaredFields(); List<String> fields = new ArrayList<String>(); for(Field classField : classFields) { String fieldName = classField.getName(); if(!in(fieldName, excludedFields) && !Modifier.isStatic(classField.getModifiers())) { fields.add(fieldName); } } return invokeSettersAndGetters(fields.toArray(new String[fields.size()])); } private boolean in(String field, String[] fields) { for(String arrayField : fields) { if( arrayField.equals(field)) { return true; } } return false; } private boolean invokeSettersAndGetters(String[] fields) { for (String field : fields) { LOG.debug("Invoking getter/setter for " + field); String capitalizedFieldName = Character.toUpperCase(field.charAt(0)) + field.substring(1); try { Object testValue = invokeSetter(methodMap.get("set" + capitalizedFieldName)); Method getterMethod = methodMap.get(constructMethodName(capitalizedFieldName, testValue)); Object result = getterMethod.invoke(testTarget); if (!testValue.equals(result)) { LOG.error("Getter result did not match set value for field:" + field); return false; } } catch (Exception e) { LOG.error("Unable to invoke both getter and setter for " + field, e); return false; } } return true; } private Object invokeSetter(Method method) throws Exception { Class<?>[] parameterTypes = method.getParameterTypes(); if (parameterTypes.length == 1) { Object testValue = valueFactory.createValue(parameterTypes[0]); method.invoke(testTarget, testValue); return testValue; } return null; } private String constructMethodName(String capitalizedFieldName, Object testValue) { String base = "get"; if(testValue instanceof Boolean && testMethodNameDoesNotExists(base + capitalizedFieldName)) { return "is" + capitalizedFieldName; } return base + capitalizedFieldName; } private boolean testMethodNameDoesNotExists(String methodName) { try { testClass.getMethod(methodName, null); return false; } catch (NoSuchMethodException e) { return true; } } }
The GetterSetterInovker relies on the ValueFactory to create values for setting and getting. For non-final classes, it simply returns a Mockito mock of the type. Plenty of the Java core types are final however so there is a little ugliness here.
package com.express.testutils; import java.lang.reflect.Modifier; import java.util.HashMap; import java.util.Map; import static org.mockito.Mockito.mock; public class ValueFactory { private static final int A_NUMBER = 42; private static final int HALF_A_NUMBER = 21; private Map<Class<?>, Object> typeValues; public ValueFactory() { typeValues = new HashMap<Class<?>, Object>(); typeValues.put(String.class, "test"); typeValues.put(Long.class, new Long(A_NUMBER)); typeValues.put(Long.TYPE, new Long(A_NUMBER)); typeValues.put(Integer.class, Integer.valueOf(HALF_A_NUMBER)); typeValues.put(Integer.TYPE, Integer.valueOf(HALF_A_NUMBER)); typeValues.put(Boolean.class, Boolean.TRUE); typeValues.put(Boolean.TYPE, Boolean.TRUE); typeValues.put(Long[].class, new Long[0]); typeValues.put(Integer[].class, new Integer[0]); typeValues.put(String[].class, new String[0]); typeValues.put(Boolean[].class, new Boolean[0]); } public Object createValue(Class<?> type) { if(Modifier.isFinal(type.getModifiers())) { return typeValues.get(type); } return mock(type); } }
All the nastiness of getting and setting gets hidden away behind a pair of Hamcrest Matchers. As described above one invokes getters and setters for all class member variables except those specified, the other will invoke getters and setters for only the member variables specified. Here is the one that invokes getters and setters for all class fields except those specified:
package com.express.matcher; import com.express.testutils.SetterGetterInvoker; import org.hamcrest.Description; import org.hamcrest.TypeSafeMatcher; public class HasValidSettersAndGettersExcludingMatcher<T> extends TypeSafeMatcher<T> { private String[] fields; public HasValidSettersAndGettersExcludingMatcher(String... fields) { this.fields = fields; } @Override public boolean matchesSafely(T subject) { SetterGetterInvoker<T> setterGetterInvoker = new SetterGetterInvoker<T>(subject); return setterGetterInvoker.invokeSettersAndGettersExcluding(fields); } public void describeTo(Description description) { description.appendText("Unable to match setter and getter results for class fields excluding "); description.appendValue(fields); } }
And this one invokes only those specified:
package com.express.matcher; import com.express.testutils.SetterGetterInvoker; import org.hamcrest.Description; import org.hamcrest.TypeSafeMatcher; public class HasValidSettersAndGettersForMatcher<T> extends TypeSafeMatcher<T> { private String[] fields; public HasValidSettersAndGettersForMatcher(String... fields) { this.fields = fields; } @Override public boolean matchesSafely(T subject) { SetterGetterInvoker<T> setterGetterInvoker = new SetterGetterInvoker<T>(subject); return setterGetterInvoker.invokeSettersAndGettersFor(fields); } public void describeTo(Description description) { description.appendText("Unable to match setter and getter results for class fields: "); description.appendValue(fields); } }
A factory sits over the top to make it simple to acquire a matcher in the usual Hamcrest way:
package com.express.matcher; import org.hamcrest.Factory; public final class BeanMatchers { private BeanMatchers() { } @Factory public static <T> HasValidSettersAndGettersForMatcher hasValidSettersAndGettersFor(String... fields) { return new HasValidSettersAndGettersForMatcher(fields); } @Factory public static <T> HasValidSettersAndGettersExcludingMatcher hasValidSettersAndGettersExcluding(String... fields) { return new HasValidSettersAndGettersExcludingMatcher(fields); } }
Agile Tools
I read a bunch of Agile and Scrum user groups, since I am fairly involved both in Agile development and tool support for Agile development it seems sensible to stay in touch with what people are talking about and what questions they are asking.
It seems like about once a month someone will post a question to one of these groups asking for advice on what backlog management tool they should use or what is everyone else using. Often they are looking for open source tools, but not always. More often than not, whether they are requesting open source tools or not, sales people from the various agile tool companies jump on these posts to push their product, often bringing up the old "how free is free" argument, implying that open source means poorly made and you will spend more time/money trying to get it working than you would just buying their fantastic tool.
I often wonder about these types of questions. They are so unspecific about what the asker is actually looking for, I often want to ask: "What makes you think you need a tool?" or "What is the problem you are trying to solve?". Obviously I would like people to check out Express, like it, and get a lot of value from it, but it seems hard to imagine that anyone is going to be able to select a tool unless they are clear on what they need from it and how it is going to improve their process.
For many groups there is no need for anything more sophisticated than a whiteboard and some type of shared document space, like a wiki or a google doc/site. For other teams this is nowhere near adequate. What are the factors that make this nowhere near adequate? I thought I might explore them as I have some experience in this and it is why I began developing Express in the first place.
1. You have stakeholders who are not co-located with the development team: This is really common for external development companies, outsourcers, and consultancies. There is nothing unagile about not having all stakeholders co-located. Of course there needs to be a customer representative but that person will likely be representing all sorts of interested parties who would like to be able to get information on how the project is going. Tools can help here to keep the remote parties in the loop about what is going on and allow them to review the stories as well as progress.
2. Your team is distributed. There is nothing unagile about this either. Many teams have an off-site component to their development or have entire sprint teams off-site or off-shore. Of course everyone being in the same room has immense benefits in terms of informal communication and information sharing but that is not always an option and in some cases, not even desirable. Teams who are not physically co-located need to work hard to maintain communication and their commitment to tasks assigned to them. The best way to do this is with a virtual wall. This allows everyone to view the current state of the iteration stories and tasks.
3. You need to manage and track project metrics. Some agile teams are not too concerned about detailed metrics. They know their velocity and that is enough. Other teams like to keep detailed metrics on what happened during a sprint. This can be very onerous unless the team has some tools to help. Tracking this kind of thing on a spreadsheet can quickly become problematic. An Agile project management tool can be really helpful here to track how many points were delivered in each iteration, how the total count of story points delivered compares to the number of story points required to release, etc.
4. Your team has stakeholders who would like to know when you're likely to be finished. Okay, that is probably every team but it can be very difficult to say when you are likely to be finished when you don't have a tool to help with estimation and time tracking. If you are using a backlog management tool you should be able to get a more and more accurate picture of how you are tracking and when you are likely to deliver.
In general I think that knowing what you are looking for is going to make a big difference as to the tool that you select to manage your backlog. Other considerations are around integration with existing tools (if you have them) such as defect management tools and version control. It is well worth just thinking about whether you need this type of integration and exactly what benefit you see your team getting from it.