Sunday, October 20, 2013

Tygers - Chapter 2

When Sophie the Tiger was big enough, she loved to go on adventures with her big sister Rosie. Rosie loved to teach Sophie all the things she’d learnt, like how to prowl, how to growl as ferociously as possible, how to catch fish and which ones are the tastiest, and the best places to lie in the sun on a warm day. Sophie loved to follow Rosie around and try to do whatever she did. But mostly the two sisters loved to be near each other.

One sunny autumn afternoon, after they had spent the morning practicing their ferocious growls on a small family of beetles (or perhaps it was a family of small beetles, I don’t quite recall), and the beetles had gotten so upset that they had scurried into a crack in the ground and refused to emerge until the following spring, Rosie and Sophie were lying on a grassy bank watching the stream flow lazily by, and wondering what to do next.

Rosie lay on her back in the warm grass, feeling the sun on her tummy fur. A mother duck was swimming downstream, leading her family of five fluffy ducklings. She gave a cautions “quack” when she saw Rose and Sophie, and lead her ducklings towards the middle of the river. The last and littlest duckling swam curiously towards the bank. Sophie turned and sniffed at the duckling who paddled away furiously to catch up with his mother and siblings.

“I know,” said Rosie, “Let’s go and look for dinosaur bones.”

Sophie jumped to her paws. “Are there any dinosaurs?” she asked, her ears quivering with excitement. “I’ve never seen one.”

“No,” said Rosie, “the dinosaurs are all gone. But some of their bones have turned into fossils. I learnt about them at pre-school when we went to the Hall of Science.”

“What’s a fussel?” asked Sophie.

“They’re things you dig up,” said Rosie, not being quite sure herself.

“You mean like a carrot?” asked Sophie. “I dug up a carrot once. But I gave it to a bunny rabbit.”

“I think so,” said Rosie. “Only I think you have to dig deeper. More like a potato.”

“Oh,” said Sophie, and thought a little. “They have potatoes at that people-village up in the hills,” she said. “Maybe they grow fussels there too?”

“Yes,” said Rosie. “They’re sure to”, she said. “Come on. Let’s go.”

And so the two tigers headed off together.

Rosie and Sophie walked side by side along the path that ran by the side of the river. After a while the path became narrower and Sophie and Rosie had to walk one in front of the other. They came to a place where a narrow path lead off to the right. There was a post in the ground, and a board nailed to it upon which was written “Little Pig”. Underneath was a smaller board saying “No soliciting”, and underneath that was a piece of note paper, upon which was written, “No Big Bad Wolves”. Rosie turned to Sophie and asked “Shall we see what’s down here?”. Sophie nodded.

The path ended in front of a small brick house. On the porch, a small, pink pig sat in a rocking chair, chewing a piece of grass. “Hello little Tigers,” said the pig.

“Hello,” said Rosie curiously, “Do you live here by yourself?”

The pig explained that his two brothers used to live next door, but that their houses had been demolished due to not meeting the local building codes, and they had moved to Florida, so he was feeling a little lonely. “Would you like some tea?” he asked.

Rosie and Sophie followed the pig into his house. He made a large pot of tea and put out a plate full of scones with a bowl jam and and another of clotted cream.

“We’re looking for dinosaur bones,” said Sophie after she’d finished her fourth scone. “Would you like to come?” she asked, and licked a blob of clotted cream of her nose.

“Oh yes!” said the piggy. “I have a a very good nose for finding things that are buried.”

The two tigers and the pig set off through the forest. The pig put his snout to the ground and sniffed and the tigers followed. They threaded their way between the trees and up a muddy river bank. Suddenly Rosie stopped. “Look!” she called. Sophie came to stand beside her and the pig scurried back to see what she had found.

Rosie pointed with her nose at the mud in front of her. “Tracks,” she said. The other animals peered at the paw prints in the mud.

“What are they?” asked Sophie. “They look like tiger paws, but they’re too small.”

“But they’re much too big for a kitty-cat,” said the piggy.

“Maybe a leopard,” suggested Rosie thoughtfully. “Let’s follow them and see where they lead.”

The two tigers and the pig set off, this time with Rosie in the front. The tracks continued through the trees for a little while longer and then climbed a hill. At the top was a mound of loose earth. The pig sniffed. “There’s something buried underneath,” he said. Sophie and Rose started to dig with their big front paws.

After about ten minutes, Sophie’s claw hit something hard. There was a small metalic “clink” sound. They carefully dug a little further and brushed the dirt away to reveal a large wooden box, fastened with iron straps and hinges. It was too heavy for either tiger to lift by herself, so Rose took one side and Sophie took the other, and together they lifted the chest out of the pit.

Fortunately there was no lock on the box. But the hinges had rusted and creaked loudly as the three animals pushed and pulled at the lid to open it. At last, with one last creak, the like swung open, and inside Rosie and Sophie saw the largest bone they had ever seen. It was a soft mottled white colour, smooth, and several inches longer than Rosie’s tail. Around one end was tied a paper tag on which the words “This dinosaur bone belongs to Baby” were written in large letters, and on the other side of the tag was written just “Baby” in big letters.

“Wow!” said Sophie, licking the tip of her nose.

“We’d better put it back,” said Rosie. “It belongs to someone else.” They put the bone back in the box, lowered it into the pit, and filled it up with earth until it was just as they had found it.

When they were done, Rosie and Sophie noticed that the pig was busy sniffing around the base of a tree and burrowing with his snout. After a minute he came up holding something black and wrinkly between his teeth. “Look!” he cried triumphantly.

“What is it?” asked Rose and Sophie together, coming a bit closer.

“It’s a truffle,” said the pig. “They’re delicious, and only we pigs can find them. C’mon!” And he trotted back to his home, holding the truffle proudly between his teeth, while Rose and Sophie followed curiously behind.

By the time Rose and Sophie reached the pig’s little brick house, he had already put on his apron and put a large pot of water on the stove to boil. The pig made three plates of fresh pasta and put shaved truffles and parmesan cheese on top. The two tigers were so hungry after all the digging they’ed done, that they both licked their plates clean. They both said “Yum!” and thanked the piggy lovingly for his hospitality, and then went on their way, purring softly to themselves.

Thursday, October 3, 2013

Making Visual Studio Solutions, CMake and Cygwin Play Nicely Together

I'm currently involved in development and maintenance of a set of command-line tools for analyzing Affymetrix micro-array data. The tools are developed in C++ and intended to be platform independent (compiled and tested on a couple of different Linux flavors, Windows using Visual C++, and OS X). We're working on version 2.0 which is the first major rewrite in several years.

Previous versions relied on separate Visual Studio solution files for Windows, and makefiles for Linux, which needed to be maintained in parallel. This was a pain, particularly in the Windows world: for Unix/Linux, Make hasn't changed much for decades, and if you learnt to use it in the 80's you'll probably still be able to get by. But, for Visual Studio things can change drastically between versions, so that the solution and project files in VS2008 and 2010 are completely different, and required a major transition. So, in version 2.0 we decided to switch to CMake to generate our build scripts across all platforms.

We've also been using Bamboo to run our continuous integration builds and nightly regression tests. For Linux this meant having Bamboo execute a bash script. So as to make things as uniform as possible between platforms, we opted to use a bash script under cygwin to run the builds and tests on Windows.

I found a number of issues and complications in getting cygwin to work with Visual Studio 2010/2012 solutions and CMake - some I found details of on the web, and others required experimentation to fix. Since there wasn't one place I could go to find out all these things, and much of the information I found was outdated, I'm going to gather them all together in one place as a public service. The intended audience is people who are familiar with CMake and bash, but struggling with getting things working in the Windows world.

Setting environment variables for MSBuild/VC++

There are a bunch of environment variables needed for running Visual Studio solutions in the command line, which can be set by one of the bat files vcvars*.bat (see, Since were doing 64 bit only, we want the ones called vcvarsx86_amd64.bat. We need to call the right one for the version of Visual Studio we're using:

# Read vcvars.bat file to set Windows tools settings
# VS2012 version
${CMD} /c "C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\x86_amd64\vcvarsx86_amd64.bat"

Or for Visual Studio 2010:

# VS2010 version

${CMD} /c "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\x86_amd64\vcvarsx86_amd64.bat"

Running CMake from cygwin

First we need to pick the appropriate executables for CMake and CPack:

CMAKE="/cygdrive/c/Program Files (x86)/CMake 2.8/bin/cmake.exe"
CTEST="/cygdrive/c/Program Files (x86)/CMake 2.8/bin/ctest.exe"

We're doing an out-of-source build, so we use two environment variables, SRCROOT for the source directory (in our case an svn checkout), and CMAKEROOT for the build directory. We may have to create the CMAKEROOT directory if it's not there already:

# Create cmake directory if not present
if [[ ! -d "${CMAKEROOT}" ]]
  mkdir "${CMAKEROOT}"

In order to run CMake we'll need to specify the Generator, and to feed it the source directory in Windows path format. If you do "cmake --help" it will provide a list of available generators. In our case we want "Visual Studio 11 Win64". To get the directory name in the format CMake wants, rather than in cygwin format we use the "cygpath -w" utility.

This next bit is important:

We now come to a particularly annoying and hard to diagnose bug (see cygwin by default declares certain environment variables like TEMP and TMP. CMake will use certain environment variables, including tmp and temp,  In cygwin this is fine and good, but Windows applications, and particularly MSBuild, do not distinguish between environment variables based on case. So when you run MSBuild after CMake in the cygwin environment, MSBuild will die because it sees that there are multiply defined environment variables.

The worst thing about this is that, once you've encountered the problem, simply undef-ing the relevant variables won't fix it. MSBuild starts some processes the first time it's run and those will continue to break. At this point you have to go into the Task Manager and kill any MSBuild processes before you can contiue. The work around is to undef all the variables before the first call of CMake.

This is the code from our scripts which runs CMake:

# Set the generator - VS2012 version
CMAKE_GENERATOR='Visual Studio 11 Win64'
# Have to unset some environment variables because of a weird bug with cmake under cygwin
# - see
unset tmp TMP temp TEMP
# Now call CMake
"$CMAKE" -G "$CMAKE_GENERATOR" "$(cygpath -w ${SRCROOT})"
if [ "${cmake_rv}" != 0 ]
  echo "CMake encountered errors"
  exit 1

Using MSBuild to build a Visual Studio solution file

CMake should create a .sln file and a bunch of vcxproj files (one for each target). First we need to pick the right MSBuild executable, and also decide on the configuration and platform for our build:

# This maybe system dependent
# We're going to do a 64 bit release build

Checking for errors in MSBuild

With Make in the Linux/Unix world, we can check the output value ("$?") to see if the build was successful.  With MSBuild this doesn't necessarily work. Consequently, to check for a successful build we need to generate a log file, and scan the log file for an errors message - the log file should contain the line '0 Error(s)' if it was successful. (In the following the environment variable SLN_FILE is set to the name of the solution file).

"${MSBUILD}" ${SLN_FILE} /fl /property:Configuration=${BUILD_CONFIG};Platform=${PLATFORM}
if [ ${msbuild_rv} != 0 ]
    echo "MSBuild ${SLN_FILE} exited with status '${msbuild_rv}'"
    exit 1
grep -q '0 Error(s)' ${MSBUILD_LOG}
if [ $? != 0 ]
    echo "MSBuild ${SLN_FILE} detected errors"
    exit 1

(The /fl option causes MSBuild to write a log file).

Using MSBuild to build a Project file

Our CMake scripts are configured to create a number of additional custom targets which are not part of the default build target (do not have ALL set), such as a target regression which will update the regression test data and run the regression tests. In Linux these translate to Makefile targets which are not called as part of the default target. In Visual Studio, these give rise to .vcxproj files which are not built by default when one builds the main project .sln file.

To build these we can call MSBuild directly on a project file:

# Build regression project
"${MSBUILD}" ${REGRESSION_PROJ_FILE} /fl /property:Configuration=${BUILD_CONFIG};Platform=${PLATFORM}

And that's all. Hope this is useful to someone out there.