Getting a PhD, Week 5: The Joy of Sextractor
Photo by Fotis Fotopoulos, via Unsplash
Week 5: The Joy of SExtractor
I love programming. There is little more satisfying than watching your hard-written program cascade into reality. It's the closest I've ever come to feeling like a god. However, this rarely - perhaps never - happens on the first try. I present the following as an example of the numerous frustrations that can encumber scientific research:
Part of my research involves getting photometry of distant galaxies - essentially calculating how bright they are, using images taken by telescopes (like the Very Large Telescope, the one I work with most often). For this, I use a package called Source Extractor (usually contracted to SExtractor; go on, get it out of your system). This software has its issues, but installation had never been one of them for me - it always installed cleanly and easily, and I could use it without much issue (aside from my own ignorance of its inner workings, which came back to bite me more than once during my Masters). However, I moved to a new computer when I started my PhD, which means that I had to reinstall everything, SExtractor included. No problem, I thought, easily done. I typed 'sudo apt install sextractor' into the terminal and pressed enter, which is the way such things are usually done on Ubuntu; the usual stream of text slid by, notifying me that installation had occurred and that the package was ready to use. I then ran some of my own code, which invokes SExtractor as a part of it, on one of our newly-observed galaxies, expecting everything to run smoothly. It did not. Instead, it failed spectacularly. Inspecting the output, I noticed that SExtractor had not run at all. I typed 'sex' into the terminal (this is the command for running SExtractor; yes, they knew what they were doing), only to be served a 'command not found' error.
It turns out that, although my computer believed it had installed SExtractor, it really hadn't; the binary files it runs from were nowhere to be found in my system (and I hunted in every place I could think of). The quick, clean, one-line installation method I had used previously no longer worked. I still don't know why; it might be because I'm using Ubuntu 20 when previously I was on 18, or that SExtractor had upgraded to version 2.25 while I wasn't looking, or some arcane interaction between these two factors. As a result, I needed to install SExtractor manually. No problem, I thought, shouldn't be too hard. It's a little annoying, but I've installed things before.
Not SExtractor, I hadn't. It turns out that this package has a peculiar set of dependencies, some of which are nowhere to be found in the standard Ubuntu repos and must themselves be installed manually (and non-trivially). One of these - ATLAS - required me to fiddle with my PC’s CPU throttling settings, not a thing I knew even existed. Two days later, and I had just managed to get Sextractor working. A process that should have taken less than a minute took up a huge chunk of my working week. This is what you get, working with scientific software on Linux. Usually, it just works; but when it doesn't, expect to spend FAR too long working out why.
I’ve heard it said that if a software developer tells you how long a project will take, they’re lying. I think the same is true of scientific research. This is perhaps the most important lesson I took from my Masters: that research is filled with this kind of landmine. That research plan that I so carefully arranged in my calendar slipped through my fingers one date at a time. There were periods in which I seemed to achieve not much at all, when I was either bashing my head on the same problem for weeks at a time or (felt like) I was just chasing my tail; and then there were great surges of productivity, when things seemed to all click together, to work, and I would make more progress in a week than I had in the last month. The most notable of these occurred in the final weeks before my thesis was due; I finished writing the code to collect key data perhaps three weeks before that looming Darth Vader of a deadline, and didn't actually finish collecting the data until the very week of the due date. The result, I was at least happy with, and so were my markers; enough to admit me to the degree, in any case.
So: if you imagine astronomers looking through telescopes through most of the working day, you’re picturing it wrong. We spend far more time wishing that we could strangle our computers.