Using Makefiles for analysis pipelines

Got a mess-of-data. Whole load of it. Need to analyze it. Again and again and again.

Don’t want to cut n paste. Or write too much code. Need to automate plot generation.

This reminded me of my thesis many (*cough cough*) years ago. I used a Makefile to automate driving TeX. And image formatting, and final document assembly. Yes, to write my thesis, I typed “make”.

Sure enough, same type of process, different decade (*cough millennium*).

Started throwing it together and …

Damn, thats nice. I forgot how much fun it used to be to do this stuff.

Viewed 49047 times by 5341 viewers

Facebooktwittergoogle_plusredditpinterestlinkedinmail

3 thoughts on “Using Makefiles for analysis pipelines

  1. @Damien

    … so you’ve not used Fortran or Python I see …

    Indentation as a scoping method in Python. Indentation as demanded by f77 (f90 and beyond largely did away with that sillyness).

    I don’t necessarily like the tabs as part of the syntax. But the concept? Absolutely brilliant. Makes repeatable analyses real easy to build. Its kind of a shame that we can’t easily build a parallel distributed make code that distributed the computation among multiple nodes and used pipes or shared fs as an IPC method. Would make pipeline building easy. No wait …

  2. Back in the UK (10 years ago) I wasted a week helping some folks out with a Python project and working around a very dumb urllib. Using the language with no introduction (or help) I just couldn’t get it to work, though the code looked fine.

    Then they told me that the code was fine but my indentation wasn’t right. My response was “but that’s how I want it indented”. Sadly the developers preference doesn’t seem to matter for much there.

    I’ll stick to Perl I think..

    Regarding the problem at hand – this person seems to be trying to scratch a very similar itch to you…

    http://stackoverflow.com/questions/399409/free-secure-distributed-make-system-for-linux

Comments are closed.