Archive For November 16, 2015

The amazing GNU parallel

By |

The GNU parallel allows you take advantage of multiple cores on your machine Nice (bioinformatics heavy) explanation here

Read more »

first scratch at spark

By |

This follows from the previous post where i tried out Hadoop. This uses spark on the same comet cluster. To run interactively, the best thing to do first off is add this to my .bashrc

Then in the folder with the spark code (get it here), run this

Basically, it sleeps for 4…

Read more »

First scratch at hadoop

By |

Trying out comet at SDSC, thanks to the XSEDE folks. I’m not familiar with SLURM, but it was fun especially since I’ve been interseted in Hadoop and Spark for awhile. This is a run through using Hadoop Map Reduce using a Java program that looks for anagrams in a list of words. (link to data/code…

Read more »