Get the task attempt ID for the currently running Hadoop task

Tag: hadoop Author: xiao_san_shao Date: 2009-09-13

The Task Side-Effect Files section of the Hadoop tutorial mentions using the "attemptid" of the task as a unique name. How do I get this attempt ID in my mapper or reducer?

Best Answer

If you need a unique id for a side effect file in hadoop, you can leverage the attempt unique id in the job with this code:

   public static String getAttemptId(Configuration conf) throws IllegalArgumentException
   {
       if (conf == null) {
           throw new NullPointerException("conf is null");
       }

       String taskId = conf.get("mapred.task.id");
       if (taskId == null) {
           throw new IllegalArgumentException("Configutaion does not contain the property mapred.task.id");
       }

       String[] parts = taskId.split("_");
       if (parts.length != 6 ||
               !parts[0].equals("attempt") ||
               (!"m".equals(parts[3]) && !"r".equals(parts[3]))) {
           throw new IllegalArgumentException("TaskAttemptId string : " + taskId + " is not properly formed");
       }

       return parts[4] + "-" + parts[5];
   }

Other Answer1

With the new Hadoop API:

context.getTaskAttemptID().getTaskID().getId()

Other Answer2

Late to the party, but you can use the TaskAttemptID class to parse the mapred.task.id property.

In my case, I wanted the numeric attempt value itself and used the following in my Mapper:

int _attemptID;

@Override
public void configure(JobConf conf) {
    TaskAttemptID attempt = TaskAttemptID.forName(conf.get("mapred.task.id"));
    _attemptID = attempt.id();
}