Very basic question about Hadoop and compressed input files

Tag: compression , hadoop Author: dlw1123 Date: 2009-12-31

I have started to look into Hadoop. If my understanding is right i could process a very big file and it would get split over different nodes, however if the file is compressed then the file could not be split and wold need to be processed by a single node (effectively destroying the advantage of running a mapreduce ver a cluster of parallel machines).

My question is, assuming the above is correct, is it possible to split a large file manually in fixed-size chunks, or daily chunks, compress them and then pass a list of compressed input files to perform a mapreduce?

Other Answer1

Consider using LZO compression. It's splittable. That means a big .lzo file can be processed by many mappers. Bzip2 can do that, but it's slow.

Cloudera had an introduction about it. For MapReduce, LZO sounds a good balance between compression ratio and compress/decompress speed.


LZO is not splittable alone. You must run a separate process to index the LZO files so that compressed blocks line correctly with input splits. See the tiny baby of a line at the very end of the page:
@Luis But do keep in mind that LZO is GPL licensed so the regular terms and conditions apply. Another alternative will be to use Google's Snappy compression. Google Snappy It's packaged by default with Hadoop (I use 0.20.x) and other ecosystem frameworks like Apache Flume, etc also understand it well by default.

Other Answer2

yes, you could have one large compressed file, or multiple compressed files (multiple files specified with -files or the api).

TextInputFormat and descendants should automatically handle .gz compressed files. you can also implement your own InputFormat (which will split the input file into chunks for processing) and RecordReader (which extract one record at a time from the chunk)

another alternative for generic copmression might be to use a compressed file system (such as ext3 with the compression patch, zfs, compFUSEd, or FuseCompress...)


Thanks that sound great.

Other Answer3

You can use bz2 as your compress codec, and this format also can been split.