Search

Can we compress the output from Map tasks ?



Yes, Mapreduce allows us to compress the intermediate map output files on local file system to reduce network traffic.
Below are the lines to add to enable Snappy map output compression in your job:


Java

Configuration conf = new Configuration();
conf.setBoolean("mapred.compress.map.output", true);
conf.setClass("mapred.map.output.compression.codec", SnappyCodec.class,
CompressionCodec.class);
Job job = new Job(conf);