Hadoop setJarByClass not working -
my wordcount example following structure:
public class wordcount extends configured implements tool { public static class map extends mapper<longwritable, text, text, intwritable> {} public static class reduce extends reducer<text, intwritable, text, intwritable> {} public static void main(string[] args) throws exception { basicconfigurator.configure(); logger.getrootlogger().setlevel(level.warn); int res = toolrunner.run(new configuration(), new wordcount(), args); system.exit(res); } @override public int run(string[] args) throws exception { configuration conf = new configuration(); filesystem fs = filesystem.get(conf); if (fs.exists(new path(args[1]))) { fs.delete(new path(args[1]), true); } job job = job.getinstance(conf, "wordcount"); long starttime = system.currenttimemillis(); job.setoutputkeyclass(text.class); job.setoutputvalueclass(intwritable.class); job.setmapperclass(map.class); job.setreducerclass(reduce.class); job.setinputformatclass(textinputformat.class); job.setoutputformatclass(textoutputformat.class); fileinputformat.addinputpath(job, new path(args[0])); fileoutputformat.setoutputpath(job, new path(args[1])); job.setjarbyclass(wordcount.class); // job.setjar(wordcount.class.getsimplename()); job.waitforcompletion(true); system.out.println("job finished in " + (system.currenttimemillis() - starttime) / 1000.0 + " seconds"); return 0; } }
the job.setjarbyclass()
call not working, , "no job jar file set" message. also, job.getjar() after call shows "null" value. knows what's problem here?
i tried job.setjarbyclass(this.getclass())
, job.setjar("wordcount")
, job.setjar(wordcount.class.getsimplename())
. first 1 has no effect, job.getjar()
returns null, second , third both give me filenotfoundexception
: file wordcount not exist. tried job.setjar("src/wordcount/wordcount.java")
, job.setjar("bin/wordcount/wordcount.class")
, both succeed within eclipse (without warning message), still fail filenotfoundexception
when executed standalone jar file on command line. guess problem may relate class path setting if not unresolved dependencies.
think should add appropriate jar files.
in case must have jar org.apache.hadoop.mapreduce.job
in project file.
i imported following classes , interfaces
import org.apache.hadoop.conf.configuration; import org.apache.hadoop.conf.configured; import org.apache.hadoop.fs.filesystem; import org.apache.hadoop.fs.path; import org.apache.hadoop.io.intwritable; import org.apache.hadoop.io.longwritable; import org.apache.hadoop.io.text; import org.apache.hadoop.mapreduce.job; import org.apache.hadoop.mapreduce.mapper; import org.apache.hadoop.mapreduce.mapper.context; import org.apache.hadoop.mapreduce.reducer; import org.apache.hadoop.mapreduce.lib.input.fileinputformat; import org.apache.hadoop.mapreduce.lib.input.textinputformat; import org.apache.hadoop.mapreduce.lib.output.fileoutputformat; import org.apache.hadoop.mapreduce.lib.output.textoutputformat; import org.apache.hadoop.util.tool; import org.apache.hadoop.util.toolrunner; import org.apache.log4j.basicconfigurator; import org.apache.log4j.level; import org.apache.log4j.logger;
and project working fine. check after importing above mentioned classes. if problem, give me comment.
Comments
Post a Comment