問題描述
Lỗi phân vùng tùy chỉnh (Custom Partitioner Error)
I am writing my own custom Partitioner(Old Api) below is the code where I am extending Partitioner class:
public static class WordPairPartitioner extends Partitioner<WordPair,IntWritable> {
@Override
public int getPartition(WordPair wordPair, IntWritable intWritable, int numPartitions) {
return wordPair.getWord().hashCode() % numPartitions;
}
}
Setting the JobConf:
conf.setPartitionerClass(WordPairPartitioner.class);
WordPair Class contains: private Text word; private Text neighbor;
Questions: 1. I am getting error:"actual argument class (WordPairPartitioner) cannot convert to Class (?extends Partitioner). 2. Is this a right way to write the custom partitioner or do I need to override some other functionality as well?
參考解法
方法 1:
I believe you are mixing up old API(classes from org.apache.hadoop.mapred.*) and new API(classes from org.apache.hadoop.mapreduce.*)
Using old API, you may do as follows:
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.Partitioner;
public static class WordPairPartitioner implements Partitioner<WordPair,IntWritable> {
@Override
public int getPartition(WordPair wordPair, IntWritable intWritable, int numPartitions) {
return wordPair.getWord().hashCode() % numPartitions;
}
@Override
public void configure(JobConf arg0) {
}
}
方法 2:
In addition to Amar's answer, you should handle the eventuality of hashCode returning a negative number by bit masking:
@Override
public int getPartition(WordPair wordPair, IntWritable intWritable, int numPartitions) {
return (wordPair.getWord().hashCode() % numPartitions) & 0x7FFFFFFF;
}
(by JackSparrow、Amar、Chris White)