hadoop 地图reduce鍒板簳鏄浣曡繍琛岀殑
聽
聽
聽
棣栧厛鏄鎴风瑕佺紪鍐欏ソmapreduce绋嬪簭锛岄厤缃ソmapreduce鐨勪綔涓氫篃灏辨槸job锛屾帴涓嬫潵灏辨槸鎻愪氦job浜嗭紝鎻愪氦job鏄彁浜ゅ埌JobTracker涓婄殑锛岃繖涓椂鍊橨obTracker灏变細鏋勫缓杩欎釜job锛屽叿浣撳氨鏄垎閰嶄竴涓柊鐨刯ob浠诲姟鐨処D鍊硷紝鎺ヤ笅鏉ュ畠浼氬仛妫€鏌ユ搷浣滐紝杩欎釜妫€鏌ュ氨鏄‘瀹氳緭鍑虹洰褰曟槸鍚﹀瓨鍦紝濡傛灉瀛樺湪閭ｄ箞job灏变笉鑳芥甯歌繍琛屼笅鍘伙紝JobTracker浼氭姏鍑洪敊璇粰瀹㈡埛绔紝鎺ヤ笅鏉ヨ繕瑕佹鏌ヨ緭鍏ョ洰褰曟槸鍚﹀瓨鍦紝濡傛灉涓嶅瓨鍦ㄥ悓鏍锋姏鍑洪敊璇紝濡傛灉瀛樺湪JobTracker浼氭牴鎹緭鍏ヨ绠楄緭鍏ュ垎鐗囷紙Input Split锛夛紝濡傛灉鍒嗙墖璁＄畻涓嶅嚭鏉ヤ篃浼氭姏鍑洪敊璇紝鑷充簬杈撳叆鍒嗙墖鎴戝悗闈細鍋氳瑙ｇ殑锛岃繖浜涢兘鍋氬ソ浜咼obTracker灏变細閰嶇疆Job闇€瑕佺殑璧勬簮浜嗐€傚垎閰嶅ソ璧勬簮鍚庯紝JobTracker灏变細鍒濆鍖栦綔涓氾紝鍒濆鍖栦富瑕佸仛鐨勬槸灏咼ob鏀惧叆涓€涓唴閮ㄧ殑闃熷垪锛岃閰嶇疆濂界殑浣滀笟璋冨害鍣ㄨ兘璋冨害鍒拌繖涓綔涓氾紝浣滀笟璋冨害鍣ㄤ細鍒濆鍖栬繖涓猨ob锛屽垵濮嬪寲灏辨槸鍒涘缓涓€涓鍦ㄨ繍琛岀殑job瀵硅薄锛堝皝瑁呬换鍔″拰璁板綍淇℃伅锛夛紝浠ヤ究JobTracker璺熻釜job鐨勭姸鎬佸拰杩涚▼銆傚垵濮嬪寲瀹屾瘯鍚庯紝浣滀笟璋冨害鍣ㄤ細鑾峰彇杈撳叆鍒嗙墖淇℃伅锛坕nput split锛夛紝姣忎釜鍒嗙墖鍒涘缓涓€涓猰ap浠诲姟銆傛帴涓嬫潵灏辨槸浠诲姟鍒嗛厤浜嗭紝杩欎釜鏃跺€檛asktracker浼氳繍琛屼竴涓畝鍗曠殑寰幆鏈哄埗瀹氭湡鍙戦€佸績璺崇粰jobtracker锛屽績璺抽棿闅旀槸5绉掞紝绋嬪簭鍛樺彲浠ラ厤缃繖涓椂闂达紝蹇冭烦灏辨槸jobtracker鍜宼asktracker娌熼€氱殑妗ユ锛岄€氳繃蹇冭烦锛宩obtracker鍙互鐩戞帶tasktracker鏄惁瀛樻椿锛屼篃鍙互鑾峰彇tasktracker澶勭悊鐨勭姸鎬佸拰闂锛屽悓鏃秚asktracker涔熷彲浠ラ€氳繃蹇冭烦閲岀殑杩斿洖鍊艰幏鍙杍obtracker缁欏畠鐨勬搷浣滄寚浠ゃ€備换鍔″垎閰嶅ソ鍚庡氨鏄墽琛屼换鍔′簡銆傚湪浠诲姟鎵ц鏃跺€檍obtracker鍙互閫氳繃蹇冭烦鏈哄埗鐩戞帶tasktracker鐨勭姸鎬佸拰杩涘害锛屽悓鏃朵篃鑳借绠楀嚭鏁翠釜job鐨勭姸鎬佸拰杩涘害锛岃€宼asktracker涔熷彲浠ユ湰鍦扮洃鎺ц嚜宸辩殑鐘舵€佸拰杩涘害銆傚綋jobtracker鑾峰緱浜嗘渶鍚庝竴涓畬鎴愭寚瀹氫换鍔＄殑tasktracker鎿嶄綔鎴愬姛鐨勯€氱煡鏃跺€欙紝jobtracker浼氭妸鏁翠釜job鐘舵€佺疆涓烘垚鍔燂紝鐒跺悗褰撳鎴风鏌ヨjob杩愯鐘舵€佹椂鍊欙紙娉ㄦ剰锛氳繖涓槸寮傛鎿嶄綔锛夛紝瀹㈡埛绔細鏌ュ埌job瀹屾垚鐨勯€氱煡鐨勩€傚鏋渏ob涓€斿け璐ワ紝mapreduce涔熶細鏈夌浉搴旀満鍒跺鐞嗭紝涓€鑸€岃█濡傛灉涓嶆槸绋嬪簭鍛樼▼搴忔湰韬湁bug锛宮apreduce閿欒澶勭悊鏈哄埗閮借兘淇濊瘉鎻愪氦鐨刯ob鑳芥甯稿畬鎴愩€?/p>
聽
銆€銆€涓嬮潰鎴戜粠閫昏緫瀹炰綋鐨勮搴﹁瑙apreduce杩愯鏈哄埗锛岃繖浜涙寜鐓ф椂闂撮『搴忓寘鎷細杈撳叆鍒嗙墖锛?/em>input split锛夈€乵ap闃舵銆乧ombiner闃舵銆乻huffle闃舵鍜宺educe闃舵銆?/p>
- 杈撳叆鍒嗙墖锛坕nput split锛夛細鍦ㄨ繘琛宮ap璁＄畻涔嬪墠锛宮apreduce浼氭牴鎹緭鍏ユ枃浠惰绠楄緭鍏ュ垎鐗囷紙input split锛夛紝姣忎釜杈撳叆鍒嗙墖锛坕nput split锛夐拡瀵逛竴涓猰ap浠诲姟锛岃緭鍏ュ垎鐗囷紙input split锛夊瓨鍌ㄧ殑骞堕潪鏁版嵁鏈韩锛岃€屾槸涓€涓垎鐗囬暱搴﹀拰涓€涓褰曟暟鎹殑浣嶇疆鐨勬暟缁勶紝杈撳叆鍒嗙墖锛坕nput split锛夊線寰€鍜宧dfs鐨刡lock锛堝潡锛夊叧绯诲緢瀵嗗垏锛屽亣濡傛垜浠瀹歨dfs鐨勫潡鐨勫ぇ灏忔槸64mb锛屽鏋滄垜浠緭鍏ユ湁涓変釜鏂囦欢锛屽ぇ灏忓垎鍒槸3mb銆?5mb鍜?27mb锛岄偅涔坢apreduce浼氭妸3mb鏂囦欢鍒嗕负涓€涓緭鍏ュ垎鐗囷紙input split锛夛紝65mb鍒欐槸涓や釜杈撳叆鍒嗙墖锛坕nput split锛夎€?27mb涔熸槸涓や釜杈撳叆鍒嗙墖锛坕nput split锛夛紝鎹㈠彞璇濊鎴戜滑濡傛灉鍦╩ap璁＄畻鍓嶅仛杈撳叆鍒嗙墖璋冩暣锛屼緥濡傚悎骞跺皬鏂囦欢锛岄偅涔堝氨浼氭湁5涓猰ap浠诲姟灏嗘墽琛岋紝鑰屼笖姣忎釜map鎵ц鐨勬暟鎹ぇ灏忎笉鍧囷紝杩欎釜涔熸槸mapreduce浼樺寲璁＄畻鐨勪竴涓叧閿偣銆?/li>
-
map闃舵锛?/strong>灏辨槸绋嬪簭鍛樼紪鍐欏ソ鐨刴ap鍑芥暟浜嗭紝鍥犳map鍑芥暟鏁堢巼鐩稿濂芥帶鍒讹紝鑰屼笖涓€鑸琺ap鎿嶄綔閮芥槸鏈湴鍖栨搷浣滀篃灏辨槸鍦ㄦ暟鎹瓨鍌ㄨ妭鐐逛笂杩涜锛?/li>
- combiner闃舵锛歝ombiner闃舵鏄▼搴忓憳鍙互閫夋嫨鐨勶紝combiner鍏跺疄涔熸槸涓€绉峳educe鎿嶄綔锛屽洜姝ゆ垜浠湅瑙乄ordCount绫婚噷鏄敤reduce杩涜鍔犺浇鐨勩€侰ombiner鏄竴涓湰鍦板寲鐨剅educe鎿嶄綔锛屽畠鏄痬ap杩愮畻鐨勫悗缁搷浣滐紝涓昏鏄湪map璁＄畻鍑轰腑闂存枃浠跺墠鍋氫竴涓畝鍗曠殑鍚堝苟閲嶅key鍊肩殑鎿嶄綔锛屼緥濡傛垜浠鏂囦欢閲岀殑鍗曡瘝棰戠巼鍋氱粺璁★紝map璁＄畻鏃跺€欏鏋滅鍒颁竴涓猦adoop鐨勫崟璇嶅氨浼氳褰曚负1锛屼絾鏄繖绡囨枃绔犻噷hadoop鍙兘浼氬嚭鐜皀澶氭锛岄偅涔坢ap杈撳嚭鏂囦欢鍐椾綑灏变細寰堝锛屽洜姝ゅ湪reduce璁＄畻鍓嶅鐩稿悓鐨刱ey鍋氫竴涓悎骞舵搷浣滐紝閭ｄ箞鏂囦欢浼氬彉灏忥紝杩欐牱灏辨彁楂樹簡瀹藉甫鐨勪紶杈撴晥鐜囷紝姣曠珶hadoop璁＄畻鍔涘甯﹁祫婧愬線寰€鏄绠楃殑鐡堕涔熸槸鏈€涓哄疂璐电殑璧勬簮锛屼絾鏄痗ombiner鎿嶄綔鏄湁椋庨櫓鐨勶紝浣跨敤瀹冪殑鍘熷垯鏄痗ombiner鐨勮緭鍏ヤ笉浼氬奖鍝嶅埌reduce璁＄畻鐨勬渶缁堣緭鍏ワ紝渚嬪锛氬鏋滆绠楀彧鏄眰鎬绘暟锛屾渶澶у€硷紝鏈€灏忓€煎彲浠ヤ娇鐢╟ombiner锛屼絾鏄仛骞冲潎鍊艰绠椾娇鐢╟ombiner鐨勮瘽锛屾渶缁堢殑reduce璁＄畻缁撴灉灏变細鍑洪敊銆?/li>
- shuffle闃舵锛?/strong>灏唌ap鐨勮緭鍑轰綔涓簉educe鐨勮緭鍏ョ殑杩囩▼灏辨槸shuffle浜嗭紝杩欎釜鏄痬apreduce浼樺寲鐨勯噸鐐瑰湴鏂广€傝繖閲屾垜涓嶈鎬庝箞浼樺寲shuffle闃舵锛岃璁瞫huffle闃舵鐨勫師鐞嗭紝鍥犱负澶ч儴鍒嗙殑涔︾睄閲岄兘娌¤娓呮shuffle闃舵銆係huffle涓€寮€濮嬪氨鏄痬ap闃舵鍋氳緭鍑烘搷浣滐紝涓€鑸琺apreduce璁＄畻鐨勯兘鏄捣閲忔暟鎹紝map杈撳嚭鏃跺€欎笉鍙兘鎶婃墍鏈夋枃浠堕兘鏀惧埌鍐呭瓨鎿嶄綔锛屽洜姝ap鍐欏叆纾佺洏鐨勮繃绋嬪崄鍒嗙殑澶嶆潅锛屾洿浣曞喌map杈撳嚭鏃跺€欒瀵圭粨鏋滆繘琛屾帓搴忥紝鍐呭瓨寮€閿€鏄緢澶х殑锛宮ap鍦ㄥ仛杈撳嚭鏃跺€欎細鍦ㄥ唴瀛橀噷寮€鍚竴涓幆褰㈠唴瀛樼紦鍐插尯锛岃繖涓紦鍐插尯涓撻棬鐢ㄦ潵杈撳嚭鐨勶紝榛樿澶у皬鏄?00mb锛屽苟涓斿湪閰嶇疆鏂囦欢閲屼负杩欎釜缂撳啿鍖鸿瀹氫簡涓€涓榾鍊硷紝榛樿鏄?.80锛堣繖涓ぇ灏忓拰闃€鍊奸兘鏄彲浠ュ湪閰嶇疆鏂囦欢閲岃繘琛岄厤缃殑锛夛紝鍚屾椂map杩樹細涓鸿緭鍑烘搷浣滃惎鍔ㄤ竴涓畧鎶ょ嚎绋嬶紝濡傛灉缂撳啿鍖虹殑鍐呭瓨杈惧埌浜嗛榾鍊肩殑80%鏃跺€欙紝杩欎釜瀹堟姢绾跨▼灏变細鎶婂唴瀹瑰啓鍒扮鐩樹笂锛岃繖涓繃绋嬪彨spill锛屽彟澶栫殑20%鍐呭瓨鍙互缁х画鍐欏叆瑕佸啓杩涚鐩樼殑鏁版嵁锛屽啓鍏ョ鐩樺拰鍐欏叆鍐呭瓨鎿嶄綔鏄簰涓嶅共鎵扮殑锛屽鏋滅紦瀛樺尯琚拺婊′簡锛岄偅涔坢ap灏变細闃诲鍐欏叆鍐呭瓨鐨勬搷浣滐紝璁╁啓鍏ョ鐩樻搷浣滃畬鎴愬悗鍐嶇户缁墽琛屽啓鍏ュ唴瀛樻搷浣滐紝鍓嶉潰鎴戣鍒板啓鍏ョ鐩樺墠浼氭湁涓帓搴忔搷浣滐紝杩欎釜鏄湪鍐欏叆纾佺洏鎿嶄綔鏃跺€欒繘琛岋紝涓嶆槸鍦ㄥ啓鍏ュ唴瀛樻椂鍊欒繘琛岀殑锛屽鏋滄垜浠畾涔変簡combiner鍑芥暟锛岄偅涔堟帓搴忓墠杩樹細鎵цcombiner鎿嶄綔銆傛瘡娆pill鎿嶄綔涔熷氨鏄啓鍏ョ鐩樻搷浣滄椂鍊欏氨浼氬啓涓€涓孩鍑烘枃浠讹紝涔熷氨鏄鍦ㄥ仛map杈撳嚭鏈夊嚑娆pill灏变細浜х敓澶氬皯涓孩鍑烘枃浠讹紝绛塵ap杈撳嚭鍏ㄩ儴鍋氬畬鍚庯紝map浼氬悎骞惰繖浜涜緭鍑烘枃浠躲€傝繖涓繃绋嬮噷杩樹細鏈変竴涓狿artitioner鎿嶄綔锛屽浜庤繖涓搷浣滃緢澶氫汉閮藉緢杩风硦锛屽叾瀹濸artitioner鎿嶄綔鍜宮ap闃舵鐨勮緭鍏ュ垎鐗囷紙Input split锛夊緢鍍忥紝涓€涓狿artitioner瀵瑰簲涓€涓猺educe浣滀笟锛屽鏋滄垜浠琺apreduce鎿嶄綔鍙湁涓€涓猺educe鎿嶄綔锛岄偅涔圥artitioner灏卞彧鏈変竴涓紝濡傛灉鎴戜滑鏈夊涓猺educe鎿嶄綔锛岄偅涔圥artitioner瀵瑰簲鐨勫氨浼氭湁澶氫釜锛孭artitioner鍥犳灏辨槸reduce鐨勮緭鍏ュ垎鐗囷紝杩欎釜绋嬪簭鍛樺彲浠ョ紪绋嬫帶鍒讹紝涓昏鏄牴鎹疄闄卥ey鍜寁alue鐨勫€硷紝鏍规嵁瀹為檯涓氬姟绫诲瀷鎴栬€呬负浜嗘洿濂界殑reduce璐熻浇鍧囪 瑕佹眰杩涜锛岃繖鏄彁楂榬educe鏁堢巼鐨勪竴涓叧閿墍鍦ㄣ€傚埌浜唕educe闃舵灏辨槸鍚堝苟map杈撳嚭鏂囦欢浜嗭紝Partitioner浼氭壘鍒板搴旂殑map杈撳嚭鏂囦欢锛岀劧鍚庤繘琛屽鍒舵搷浣滐紝澶嶅埗鎿嶄綔鏃秗educe浼氬紑鍚嚑涓鍒剁嚎绋嬶紝杩欎簺绾跨▼榛樿涓暟鏄?涓紝绋嬪簭鍛樹篃鍙互鍦ㄩ厤缃枃浠舵洿鏀瑰鍒剁嚎绋嬬殑涓暟锛岃繖涓鍒惰繃绋嬪拰map鍐欏叆纾佺洏杩囩▼绫讳技锛屼篃鏈夐榾鍊煎拰鍐呭瓨澶у皬锛岄榾鍊间竴鏍峰彲浠ュ湪閰嶇疆鏂囦欢閲岄厤缃紝鑰屽唴瀛樺ぇ灏忔槸鐩存帴浣跨敤reduce鐨則asktracker鐨勫唴瀛樺ぇ灏忥紝澶嶅埗鏃跺€檙educe杩樹細杩涜鎺掑簭鎿嶄綔鍜屽悎骞舵枃浠舵搷浣滐紝杩欎簺鎿嶄綔瀹屼簡灏变細杩涜reduce璁＄畻浜嗐€?/li>
- reduce闃舵锛氬拰map鍑芥暟涓€鏍蜂篃鏄▼搴忓憳缂栧啓鐨勶紝鏈€缁堢粨鏋滄槸瀛樺偍鍦╤dfs涓婄殑銆?/li>
銆€銆€Mapreduce鐨勭浉鍏抽棶棰?/strong>
銆€銆€杩欓噷鎴戣璋堣皥鎴戝涔爉apreduce鎬濊€冪殑涓€浜涢棶棰橈紝閮芥槸鎴戣嚜宸辨兂鍑鸿В閲婄殑闂锛屼絾鏄煇浜涢棶棰樺埌搴曞涓嶅锛屽氨瑕佸箍澶х闉嬪府鎴戠‘璁や簡銆?/p>
- jobtracker鐨勫崟鐐规晠闅滐細jobtracker鍜宧dfs鐨刵amenode涓€鏍蜂篃瀛樺湪鍗曠偣鏁呴殰锛屽崟鐐规晠闅滀竴鐩存槸hadoop琚汉璇熺梾鐨勫ぇ闂锛屼负浠€涔坔adoop鐨勫仛鐨勬枃浠剁郴缁熷拰mapreduce璁＄畻妗嗘灦閮芥槸楂樺閿欑殑锛屼絾鏄渶閲嶈鐨勭鐞嗚妭鐐圭殑鏁呴殰鏈哄埗鍗村姝や笉濂斤紝鎴戣涓轰富瑕佹槸namenode鍜宩obtracker鍦ㄥ疄闄呰繍琛屼腑閮芥槸鍦ㄥ唴瀛樻搷浣滐紝鑰屽仛鍒板唴瀛樼殑瀹归敊灏辨瘮杈冨鏉備簡锛屽彧鏈夊綋鍐呭瓨鏁版嵁琚寔涔呭寲鍚庡閿欐墠濂藉仛锛宯amenode鍜宩obtracker閮藉彲浠ュ浠借嚜宸辨寔涔呭寲鐨勬枃浠讹紝浣嗘槸杩欎釜鎸佷箙鍖栭兘浼氭湁寤惰繜锛屽洜姝ょ湡鐨勫嚭鏁呴殰锛屼换鐒朵笉鑳芥暣浣撴仮澶嶏紝鍙﹀hadoop妗嗘灦閲屽寘鍚珃ookeeper妗嗘灦锛寊ookeeper鍙互缁撳悎jobtracker锛岀敤鍑犲彴鏈哄櫒鍚屾椂閮ㄧ讲jobtracker锛屼繚璇佷竴鍙板嚭鏁呴殰锛屾湁涓€鍙伴┈涓婅兘琛ュ厖涓婏紝涓嶈繃杩欑鏂瑰紡涔熸病娉曟仮澶嶆鍦ㄨ窇鐨刴apreduce浠诲姟銆?/li>
- 鍋歮apreduce璁＄畻鏃跺€欙紝杈撳嚭涓€鑸槸涓€涓枃浠跺す锛岃€屼笖璇ユ枃浠跺す鏄笉鑳藉瓨鍦紝鎴戝湪鍑洪潰璇曢鏃跺€欐彁鍒颁簡杩欎釜闂锛岃€屼笖杩欎釜妫€鏌ュ仛鐨勫緢鏃╋紝褰撴垜浠彁浜ob鏃跺€欏氨浼氳繘琛岋紝mapreduce涔嬫墍浠ヨ繖涔堣璁℃槸淇濊瘉鏁版嵁鍙潬鎬э紝濡傛灉杈撳嚭鐩綍瀛樺湪reduce灏辨悶涓嶆竻妤氫綘鍒板簳鏄杩藉姞杩樻槸瑕嗙洊锛屼笉绠℃槸杩藉姞鍜岃鐩栨搷浣滈兘浼氭湁鍙兘瀵艰嚧鏈€缁堢粨鏋滃嚭闂锛宮apreduce鏄仛娴烽噺鏁版嵁璁＄畻锛屼竴涓敓浜ц绠楃殑鎴愭湰寰堥珮锛屼緥濡備竴涓猨ob瀹屽叏鎵ц瀹屽彲鑳借鍑犱釜灏忔椂锛屽洜姝や竴鍒囧奖鍝嶉敊璇殑鎯呭喌mapreduce鏄浂瀹瑰繊鐨勩€?/li>
- Mapreduce杩樻湁涓€涓狪nputFormat鍜孫utputFormat锛屾垜浠湪缂栧啓map鍑芥暟鏃跺€欏彂鐜癿ap鏂规硶鐨勫弬鏁版槸涔嬮棿鎿嶄綔琛屾暟鎹紝娌℃湁鐗垫秹鍒癐nputFormat锛岃繖浜涗簨鎯呭湪鎴戜滑new Path鏃跺€檓apreduce璁＄畻妗嗘灦甯垜浠仛濂戒簡锛岃€孫utputFormat涔熸槸reduce甯垜浠仛濂戒簡锛屾垜浠娇鐢ㄤ粈涔堟牱鐨勮緭鍏ユ枃浠讹紝灏辫璋冪敤浠€涔堟牱鐨処nputFormat锛孖nputFormat鏄拰鎴戜滑杈撳叆鐨勬枃浠剁被鍨嬬浉鍏崇殑锛宮apreduce閲屽父鐢ㄧ殑InputFormat鏈塅ileInputFormat鏅€氭枃鏈枃浠讹紝SequenceFileInputFormat鏄寚hadoop鐨勫簭鍒楀寲鏂囦欢锛屽彟澶栬繕鏈塊eyValueTextInputFormat銆侽utputFormat灏辨槸鎴戜滑鎯虫渶缁堝瓨鍌ㄥ埌hdfs绯荤粺涓婄殑鏂囦欢鏍煎紡浜嗭紝杩欎釜鏍规嵁浣犻渶瑕佸畾涔変簡锛宧adoop鏈夋敮鎸佸緢澶氭枃浠舵牸寮忥紝杩欓噷涓嶄竴涓€鍒椾妇锛屾兂鐭ラ亾鐧惧害涓嬪氨鐪嬪埌浜嗐€?/li>