MaxCompute flighting 鈥斺€?Task鐏板益鍙戝竷
MaxCompute flighting 鈥斺€?Task鐏板害鍙戝竷
鍓嶈█
鎸夌収鐧剧璇嶆潯鐨勮В閲婏細鐏板害鍙戝竷鏄寚鍦ㄩ粦涓庣櫧涔嬮棿锛岃兘澶熷钩婊戣繃娓＄殑涓€绉嶅彂甯冩柟寮忋€傜伆搴﹀彂甯冨彲浠ヤ繚璇佹暣浣撶郴缁熺殑绋冲畾锛屽湪鍒濆鐏板害鐨勬椂鍊欏氨鍙互鍙戠幇銆佽皟鏁撮棶棰橈紝浠ヤ繚璇佸叾褰卞搷搴︺€傞檷浣庡彂甯冧笂绾跨殑椋庨櫓銆?
鐏板害鍙戝竷鐨勫叧閿槸閫夋嫨鍚堥€傜殑鐏板害绛栫暐锛屾妸绗﹀悎绛栫暐鐨勬祦閲忓紩鍏ュ埌鏂扮増鏈笂銆傜伆搴︾瓥鐣ョ殑閫夋嫨闇€瑕佽€冭檻寰呭彂甯冨璞″湪閾捐矾鎴栫郴缁熶腑鐨勪綅缃鑹诧紙濡傚鎴风銆佺Щ鍔ˋPP銆佹湇鍔″悗鍙扮瓑锛夈€佺敤鎴峰睘鎬у拰鍙戝竷闇€姹傜洰鏍囩瓑銆傛湰鏂囧皢瀵筂axCompute鐨凾ask鐏板害鍙戝竷绯荤粺杩涜浠嬬粛銆?
MaxCompute Task鐏板害鍙戝竷
MaxCompute flighting锛圫print21锛夊疄鐜颁簡MaxCompute妗嗘灦鍜孴ask鐨勮В鑰︼紝MaxCompute妗嗘灦鍜孴ask鍙互鏍规嵁鍚勮嚜鐨勭爺鍙戣鍒掕繘琛岀嫭绔嬬殑鍙戝竷銆傚悓鏃跺Task寮曞叆浜嗙増鏈寲绠＄悊锛堝ぇ鐗堟湰/灏忕増鏈級锛孧axCompute妗嗘灦鏀寔鍚岀Task鐨勫涓ぇ鐗堟湰骞跺瓨涓旂嫭绔嬫彁渚涙湇鍔°€傝繖浜涚壒鎬т负Task鐏板害鍙戝竷鎻愪緵浜嗗熀纭€鏀寔銆?
MaxComputer_framework
鍥?MaxCompute flighting 鏋舵瀯鍥?
MaxCompute鐩墠鏀寔SQL_RELATIVE_TASKS, PS_TASK绛塗ask Package绫诲瀷, Task鐨勭伆搴﹀彂甯冧究鏄杩欎簺Task Package鍙戝竷锛屽笇鏈涳細
1锛?鍚凾ask Package鐙珛杩涜鐏帮紝浜掍笉骞叉壈锛?
2锛?鍙戝竷杩囩▼瀵圭敤鎴烽€忔槑锛?
3锛?鐏板害鑼冨洿銆佸奖鍝嶉潰鍜岀粨鏋滃彲鎺э紱
4锛?鍏锋湁蹇€熷彲闈犵殑鐏板害缁撴灉璇勪及鏈哄埗锛?
5锛?鎿嶄綔绠€鍗曪紝鍙嚜鍔ㄥ寲/鍗婅嚜鍔ㄥ寲杩愮淮锛?
6锛?鐗堟湰绠＄悊绠€鍗曪紝鏄撲簬杩涜鐗堟湰缁存姢锛?
7锛?鐗规畩鍦哄悎锛屽厑璁稿紑鍙戠嚎涓婅繘琛屽熀浜庣伆搴︽祴璇曪紱
MaxCompute Task鐏板害鍙戝竷鍘熺悊
鍦ㄤ粙缁峂axCompute鐨凾ask鐏板害鍘熺悊鍓嶏紝鍏堜簡瑙ｄ笅鍏朵笟鍔℃祦绋嬨€傜敤鎴锋彁浜や綔涓氬悗锛岃姹傞鍏堝埌worker锛寃orker涓烘瘡涓猨ob鍒涘缓instanceId锛岀劧鍚嶳PC鍒癊xecutor杩涜TryRun锛孴ryRun鏃剁敱鏀寔璇askType鐨凱ackage澶勭悊锛屽鏋滄槸鍚屾Task锛屽垯鐩存帴澶勭悊骞跺皢缁撴灉杩斿洖锛屽鏋滀负寮傛Task鍒欒繑鍥炵粰Worker锛學orker鍙戦€佺粰Scheduler锛屽湪Executor杞璇锋眰Task鏃跺彂閫佺粰鍏跺苟缁х画澶勭悊銆傚涓嬪浘鎵€绀猴細
screenshot
鍥?MaxCompute flighting 璇锋眰涓氬姟閫昏緫
娉細
鍚屾Task锛?1 -> 2 -> 3
寮傛Task锛?1 -> 2 -> 3 -> 4 -> 5 -> 6
涓轰簡淇濊瘉Task鎵ц涓€鑷存€э紝Worker鍚慐xecutor鍙戦€佽姹傚墠闇€瑕侀€夊畾package鐨勭増鏈紝骞朵笖鍚庣画璇锋眰锛堝紓姝ワ級涔熼渶瑕佽鐗堟湰package鎻愪緵锛岀粦瀹氳繃绋嬪湪绗簩姝ュ畬鎴愩€傝€冭檻浠ヤ笂浜嬪疄锛屽彲閫氳繃鍦╓orker璇诲彇鐏板害娴侀噺閰嶇疆锛屾牴鎹敤鎴峰睘鎬х壒寰佸拰娴侀噺閰嶇疆鍐冲畾鎵цTask鐨刾ackage鐗堟湰閫夋嫨缁撴灉銆傜伆搴﹀師鐞嗗浘濡備笅锛?
screenshot
鍥?Task鐏板害鍙戝竷鍘熺悊
鐏板害鍙戝竷鍓嶏紝PE鎴栬€呭紑鍙戜汉鍛樺垵濮嬪寲鐏板害閰嶇疆锛岀伆搴﹂厤缃俊鎭寘鍚杩涜鐏板害鍙戝竷鐨勭増鏈€佸弬涓庣伆搴︾殑鐢ㄦ埛鐗瑰緛浠ュ強鐏板害鏃剁殑娴侀噺鍒嗛厤锛岃瑙乀ask鐏板害鍏冩暟鎹€傚噯澶囧伐浣滃畬鎴愬悗鍙繘琛岀伆搴﹀彂甯冦€傚湪鐏板害鍙戝竷杩囩▼涓紝Worker瀹氭椂涓嶰TS鍚屾鐏板害閰嶇疆骞舵牴鎹綋鍓嶇伆搴﹂厤缃紝閫夊彇婊¤冻鐗瑰緛瑕佹眰鐨勭敤鎴疯姹傚弬涓庣伆搴︼紱鎸夌収鐏板害娴侀噺閰嶇疆寮曞娴侀噺鍒扮伆搴﹀彂甯冪殑Package鐗堟湰涓婏紝鍦ㄧ伆搴﹀彂甯冭繃绋嬩腑锛孭E鎴栬€呭紑鍙戜汉鍛樻牴鎹伆搴﹁繘灞曠姸鍐靛喅瀹氱户缁伆搴﹁繕鏄仠姝㈢伆搴︼紝鍐冲畾灏嗗弽搴斿湪鐏板害閰嶇疆鐨勪慨鏀广€?
Task鐏板害鍏冩暟鎹?
鍩轰簬濡備笂鐨勮€冭檻锛孧axCompute璁捐浜嗗悇Task鐙珛鐨勭伆搴﹀彂甯冩祦绋嬶紝浣撶幇鍦ㄧ嫭绔嬬殑鐏板害鍏冩暟鎹璁′笂锛屽涓嬶細
{
"grayInfo": {
"MaxComputeServiceTest1": {
"reserved": {},
"taskVersion": {
"defaultVersion": "default",
"flightingVersion": "flighting"},
"tierGrayInfo": {
"DEFAULTTIER": {
"grayInfo": {
"defaultVersion": 100,
"flightingVersion": 0},
"reserved": {}
}
},
"MaxComputeServiceTest2": {
"reserved": {},
"taskVersion": {
"defaultVersion": "default",
"flightingVersion": "flighting"},
"tierGrayInfo": {
"DEFAULTTIER": {
"grayInfo": {
"defaultVersion": 10,
"flightingVersion": 90},
"reserved": {}
}
}
}
},
"projectTier": {
"DEFAULTTIER": [
"admin_task_project",
"algo_public",
"algo_test"]
},
"reserved": {},
"taskType": "SQL_RELATIVE_TASKS"
}
姣忕Task閮芥湁鐙珛鐨勭敤鎴风壒寰佸垎缁勪俊鎭紙Tier锛夛紱鑰冭檻MaxCompute鐨勫鎺у満鏅拰鍚勬帶鍒堕泦缇ょ嫭绔嬬殑鍖呯鐞嗗櫒鏂瑰紡锛岀伆搴﹂厤缃俊鎭笂澶氬姞浜嗕竴灞傛帶鍒堕泦缇ょ淮搴︼紝姣忎釜鎺у埗闆嗙兢瀵筎ask鐏板害鍙戝竷鍏变韩Tier鍒嗙粍淇℃伅锛岀嫭浜伆搴﹂厤缃俊鎭紱
Tier閰嶇疆
Tier鏄疢axCompute鐏板害閰嶇疆鐨勬渶灏忕矑搴︼紝鍙互瀵规瘡涓猅ier璁剧疆涓嶅悓鐨勬祦閲忓垎娴佷俊鎭痝rayInfo鍜屾帶鍒堕泦缇ょ淮搴︿笅鐨勫叡浜玀apping淇℃伅銆俫rayInfo淇℃伅閰嶇疆浜嗘祦閲忓湪涓嶅悓鐗堟湰涓婄殑鍒嗘祦鎯呭喌锛屾墍鏈夐毝灞炰簬璇ier鐨刾roject鐨勮绉峧ob閮芥寜鐓ц繖涓厤缃繘琛屾祦閲忓垎娴侊紱Mapping淇℃伅璁剧疆浜嗙増鏈琓ag鍒板疄闄呯殑Task Package version鐨勬槧灏勶紝鐗瑰埆鐨勶紝姣忎釜Tier涔熷彲浠ヨ缃嫭绔嬬殑Mapping淇℃伅锛孡oad鐏板害閰嶇疆鏃讹紝浼樺厛浣跨敤Tier鐨凪apping淇℃伅銆俆ier鐨勭鏈塎apping璁剧疆鏄€氳繃淇濈暀瀛楁杩涜鎵╁睍鐨勶紝濡備笅锛?
"tierGrayInfo": {
"DEFAULTTIER": {
"grayInfo": {
"defaultVersion": 80,
"flightingVersion": 20},
"reserved": {
"defaultVersion": "tom",
"flightingVersion": "jerry"}
}
}
鍗囩骇/鍥炴粴
閴翠簬绾夸笂MaxCompute澶氭帶闆嗙兢鐜扮姸锛屼笖姣忎釜鎺у埗闆嗙兢閮芥湁鐙珛鐨勫寘绠＄悊鍣紝鍦ㄥ崌绾?鍥炴粴鏃堕渶瑕佽€冭檻MaxCompute妗嗘灦鍜孴ask package鐨勫吋瀹规€х瓑闂锛岄渶瑕佸鐏板害鍏冧俊鎭繘琛屼竴瀹氱殑澶勭悊锛?
鍏冩暟鎹?鍗囩骇&鍥炴粴
Tier鍒嗙粍淇℃伅 淇濈暀
Tier娴侀噺鍒嗘祦 閲嶇疆鐩爣闆嗙兢
鐗堟湰绠＄悊
MaxCompute flighting瑙ｈ€︿簡MaxCompute妗嗘灦鍜孴ask锛岀悊璁轰笂鍚勫姛鑳藉潡鍙嫭绔嬬鐞嗗彂甯冿紝杩欑粰MaxCompute鐨勫彂甯冪鐞嗗甫鏉ヤ簡鎸戞垬锛?鑰冭檻鍏煎鎬э紝MaxCompute瀵圭増鏈鐞嗗仛浜嗗涓嬬害瀹氾細
1锛?MaxCompute妗嗘灦闅廠print澶х増鏈繘琛屽彂甯冿紝骞朵笖鍦⊿print鍙戝竷鏃跺己琛岃繘琛屼竴娆′笌鍚凾ask鐨勭増鏈榻愶紝鎼哄甫鍚凾ask Package鐨刣efault鐗堟湰锛?
2锛?鍚庡彂甯冪殑瀵瑰吋瀹规€ц礋璐ｏ紱
3锛?姣忕Task鏈変袱涓寮忓ぇ鐗堟湰default鍜宖lighting锛宒efault闅忔瘡涓猻print鍙戝竷锛宖lighting鍦ㄩ渶瑕佸Task杩涜鐏板害鍙戝竷鏃跺彂甯冿紱
4锛?鍥哄畾浣跨敤default璁板綍Task鍙戝竷鍘嗗彶锛岀伆搴﹀彂甯冩垚鍔熺殑鐗堟湰闇€瑕侀噸鏂版彁浜ゆ垚default鐨勪竴涓柊鐨勫皬鐗堟湰锛?
閰嶅璁炬柦
MaxCompute Task鐏板害鍙戝竷娑夊強鍒板涓伐鍏凤紝鍖呮嫭Task Package鐨勯儴缃插伐鍏凤紝鐏板害閰嶇疆宸ュ叿锛圓dmin Task锛変互鍙婄伆搴﹁繃绋嬩腑琛￠噺鐏板害缁撴灉鐨勭粺璁″伐鍏峰拰鐩戞帶鎶ヨ宸ュ叿銆?
鍖呴儴缃插伐鍏凤細
OPDS flighting寮€鍙戜簡鐙珛鐨凾ask Package閮ㄧ讲宸ュ叿package_util锛屽彲鐢ㄤ簬绠＄悊Task鐨勬帶鍒堕泦缇ゅ寘鍜岃绠楅泦缇ゅ寘锛涢€氳繃瀹冨彲瀵筎ask Package杩涜deploy & rollback绛夋搷浣滐紱
鐏板害閰嶇疆宸ュ叿锛?
OPDS鐨凾ask鐏板害閰嶇疆鍏冧俊鎭瓨鍌ㄥ湪OTS涓紝闇€瑕佷娇鐢ˋdmin Task鍛戒护瀹炵幇瀵圭伆搴﹂厤缃慨鏀圭殑鐩殑锛岀洰鍓嶆敮鎸佺殑鍛戒护濡備笅锛?
鍒濆鍖栨煇绫诲瀷鐨凾ask鐏板害閰嶇疆锛?
Tier鐨凜RUD锛?
淇敼闆嗙兢鏍囩涓嬬殑鍏变韩Mapping锛?
淇敼Tier鐨勬祦閲忛厤缃紱
淇敼Tier鐗瑰畾鐨凪apping锛?
鍗囩骇/鍥炴粴鏃舵槸鍚︽竻妤氱伆搴﹁繘搴︿俊鎭紱
琛￠噺鐏板害鐨勬寚鏍囧拰鐩戞帶鎶ヨ锛?
MaxCompute Task鐏板害鍙戝竷鐨勭粨鏋滆 閲忔湁涓や釜灞傞潰锛屼竴涓槸妗嗘灦灞傞潰鏅亶涓€鑸殑缁熻锛屽task鐨勬垚鍔熺巼/澶辫触鐜囷紝閫氳繃涓巇efault鐗堟湰鐨勫姣旓紝鏉ヨ繘琛岃瘎浼帮紱鍙︿竴涓槸Task灞傞潰鍋氱殑涓撲笟鐨勮瘎浼般€傛鏋跺眰闈㈢殑璇勪及浼氭瘮杈冪缁熺畝鍗曪紝鏇翠笓涓氥€佸鏉傜殑璇勪及闇€瑕乀ask灞傞潰鍘昏繘琛屻€侻axCompute妗嗘灦浣跨敤shennong鈥濆疄鏃垛€滅殑杈撳嚭缁熻锛屽涓嬶細
screenshot
screenshot
鐏板害浣跨敤鍦烘櫙
Task鐏板害鍙戝竷锛?
1锛?PE鍒濆鍖栫伆搴︿俊鎭紱瀵筽roject杩涜Tier鍒嗙粍锛涘埗瀹氱伆搴﹀彂甯冩柟妗堝拰搴旀€ュ洖婊氭柟妗堬紱
2锛?PE浣跨敤flighting鐗堟湰锛岄儴缃查杩涜鐏板害鍙戝竷鐨凾ask Package锛?
3锛?PE鎸夌収鏂规鎿嶄綔鏇存柊Tier鐏板害閰嶇疆锛屾寜閮ㄥ氨鐝殑瀹炴柦鐏板害鍙戝竷
4锛?寮€鍙戝拰PE閫氳繃缁熻鍜岀洃鎺у伐鍏峰疄鏃跺叧娉ㄧ伆搴﹁繘灞曪紝鏍规嵁鐏板害缁撴灉鍒ゆ柇缁х画鐏板害杩樻槸鍋滄鐏板害锛?
绾夸笂鐏板害娴嬭瘯锛?
1锛?PE鍒濆鍖栫伆搴︿俊鎭紱鍒涘缓鐏板害娴嬭瘯涓撶敤鐨凾ier鍒嗙粍锛屽埗瀹氭祴璇曟柟妗堝拰搴旀€ュ洖婊氭柟妗堬紱
2锛?PE閮ㄧ讲棰勮繘琛岀伆搴︽祴璇曠殑Task Package锛?
3锛?PE鎸夌収娴嬭瘯鏂规璁剧疆Tier鐏板害閰嶇疆锛屽寘鎷琈apping锛屽垎娴佷俊鎭紱
4锛?寮€鍙戦€氳繃缁熻鍜岀洃鎺у伐鍏峰疄鏃跺叧娉ㄧ伆搴︽祴璇曡繘灞曪紝鏍规嵁鐏板害缁撴灉鍒ゆ柇娴嬭瘯缁х画杩樻槸鍋滄鐏板害
寮€鍙戠幆澧冨彲浠ヤ娇鐢╯et蹇界暐鐏板害閰嶇疆锛屾寚瀹氫娇鐢ㄥ浐瀹氱増鏈?
set MaxCompute.task.major.version=tom;
鐏板害淇℃伅浣跨敤锛?
MaxCompute MaxCompute_worker瀹氭椂璇诲彇鐏板害閰嶇疆鍒扮伆搴﹀垽鍐砪ache涓紝闂撮殧涓? min锛屽綋鏈塲ob璇锋眰鍒癕axCompute_worker鏃讹紝鍦ㄥ悜executor鍙戝嚭TryRun鍓嶄娇鐢ㄧ伆搴︾紦瀛樹腑鐨勪俊鎭垽鏂璇锋眰鍘诲線鐨勫叿浣揟ask package鐗堟湰锛屼娇鐢ㄨ鍒欏涓嬶細
1锛?浼樺厛鍒ゆ柇鏄惁浣跨敤SET optin 璁剧疆task鐗堟湰锛屽鏄娇鐢ㄦ寚瀹氱増鏈繘琛屽悗缁姹傦紱
2锛?鏈娇鐢⊿ET锛屾牴鎹畃roject鎵€鍦ㄧ殑Tier锛岃幏鍙朤ier鐨凪apping鍜屽垎娴佷俊鎭紝鏍规嵁鍒嗘祦淇℃伅姣斾緥纭畾鐩爣Task鐗堟湰锛?
3锛?濡傛灉鐩爣鐗堟湰鏈儴缃插垯浼氬鍒癲efault
缁撹
MaxCompute 鐨凾ask鐏板害鍙戝竷鎬荤殑鏉ヨ锛岄厤缃伒娲绘搷浣滀究鎹凤紝鑳藉婊¤冻鐩墠MaxCompute鐨凾ask鐨勭伆搴﹀彂甯冮渶姹傦紝鐏板害鍏冩暟鎹湪璁捐涓婅€冭檻浜嗗悗缁殑鎵╁睍锛屼繚鐣欎簡瓒冲鐨勬墿灞曠殑鑳藉姏锛涘湪MaxCompute Task鐏板害鍙戝竷瀹炶返涓婏紝MaxCompute鍙戝竷绠＄悊鍥㈤槦鎻愬嚭浜嗘祦閲忔《鐨勭伆搴﹀彂甯冩柟寮忥紝鍥哄畾涓猅ier鐨勬祦閲忛厤缃俊鎭紝灏嗛渶瑕佺伆搴︾殑project閫愭壒娆＄殑淇敼褰掑睘鐨凾ier锛屽舰璞＄偣灏辨槸灏嗕笉鍚岀殑project鏀惧叆涓嶅悓娴侀噺鐨勬《涓€侻axCompute 鐏板害鍙戝竷鐨勫悗缁睍鏈涳紝甯屾湜寮€鍙戣兘澶熷湪涓€瀹氬鏍镐笅鑷富鐨勯儴缃叉棩甯哥伆搴︽祴璇曠殑package锛岀畝鍖栧綋鍓嶅鎵归儴缃叉祦绋嬶紝鎻愰珮鏁堢巼銆?
闃呰鍏ㄦ枃锛歨ttp://click.aliyun.com/m/13996/
鍓嶈█
鎸夌収鐧剧璇嶆潯鐨勮В閲婏細鐏板害鍙戝竷鏄寚鍦ㄩ粦涓庣櫧涔嬮棿锛岃兘澶熷钩婊戣繃娓＄殑涓€绉嶅彂甯冩柟寮忋€傜伆搴﹀彂甯冨彲浠ヤ繚璇佹暣浣撶郴缁熺殑绋冲畾锛屽湪鍒濆鐏板害鐨勬椂鍊欏氨鍙互鍙戠幇銆佽皟鏁撮棶棰橈紝浠ヤ繚璇佸叾褰卞搷搴︺€傞檷浣庡彂甯冧笂绾跨殑椋庨櫓銆?
鐏板害鍙戝竷鐨勫叧閿槸閫夋嫨鍚堥€傜殑鐏板害绛栫暐锛屾妸绗﹀悎绛栫暐鐨勬祦閲忓紩鍏ュ埌鏂扮増鏈笂銆傜伆搴︾瓥鐣ョ殑閫夋嫨闇€瑕佽€冭檻寰呭彂甯冨璞″湪閾捐矾鎴栫郴缁熶腑鐨勪綅缃鑹诧紙濡傚鎴风銆佺Щ鍔ˋPP銆佹湇鍔″悗鍙扮瓑锛夈€佺敤鎴峰睘鎬у拰鍙戝竷闇€姹傜洰鏍囩瓑銆傛湰鏂囧皢瀵筂axCompute鐨凾ask鐏板害鍙戝竷绯荤粺杩涜浠嬬粛銆?
MaxCompute Task鐏板害鍙戝竷
MaxCompute flighting锛圫print21锛夊疄鐜颁簡MaxCompute妗嗘灦鍜孴ask鐨勮В鑰︼紝MaxCompute妗嗘灦鍜孴ask鍙互鏍规嵁鍚勮嚜鐨勭爺鍙戣鍒掕繘琛岀嫭绔嬬殑鍙戝竷銆傚悓鏃跺Task寮曞叆浜嗙増鏈寲绠＄悊锛堝ぇ鐗堟湰/灏忕増鏈級锛孧axCompute妗嗘灦鏀寔鍚岀Task鐨勫涓ぇ鐗堟湰骞跺瓨涓旂嫭绔嬫彁渚涙湇鍔°€傝繖浜涚壒鎬т负Task鐏板害鍙戝竷鎻愪緵浜嗗熀纭€鏀寔銆?
MaxComputer_framework
鍥?MaxCompute flighting 鏋舵瀯鍥?
MaxCompute鐩墠鏀寔SQL_RELATIVE_TASKS, PS_TASK绛塗ask Package绫诲瀷, Task鐨勭伆搴﹀彂甯冧究鏄杩欎簺Task Package鍙戝竷锛屽笇鏈涳細
1锛?鍚凾ask Package鐙珛杩涜鐏帮紝浜掍笉骞叉壈锛?
2锛?鍙戝竷杩囩▼瀵圭敤鎴烽€忔槑锛?
3锛?鐏板害鑼冨洿銆佸奖鍝嶉潰鍜岀粨鏋滃彲鎺э紱
4锛?鍏锋湁蹇€熷彲闈犵殑鐏板害缁撴灉璇勪及鏈哄埗锛?
5锛?鎿嶄綔绠€鍗曪紝鍙嚜鍔ㄥ寲/鍗婅嚜鍔ㄥ寲杩愮淮锛?
6锛?鐗堟湰绠＄悊绠€鍗曪紝鏄撲簬杩涜鐗堟湰缁存姢锛?
7锛?鐗规畩鍦哄悎锛屽厑璁稿紑鍙戠嚎涓婅繘琛屽熀浜庣伆搴︽祴璇曪紱
MaxCompute Task鐏板害鍙戝竷鍘熺悊
鍦ㄤ粙缁峂axCompute鐨凾ask鐏板害鍘熺悊鍓嶏紝鍏堜簡瑙ｄ笅鍏朵笟鍔℃祦绋嬨€傜敤鎴锋彁浜や綔涓氬悗锛岃姹傞鍏堝埌worker锛寃orker涓烘瘡涓猨ob鍒涘缓instanceId锛岀劧鍚嶳PC鍒癊xecutor杩涜TryRun锛孴ryRun鏃剁敱鏀寔璇askType鐨凱ackage澶勭悊锛屽鏋滄槸鍚屾Task锛屽垯鐩存帴澶勭悊骞跺皢缁撴灉杩斿洖锛屽鏋滀负寮傛Task鍒欒繑鍥炵粰Worker锛學orker鍙戦€佺粰Scheduler锛屽湪Executor杞璇锋眰Task鏃跺彂閫佺粰鍏跺苟缁х画澶勭悊銆傚涓嬪浘鎵€绀猴細
screenshot
鍥?MaxCompute flighting 璇锋眰涓氬姟閫昏緫
娉細
鍚屾Task锛?1 -> 2 -> 3
寮傛Task锛?1 -> 2 -> 3 -> 4 -> 5 -> 6
涓轰簡淇濊瘉Task鎵ц涓€鑷存€э紝Worker鍚慐xecutor鍙戦€佽姹傚墠闇€瑕侀€夊畾package鐨勭増鏈紝骞朵笖鍚庣画璇锋眰锛堝紓姝ワ級涔熼渶瑕佽鐗堟湰package鎻愪緵锛岀粦瀹氳繃绋嬪湪绗簩姝ュ畬鎴愩€傝€冭檻浠ヤ笂浜嬪疄锛屽彲閫氳繃鍦╓orker璇诲彇鐏板害娴侀噺閰嶇疆锛屾牴鎹敤鎴峰睘鎬х壒寰佸拰娴侀噺閰嶇疆鍐冲畾鎵цTask鐨刾ackage鐗堟湰閫夋嫨缁撴灉銆傜伆搴﹀師鐞嗗浘濡備笅锛?
screenshot
鍥?Task鐏板害鍙戝竷鍘熺悊
鐏板害鍙戝竷鍓嶏紝PE鎴栬€呭紑鍙戜汉鍛樺垵濮嬪寲鐏板害閰嶇疆锛岀伆搴﹂厤缃俊鎭寘鍚杩涜鐏板害鍙戝竷鐨勭増鏈€佸弬涓庣伆搴︾殑鐢ㄦ埛鐗瑰緛浠ュ強鐏板害鏃剁殑娴侀噺鍒嗛厤锛岃瑙乀ask鐏板害鍏冩暟鎹€傚噯澶囧伐浣滃畬鎴愬悗鍙繘琛岀伆搴﹀彂甯冦€傚湪鐏板害鍙戝竷杩囩▼涓紝Worker瀹氭椂涓嶰TS鍚屾鐏板害閰嶇疆骞舵牴鎹綋鍓嶇伆搴﹂厤缃紝閫夊彇婊¤冻鐗瑰緛瑕佹眰鐨勭敤鎴疯姹傚弬涓庣伆搴︼紱鎸夌収鐏板害娴侀噺閰嶇疆寮曞娴侀噺鍒扮伆搴﹀彂甯冪殑Package鐗堟湰涓婏紝鍦ㄧ伆搴﹀彂甯冭繃绋嬩腑锛孭E鎴栬€呭紑鍙戜汉鍛樻牴鎹伆搴﹁繘灞曠姸鍐靛喅瀹氱户缁伆搴﹁繕鏄仠姝㈢伆搴︼紝鍐冲畾灏嗗弽搴斿湪鐏板害閰嶇疆鐨勪慨鏀广€?
Task鐏板害鍏冩暟鎹?
鍩轰簬濡備笂鐨勮€冭檻锛孧axCompute璁捐浜嗗悇Task鐙珛鐨勭伆搴﹀彂甯冩祦绋嬶紝浣撶幇鍦ㄧ嫭绔嬬殑鐏板害鍏冩暟鎹璁′笂锛屽涓嬶細
{
"grayInfo": {
"MaxComputeServiceTest1": {
"reserved": {},
"taskVersion": {
"defaultVersion": "default",
"flightingVersion": "flighting"},
"tierGrayInfo": {
"DEFAULTTIER": {
"grayInfo": {
"defaultVersion": 100,
"flightingVersion": 0},
"reserved": {}
}
},
"MaxComputeServiceTest2": {
"reserved": {},
"taskVersion": {
"defaultVersion": "default",
"flightingVersion": "flighting"},
"tierGrayInfo": {
"DEFAULTTIER": {
"grayInfo": {
"defaultVersion": 10,
"flightingVersion": 90},
"reserved": {}
}
}
}
},
"projectTier": {
"DEFAULTTIER": [
"admin_task_project",
"algo_public",
"algo_test"]
},
"reserved": {},
"taskType": "SQL_RELATIVE_TASKS"
}
姣忕Task閮芥湁鐙珛鐨勭敤鎴风壒寰佸垎缁勪俊鎭紙Tier锛夛紱鑰冭檻MaxCompute鐨勫鎺у満鏅拰鍚勬帶鍒堕泦缇ょ嫭绔嬬殑鍖呯鐞嗗櫒鏂瑰紡锛岀伆搴﹂厤缃俊鎭笂澶氬姞浜嗕竴灞傛帶鍒堕泦缇ょ淮搴︼紝姣忎釜鎺у埗闆嗙兢瀵筎ask鐏板害鍙戝竷鍏变韩Tier鍒嗙粍淇℃伅锛岀嫭浜伆搴﹂厤缃俊鎭紱
Tier閰嶇疆
Tier鏄疢axCompute鐏板害閰嶇疆鐨勬渶灏忕矑搴︼紝鍙互瀵规瘡涓猅ier璁剧疆涓嶅悓鐨勬祦閲忓垎娴佷俊鎭痝rayInfo鍜屾帶鍒堕泦缇ょ淮搴︿笅鐨勫叡浜玀apping淇℃伅銆俫rayInfo淇℃伅閰嶇疆浜嗘祦閲忓湪涓嶅悓鐗堟湰涓婄殑鍒嗘祦鎯呭喌锛屾墍鏈夐毝灞炰簬璇ier鐨刾roject鐨勮绉峧ob閮芥寜鐓ц繖涓厤缃繘琛屾祦閲忓垎娴侊紱Mapping淇℃伅璁剧疆浜嗙増鏈琓ag鍒板疄闄呯殑Task Package version鐨勬槧灏勶紝鐗瑰埆鐨勶紝姣忎釜Tier涔熷彲浠ヨ缃嫭绔嬬殑Mapping淇℃伅锛孡oad鐏板害閰嶇疆鏃讹紝浼樺厛浣跨敤Tier鐨凪apping淇℃伅銆俆ier鐨勭鏈塎apping璁剧疆鏄€氳繃淇濈暀瀛楁杩涜鎵╁睍鐨勶紝濡備笅锛?
"tierGrayInfo": {
"DEFAULTTIER": {
"grayInfo": {
"defaultVersion": 80,
"flightingVersion": 20},
"reserved": {
"defaultVersion": "tom",
"flightingVersion": "jerry"}
}
}
鍗囩骇/鍥炴粴
閴翠簬绾夸笂MaxCompute澶氭帶闆嗙兢鐜扮姸锛屼笖姣忎釜鎺у埗闆嗙兢閮芥湁鐙珛鐨勫寘绠＄悊鍣紝鍦ㄥ崌绾?鍥炴粴鏃堕渶瑕佽€冭檻MaxCompute妗嗘灦鍜孴ask package鐨勫吋瀹规€х瓑闂锛岄渶瑕佸鐏板害鍏冧俊鎭繘琛屼竴瀹氱殑澶勭悊锛?
鍏冩暟鎹?鍗囩骇&鍥炴粴
Tier鍒嗙粍淇℃伅 淇濈暀
Tier娴侀噺鍒嗘祦 閲嶇疆鐩爣闆嗙兢
鐗堟湰绠＄悊
MaxCompute flighting瑙ｈ€︿簡MaxCompute妗嗘灦鍜孴ask锛岀悊璁轰笂鍚勫姛鑳藉潡鍙嫭绔嬬鐞嗗彂甯冿紝杩欑粰MaxCompute鐨勫彂甯冪鐞嗗甫鏉ヤ簡鎸戞垬锛?鑰冭檻鍏煎鎬э紝MaxCompute瀵圭増鏈鐞嗗仛浜嗗涓嬬害瀹氾細
1锛?MaxCompute妗嗘灦闅廠print澶х増鏈繘琛屽彂甯冿紝骞朵笖鍦⊿print鍙戝竷鏃跺己琛岃繘琛屼竴娆′笌鍚凾ask鐨勭増鏈榻愶紝鎼哄甫鍚凾ask Package鐨刣efault鐗堟湰锛?
2锛?鍚庡彂甯冪殑瀵瑰吋瀹规€ц礋璐ｏ紱
3锛?姣忕Task鏈変袱涓寮忓ぇ鐗堟湰default鍜宖lighting锛宒efault闅忔瘡涓猻print鍙戝竷锛宖lighting鍦ㄩ渶瑕佸Task杩涜鐏板害鍙戝竷鏃跺彂甯冿紱
4锛?鍥哄畾浣跨敤default璁板綍Task鍙戝竷鍘嗗彶锛岀伆搴﹀彂甯冩垚鍔熺殑鐗堟湰闇€瑕侀噸鏂版彁浜ゆ垚default鐨勪竴涓柊鐨勫皬鐗堟湰锛?
閰嶅璁炬柦
MaxCompute Task鐏板害鍙戝竷娑夊強鍒板涓伐鍏凤紝鍖呮嫭Task Package鐨勯儴缃插伐鍏凤紝鐏板害閰嶇疆宸ュ叿锛圓dmin Task锛変互鍙婄伆搴﹁繃绋嬩腑琛￠噺鐏板害缁撴灉鐨勭粺璁″伐鍏峰拰鐩戞帶鎶ヨ宸ュ叿銆?
鍖呴儴缃插伐鍏凤細
OPDS flighting寮€鍙戜簡鐙珛鐨凾ask Package閮ㄧ讲宸ュ叿package_util锛屽彲鐢ㄤ簬绠＄悊Task鐨勬帶鍒堕泦缇ゅ寘鍜岃绠楅泦缇ゅ寘锛涢€氳繃瀹冨彲瀵筎ask Package杩涜deploy & rollback绛夋搷浣滐紱
鐏板害閰嶇疆宸ュ叿锛?
OPDS鐨凾ask鐏板害閰嶇疆鍏冧俊鎭瓨鍌ㄥ湪OTS涓紝闇€瑕佷娇鐢ˋdmin Task鍛戒护瀹炵幇瀵圭伆搴﹂厤缃慨鏀圭殑鐩殑锛岀洰鍓嶆敮鎸佺殑鍛戒护濡備笅锛?
鍒濆鍖栨煇绫诲瀷鐨凾ask鐏板害閰嶇疆锛?
Tier鐨凜RUD锛?
淇敼闆嗙兢鏍囩涓嬬殑鍏变韩Mapping锛?
淇敼Tier鐨勬祦閲忛厤缃紱
淇敼Tier鐗瑰畾鐨凪apping锛?
鍗囩骇/鍥炴粴鏃舵槸鍚︽竻妤氱伆搴﹁繘搴︿俊鎭紱
琛￠噺鐏板害鐨勬寚鏍囧拰鐩戞帶鎶ヨ锛?
MaxCompute Task鐏板害鍙戝竷鐨勭粨鏋滆 閲忔湁涓や釜灞傞潰锛屼竴涓槸妗嗘灦灞傞潰鏅亶涓€鑸殑缁熻锛屽task鐨勬垚鍔熺巼/澶辫触鐜囷紝閫氳繃涓巇efault鐗堟湰鐨勫姣旓紝鏉ヨ繘琛岃瘎浼帮紱鍙︿竴涓槸Task灞傞潰鍋氱殑涓撲笟鐨勮瘎浼般€傛鏋跺眰闈㈢殑璇勪及浼氭瘮杈冪缁熺畝鍗曪紝鏇翠笓涓氥€佸鏉傜殑璇勪及闇€瑕乀ask灞傞潰鍘昏繘琛屻€侻axCompute妗嗘灦浣跨敤shennong鈥濆疄鏃垛€滅殑杈撳嚭缁熻锛屽涓嬶細
screenshot
screenshot
鐏板害浣跨敤鍦烘櫙
Task鐏板害鍙戝竷锛?
1锛?PE鍒濆鍖栫伆搴︿俊鎭紱瀵筽roject杩涜Tier鍒嗙粍锛涘埗瀹氱伆搴﹀彂甯冩柟妗堝拰搴旀€ュ洖婊氭柟妗堬紱
2锛?PE浣跨敤flighting鐗堟湰锛岄儴缃查杩涜鐏板害鍙戝竷鐨凾ask Package锛?
3锛?PE鎸夌収鏂规鎿嶄綔鏇存柊Tier鐏板害閰嶇疆锛屾寜閮ㄥ氨鐝殑瀹炴柦鐏板害鍙戝竷
4锛?寮€鍙戝拰PE閫氳繃缁熻鍜岀洃鎺у伐鍏峰疄鏃跺叧娉ㄧ伆搴﹁繘灞曪紝鏍规嵁鐏板害缁撴灉鍒ゆ柇缁х画鐏板害杩樻槸鍋滄鐏板害锛?
绾夸笂鐏板害娴嬭瘯锛?
1锛?PE鍒濆鍖栫伆搴︿俊鎭紱鍒涘缓鐏板害娴嬭瘯涓撶敤鐨凾ier鍒嗙粍锛屽埗瀹氭祴璇曟柟妗堝拰搴旀€ュ洖婊氭柟妗堬紱
2锛?PE閮ㄧ讲棰勮繘琛岀伆搴︽祴璇曠殑Task Package锛?
3锛?PE鎸夌収娴嬭瘯鏂规璁剧疆Tier鐏板害閰嶇疆锛屽寘鎷琈apping锛屽垎娴佷俊鎭紱
4锛?寮€鍙戦€氳繃缁熻鍜岀洃鎺у伐鍏峰疄鏃跺叧娉ㄧ伆搴︽祴璇曡繘灞曪紝鏍规嵁鐏板害缁撴灉鍒ゆ柇娴嬭瘯缁х画杩樻槸鍋滄鐏板害
寮€鍙戠幆澧冨彲浠ヤ娇鐢╯et蹇界暐鐏板害閰嶇疆锛屾寚瀹氫娇鐢ㄥ浐瀹氱増鏈?
set MaxCompute.task.major.version=tom;
鐏板害淇℃伅浣跨敤锛?
MaxCompute MaxCompute_worker瀹氭椂璇诲彇鐏板害閰嶇疆鍒扮伆搴﹀垽鍐砪ache涓紝闂撮殧涓? min锛屽綋鏈塲ob璇锋眰鍒癕axCompute_worker鏃讹紝鍦ㄥ悜executor鍙戝嚭TryRun鍓嶄娇鐢ㄧ伆搴︾紦瀛樹腑鐨勪俊鎭垽鏂璇锋眰鍘诲線鐨勫叿浣揟ask package鐗堟湰锛屼娇鐢ㄨ鍒欏涓嬶細
1锛?浼樺厛鍒ゆ柇鏄惁浣跨敤SET optin 璁剧疆task鐗堟湰锛屽鏄娇鐢ㄦ寚瀹氱増鏈繘琛屽悗缁姹傦紱
2锛?鏈娇鐢⊿ET锛屾牴鎹畃roject鎵€鍦ㄧ殑Tier锛岃幏鍙朤ier鐨凪apping鍜屽垎娴佷俊鎭紝鏍规嵁鍒嗘祦淇℃伅姣斾緥纭畾鐩爣Task鐗堟湰锛?
3锛?濡傛灉鐩爣鐗堟湰鏈儴缃插垯浼氬鍒癲efault
缁撹
MaxCompute 鐨凾ask鐏板害鍙戝竷鎬荤殑鏉ヨ锛岄厤缃伒娲绘搷浣滀究鎹凤紝鑳藉婊¤冻鐩墠MaxCompute鐨凾ask鐨勭伆搴﹀彂甯冮渶姹傦紝鐏板害鍏冩暟鎹湪璁捐涓婅€冭檻浜嗗悗缁殑鎵╁睍锛屼繚鐣欎簡瓒冲鐨勬墿灞曠殑鑳藉姏锛涘湪MaxCompute Task鐏板害鍙戝竷瀹炶返涓婏紝MaxCompute鍙戝竷绠＄悊鍥㈤槦鎻愬嚭浜嗘祦閲忔《鐨勭伆搴﹀彂甯冩柟寮忥紝鍥哄畾涓猅ier鐨勬祦閲忛厤缃俊鎭紝灏嗛渶瑕佺伆搴︾殑project閫愭壒娆＄殑淇敼褰掑睘鐨凾ier锛屽舰璞＄偣灏辨槸灏嗕笉鍚岀殑project鏀惧叆涓嶅悓娴侀噺鐨勬《涓€侻axCompute 鐏板害鍙戝竷鐨勫悗缁睍鏈涳紝甯屾湜寮€鍙戣兘澶熷湪涓€瀹氬鏍镐笅鑷富鐨勯儴缃叉棩甯哥伆搴︽祴璇曠殑package锛岀畝鍖栧綋鍓嶅鎵归儴缃叉祦绋嬶紝鎻愰珮鏁堢巼銆?
闃呰鍏ㄦ枃锛歨ttp://click.aliyun.com/m/13996/