780      iPhone_iPad_Mac_手機_平板_蘋果apple

使用shell執行datax任務__常見問題_大數據開發套件-阿裏雲

1. 如果沒有配置好shell環境，請參考此鏈接的文檔：Q：怎麼順利執行Shell/MR類型任務

2. 配置完成後，在shell節點填寫代碼。可以參考下麵的demo代碼

   • 代碼說明
     1. 生成臨時datax配置文件（您隻需要修改配置文件內容即可，其他不用更改。參考文檔：https://github.com/alibaba/DataX/wiki/Quick-Start）
     2. 讀取調度參數，分別為$1, $2
        • ${bdp.system.bizdate}，${bdp.system.cyctime} 不需要配置，具體含義參考：係統參數
     3. 執行datax任務，進行數據同步
     4. 刪除臨時文件
     5. 判斷任務成功失敗，進行返回。0代表成功

#!/bin/bash
#********************************************************************#
##author:yixiao  streamreader->streamwriter
##create time:2016-03-28 10:00:00
#********************************************************************#
shell_datax_home='/home/admin/shell_datax'
mkdir -p ${shell_datax_home}
shell_datax_config=${shell_datax_home}/${ALISA_TASK_ID}
echo '''
{
    "job": {
        "setting": {
            "speed": {
                "channel": 1
            },
            "errorLimit": {
                "record": 0,
                "percentage": 0.02
            }
        },
        "content": [
            {
                "reader": {
                    "name": "streamreader",
                    "parameter": {
                        "column": [
                            {
                                "value": "${bdp.system.bizdate}",
                                "type": "string"
                            },
                            {
                                "value": "${bdp.system.cyctime}",
                                "type": "string"
                            },
                            {
                                "value": "${params1}__${params2}",
                                "type": "string"
                            },
                            {
                                "value": 19890427,
                                "type": "long"
                            },
                            {
                                "value": "1989-06-04 00:00:00",
                                "type": "date"
                            },
                            {
                                "value": true,
                                "type": "bool"
                            },
                            {
                                "value": "test",
                                "type": "bytes"
                            }
                        ],
                        "sliceRecordCount": 10
                    }
                },
                "writer": {
                    "name": "streamwriter",
                    "parameter": {
                        "print": true,
                        "encoding": "UTF-8"
                    }
                }
            }
        ]
    }
}
''' > ${shell_datax_config}
params1=$1
params2=$2
datax_params='-p "-Dparams1=${params1} -Dparams2=${params2}"'
echo "`date '+%Y-%m-%d %T'` shell datax config: ${shell_datax_config}"
echo "`date '+%Y-%m-%d %T'` shell datax params: -p "-Dparams1=${params1} -Dparams2=${params2}""
/home/admin/datax3/bin/datax.py ${shell_datax_config} -p "-Dparams1=${params1} -Dparams2=${params2}"
shell_datax_run_result=$?
rm ${shell_datax_config}
if [${shell_datax_run_result} -ne 0]
then
    echo "`date '+%Y-%m-%d %T'` shell datax ended failed :("
    exit -1
fi
echo "`date '+%Y-%m-%d %T'` shell datax ended success～"

3。 調度係統配置參數