780      阿里云  技术社区[云栖]

使用shell执行datax任务__常见问题_大数据开发套件-阿里云

1. 如果没有配置好shell环境，请参考此链接的文档：Q：怎么顺利执行Shell/MR类型任务

2. 配置完成后，在shell节点填写代码。可以参考下面的demo代码

   • 代码说明
     1. 生成临时datax配置文件（您只需要修改配置文件内容即可，其他不用更改。参考文档：https://github.com/alibaba/DataX/wiki/Quick-Start）
     2. 读取调度参数，分别为$1, $2
        • ${bdp.system.bizdate}，${bdp.system.cyctime} 不需要配置，具体含义参考：系统参数
     3. 执行datax任务，进行数据同步
     4. 删除临时文件
     5. 判断任务成功失败，进行返回。0代表成功

#!/bin/bash
#********************************************************************#
##author:yixiao  streamreader->streamwriter
##create time:2016-03-28 10:00:00
#********************************************************************#
shell_datax_home='/home/admin/shell_datax'
mkdir -p ${shell_datax_home}
shell_datax_config=${shell_datax_home}/${ALISA_TASK_ID}
echo '''
{
    "job": {
        "setting": {
            "speed": {
                "channel": 1
            },
            "errorLimit": {
                "record": 0,
                "percentage": 0.02
            }
        },
        "content": [
            {
                "reader": {
                    "name": "streamreader",
                    "parameter": {
                        "column": [
                            {
                                "value": "${bdp.system.bizdate}",
                                "type": "string"
                            },
                            {
                                "value": "${bdp.system.cyctime}",
                                "type": "string"
                            },
                            {
                                "value": "${params1}__${params2}",
                                "type": "string"
                            },
                            {
                                "value": 19890427,
                                "type": "long"
                            },
                            {
                                "value": "1989-06-04 00:00:00",
                                "type": "date"
                            },
                            {
                                "value": true,
                                "type": "bool"
                            },
                            {
                                "value": "test",
                                "type": "bytes"
                            }
                        ],
                        "sliceRecordCount": 10
                    }
                },
                "writer": {
                    "name": "streamwriter",
                    "parameter": {
                        "print": true,
                        "encoding": "UTF-8"
                    }
                }
            }
        ]
    }
}
''' > ${shell_datax_config}
params1=$1
params2=$2
datax_params='-p "-Dparams1=${params1} -Dparams2=${params2}"'
echo "`date '+%Y-%m-%d %T'` shell datax config: ${shell_datax_config}"
echo "`date '+%Y-%m-%d %T'` shell datax params: -p "-Dparams1=${params1} -Dparams2=${params2}""
/home/admin/datax3/bin/datax.py ${shell_datax_config} -p "-Dparams1=${params1} -Dparams2=${params2}"
shell_datax_run_result=$?
rm ${shell_datax_config}
if [${shell_datax_run_result} -ne 0]
then
    echo "`date '+%Y-%m-%d %T'` shell datax ended failed :("
    exit -1
fi
echo "`date '+%Y-%m-%d %T'` shell datax ended success～"

3。 调度系统配置参数