114      阿裏雲

並發__操作指南_批量計算-阿裏雲

一個作業(Job)中可以有多個任務(Task)，一個任務可以指定在多個實例(Instance)上運行程序。

如何並發？

請看下麵 job description json 例子：

{
    "DAG": {
        ...
        "Tasks": {
            ...
            "count": {
                "InstanceCount": 3,  //指定需要實例數：3台VM
                "LogMapping": {},
                "AutoCluster": {
                    "ResourceType": "OnDemand",
                    "ImageId": "img-ubuntu",
                    "InstanceType": "bcs.a2.large"
                },
                "Parameters": {
                    "Command": {
                        "EnvVars": {},
                        "CommandLine": "python count.py",
                        "PackagePath": "oss://your-bucket/log-count/worker.tar.gz"
                    },
                    "InputMappingConfig": {
                        "Lock": true
                    },
                    "StdoutRedirectPath": "oss://your-bucket/log-count/logs/",
                    "StderrRedirectPath": "oss://your-bucket/log-count/logs/"
                },
                "OutputMapping": {},
                "MaxRetryCount": 0,
                "Timeout": 21600,
                "InputMapping": {}
            }
        }
    },
    "Description": "batchcompute job",
    "Priority": 0,
    "JobFailOnInstanceFail": true,
    "Type": "DAG",
    "Name": "log-count"
}

任務count中配置了InstanceCount為3， 表示需要實例數3台, 即在3台VM上運行這個任務的程序。

並發處理不同片段的數據

3台VM上運行的任務程序都是一樣的，如何讓它處理不同的數據呢？

在任務程序中使用環境變量: BATCH_COMPUTE_DAG_INSTANCE_ID(實例ID) 來區分，可以處理不同片段的數據。

以下是 count.py 代碼片段:

...
# instance_id: should start from 0
instance_id = os.environ['BATCH_COMPUTE_DAG_INSTANCE_ID']
...
filename = 'part_%s.txt' %  instance_id
...
# 1. download a part
oss_tool.download_file('%s/%s/%s.txt' % (pre, split_results, instance_id ), filename)
...
# 3. upload result to oss
upload_to = '%s/count_results/%s.json' % (pre, instance_id )
print('upload to %s' % upload_to)
oss_tool.put_data(json.dumps(m), upload_to)
...