304      財經資訊

數據預處理__使用手冊(new)_機器學習-阿裏雲

目錄

• 加權采樣

• 隨機采樣

• 過濾與映射

• 分層采樣

• join

• 合並列

• UNION

• 增加序號列

• 拆分

• 缺失值填充

• 歸一化

• 標準化

• 類型轉換

加權采樣

以加權方式生成采樣數據；權重列必須為double或int類型，按照該列的value大小采樣；如col的值是1.2和1.0；則value=1.2所屬樣本的被采樣的概率就大一些。

參數設置

參數框

• 可手動輸入采樣個數（或者采樣比例）
• 可以選擇放回采樣或者不放回采樣，默認為不放回，勾選後變為放回
• 下拉框選擇加權列，加權列支持double型和bigint型
• 可配置隨機數種子，默認係統自動分配

PAI 命令

PAI -name WeightedSample -project algo_public -DprobCol="previous" -DsampleSize="500" 
    -DoutputTableName="test2" -DinputPartitions="pt=20150501" -DinputTableName="bank_data_partition";

算法參數

注：當sampleSize 與 sampleRatio都為空時，報錯； 當sampleSize 與 sampleRatio都不為空時，以sampleSize為準。

隨機采樣

以隨機方式生成采樣數據，每次采樣是各自獨立的。

參數設置

• 可手動輸入采樣個數（或者采樣比例）
• 可以選擇放回采樣或者不放回采樣，默認為不放回，勾選後變為放回
• 可配置隨機數種子，默認係統自動分配
• 可以自己配置並發計算核心數目與內存，默認係統自動分配

PAI 命令

pai -name RandomSample -project algo_public 
    -DinputTableName=wbpc 
    -DoutputTableName=wpbc_sample 
    -DsampleSize=100 
    -Dreplace=false 
    -DrandomSeed=1007;

算法參數

注：當sampleSize 與 sampleRatio都為空時，報錯； 當sampleSize 與 sampleRatio都不為空時，以sampleSize為準。

過濾與映射

對數據按照過濾表達式進行篩選；可以重命名字段名；

參數設置

1、通過where條件實現數據過濾，與SQL類似，如：

- 篩選條件：目前操作符支持”=”，”!=”，”>”，”<”， “>=”和“<=”，like，rlike    

2、重命名字段

PAI命令

PAI -name Filter -project algo_public -DoutTableName="test_9" -DinputPartitions="pt=20150501"
    -DinputTableName="bank_data_partition" -Dfilter="age>=40";

• name: 組件名字
• project: project名字，用於指定算法所在空間。係統默認是algo_public，用戶自己更改後係統會報錯
• outTableName: 輸出表的名字
• inputPartitions：（可選）訓練輸入表分區。輸入表對應的輸入分區，選中全表則為None
• inputTableName: 輸入表的名字
• filter: where篩選條件，目前操作符支持”=”，”!=”，”>”，”<”， “>=”，“<=”,“like”和“rlike”

分層采樣

數據集分層抽取一定比例或者一定數據的隨機樣本

參數設置

參數框

• 下拉框選擇分組列（最大支持100個分組）
• 可手動輸入分組的采樣個數（或者采樣比例）
• 可配置隨機數種子，默認1234567
• 可以自己配置並發計算核心數目與內存，默認係統自動分配

pai 命令

PAI -name StratifiedSample -project algo_public 
    -DinputTableName="test_input" 
    -DoutputTableName="test_output" 
    -DstrataColName="label" 
    -DsampleSize="A:200,B:300,C:500" 
    -DrandomSeed=1007 
    -Dlifecycle=30

算法參數

注：當sampleSize 與 sampleRatio都為空時，報錯； 當sampleSize 與 sampleRatio都不為空時，以sampleSize為準。

示例

源數據

1. 創建實驗

2. 選取分組列

• 選擇play列，做為分組列

3. 配置采樣個數

• play=Yes的分組采4條；play=No有分組采3條

4. 采樣結果

join

兩張表通過關聯信息，合成一張表，並決定輸出的字段;與SQL的join語句功能類似

參數設置

填寫參數

• 連接類型支持：左連接，內連接，右連接，全連接
• 關聯條件目前隻支持等式
• 可手動添加或刪除關聯條件

PAI 命令

不提供pai命令

合並列

將兩張表的數據按列合並，需要表的行數保持一致

參數設置

如圖：左表選擇輸入列右表選擇輸入列

• 選擇的兩張表行數需保持一致

• 左表與右表選擇的輸出列名不能重複

• 選擇輸出字段列時可手動更改輸出字段名

• 若左表右表均不選擇輸出列，則默認輸出全表。此時勾選‘是否自動重命名輸出列’，會將重複列重命名後輸出

PAI 命令

PAI -name AppendColumns -project algo_public -DoutputTableColNames="petal_length,petal_width,petal_length2,petal_width2"
    -DautoRenameCol="false" -DoutputTableName="pai_temp_770_6840_1" -DinputTableNames="iris_twopartition,iris_twopartition"
    -DinputPartitionsInfoList="dt=20150125/dp=20150124;dt=20150124/dp=20150123" 
    -DselectedColNamesList="petal_length,petal_width;sepal_length,sepal_width";

• name: 組件名字
• project: project名字，用於指定算法所在空間。係統默認是algo_public，用戶自己更改後係統會報錯
• outputTableColNames: 新表中各列的列名。逗號分隔，如果autoRenameCol為true，則此參數無效
• autoRenameCol: （可選）輸出表是否自動重命名列，true為重命名，false不進行重命名，默認false
• outputTableName: 輸出的表名
• inputTableNames：輸入的表名，如果有多個，逗號分隔
• inputPartitionsInfoList：（可選）輸入表對應選擇的partition列表，同一個表的各partition按逗號分隔，不同表的partition分號分隔
• selectedColNamesList：選擇輸入的列名，同張表之間列名用逗號分隔，不同表之間用分號分隔

UNION

將兩張表的數據按行合並，左表及右表選擇輸出的字段個數以及類型應保持一致。整合了union和union all的功能。

參數設置

調整參數，如下：

• 進行聯合操作時，左右表選擇的列數需相同，對應列的類型需保證一致
• 可根據實際情況在條件框中手動輸入已選字段的過濾條件，默認情況下全表，目前操作符支持”=”，”!=”，”>”，”<”， “>=”，“<=”，“like” 和 “rlike”
• 係統默認勾選去重。勾選後將會對生成的數據表的重複行進行去重操作（distinct）左表聯合列右表聯合列

PAI 命令

不提供pai命令

增加序號列

在數據表第一列追加ID列，並將append id後的內容存為新表

參數設置

略

PAI命令

PAI -name AppendId -project algo_public -DIDColName="append_id" -DoutputTableName="test_11" -DinputTableName="bank_data" 
    -DselectedColNames="age,campaign,cons_conf_idx,cons_price_idx,emp_var_rate,euribor3m,nr_employed,pdays,poutcome,previous,y";

• name: 組件名字
• project: project名字，用於指定算法所在空間。係統默認是algo_public，用戶自己更改後係統會報錯
• IDColName：追加的ID列列名，從0開始編號，依次為0,1,2,3…
• outputTableNames: 輸出表的名字
• inputTableNmae: 輸入表的名字
• selectedColNames: 要保留的字段名，多個用逗號分隔

拆分

背景

• 對輸入表或分區進行按比例拆分，分別寫入兩張輸出表。

算法組件

• 拆分組件，對應兩個輸出樁
• 如圖所示配置0.6，則左邊的輸出樁對應60%的數據，右邊對應40%的數據

PAI命令

pai -name split -project algo_public 
    -DinputTableName=wbpc 
    -Doutput1TableName=wpbc_split1 
    -Doutput2TableName=wpbc_split2 
    -Dfraction=0.25;

參數設置

缺失值填充

缺失值填充用來將空值或者一個指定的值替換為最大值，最小值，均值或者一個自定義的值。可以通過給定一個缺失值的配置列表，來實現將輸入表的缺失值用指定的值來填充。

• 可以將數值型的空值替換為最大值，最小值，均值或者一個自定義的值
• 可以將字符型的空值，空字符串，空值和空字符串，指定值替換為一個自定義的值
• 待填充的缺失值可以選擇空值或空字符，也可以自定義
• 缺失值若選擇空字符，則填充的目標列應是string型
• 數值型替換可以自定義，也可以直接選擇替換成數值最大值，最小值或者均值

缺失值填充界麵

兩個輸入樁依次對應參數為：

• inputTableName 輸入表，即要填充的表

• inputParaTableName 配置輸入表，即缺失值填充節點生成的參數列表，通過此參數，可以將一張表的配置參數應用到一張新的表

兩個輸出樁依次對應參數為：

• outputTableName 輸出表，即填充完成的表

• outputParaTableName 輸出配置表，用於應用到其他的數據集上

缺失值填充參數界麵

• 填充的字段，原值，替換為三個部分組成了config參數，分別對應config參數的三個部分：列名，原值，替換值

PAI 命令

PAI -name FillMissingValues -project algo_public -Dconfigs="poutcome,null-empty,testing" 
    -DoutputTableName="test_3" -DinputPartitions="pt=20150501" -DinputTableName="bank_data_partition";

算法參數

實例

測試數據

新建數據SQL

drop table if exists fill_missing_values_test_input;
create table fill_missing_values_test_input(
    col_string string,
    col_bigint bigint,
    col_double double,
    col_boolean boolean,
    col_datetime datetime);
insert overwrite table fill_missing_values_test_input
select
    *
from
(
    select
        '01' as col_string,
        10 as col_bigint,
        10.1 as col_double,
        True as col_boolean,
        cast('2016-07-01 10:00:00' as datetime) as col_datetime
    from dual
    union all
        select
            cast(null as string) as col_string,
            11 as col_bigint,
            10.2 as col_double,
            False as col_boolean,
            cast('2016-07-02 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '02' as col_string,
            cast(null as bigint) as col_bigint,
            10.3 as col_double,
            True as col_boolean,
            cast('2016-07-03 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '03' as col_string,
            12 as col_bigint,
            cast(null as double) as col_double,
            False as col_boolean,
            cast('2016-07-04 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '04' as col_string,
            13 as col_bigint,
            10.4 as col_double,
            cast(null as boolean) as col_boolean,
            cast('2016-07-05 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '05' as col_string,
            14 as col_bigint,
            10.5 as col_double,
            True as col_boolean,
            cast(null as datetime) as col_datetime
        from dual
) tmp;

輸入數據說明

+------------+------------+------------+-------------+--------------+
| col_string | col_bigint | col_double | col_boolean | col_datetime |
+------------+------------+------------+-------------+--------------+
| 04         | 13         | 10.4       | NULL        | 2016-07-05 10:00:00 |
| 02         | NULL       | 10.3       | true        | 2016-07-03 10:00:00 |
| 03         | 12         | NULL       | false       | 2016-07-04 10:00:00 |
| NULL       | 11         | 10.2       | false       | 2016-07-02 10:00:00 |
| 01         | 10         | 10.1       | true        | 2016-07-01 10:00:00 |
| 05         | 14         | 10.5       | true        | NULL         |
+------------+------------+------------+-------------+--------------+

運行命令

drop table if exists fill_missing_values_test_input_output;
drop table if exists fill_missing_values_test_input_model_output;
PAI -name FillMissingValues
-project algo_public
-Dconfigs="col_double,null,mean;col_string,null-empty,str_type_empty;col_bigint,null,max;col_boolean,null,true;col_datetime,null,2016-07-06 10:00:00"
-DoutputParaTableName="fill_missing_values_test_input_model_output"
-Dlifecycle="28"
-DoutputTableName="fill_missing_values_test_input_output"
-DinputTableName="fill_missing_values_test_input";
drop table if exists fill_missing_values_test_input_output_using_model;
drop table if exists fill_missing_values_test_input_output_using_model_model_output;
PAI -name FillMissingValues
-project algo_public
-DoutputParaTableName="fill_missing_values_test_input_output_using_model_model_output"
-DinputParaTableName="fill_missing_values_test_input_model_output"
-Dlifecycle="28"
-DoutputTableName="fill_missing_values_test_input_output_using_model"
-DinputTableName="fill_missing_values_test_input";

運行結果

+------------+------------+------------+-------------+--------------+
| col_string | col_bigint | col_double | col_boolean | col_datetime |
+------------+------------+------------+-------------+--------------+
| 04         | 13         | 10.4       | true        | 2016-07-05 10:00:00 |
| 02         | 14         | 10.3       | true        | 2016-07-03 10:00:00 |
| 03         | 12         | 10.3       | false       | 2016-07-04 10:00:00 |
| str_type_empty | 11         | 10.2       | false       | 2016-07-02 10:00:00 |
| 01         | 10         | 10.1       | true        | 2016-07-01 10:00:00 |
| 05         | 14         | 10.5       | true        | 2016-07-06 10:00:00 |
+------------+------------+------------+-------------+--------------+

+------------+------------+
| feature    | json       |
+------------+------------+
| col_string | {"name": "fillMissingValues", "type": "string", "paras":{"missing_value_type": "null-empty",  "replaced_value": "str_type_empty"}} |
| col_bigint | {"name": "fillMissingValues", "type": "bigint", "paras":{"missing_value_type": "null",  "replaced_value": 14}} |
| col_double | {"name": "fillMissingValues", "type": "double", "paras":{"missing_value_type": "null",  "replaced_value": 10.3}} |
| col_boolean | {"name": "fillMissingValues", "type": "boolean", "paras":{"missing_value_type": "null",  "replaced_value": 1}} |
| col_datetime | {"name": "fillMissingValues", "type": "datetime", "paras":{"missing_value_type": "null",  "replaced_value": 1467770400000}} |
+------------+------------+

+------------+------------+------------+-------------+--------------+
| col_string | col_bigint | col_double | col_boolean | col_datetime |
+------------+------------+------------+-------------+--------------+
| 04         | 13         | 10.4       | true        | 2016-07-05 10:00:00 |
| 02         | 14         | 10.3       | true        | 2016-07-03 10:00:00 |
| 03         | 12         | 10.3       | false       | 2016-07-04 10:00:00 |
| str_type_empty | 11         | 10.2       | false       | 2016-07-02 10:00:00 |
| 01         | 10         | 10.1       | true        | 2016-07-01 10:00:00 |
| 05         | 14         | 10.5       | true        | 2016-07-06 10:00:00 |
+------------+------------+------------+-------------+--------------+

+------------+------------+
| feature    | json       |
+------------+------------+
| col_string | {"name": "fillMissingValues", "type": "string", "paras":{"missing_value_type": "null-empty",  "replaced_value": "str_type_empty"}} |
| col_bigint | {"name": "fillMissingValues", "type": "bigint", "paras":{"missing_value_type": "null",  "replaced_value": 14}} |
| col_double | {"name": "fillMissingValues", "type": "double", "paras":{"missing_value_type": "null",  "replaced_value": 10.3}} |
| col_boolean | {"name": "fillMissingValues", "type": "boolean", "paras":{"missing_value_type": "null",  "replaced_value": 1}} |
| col_datetime | {"name": "fillMissingValues", "type": "datetime", "paras":{"missing_value_type": "null",  "replaced_value": 1467770400000}} |
+------------+------------+

PAI -name Normalize -project algo_public -DkeepOriginal="true" -DoutputTableName="test_4" -DinputPartitions="pt=20150501" 
    -DinputTableName="bank_data_partition" -DselectedColNames="emp_var_rate,euribor3m";

drop table if exists normalize_test_input;
create table normalize_test_input(
    col_string string,
    col_bigint bigint,
    col_double double,
    col_boolean boolean,
    col_datetime datetime);
insert overwrite table normalize_test_input 
select 
    *
from 
(
    select
        '01' as col_string,
        10 as col_bigint,
        10.1 as col_double,
        True as col_boolean,
        cast('2016-07-01 10:00:00' as datetime) as col_datetime
    from dual
    union all
        select
            cast(null as string) as col_string,
            11 as col_bigint,
            10.2 as col_double,
            False as col_boolean,
            cast('2016-07-02 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '02' as col_string,
            cast(null as bigint) as col_bigint,
            10.3 as col_double,
            True as col_boolean,
            cast('2016-07-03 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '03' as col_string,
            12 as col_bigint,
            cast(null as double) as col_double,
            False as col_boolean,
            cast('2016-07-04 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '04' as col_string,
            13 as col_bigint,
            10.4 as col_double,
            cast(null as boolean) as col_boolean,
            cast('2016-07-05 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '05' as col_string,
            14 as col_bigint,
            10.5 as col_double,
            True as col_boolean,
            cast(null as datetime) as col_datetime
        from dual
) tmp;

+------------+------------+------------+-------------+--------------+
| col_string | col_bigint | col_double | col_boolean | col_datetime |
+------------+------------+------------+-------------+--------------+
| 01         | 10         | 10.1       | true        | 2016-07-01 10:00:00 |
| NULL       | 11         | 10.2       | false       | 2016-07-02 10:00:00 |
| 02         | NULL       | 10.3       | true        | 2016-07-03 10:00:00 |
| 03         | 12         | NULL       | false       | 2016-07-04 10:00:00 |
| 04         | 13         | 10.4       | NULL        | 2016-07-05 10:00:00 |
| 05         | 14         | 10.5       | true        | NULL         |
+------------+------------+------------+-------------+--------------+

drop table if exists normalize_test_input_output;
drop table if exists normalize_test_input_model_output;
PAI -name Normalize
-project algo_public
-DoutputParaTableName="normalize_test_input_model_output"
-Dlifecycle="28"
-DoutputTableName="normalize_test_input_output"
-DinputTableName="normalize_test_input"
-DselectedColNames="col_double,col_bigint"
-DkeepOriginal="true";
drop table if exists normalize_test_input_output_using_model;
drop table if exists normalize_test_input_output_using_model_model_output;
PAI -name Normalize
-project algo_public
-DoutputParaTableName="normalize_test_input_output_using_model_model_output"
-DinputParaTableName="normalize_test_input_model_output"
-Dlifecycle="28"
-DoutputTableName="normalize_test_input_output_using_model"
-DinputTableName="normalize_test_input";

+------------+------------+------------+-------------+--------------+-----------------------+-----------------------+
| col_string | col_bigint | col_double | col_boolean | col_datetime | normalized_col_bigint | normalized_col_double |
+------------+------------+------------+-------------+--------------+-----------------------+-----------------------+
| 01         | 10         | 10.1       | true        | 2016-07-01 10:00:00 | 0.0                   | 0.0                   |
| NULL       | 11         | 10.2       | false       | 2016-07-02 10:00:00 | 0.25                  | 0.2499999999999989    |
| 02         | NULL       | 10.3       | true        | 2016-07-03 10:00:00 | NULL                  | 0.5000000000000022    |
| 03         | 12         | NULL       | false       | 2016-07-04 10:00:00 | 0.5                   | NULL                  |
| 04         | 13         | 10.4       | NULL        | 2016-07-05 10:00:00 | 0.75                  | 0.7500000000000011    |
| 05         | 14         | 10.5       | true        | NULL         | 1.0                   | 1.0                   |
+------------+------------+------------+-------------+--------------+-----------------------+-----------------------+

+------------+------------+
| feature    | json       |
+------------+------------+
| col_bigint | {"name": "normalize", "type":"bigint", "paras":{"min":10, "max": 14}} |
| col_double | {"name": "normalize", "type":"double", "paras":{"min":10.1, "max": 10.5}} |
+------------+------------+

+------------+------------+------------+-------------+--------------+
| col_string | col_bigint | col_double | col_boolean | col_datetime |
+------------+------------+------------+-------------+--------------+
| 01         | 0.0        | 0.0        | true        | 2016-07-01 10:00:00 |
| NULL       | 0.25       | 0.2499999999999989 | false       | 2016-07-02 10:00:00 |
| 02         | NULL       | 0.5000000000000022 | true        | 2016-07-03 10:00:00 |
| 03         | 0.5        | NULL       | false       | 2016-07-04 10:00:00 |
| 04         | 0.75       | 0.7500000000000011 | NULL        | 2016-07-05 10:00:00 |
| 05         | 1.0        | 1.0        | true        | NULL         |
+------------+------------+------------+-------------+--------------+

+------------+------------+
| feature    | json       |
+------------+------------+
| col_bigint | {"name": "normalize", "type":"bigint", "paras":{"min":10, "max": 14}} |
| col_double | {"name": "normalize", "type":"double", "paras":{"min":10.1, "max": 10.5}} |
+------------+------------+

PAI -name Standardize -project algo_public -DkeepOriginal="false" -DoutputTableName="test_5" 
    -DinputPartitions="pt=20150501" -DinputTableName="bank_data_partition" -DselectedColNames="euribor3m,pdays";

drop table if exists standardize_test_input;
create table standardize_test_input(
    col_string string,
    col_bigint bigint,
    col_double double,
    col_boolean boolean,
    col_datetime datetime);
insert overwrite table standardize_test_input 
select 
    *
from 
(
    select
        '01' as col_string,
        10 as col_bigint,
        10.1 as col_double,
        True as col_boolean,
        cast('2016-07-01 10:00:00' as datetime) as col_datetime
    from dual
    union all
        select
            cast(null as string) as col_string,
            11 as col_bigint,
            10.2 as col_double,
            False as col_boolean,
            cast('2016-07-02 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '02' as col_string,
            cast(null as bigint) as col_bigint,
            10.3 as col_double,
            True as col_boolean,
            cast('2016-07-03 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '03' as col_string,
            12 as col_bigint,
            cast(null as double) as col_double,
            False as col_boolean,
            cast('2016-07-04 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '04' as col_string,
            13 as col_bigint,
            10.4 as col_double,
            cast(null as boolean) as col_boolean,
            cast('2016-07-05 10:00:00' as datetime) as col_datetime
        from dual
    union all
        select
            '05' as col_string,
            14 as col_bigint,
            10.5 as col_double,
            True as col_boolean,
            cast(null as datetime) as col_datetime
        from dual
) tmp;

+------------+------------+------------+-------------+--------------+
| col_string | col_bigint | col_double | col_boolean | col_datetime |
+------------+------------+------------+-------------+--------------+
| 01         | 10         | 10.1       | true        | 2016-07-01 10:00:00 |
| NULL       | 11         | 10.2       | false       | 2016-07-02 10:00:00 |
| 02         | NULL       | 10.3       | true        | 2016-07-03 10:00:00 |
| 03         | 12         | NULL       | false       | 2016-07-04 10:00:00 |
| 04         | 13         | 10.4       | NULL        | 2016-07-05 10:00:00 |
| 05         | 14         | 10.5       | true        | NULL         |
+------------+------------+------------+-------------+--------------+

drop table if exists standardize_test_input_output;
drop table if exists standardize_test_input_model_output;
PAI -name Standardize
-project algo_public
-DoutputParaTableName="standardize_test_input_model_output"
-Dlifecycle="28"
-DoutputTableName="standardize_test_input_output"
-DinputTableName="standardize_test_input"
-DselectedColNames="col_double,col_bigint"
-DkeepOriginal="true";
drop table if exists standardize_test_input_output_using_model;
drop table if exists standardize_test_input_output_using_model_model_output;
PAI -name Standardize
-project algo_public
-DoutputParaTableName="standardize_test_input_output_using_model_model_output"
-DinputParaTableName="standardize_test_input_model_output"
-Dlifecycle="28"
-DoutputTableName="standardize_test_input_output_using_model"
-DinputTableName="standardize_test_input";

+------------+------------+------------+-------------+--------------+--------------------+--------------------+
| col_string | col_bigint | col_double | col_boolean | col_datetime | stdized_col_bigint | stdized_col_double |
+------------+------------+------------+-------------+--------------+--------------------+--------------------+
| 01         | 10         | 10.1       | true        | 2016-07-01 10:00:00 | -1.2649110640673518 | -1.2649110640683832 |
| NULL       | 11         | 10.2       | false       | 2016-07-02 10:00:00 | -0.6324555320336759 | -0.6324555320341972 |
| 02         | NULL       | 10.3       | true        | 2016-07-03 10:00:00 | NULL               | 0.0                |
| 03         | 12         | NULL       | false       | 2016-07-04 10:00:00 | 0.0                | NULL               |
| 04         | 13         | 10.4       | NULL        | 2016-07-05 10:00:00 | 0.6324555320336759 | 0.6324555320341859 |
| 05         | 14         | 10.5       | true        | NULL         | 1.2649110640673518 | 1.2649110640683718 |
+------------+------------+------------+-------------+--------------+--------------------+--------------------+

+------------+------------+
| feature    | json       |
+------------+------------+
| col_bigint | {"name": "standardize", "type":"bigint", "paras":{"mean":12, "std": 1.58113883008419}} |
| col_double | {"name": "standardize", "type":"double", "paras":{"mean":10.3, "std": 0.1581138830082909}} |
+------------+------------+

+------------+------------+------------+-------------+--------------+
| col_string | col_bigint | col_double | col_boolean | col_datetime |
+------------+------------+------------+-------------+--------------+
| 01         | -1.2649110640673515 | -1.264911064068383 | true        | 2016-07-01 10:00:00 |
| NULL       | -0.6324555320336758 | -0.6324555320341971 | false       | 2016-07-02 10:00:00 |
| 02         | NULL       | 0.0        | true        | 2016-07-03 10:00:00 |
| 03         | 0.0        | NULL       | false       | 2016-07-04 10:00:00 |
| 04         | 0.6324555320336758 | 0.6324555320341858 | NULL        | 2016-07-05 10:00:00 |
| 05         | 1.2649110640673515 | 1.2649110640683716 | true        | NULL         |
+------------+------------+------------+-------------+--------------+

+------------+------------+
| feature    | json       |
+------------+------------+
| col_bigint | {"name": "standardize", "type":"bigint", "paras":{"mean":12, "std": 1.58113883008419}} |
| col_double | {"name": "standardize", "type":"double", "paras":{"mean":10.3, "std": 0.1581138830082909}} |
+------------+------------+

PAI -name type_transform 
    -project algo_public 
    -DinputTable="pai_dense"
    -DselectedCols="gail,loss,work_year" 
    -Dpre_type="double"
    -Dnew_type="bigint" 
    -DoutputTable="pai_temp_2250_20272_1"
    -Dlifecycle="28"