944      技術社區[雲棲]

Spark + HBase__Spark_開發人員指南_E-MapReduce-阿裏雲

Spark + Hbase

Spark 接入 Hbase

下麵這個例子演示了 Spark 如何向 Hbase 寫數據。需要指出的是，計算集群需要和 Hbase 集群處於一個安全組內，否則網絡無法打通。在 E-Mapreduce 創建集群時，請注意選擇 Hbase 集群所處的安全組。

    object ConnectionUtil extends Serializable {
      private val conf = HBaseConfiguration.create()
      conf.set(HConstants.ZOOKEEPER_QUORUM,"ecs1,ecs1,ecs3")
      conf.set(HConstants.ZOOKEEPER_ZNODE_PARENT, "/hbase")
      private val connection = ConnectionFactory.createConnection(conf)
      def getDefaultConn: Connection = connection
    }
    //創建數據流 unionStreams
    unionStreams.foreachRDD(rdd => {
      rdd.map(bytes => new String(bytes))
        .flatMap(line => line.split(" "))
        .map(word => (word, 1))
        .reduceByKey(_ + _)
        .mapPartitions {words => {
          val conn = ConnectionUtil.getDefaultConn
          val tableName = TableName.valueOf(tname)
          val t = conn.getTable(tableName)
          try {
            words.sliding(100, 100).foreach(slice => {
              val puts = slice.map(word => {
                println(s"word: $word")
                val put = new Put(Bytes.toBytes(word._1 + System.currentTimeMillis()))
                put.addColumn(COLUMN_FAMILY_BYTES, COLUMN_QUALIFIER_BYTES,
                  System.currentTimeMillis(), Bytes.toBytes(word._2))
                put
              }).toList
              t.put(puts)
            })
          } finally {
            t.close()
          }
          Iterator.empty
        }}.count()
    })
    ssc.start()
    ssc.awaitTermination()

附錄

完整示例代碼請看: