mac上安装单机版Spark

下载安装测试

下载地址:
http://spark.apache.org/downloads.html

1. 点击下载地址,现在最新版本是spark-2.0.1-bin-hadoop2.7.tgz ,大概180MB.
2. 解压下载文件,并将这个文件放大你想要的位置.
3. 然后这就算安装好了..
   mv spark-2.0.1-bin-hadoop /usr/local/
4. 测试

--进入安装目录

cd /usr/local/spark

--启动

./sbin/start-master.sh

--打开浏览器输入,查看主机运行情况

http://localhost:8080

--加子嗣(现在主机是司令,要给他派点兵.)

./bin/spark-class org.apache.spark.deploy.worker.Worker spark://IP:PORT

--这是后台启动

nohup spark-class org.apache.spark.deploy.worker.Worker spark:/192.168.70.145:7077 &

--回到浏览器

http://localhost:8080

在“Workers”列表下赫然出现了你的第一个子嗣。其状态State为ALIVE。表示它正在运作。这时候你需要记住，当前Terminal的窗口对应就是那个刚加入的Worker Id。

删除子嗣

工作完成后吧小崽子们放出去玩,只需要在哪个看似死机的teminal 里按下CTRL+C

关闭主机

./sbin/stop-master.sh

WordCount脚本

进入spark-shell

./bin/spark-shell

import org.apache.spark.SparkConf

import org.apache.spark.SparkContext

import org.apache.spark.SparkContext._

var conf=new SparkConf().setMaster("local").setAppName("MyApp")

val sc=new SparkContext(conf)

val input=sc.textFile("README.md") //这里可以设置为绝对路径

val words=input.flatMap(line=>line.split(" "))

val counts=words.map(word=>(word,1)).reduceByKey{case (x,y)=>x+y}

counts.saveAsTextFile(outputFile) //这里可以设置为绝对路径

在Intellij IDEA里执行WordCount

import org.apache.spark.SparkContext

import org.apache.spark.SparkContext._

import org.apache.spark.SparkConf

import org.scalatest._

object SimpleApp {

def main(args: Array[String]) {

var logFile="/usr/local/spark/README.md"

var conf=new SparkConf().setAppName("Simple Application").setMaster("local")

var sc=new SparkContext(conf)

val data=Array(1,2,3,4,5)

val distData=sc.parallelize(data)

val sumArray=distData.reduce((a,b)=>a+b)

val logData=sc.textFile(logFile,2).cache()

val numAs=logData.filter(line=>line.contains("a")).count()

val numBs=logData.filter(line=>line.contains("b")).count()

println("Lines with a:%s,Lines with b:%s".format(numAs,numBs))

println("sumArray is:"+sumArray)

}

}