IT Lab

$ spark-shell --packages "io.lamma:lamma_2.11:2.3.0"

Ivy Default Cache set to: /home/hadoop/.ivy2/cache

The jars for the packages stored in: /home/hadoop/.ivy2/jars

:: loading settings :: url = jar:file:/usr/lib/spark/lib/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar!/org/apache/ivy/core/settings/ivysettings.xml

joda-time#joda-time added as a dependency

io.lamma#lamma_2.11 added as a dependency

:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0

        confs: [default]

        found joda-time#joda-time;2.9.2 in central

        found io.lamma#lamma_2.11;2.3.0 in central

downloading https://repo1.maven.org/maven2/io/lamma/lamma_2.11/2.3.0/lamma_2.11-2.3.0.jar ...

        [SUCCESSFUL ] io.lamma#lamma_2.11;2.3.0!lamma_2.11.jar (68ms)

:: resolution report :: resolve 1799ms :: artifacts dl 78ms

        :: modules in use:

        io.lamma#lamma_2.11;2.3.0 from central in [default]

        joda-time#joda-time;2.9.2 from central in [default]

        ---------------------------------------------------------------------

        |                  |            modules            ||   artifacts   |

        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|

        ---------------------------------------------------------------------

        |      default     |   2   |   1   |   1   |   0   ||   2   |   1   |

        ---------------------------------------------------------------------

:: retrieving :: org.apache.spark#spark-submit-parent

        confs: [default]

        1 artifacts copied, 1 already retrieved (338kB/13ms)

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0

      /_/

Using Scala version 2.10.5 (OpenJDK 64-Bit Server VM, Java 1.7.0_91)

Type in expressions to have them evaluated.

Type :help for more information.

16/02/11 05:49:39 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.

Spark context available as sc.

Thu Feb 11 05:49:54 UTC 2016 Thread[main,5,main] java.io.FileNotFoundException: derby.log (Permission denied)

----------------------------------------------------------------

Thu Feb 11 05:49:55 UTC 2016:

Booting Derby version The Apache Software Foundation - Apache Derby - 10.10.1.1 - (1458268): instance a816c00e-0152-cee0-f31c-000025557508

on database directory /mnt/tmp/spark-9e5a20ed-acea-4dc8-ab4b-5803c4a3dca9/metastore with class loader sun.misc.Launcher$AppClassLoader@753d556f

Loaded from file:/usr/lib/spark/lib/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar

java.vendor=Oracle Corporation

java.runtime.version=1.7.0_91-mockbuild_2015_10_27_19_01-b00

user.dir=/etc/spark/conf.dist

os.name=Linux

os.arch=amd64

os.version=4.1.13-19.30.amzn1.x86_64

derby.system.home=null

Database Class Loader started - derby.database.classpath=''

16/02/11 05:50:04 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0

16/02/11 05:50:04 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException

SQL context available as sqlContext.

scala> import io.lamma.Date

import io.lamma.Date

scala> Date("2016-02-01") to Date("2016-02-11") foreach println

Date(2016,2,1)

Date(2016,2,2)

Date(2016,2,3)

Date(2016,2,4)

Date(2016,2,5)

Date(2016,2,6)

Date(2016,2,7)

Date(2016,2,8)

Date(2016,2,9)

Date(2016,2,10)

Date(2016,2,11)

scala>