Tag Archives: Spark

Spark can infer the data structure, but you can explicitly specify the data by providing the Schema to the DataFrame.

Continue reading “[Spark By Example] Schema”

The following sample code (by Python and C#) shows how to use SparkSession.

SparkSession

• SparkSession is an entry point to your spark application since Spark version 2.
• SparkSession wraps all different contexts (SparkContext, SQLContext, HiveContext, …) to a single entry point.
• You can create as many SparkSessions as you want.
• In the Spark shell, such as PySpark shell, the SparkSession object (named as “spark”) is created for you.
• In the application, you need to create a SparkSession object.

Continue reading “[Spark By Example] SparkSession”

In Spark, you dealing with array or map data as they come. The following example shows how to convert collection data to rows and vice versa.

Continue reading “[Spark By Example] Explode and Collect”

In Spark SQL, you can define your custom functions and use them in the SQL statement. The following example shows how to create a very simple UDF, register it, and use it in the SQL.

Continue reading “[Spark By Example] Spark SQL – UDFs”

DataFrameReader is an interface to load a DataFrame from external sources.

You cannot create the DataFrameReader object, but you can access it through the “SparkSession.read” property.

Continue reading “[Spark By Example] DataFrameReader”

Let’s play with Spark SQL more.

[Note] When the underlying DataFrame schema is changed, the view should be updated again.

Continue reading “[Spark By Example] Spark SQL – Grouping”

With Spark SQL, you can use the familiar SQL syntax to query the data.

Continue reading “[Spark By Example] Spark SQL – TempView”