Python – Page 2 – Scriptorium

[Spark By Example] DataFrameReader

DataFrameReader is an interface to load a DataFrame from external sources.

You cannot create the DataFrameReader object, but you can access it through the “SparkSession.read” property.

[Spark By Example] Spark SQL – Grouping

Let’s play with Spark SQL more.

[Note] When the underlying DataFrame schema is changed, the view should be updated again.

[Spark By Example] Spark SQL – TempView

With Spark SQL, you can use the familiar SQL syntax to query the data.

[Spark By Example] Aggregation

The following sample code (by Python and C#) shows how to group DataFrame and compute the aggregated values such as Max, Min, or Average.

[Spark By Example] Date & Timestamp

Handling date and time is one of most important part of data processing. But due to its complex formats, you need to how to convert types and manipulate date and time.

[Spark By Example] DataFrame Columns

The following sample code (by Python and C#) shows how to handle columns in the DataFrame. You can check the number of columns, add a new column, rename the existing column, or even remove columns.

[Spark By Example] DataFrame Query

The following sample code (by Python and C#) shows how to read query data in the DataFrame. Once you have a DataFrame object, you can query the data using the SQL-like syntax regardless of the origin of data.

[Spark By Example] Read JSON – Complex Type

The following sample code (by Python and C#) shows how to read JSON file with complex objects.

CSV handles the flat data structure. But with JSON, you can read complex data structure into the DataFrame.