The following sample code (by Python and C#) shows how to read query data in the DataFrame. Once you have a DataFrame object, you can query the data using the SQL-like syntax regardless of the origin of data.
Continue reading “[Spark By Example] DataFrame Query”[Spark By Example] Read JSON – Complex Type
The following sample code (by Python and C#) shows how to read JSON file with complex objects.
CSV handles the flat data structure. But with JSON, you can read complex data structure into the DataFrame.
Continue reading “[Spark By Example] Read JSON – Complex Type”[Spark By Example] Read JSON – Array Type
The following sample code (by Python and C#) shows how to read JSON file with array data.
With JSON, it is easy to specify the schema. Therefore, you can directly parse the array data into the DataFrame.
Continue reading “[Spark By Example] Read JSON – Array Type”[Spark By Example] Read JSON with Schema
The following sample code (by Python and C#) shows how to read JSON file with schema. With JSON, it is always a good idea to provide the schema for your data.
Continue reading “[Spark By Example] Read JSON with Schema”[Spark By Example] Read CSV – Array Type
The following sample code (by Python and C#) shows how to read CSV file with a column of array.
CSV does not support complex objects such as an array. To make it work, you need to pass a JSON array in CSV and parse it. You will learn this trick here.
JSON array format is like this.
["Ford", "Toyota", "BMW", "Fiat"]
[Spark By Example] Read CSV with Schema
The following sample code (by Python and C#) shows how to read CSV file with schema. You will see how you can specify the schema explicitly.
Continue reading “[Spark By Example] Read CSV with Schema”[Spark By Example] Read CSV
The following sample code (by Python and C#) shows how to read CSV file without schema.
Continue reading “[Spark By Example] Read CSV”[Spark By Example] Word Count
The following sample code (by Python and C#) shows how to count the word in a text file.
Continue reading “[Spark By Example] Word Count”