SQLContext

SQLContext

The entry point for working with structured data (rows and columns) in Spark. Allows the creation of DataFrame objects as well as the execution of SQL queries.

Constructor

new SQLContext()

Note: Do not use directly. Access via sqlContext.

Source:

Methods

createDataFrame(jsonArray)

Creates a DataFrame from an array of rows, represented as javascript objects. These objects should be serializable and deserializable to/from json. This function goes through the input once to determine the input schema.

Parameters:
Name Type Description
jsonArray

Array of JSON objects.
Since:
  • 1.3.0
Source:

emptyDataFrame()

Returns a DataFrame with no rows or columns.

Since:
  • 1.3.0
Source:

range(start_or_end, endopt, stepopt)

Creates a DataFrame with a single LongType column named id, containing elements in a range with step value 1. If end is provided, the range is from start_or_end to end. Otherwise it is from 0 to start_or_end.

Parameters:
Name Type Attributes Default Description
start_or_end

Start (if end is provided) or end of range.
end <optional>
 null

End of range.
step <optional>
 1

Step.
Since:
  • 1.4.1
Source:

read()

Returns a DataFrameReader that can be used to read data in as a DataFrame.

Since:
  • 1.4.0
Source:

sql(sqlText)

Executes a SQL query using Spark, returning the result as a DataFrame. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.

Parameters:
Name Type Description
sqlText

SQL query.
Since:
  • 1.3.0
Source: