JSON file

You can read JSON files in single-line or multi-line mode. In single-line mode, a file can be split into many parts and read in parallel. In multi-line mode, a file is loaded as a whole entity and cannot be split.

For further information, see JSON Files.

Options

See the following Apache Spark reference articles for supported read and write options.

Examples

Single-line mode

In this example, there is one JSON object per line:

{"string":"string1","int":1,"array":[1,2,3],"dict": {"key": "value1"}}
{"string":"string2","int":2,"array":[2,4,6],"dict": {"key": "value2"}}
{"string":"string3","int":3,"array":[3,6,9],"dict": {"key": "value3", "extra_key": "extra_value3"}}

To read the JSON data, use:

val df = spark.read.json("example.json")

Spark infers the schema automatically.

df.printSchema
root
 |-- array: array (nullable = true)
 |    |-- element: long (containsNull = true)
 |-- dict: struct (nullable = true)
 |    |-- extra_key: string (nullable = true)
 |    |-- key: string (nullable = true)
 |-- int: long (nullable = true)
 |-- string: string (nullable = true)

Multi-line mode

This JSON object occupies multiple lines:

[
  {"string":"string1","int":1,"array":[1,2,3],"dict": {"key": "value1"}},
  {"string":"string2","int":2,"array":[2,4,6],"dict": {"key": "value2"}},
  {
    "string": "string3",
    "int": 3,
    "array": [
        3,
        6,
        9
    ],
    "dict": {
        "key": "value3",
        "extra_key": "extra_value3"
    }
  }
]

To read this object, enable multi-line mode:

CREATE TEMPORARY VIEW multiLineJsonTable
USING json
OPTIONS (path="/tmp/multi-line.json",multiline=true)
val mdf = spark.read.option("multiline", "true").json("/tmp/multi-line.json")
mdf.show(false)

Charset auto-detection

By default, the charset of input files is detected automatically. You can specify the charset explicitly using the charset option:

spark.read.option("charset", "UTF-16BE").json("fileInUTF16.json")

Some supported charsets include: UTF-8, UTF-16BE, UTF-16LE, UTF-16, UTF-32BE, UTF-32LE, UTF-32. For the full list of charsets supported by Oracle Java SE, see Supported Encodings.

Notebook

The following notebook demonstrates single line and multi-line mode.

Read JSON files notebook

Open notebook in new tab