%md ## How to Process IoT Device JSON Data Using Datasets

%md Datasets in Apache Spark 2 and 3 provide high-level domain specific APIs as well as provide structure and compile-time type safety. You can read your
JSON data file into a DataFrame, which is essentially a generic row of JVM objects, and convert them into a type-specific collection of JVM objects. 

This notebook shows you how to read a JSON file, convert your semi-structured JSON data into a collection of Datasets[T], where `T` is a class, and provides an introduction to some high-level Spark Dataset APIs.

%md ####Reading JSON as a Dataset

%md Use the Scala case class `DeviceIoTData` to convert the JSON device data into a Scala object. In addition to the IP address, the file contains the geographic information for each device entry:
* ISO-3166-1 two and three letter codes
* Country Name
* Latitude and longitude

For each IP address, you can obtain these attributes from a webservice at https://ipstack.com/. With these attributes, you can create map visualizations of the data.

`{"device_id": 198164, "device_name": "sensor-pad-198164owomcJZ", "ip": "80.55.20.25", "cca2": "PL", "cca3": "POL", "cn": "Poland", "latitude": 53.080000, "longitude": 18.620000, "scale": "Celsius", "temp": 21, "humidity": 65, "battery_level": 8, "c02_level": 1408, "lcd": "red", "timestamp" :1458081226051 }`

%md Create a case class to represent your IoT Device Data

%md Count the number of devices

%md Display your Dataset

%md #### Iterating, transforming, and filtering Dataset

%md Iterate over the first 10 entries with the `foreach()` method and print them

%md  For all relational expressions, Apache Spark formulates an optimized logical and physical plan for execution, and optimizes the generated code. For our `DeviceIoTData`, it will use its standard encoders to optimize its binary internal representation, hence decrease the size of generated code, minimize the bytes transferred over the networks between nodes, and execute faster.

First filter the device dataset on `temp` and `humidity` attributes with a predicate.

%md Use Dataset APIs for filtering: `take(10)` returns an `Array[DeviceIoTData]`; using a `foreach()` method on the Array collection, print each item.

%md Filter out dataset using the high-level and readable method `where()`. `filter()` and `where()` are equivalent.

%md Both `where()` and `map()` return a Dataset

%md Use the `filter()` method that is equivalent as the `where()` method used above.

%md Select individual fields using the Dataset method `select()` where battery_level is greater than 6, sort in ascending order on C02_level. This high-level domain specific language API reads like a SQL query.

%md Apply higher-level Dataset API methods such as `groupBy()` and `avg()`. In order words, take all temperatures readings > 25, along with their corresponding devices' humidity, groupBy ccca3 country code, and compute averages. Plot the resulting Dataset.

%md #### Visualizing datasets

%md

Data without visualization without a narrative arc, to infer insights or to see a trend, is useless. By saving a Dataset as a temporary table, you can issue complex SQL queries against it and visualize the results, using notebook's myriad plotting options.