%md

# **Chart and Graph Types with Scala**

This notebook covers the various charts and graphs that are built into Databricks.

While Scala is used to generate the test data displayed in the visualizations in this notebook, all the information about how to configure these charts & graphs applies to all notebooks.

%md  ### A **Table View** is the most basic way to view data.

* Only the first 1000 rows will be displayed in the table view.

%md ### Tables in Databricks Cloud are configured with **Plot Options...**.

* The **Keys** section is for specifying the control variable which is typically displayed as the X-Axis on many of the graph types.  Most graphs can plot about 1000 values for the keys, but again - it varies for different graphs.

* The **Values** section is for specifying the observed variable and is typically displayed on the Y-Axis.  This also tends to be an observed numerical value on most graph types.

* The **Series groupings** section is for specifying ways to break out the data - for a bar graph - each series grouping has a different color for the bars with a legend to denote that value of each series grouping.  Many of the graph types can only handle series groupings that has 10 or less unique values. 

**Some graph types also specifying even more options - and those will be discussed as applicable.**

%md ### A **Pivot Table** is another way to view data in a table format.

Instead of just returning the raw results of the table - it can automatically sort, count total or give the average of the data stored in the table.

* Read more about Pivot Tables here: http://en.wikipedia.org/wiki/Pivot_table

* For a Pivot Table, key, series grouping and value fields can be specified.   

* The **Key** is the first column, and there will be one row per key in the Pivot Table.

* There will be additional column for each unique value for the **Series Grouping**.

* The table will contain the **Values** field in the cells.  Value must be a numerical field that can be combined using aggregation functions.

* Cell in the Pivot Table are calculated from multiple rows of the original table.

  * Select **SUM**, **AVG**, **MIN**, **MAX**, or **COUNT** as the way to combine the original rows into that cell.

* Pivoting is done on the server side of Databricks Cloud to calculate the cell values.

%md

To create a Pivot Table, click on the Graph Icon below a result and select **Pivot**:

![plot-menu-pick-pivot](https://docs.databricks.com/_static/images/notebooks/plot-menu-pick-pivot.png)

%md

Another way to think of a pivot table is that it does a group by on your original table by the key & series grouping, but instead of outputting (key, series_grouping, aggregation_function(value)) tuples, it outputs a table where the schema is the key and every unique value for the series grouping.

* See the results of group_by statement below, which contains all the data that is in the pivot table above.

%md ### A **Bar Chart** is a type of visual pivot table graph and a great basic way to visualize data.

* **Plot Options...** was used to configure the graph below.

* The **Key** is *Year* and appears on the X-Axis.

* The **Series groupings** is *Category* and there is a different color to denote each of those.

* The **Values** is *salesAmount* and appears on the Y-Axis.

* **Sum** was selected as the aggregation method, which means sales amounts are summed for all the products in the category for that year.

%md **Tip:** Hover over each bar in the chart below to see the exact values plotted.

%md ### A **Line Graph** is another example of a pivot table graph that can highlight trends for your data set.

* **Plot Options...** was used to configure the graph below.

* The **Key** is *Year* and appears on the X-Axis.

* The **Series groupings** is *Product* and there is different color to denote each of those.

* The **Values** is *salesAmount* and appears on the Y-Axis.

* **Sum** was selected as the aggregation method, which means sales amounts are summed for all the products in the category for that year.

%md ### A **Pie Chart** is pivot table graph type that can allow you to see what percentage of the whole your values represent.

* **NOTE:** As opposed to the previous examples, Key & Series Groupings have been switched.

* **Plot Options...** was used to configure the graph below.

* The **Key** is *Category* and one color is used for each product.

* The **Series groupings** is *Year* and there is different pie chart for each year.

* The **Values** is *salesAmount* and is used to calculate the percentage of the pie.

* **Sum** is selected as the aggregation method.

%md ### A **Map Graph** is a way to visualize your data on a map.

* **Plot Options...** was used to configure the graph below.

* **Keys** should contain the field with the location.

* **Series groupings** is always ignored for World Map graphs.

* **Values** should contain exactly one field with a numerical value.

* Since there can multiple rows with the same location key, choose "Sum", "Avg", "Min", "Max", "COUNT" as the way to combine the values for a single key.

* Different values are denoted by color on the map, and ranges are always spaced evenly.

  * Apply a smoothing function to your graph if your values are not evenly distributed.

%md To plot just a graph of the US by state, use US state postal codes as the key.

%md

To plot a graph of the world, use [country codes in ISO 3166-1 alpha-3 format](http://en.wikipedia.org/wiki/ISO_3166-1_alpha-3) as the key.

%md ### A **Scatter Plot** allows you to see if there is a correlation between two variables.

* **Plot Options...** was selected to configure the graph below.

* **Keys** will be used to color the points on the graphs - with a legend on the side.

* **Series Grouping** is ignored.

* **Value** must contain at least two numberical fields.  This graph has a, b, and c as the values.

* The diagonal of the resulting plot is the kernal density plot of the variable.

* The row always has the variable in the Y-Axis, and the column has the variable in the X-Axis.

%md #### LOESS Fit Curves for Scatter Plots

[LOESS](https://en.wikipedia.org/wiki/Local_regression) is a method of performing local regression on your data to produce a smooth estimation curve that describes the data trend of your scatter plot. It does this by interpolating a curve within its neighborhood of data points. The LOESS fit curve is controlled by a bandwidth parameter that specifies how many neighboring points should be used to smooth the plot. A high bandwidth parameter (close to 1) gives a very smooth curve that may miss the general trend, while a low bandwidth parameter (close to 0) does not smooth the plot much.

LOESS fit curves are now available for scatter plots. Here is an example of how you can create a LOESS fit for your scatter plots.

**NOTE:** If your dataset has more than 5000 data points, the LOESS fit is computed using the first 5000 points.

%md You can turn this data into a scatter plot using the controls on the bottom left of the display table.

![plot-menu-pick-scatter](https://docs.databricks.com/_static/images/notebooks/plot-menu-pick-scatter.png)

You can now access the LOESS fit option when you select *Plot Options*:

![screen shot 2015-10-13 at 3 43 16 pm](https://cloud.githubusercontent.com/assets/7594753/10472058/d7ce763c-71d0-11e5-91b2-4d90e9a704c9.png)

You can experiment with the bandwith parameter to see how the curve adapts to noisy data.

Once you accept the change, you will see the LOESS fit on your scatter plot!

%md ### A **Histogram** allows you to determine the distribution of values.

* **Plot Options...** was selected to configure the graph below.

* **Value** should contain exactly one field.

* **Series Grouping** is always ignored.

* **Keys** can support up to 2 fields.

  * When no key is specified, exactly one histogram is output.

  * When 2 fields are specified, then there is a trellis of histograms.

* **Aggregation** is not applicable.

* **Number of bins** is a special option that appears only for histogram plots, and controls the number of bins in the histogram.

* Bins are computed on the serverside for histograms, so it can plot all the rows in a table.