Inner join drops null values example(Python)

Loading...

Create sample table orders.

df_orders = spark.createDataFrame([('Nissan','Altima','2-door 2.5 S Coupe'), ('Nissan','Altima','4-door 3.5 SE Sedan'), ('Nissan','Altima',''), ('Nissan','Altima', None)], ["Company", "Model", "Info"])

Display sample table orders.

display(df_orders)
 
Company
Model
Info
1
2
3
4
Nissan
Altima
2-door 2.5 S Coupe
Nissan
Altima
4-door 3.5 SE Sedan
Nissan
Altima
Nissan
Altima
null

Showing all 4 rows.

Create sample table models.

df_models = spark.createDataFrame([('Nissan','Altima',''), ('Nissan','Altima','2-door 2.5 S Coupe'), ('Nissan','Altima','2-door 3.5 SE Coupe'), ('Nissan','Altima','4-door 2.5 S Sedan'), ('Nissan','Altima','4-door 3.5 SE Sedan'), ('Nissan','Altima','4-door 3.5 SL Sedan'), ('Nissan','Altima','4-door HYBRID Sedan'), ('Nissan','Altima',None)], ["Company", "Model", "Info"])

Display sample table models.

display(df_models)
 
Company
Model
Info
1
2
3
4
5
6
7
8
Nissan
Altima
Nissan
Altima
2-door 2.5 S Coupe
Nissan
Altima
2-door 3.5 SE Coupe
Nissan
Altima
4-door 2.5 S Sedan
Nissan
Altima
4-door 3.5 SE Sedan
Nissan
Altima
4-door 3.5 SL Sedan
Nissan
Altima
4-door HYBRID Sedan
Nissan
Altima
null

Showing all 8 rows.

Prepare for join.

df_orders.createOrReplaceTempView("Orders")
df_models.createOrReplaceTempView("Models")

Perform a standard inner join and the resulting table does not display any null values.

%sql
SELECT * 
FROM Orders a
INNER JOIN Models b 
ON a.Company = b.Company 
AND a.Model = b.Model 
AND a.Info = b.Info
 
Company
Model
Info
Company
Model
Info
1
2
3
Nissan
Altima
Nissan
Altima
Nissan
Altima
2-door 2.5 S Coupe
Nissan
Altima
2-door 2.5 S Coupe
Nissan
Altima
4-door 3.5 SE Sedan
Nissan
Altima
4-door 3.5 SE Sedan

Showing all 3 rows.

Perform an inner join using the null safe operator. The resulting table correctly displays null values.

%sql
SELECT * 
FROM Orders a
INNER JOIN Models b 
ON a.Company = b.Company 
AND a.Model = b.Model 
AND a.Info <=> b.Info
 
Company
Model
Info
Company
Model
Info
1
2
3
4
Nissan
Altima
Nissan
Altima
Nissan
Altima
null
Nissan
Altima
null
Nissan
Altima
2-door 2.5 S Coupe
Nissan
Altima
2-door 2.5 S Coupe
Nissan
Altima
4-door 3.5 SE Sedan
Nissan
Altima
4-door 3.5 SE Sedan

Showing all 4 rows.