sampleBy (DataFrame)

Returns a stratified sample without replacement based on the fraction given on each stratum.

Syntax

sampleBy(col: "ColumnOrName", fractions: Dict[Any, float], seed: Optional[int] = None)

Parameters

Parameter	Type	Description
col	Column or str	column that defines strata.
fractions	dict	sampling fraction for each stratum. If a stratum is not specified, we treat its fraction as zero.
seed	int, optional	random seed.

Parameter	Type	Description
col	Column or str	column that defines strata.
fractions	dict	sampling fraction for each stratum. If a stratum is not specified, we treat its fraction as zero.
seed	int, optional	random seed.

Returns

a new DataFrame that represents the stratified sample.

Examples

Python

from pyspark.sql import functions as sf
dataset = spark.range(0, 100, 1, 5).select((sf.col("id") % 3).alias("key"))
sampled = dataset.sampleBy("key", fractions={0: 0.1, 1: 0.2}, seed=0)
sampled.groupBy("key").count().orderBy("key").show()
# +---+-----+
# |key|count|
# +---+-----+
# |  0|    4|
# |  1|    9|
# +---+-----+

dataset.sampleBy(sf.col("key"), fractions={2: 1.0}, seed=0).count()
# 33