Solución:
Necesitas lanzar la columna low a la fecha de la clase y luego puedes usar datediff() en combinación con lit(). Utilizando Spark 2.2:
from pyspark.sql.functions import datediff, to_date, lit
df.withColumn("test",
datediff(to_date(lit("2017-05-02")),
to_date("low","yyyy/MM/dd"))).show()
+----------+----+------+-----+
| low|high|normal| test|
+----------+----+------+-----+
|1986/10/15| z| null|11157|
|1986/10/15| z| null|11157|
|1986/10/15| c| null|11157|
|1986/10/15|null| null|11157|
|1986/10/16|null| 4.0|11156|
+----------+----+------+-----+
Utilizando low columna a clase timestamp primero:
from pyspark.sql.functions import datediff, to_date, lit, unix_timestamp
df.withColumn("test",
datediff(to_date(lit("2017-05-02")),
to_date(unix_timestamp('low', "yyyy/MM/dd").cast("timestamp")))).show()
Alternativamente, cómo encontrar la cantidad de días transcurridos entre las acciones de dos usuarios posteriores usando pySpark:
import pyspark.sql.functions as funcs
from pyspark.sql.window import Window
window = Window.partitionBy('user_id').orderBy('action_date')
df = df.withColumn("days_passed", funcs.datediff(df.action_date,
funcs.lag(df.action_date, 1).over(window)))
+----------+-----------+-----------+
| user_id|action_date|days_passed|
+----------+-----------+-----------+
|623 |2015-10-21| null|
|623 |2015-11-19| 29|
|623 |2016-01-13| 59|
|623 |2016-01-21| 8|
|623 |2016-03-24| 63|
+----------+----------+------------+
¡Haz clic para puntuar esta entrada!
(Votos: 0 Promedio: 0)