this post was submitted on 07 Jul 2024
6 points (87.5% liked)

Data Engineering

379 readers
2 users here now

A community for discussion about data engineering

Icon base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 1 year ago
MODERATORS
ericjmorey@programming.dev
6
A guide how to adopt an existing Spark scala library for Spark Connect (semyonsinchenko.github.io)
submitted 4 months ago by sem@lemmy.ml to c/data_engineering@programming.dev
0 comments fedilink hide all child comments
 

Let me share my post with a detailed step by step guide how an exisiting Spark scala library may be adopted to work with recently introduced Spark Connect. As an example I have chosen a pupular open source data quality tool AWS Deequ. I made all the necessary protobuf messages and a Spark Connect Plugin. I tested it from PySpark Connect 3.5.1 and it works. Of course, all the code is public in git.

no comments (yet)
sorted by: hot top controversial new old
there doesn't seem to be anything here