Loading...

Nar Kumar Chhantyal - Building Data Workflows with Luigi and Kubernetes

239 views

Loading...

Loading...

Transcript

The interactive transcript could not be loaded.

Loading...

Rating is available when the video has been rented.
This feature is not available right now. Please try again later.
Published on Sep 23, 2019

"Building Data Workflows with Luigi and Kubernetes
[EuroPython 2019 - Talk - 2019-07-11 - PyCharm]
[Basel, CH]

By Nar Kumar Chhantyal

This talk will focus on how one can build complex data pipelines in Python. I will introduce Luigi and show how it solves problems while running multiple chain of batch jobs like dependency resolution, workflow management, visualisation, failure handling etc.

After that, I will present how to package Luigi pipelines as Docker image for easier testing and deployment. Finally, I will go through way to deploy them on Kubernetes cluster, thus making it possible to scale Big Data pipelines on-demand and reduce infrastructure costs. I will also give tips and tricks to make Luigi Scheduler play well with Kubernetes batch execution feature.

This talk will be accompanied by demo project. It will be very beneficial for audience who have some experience in running batch jobs (not necessarily in Python), typically people who work in Big Data sphere like data scientists, data engineers, BI devs and software developers. Familiarity with Python is helpful but not needed.



License: This video is licensed under the CC BY-NC-SA 3.0 license: https://creativecommons.org/licenses/...
Please see our speaker release agreement for details: https://ep2019.europython.eu/events/s...

Loading...

When autoplay is enabled, a suggested video will automatically play next.

Up next


to add this to Watch Later

Add to

Loading playlists...