Talk Python to Me: #519: Data Science Cloud Lessons at Scale

Today on Talk Python: What really happens when your data work outgrows your laptop. Matthew Rocklin, creator of Dask and cofounder of Coiled, and Nat Tabris a staff software engineer at Coiled join me to unpack the messy truth of cloud-scale Python. During the episode we actually spin up a 1,000 core cluster from a notebook, twice! We also discuss picking between pandas and Polars, when GPUs help, and how to avoid surprise bills. Real lessons, real tradeoffs, shared by people who have built this stuff. Stick around.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/seer'>Seer: AI Debugging, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <h2 class="links-heading">Links from the show</h2> <div><strong>Matthew Rocklin</strong>: <a href="https://x.com/mrocklin?featured_on=talkpython" target="_blank" >@mrocklin</a><br/> <strong>Nat Tabris</strong>: <a href="https://tabris.us?featured_on=talkpython" target="_blank" >tabris.us</a><br/> <br/> <strong>Dask</strong>: <a href="https://www.dask.org?featured_on=talkpython" target="_blank" >dask.org</a><br/> <strong>Coiled</strong>: <a href="https://coiled.io?featured_on=talkpython" target="_blank" >coiled.io</a><br/> <strong>Watch this episode on YouTube</strong>: <a href="https://www.youtube.com/watch?v=omBibVGLzyo" target="_blank" >youtube.com</a><br/> <strong>Episode #519 deep-dive</strong>: <a href="https://talkpython.fm/episodes/show/519/data-science-cloud-lessons-at-scale#takeaways-anchor" target="_blank" >talkpython.fm/519</a><br/> <strong>Episode transcripts</strong>: <a href="https://talkpython.fm/episodes/transcript/519/data-science-cloud-lessons-at-scale" target="_blank" >talkpython.fm</a><br/> <strong>Developer Rap Theme Song: Served in a Flask</strong>: <a href="https://talkpython.fm/flasksong" target="_blank" >talkpython.fm/flasksong</a><br/> <br/> <strong>--- Stay in touch with us ---</strong><br/> <strong>Subscribe to Talk Python on YouTube</strong>: <a href="https://talkpython.fm/youtube" target="_blank" >youtube.com</a><br/> <strong>Talk Python on Bluesky</strong>: <a href="https://bsky.app/profile/talkpython.fm" target="_blank" >@talkpython.fm at bsky.app</a><br/> <strong>Talk Python on Mastodon</strong>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <strong>Michael on Bluesky</strong>: <a href="https://bsky.app/profile/mkennedy.codes?featured_on=talkpython" target="_blank" >@mkennedy.codes at bsky.app</a><br/> <strong>Michael on Mastodon</strong>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

https://talkpython.fm/episodes/show/519/data-science-cloud-lessons-at-scale