Data Scientist's Dev Environment: pyenv & poetry on Ubuntu

Abstract: As a data scientist, your development environment plays a crucial role in your productivity. In this tutorial, we’ll guide you through setting up a powerful and efficient Python development environment using pyenv and poetry. Additionally, we’ll provide a comparison with the widely used conda environment manager to help you make an informed choice.

Keywords: Python Environment, pyenv Setup, poetry Usage, Ubuntu Development, Dependency Management, Coding Productivity

Introduction

In our comprehensive guide ‘A Comprehensive Guide to Supervised Machine Learning,’ we delved deeply into supervised ML algorithms, exploring theory and applications extensively. Additionally, in the captivating journey through ‘Unsupervised Machine Learning Algorithms,‘ we ventured into this intriguing realm, uncovering data’s latent patterns and gaining insights without the crutch of labels.

Now, as we shift our focus to this tutorial, it’s time to get our hands dirty. In the comprehensive guide, we’ve talked a lot about theory, but now it’s time to translate that knowledge into action. We’ll dive into setting up a powerful Python development environment using pyenv and poetry on Ubuntu. These tools will empower us to efficiently manage Python versions, isolate environments, streamline dependency management, and execute scripts within projects.

Benefits of pyenv and poetry

pyenv:

Python Version Management: pyenv allows you to install and switch between multiple Python versions effortlessly, ensuring compatibility for various projects.
Isolated Environments: You can create isolated virtual environments using pyenv to prevent conflicts between different project dependencies.
Global and Local Configuration: Set a default Python version globally while customizing versions on a per-project basis.
Simplified Switching: Easily switch between Python versions without affecting the system or other projects.

poetry:

Dependency Management: poetry streamlines package management by maintaining dependencies in a pyproject.toml file, promoting reproducibility.
Integrated Virtual Environments: poetry automatically manages virtual environments for each project, ensuring clean isolation.
Convenient Script Execution: Run scripts within the virtual environment created by poetry, ensuring the correct dependencies are used.

Comparing with Conda

conda:

Python and Beyond: While pyenv focuses on Python version management, conda is a versatile package and environment manager that supports multiple languages.
Cross-Platform and Extensive Libraries: conda excels in managing complex environments with different libraries across platforms.
Package Channels: conda provides access to a vast number of pre-built packages through its extensive package channels.
Anaconda vs Miniconda: Choose between Anaconda, which comes with a larger package collection out of the box, and Miniconda, which offers a lightweight installation with package flexibility.

pyenv + poetry vs conda:

Package Management: poetry simplifies package management for Python projects, focusing primarily on the Python ecosystem. conda offers more extensive multi-language support and package channels.
Environment Isolation: Both pyenv and conda provide isolation through virtual environments, but conda is more feature-rich in managing environments for different languages.
Python Version Management: pyenv excels in managing Python versions, making it easier to switch between versions, while conda provides a broader environment solution.
Simplicity: For Python-specific projects, the combination of pyenv and poetry can be simpler and more streamlined, whereas conda is suitable for complex multi-language environments.

Installation & Usage

Prerequisites

Before you start, ensure you have:

A terminal on your Ubuntu system
Basic familiarity with using the terminal

Step 1: Installing pyenv

Open a terminal by pressing Ctrl + Alt + T.
Install required dependencies:

Installation Commands


            $ sudo apt update 

            $ sudo apt install -y make build-essential git libssl-dev zlib1g-dev libbz2-dev 
                libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev 
                libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Install pyenv using git:

Installation Commands


            $ curl https://pyenv.run | bash

Add pyenv initialization to your shell profile:

Installation Commands


            $ echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> ~/.bashrc 

            $ echo 'eval "$(pyenv init --path)"' >> ~/.bashrc 

            $ echo 'eval "$(pyenv init -)"' >> ~/.bashrc 

            $ source ~/.bashrc

Verify the installation:

Command


            $ pyenv --version

Step 2: Installing Python Versions

Install desired Python versions using pyenv. For instance, to install Python 3.8 and 3.9, run:

Commands


            $ pyenv install 3.11

Set a global default Python version:

Command


            $ pyenv global 3.11

Step 3: Installing poetry

Install poetry using pip:

Command


            $ curl -sSL https://install.python-poetry.org | python3 -

Add poetry to your shell’s configuration:

Commands


            $ echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc 

            $ source ~/.bashrc

Verify the installation:

Command


            $ poetry --version

Step 4: Creating a New Project

Create a new project and create a new virtual environment:

Commands


            $ poetry new my-project 

            $ cd my-project 

            $ poetry env use 3.11

Step 5: Managing Dependencies

To add a new dependency (e.g., pandas), run:

Command


            $ poetry add pandas

To add a development-only dependency, use the --dev flag:

Command


            $ poetry add --group dev jupyter

To remove a dependency:

Command


            $ poetry remove pandas

To update dependencies:

Command


            $ poetry update

Step 6: Running Scripts and Shells

To open a shell within the virtual environment:

Command


            $ poetry shell

To run a script within the virtual environment:

Command


            $ poetry run python your_script.py

Step 7: Packaging and Distributing

Build a distributable package:

Command


            $ poetry build

Publish to PyPI (Python Package Index):

Command


            $ poetry publish

Conclusion

Congratulations! You’ve successfully set up a data scientist’s development environment using pyenv and poetry on Ubuntu. This environment will enhance your Python development experience by providing isolated environments and streamlined package management.

By comparing pyenv and poetry with conda, you’ve gained insights into their strengths and how they align with your specific needs. Consider pyenv and poetry for a streamlined Python-focused experience or conda for multi-language support and complex environments.

As you delve deeper into your data science projects, keep experimenting with these tools to find the best fit for your workflows. Happy coding and data exploration on Ubuntu!

Introduction

Benefits of pyenv and poetry

pyenv:

poetry:

Comparing with Conda

conda:

pyenv + poetry vs conda:

Installation & Usage

Prerequisites

Step 1: Installing pyenv

Step 2: Installing Python Versions

Step 3: Installing poetry

Step 4: Creating a New Project

Step 5: Managing Dependencies

Step 6: Running Scripts and Shells

Step 7: Packaging and Distributing

Conclusion

References

Like this:

Introduction

Benefits of pyenv and poetry

pyenv:

poetry:

Comparing with Conda

conda:

pyenv + poetry vs conda:

Installation & Usage

Prerequisites

Step 1: Installing pyenv

Step 2: Installing Python Versions

Step 3: Installing poetry

Step 4: Creating a New Project

Step 5: Managing Dependencies

Step 6: Running Scripts and Shells

Step 7: Packaging and Distributing

Conclusion

References

Share this:

Like this: