Introducing Python Virtual Environments in Texera
Feature Introduction · Apache Texera
Python UDFs allow users to extend Texera workflows with custom code, but many workflows depend on packages that are not available in the default system environment. To address this challenge, Texera now supports Python Virtual Environments (PVEs), enabling users to install custom dependencies and execute workflows in isolated Python environments.
By giving users control over their runtime dependencies, PVEs make it easier to develop, share, and reproduce Python-based workflows while reducing package conflicts across the platform.
01 The Challenge
Before PVEs, Python UDFs relied entirely on packages installed in the system environment. While this worked for simple workflows, it became increasingly difficult to support workflows that required specialized libraries or specific package versions.
Installing packages globally can introduce dependency conflicts and make workflow execution less reproducible. As Texera's Python ecosystem continued to grow, users needed a way to manage dependencies independently.
02 Introducing Python Virtual Environments
Python Virtual Environments provide isolated Python installations that maintain their own package dependencies. Users can create environments, install packages, and manage dependencies without affecting other workflows or users.
Creating a virtual environment, installing dependencies, and using it in a Python UDF.
Key Capabilities
- Create Python virtual environments directly within Texera
- Install custom Python packages using pip
- View and manage existing environments
- Remove environments that are no longer needed
- Reuse environments across multiple workflow executions
03 Running Python UDFs with PVEs
When configuring a Python UDF, users can select a virtual environment to use during execution. Texera launches the Python worker using the interpreter from the selected environment, giving the UDF access to all installed packages.
This allows workflows to use libraries such as pandas, numpy, scikit-learn, and many others without requiring those dependencies to be installed globally.
04 Benefits
05 Impact on Texera
The introduction of Python Virtual Environments represents an important step toward making Texera a more flexible platform for Python-based analytics and machine learning. By allowing users to manage their own dependencies, workflows become easier to share, reproduce, and extend.
Looking Ahead
Python Virtual Environments provide the foundation for more advanced environment management capabilities in Texera while giving users greater control over how their workflows are executed today.
Python Virtual Environments make custom dependency management a first-class experience in Texera.