Guide for Developers

0. Requirements

Java 11 JDK

Install Java JDK 11 (Java Development Kit) (recommend: [adoptopenjdk](https://adoptium.net/installation/)). To verify the installation, run:

java -version

Next, set JAVA_HOME. On macOS you can run:

export JAVA_HOME=$(/usr/libexec/java_home -v 11)

On Windows, add a system environment variable called JAVA_HOME that points to the JDK directory.

Python@3.12/3.11/3.10

Install Python 3.12 (or 3.11/3.10) from the official site or your preferred package manager.

Git

On Windows, install the software from https://gitforwindows.org/. Git Bash is available after installing Git.

On Mac and Linux, see https://git-scm.com/book/en/v2/Getting-Started-Installing-Git

Verify the installation by:

git --version

sbt (Scala Build Tool)

Install sbt for building the project. Please refer to sbt Reference Manual — Installing sbt. We recommend you to use sdkman to install sbt.

Verify the installation by:

sbt --version

If the above command fails on Windows after installation, it is recommended to restart your computer.

node LTS Version > 18.x

Install an LTS version (not the latest) of node. Currently, we require LTS version > 18.x.

On Windows, install from https://nodejs.org/en/.

On Mac and Linux, use NVM to install NodeJS as it avoids permission issues.

Verify the installation by:

node -v

Angular 16 Cli

Install the angular 16 cli globally:

npm install -g @angular/cli@16

Verify the installation by:

ng version

Clone and Configure Texera

In the terminal, clone the Texera repo:

git clone git@github.com:Texera/texera.git

Do the following changes to the configuration files:

• Edit common/config/src/main/resources/storage.conf to use your Postgres credentials.

    jdbc {

-        username = "postgres"
+        username = <Postgres username you have>
        username = ${?STORAGE_JDBC_USERNAME}

-        password = "postgres"
+        password = <Postgres password you have>
        password = ${?STORAGE_JDBC_PASSWORD}
    }

• Edit common/config/src/main/resources/udf.conf to use the correct python executable path(can be obtained by command which python or where python):

python {
-   path = 
+   path = "/the/executable/path/of/python"
}

Setup PostgreSQL locally

Texera uses PostgreSQL to manage the user data and system metadata. To install and configure it: Install Postgres. If you are using Mac, simply execute:

brew install postgresql

Install Pgroonga for enabling full-text search, if you are using Mac, simply execute:

brew install pgroonga

Execute sql/texera_ddl.sql to create texera_db database for storing user system data & metadata storage

psql -U postgres -f "sql/texera_ddl.sql"

Execute sql/iceberg_postgres_catalog.sql to create the database for storing Iceberg catalogs.

psql -U postgres -f "sql/iceberg_postgres_catalog.sql"

Setup the LakeFS+Minio locally

Texera requires LakeFS and S3(Minio is one of the implementations) as the dataset storage. Setting up these two storage services locally are required to make Texera’s dataset feature functioning.

Install Docker Desktop which contains both docker engine and docker compose. Make sure you launch the Docker after installing it.

In the terminal, enter the directory containing the docker-compose file:

cd file-service/src/main/resources

Edit docker-compose.yml by: search for volumes in the file and follow the instructions in the comment. This step is required otherwise your data will be lost if containers are deleted

Execute the following command to start LakeFS and Minio:

docker compose up

Import the project into IntelliJ

Before you import the project, you need to have “Scala”, and “SBT Executor” plugins installed in Intellij.

1. In Intellij, open File -> New -> Project From Existing Source, then choose the texera folder.
2. In the next window, select Import Project from external model, then select sbt.
3. In the next window, make sure Project JDK is set. Click OK.
4. IntelliJ should import and build this Scala project. In the terminal under texera, run:

sbt clean protocGenerate

This will generate proto-specified codes. And the IntelliJ indexing should start. Wait until the indexing and importing is completed. And on the right, you can open the sbt tab and check the loaded texera project and couple of sub projects:

5. When IntelliJ prompts “Scalafmt configuration detected in this project” in the bottom right corner, select “Enable”. If you missed the IntelliJ prompt, you can check the Event Log on the bottom right

6. In addition to the microservices, you need to run the JOOQ code generation using sbt DAO/jooqGenerate, make sure to provide Postgres credentials.

Run the backend micro services in IntelliJ

The easiest way to run backend services is in IntelliJ. Currently we have couple of micro services for different purposes. If one microservice failed after running, it may have dependency to another microservice, so wait for other ones to start, also make sure to run LakeFS docker compose:

To run each of the above web service, go to the corresponding scala file(i.e. for TexeraWebApplication, go find TexeraWebApplication.scala), then run the main function by pressing on the green run button and wait for the process to start up.

For TexeraWebApplication, the following message indicates that it is successfully running:

[main] [akka.remote.Remoting] Remoting now listens on addresses:
org.eclipse.jetty.server.Server: Started

• If IntelliJ displays CreateProcess error=206, the filename or extension is too long : add the -Didea.dynamic.classpath=true in Help | Edit Custom VM Options and restart the IDE

For ComputingUnitMaster, the following prompt indicates that it is successfully running:

---------Now we have 1 node in the cluster---------

Enable Python-based Operators

Texera has lots of Python-based operators like visualizations, and UDF operators. To enable them, install python dependencies by executing, you also need to install R in your system:

cd texera
pip install -r amber/requirements.txt -r amber/operator-requirements.txt

2. Launch Frontend

This is for developers that work on the frontend part of the project. This step is NOT needed if you develop the backend only.

Before you start, make sure the backend services are all running.

Install Angular CLI

cd frontend
yarn install

Ignore those warnings (warnings are usually marked in yellow color or start with WARN).

Launch Frontend in IntelliJ for local development

1. Click on the Green Run button next to the start in frontend/package.json.
2. Wait for some time and the server will get started. Open a browser and access http://localhost:4200. You should see the Texera UI with a canvas.\

Every time you save the changes to the frontend code, the browser will automatically refresh to show the latest UI. You can also run frontend using command line:

yarn start

Launch Frontend in the production environment

Run the following command

yarn run build

This command will optimize the frontend code to make it run faster. This step will take a while. After that, start the backend engine in IntelliJ and use your browser to access http://localhost:8080.

[WinError 3] The system cannot find the path specified.

New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force

Get-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled"

Caused by: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset