Only this pageAll pages
Powered by GitBook
1 of 13

v5

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Migration guides

Security consideration

Information about security considerations

1. Autolaunch Feature

The autolaunch feature empowers you to create HTTP links that automatically deploy an environment. This is an invaluable tool for initiating trainings effortlessly. However, exercise caution while using it as it could pose a security risk to the user. Consider disabling this feature if it doesn't suit your requirements or if security is a primary concern.

Disable Autolaunch

2. Group Feature

Onyxia is primarily designed to allocate resources such as a namespace and an S3 bucket to an individual user for work purposes. Additionally, it incorporates a feature that allows multiple users to share access to the same resources within a project. While this can be extremely beneficial for collaboration, be aware that it might be exploited by a malicious user within the group to leverage the privileges of another project member. Always monitor shared resources and maintain proper user access control to prevent such security breaches.

User guide

Using Onyxia (as a data scientist)

See also

It's the Onyxia user guide dedicated to our staff.

There are 3 main components accessible on the onyxia web interface :

  • catalogs and services launched by the users (Kubernetes access)

  • a file browser (S3 access)

  • secret browser (Vault access)

Following is a documentation Onyxia when configured with the default service catalogs :

This collection of charts help users to launch many IDE with various binary stacks (python , R) with or without GPU support. Docker images are built and help us to give a homogeneous stack.

This collection of charts help users to launch many databases system. Most of them are based on .

This collection of charts help users to start automation tools for their datascience activity.

Users can edit various parameters. Onyxia do some assertion based on the charts values schema and the configuration on the instance. For example some identity token can be injected by default (because Onyxia connect users to many APIs).

After launching a service, notes are shown to the user. He can retrieve those notes on the README button. Charts administrator should explain how to connect to the services (url , account) and what happens on deletion.

Users can manage their files on S3. There is no support for rename in S3 so don't be surprise. Onyxia is educational. Any action on the S3 browser in the UI is written in a console with a cli.

User can do the following S3 actions :

  • download files

  • upload files

  • delete files

Of course, in our default catalags there are all the necessary tools to connect to S3.

Our advice is to never download file to your container but directly ingest in memory the data.

Users can mange their secrets on Vault. There is also a cli console.

Onyxia use only a key value v2 secret engine in Vault. Users can store some secrets there and inject them in their services if configured by the helm chart.

Of course, in our default catalags there are all the necessary tools to connect to Vault.

Start a service

The Onyxia user experience may be very different from one catalog of service to another.

The catalog defines what options are available though Onyxia.

File browser

Secret browser

here
bitnami/charts
https://docs.sspcloud.fr
s3 browser

v4 -> v5

The primary breaking change in this release pertains to Keycloak configuration. With this update, you're no longer limited to using Keycloak; any OIDC-compliant identity provider is now supported. To accommodate this new feature, you'll need to make some adjustments to the configuration of your Onyxia instance.

Technical doc

Willing to submit PRs on the Onyxia codebase?

You don't need to specify the issuerURI in multiple locations as we have done here. If you're using just one identity server (You have only one Keycloak server for example), you can set the issuerURI solely in api->env->oidc.issuer-uri.

onyxia-values.yaml
onyxia:
   web:
     env:
-      KEYCLOAK_URL: https://auth.lab.sspcloud.fr/auth
-      KEYCLOAK_REALM: sspcloud
   api:
     env:
-      keycloak.resource: onyxia
-      keycloak.realm: sspcloud
-      keycloak.auth-server-url: https://auth.lab.sspcloud.fr/auth
-      keycloak.ssl-required: external
-      keycloak.public-client: "true"
-      keycloak.enable-basic-auth: "true"
-      keycloak.bearer-only: "true"
+      oidc.issuer-uri: "https://auth.lab.sspcloud.fr/auth/realms/sspcloud"
+      oidc.clientID: "onyxia"
+      oidc.audience: "onyxia"
       authentication.mode: "openidconnect"
     regions: 
       [
         {
           "id": "paris",
           "services": {
-              "authenticationMode": "admin",
+              "authenticationMode": "serviceAccount",
               "k8sPublicEndpoint": {
                 "URL": "https://apiserver.kub.sspcloud.fr",
-                "keycloakParams": {
-                  "URL": "https://auth.lab.sspcloud.fr/auth",
-                  "realm": "sspcloud",
-                  "clientId": "onyxia"
-                },
+                "oidcConfiguration": {
+                  "issuerURI": "https://auth.lab.sspcloud.fr/auth/realms/sspcloud",
+                  "clientID": "onyxia-k8s-apiserver",
+                }
               }
             },
           "data": {
             "S3": {
-              "keycloakParams": {
-                "URL": "https://auth.lab.sspcloud.fr/auth",
-                "realm": "sspcloud",
-                "clientId": "onyxia-minio",
-              }
+              "oidcConfiguration": {
+                "issuerURI": "https://auth.lab.sspcloud.fr/auth/realms/sspcloud",
+                "clientID": "onyxia-minio",
+              }
             }
          },
          "vault": {
              "URL": "https://vault.lab.sspcloud.fr",
-             "keycloakParams": {
-               "URL": "https://auth.lab.sspcloud.fr/auth",
-               "realm": "sspcloud",
-               "clientId": "onyxia-vault",
-             }
+             "oidcConfiguration": {
+               "URL": "https://auth.lab.sspcloud.fr/auth/realms/sspcloud",
+               "clientID": "onyxia-vault"
+             }
         }
       }
     ]

The REST API

The backend REST API in Java

This is the documentation for .

It's the part of the App that runs in the clusters. It handles the things that can't be done directly from the frontend.

The Web Application

The TypeScript App that runs in the browser.

This is the documentation for .

Roadmap

Onyxia Project Core Team Future Developments Roadmap

Data Explorer Evolution

Transforming the existing file browser into a comprehensive data explorer is a central aspect of our development roadmap. This enhancement aims to provide data scientists with immediate access to the initial rows of various file formats (including but not limited to parquet, CSV, and JSON) directly via the Onyxia-web interface. This will effectively integrate a basic SQL engine, DuckDB Wasm, for swift data access and manipulation.

Accessibility Improvement

Enhancing accessibility is a vital and immediate priority for the Onyxia project. Currently, the platform lacks certain accessibility features which we plan to implement in the immediate future.

Expanded S3 Management Options

As it stands, the Onyxia project does not have the capacity to set quotas for S3 buckets or create custom policies. These responsibilities are currently delegated to other administrators. However, we are in the process of developing an S3 operator that will simplify the process of onboarding onto S3, thereby reducing dependency on external administrators.

git clone https://github.com/InseeFrLab/onyxia
cd onyxia/web

yarn install
#Setup the var envs to tell the app to connect to the sspcloud
#Fill up with your own value to run the web app against your onyxia API.
cp .env.local.sample .env.local

# To stat the app locally
yarn start
InseeFrLab/onyxia -> web/
InseeFrLab/onyxia -> api/

Migrating to the new helm repo

Previously, the Helm chart of Onyxia was hosted on the inseefrlab/helm-charts repo and has now been moved to inseefrlab/onyxia. As a result you would now install Onyxia like this:

-helm repo add inseefrlab https://inseefrlab.github.io/helm-charts
+helm repo add onyxia https://inseefrlab.github.io/onyxia

-helm install onyxia inseefrlab/helm-charts
+helm install onyxia onyxia/onyxia

In the following we assume the current version of Onyxia is 4.1.4 but you are encorging to use the latest version instead. See releases.

If you use ArgoCD for deploying onyxia:

apps/onyxia/Chart.yaml
 apiVersion: v2
 name: onyxia
 version: 1.0.0
 dependencies:
   - name: onyxia
-    version: 4.1.0
+    version: 4.1.4
-    repository: https://inseefrlab.github.io/helm-charts/
+    repository: https://inseefrlab.github.io/onyxia/

You no longer need to manually manage the version of onyxia-web and onyxia-api, now, if you want to update Onyxia, you just update the chart version number.

helm repo add onyxia https://inseefrlab.github.io/onyxia

DOMAIN=my-domain.net

cat << EOF > ./onyxia-values.yaml
# ...
web:
  image:
-   tag: 2.29.4
api:
  image:
-   tag: v0.32   
# ...
EOF

helm install onyxia onyxia/onyxia -f onyxia-values.yaml

For the Keycloak theme, the version is now synchronized with the Onyxia version.

helm repo add codecentric https://codecentric.github.io/helm-charts

cat << EOF > ./keycloak-values.yaml
# ... See https://docs.onyxia.sh/#enabling-user-authentication
extraInitContainers: |
  - name: realm-ext-provider
    image: curlimages/curl
    imagePullPolicy: IfNotPresent
    command:
      - sh
    args:
      - -c
      - |
-       curl -L -f -S -o /extensions/onyxia.jar https://github.com/InseeFrLab/onyxia/releases/download/v2.29.4/keycloak-theme.jar
+       curl -L -f -S -o /extensions/onyxia.jar https://github.com/InseeFrLab/onyxia/releases/download/v4.1.4/keycloak-theme.jar
    volumeMounts:
      - name: extensions
        mountPath: /extensions
extraVolumeMounts: |
  - name: extensions
    mountPath: /opt/jboss/keycloak/standalone/deployments
extraVolumes: |
  - name: extensions
    emptyDir: {}
# ...
EOF

helm install keycloak codecentric/keycloak -f keycloak-values.yaml

Also note that, the theme will now appear as "onyxia" in the dropdown. Previously it was "onyxia-web"

Architecture

Main rules

  • src/ui contains the React application, it's the UI of the app.

  • src/core contains the 🧠 of the app.

    • Nothing in the src/core directory should relate to React. A concept like react hooks for example is out of scope for the src/core directory.

    • src/core should never import anything from src/ui, even types.

    • It should be possible for example to port onyxia-web to Vue.js or React Native without changing anything to the src/core directory.

    • The goal of src/core is to expose an API that serves the UI.

    • The API exposed should be reactive. We should not expose to the UI functions that returns promises, instead, the functions we expose should update states and the UI should react to these states updates.

  • Whenever we need to interact with the infrastructure we define a port in . A port is only a type definition. In our case the infrastructure is: the Keycloak server, the Vault server, the Minio server and a Kubernetes API (Onyxia-API).

  • In are the implementations of the ports. For each port we should have at least two implementations, a dummy and a real one. It enabled the app to still run, be it in degraded mode, if one piece of the infrastructure is missing. Say we don’t have a Vault server we should still be able to launch containers.

  • In

Let's say we want to create a new page in onyxia-web where users can type in a repo name and get the current number of stars the repo has on GitHub.

Now let's say we want the search to be restricted to a given GitHub organization. (Example: InseeFrLab.) The GitHub organization should be specified as an environment variable by the person in charge of deploying Onyxia. e.g.:

If no ORG_NAME is provided by the administrator, the app should always show 999 stars for any repo name queried.

Currently users can save their GitHub Personal access token in their Onyxia account but not yet their GitLab token. Let's see how we would implement that.

The easy action to take when the user selects another project is to simply reload the page (windows.location.reload()). We want to avoid doing this to enable what we call "hot projet swiping":

To implement this behavior you have to leverage the evtAction middleware from clean-redux. It enabled to register functions to be run when certain actions are dispatched.

we expose APIs for the UI to consume.
  web:
    env:
      MINIO_URL: https://minio.lab.sspcloud.fr
      VAULT_URL: https://vault.lab.sspcloud.fr
      OIDC_URL: https://auth.lab.sspcloud.fr/auth
      OIDC_REALM: sspcloud
      TITLE: SSP Cloud
      ORG_NAME: InseeFrLab #<==========
      

Architecture

In practice

UPDATE: This video remain relevant but please not that the clean archi setup have been considerably improved in latest releases. A dedicated repo have been created to explain it in detail.

Main take-way is that app have been renamed ui and lib have been renamed core.

You might wonder why some values, instead of being redux state, are returned by thunks functions.

For example, it might seem more natural to do:

const { isUserLoggedIn } = useCoreState(state => state.userAuthentication);

Instead of what we actually do, which is:

const { userAuthenticationThunks } = useThunks();
const isUserLoggedIn = userAuthenticationThunks.getIsUserLoggedIn();

However the rule is to never store as a redux state, values that are not susceptible to change. Redux states are values that we observe, any redux state changes should trigger a re-render of the React components that uses them. Conversely, there is no need to observe a value that will never change. We can get it once and never again, get it in a callback or wherever.

But, you may object, users do login and logout, isUserLoggedIn is not a constant!

Actually, from the standpoint of the web app, it is. When a user that isn't authenticated click on the login button, it is being redirected away. When he returns to the app everything is reloaded from scratch.

Another example: Recording user's GitLab token

How to deal with project switching

Unlike the other video, the following one is voiced. Find the relevant code here.

src/core/port
src/core/adapters
The page is not reloaded when changing the project
src/lib/usecases

Catalog of services

Every Onyxia instance may or may not have it's own catalog. There is three default catalogs :

This collection of charts help users to launch many IDE with various binary stacks (python , R) with or without GPU support. Docker images are built here and help us to give a homogeneous stack.

This collection of charts help users to launch many databases system. Most of them are based on bitnami/charts.

This collection of charts help users to start automation tools for their datascience activity.

You can always find the source of the catalog by clicking on the "contribute to the... " link.

https://datalab.sspcloud.fr/catalog

If you take , it has only one catalog, .

The available catalogs in a given Onyxia instance are configured at install time, example with datalab.sspcloud.fr:

In order to contribute you have to be familiar with and to be familiar with Helm you need to be familiar with .

In Onyxia we use the values.schema.json file to know what options should be displayed to the user at and what default value Onyxia should inject.

Let's consider a sample of the values.schema.json of the InseeFrLab/helm-charts-datascience's Jupyter chart:

And it translates into this:

Note the "git.name", "git.email" and "git.token", this enables to pre fill the fields.

If the user took the time to fill its profile information, know what is the Git username, email and personal access token of the user.

is defined the structure of the context that you can use in the overwriteDefaultWith field:

You can also concatenate string values using syntax.

You probably want to be able to define a limit to the amount of resources a user can request when launching a service.

It's possible to do it at the catalog level but it's best to enable the person who is deploying Onyxia to define boundaries for his deployment regions.

This is the purpose of the x-onyxia param useRegionSliderConfig

You now have all the relevent information to submit PR on the existing catalogs or even to create your own.

Remember that a helm chart repository is nothing more than a GitHub repo with a special to publish the charts on GitHub Pages.

If you are looking for a repo to start from have a look at , it has a directory where you can put the icons of your services.

helm repo add inseefrlab https://inseefrlab.github.io/helm-charts

DOMAIN=my-domain.net

cat << EOF > ./onyxia-values.yaml
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
  hosts:
    - host: onyxia.$DOMAIN
api:
  catalogs: 
    [
       {
          "id": "ide",
          "name": "Services interactifs",
          "description": "Services for datascientists.",
          "maintainer": "innovation@insee.fr",
          "location": "https://inseefrlab.github.io/helm-charts-interactive-services",
          "status": "PROD",
          "highlightedCharts": ["jupyter-python", "rstudio", "vscode-python"],
          "type": "helm",
        },
        {
          "id": "databases",
          "name": "Bases de données",
          "description": "Services for datascientists.",
          "maintainer": "innovation@insee.fr",
          "location": "https://inseefrlab.github.io/helm-charts-databases",
          "status": "PROD",
          "highlightedCharts": ["postgresql", "elastic"],
          "type": "helm",
        },
        {
          "id": "automation",
          "name": "Automatisation",
          "description": "Services for datascientists.",
          "maintainer": "innovation@insee.fr",
          "location": "https://inseefrlab.github.io/helm-charts-automation",
          "status": "PROD",
          "highlightedCharts": ["argo-cd", "argo-workflows", "mlflow"],
          "type": "helm",
        }
    ]
  regions: 
    [
      {
        "services":{
          "expose":{
            "domain":"lab.$DOMAIN"
          }
        }
      }
    ]
EOF

helm install onyxia inseefrlab/onyxia -f onyxia-values.yaml
values.schema.json
"git": {
    "description": "Git user configuration",
    "type": "object",
    "properties": {
        "enabled": {
            "type": "boolean",
            "description": "Add git config inside your environment",
            "default": true
        },
        "name": {
            "type": "string",
            "description": "user name for git",
            "default": "",
            "x-onyxia": {
                "overwriteDefaultWith": "git.name"
            },
            "hidden": {
                "value": false,
                "path": "git/enabled"
            }
        },
        "email": {
            "type": "string",
            "description": "user email for git",
            "default": "",
            "x-onyxia": {
                "overwriteDefaultWith": "git.email"
            },
            "hidden": {
                "value": false,
                "path": "git/enabled"
            }
        },
        "cache": {
            "type": "string",
            "description": "duration in seconds of the credentials cache duration",
            "default": "",
            "x-onyxia": {
                "overwriteDefaultWith": "git.credentials_cache_duration"
            },
            "hidden": {
                "value": false,
                "path": "git/enabled"
            }
        },
        "token": {
            "type": "string",
            "description": "personal access token",
            "default": "",
            "x-onyxia": {
                "overwriteDefaultWith": "git.token"
            },
            "hidden": {
                "value": false,
                "path": "git/enabled"
            }
        },
        "repository": {
            "type": "string",
            "description": "Repository url",
            "default": "",
            "hidden": {
                "value": false,
                "path": "git/enabled"
            }
        },
        "branch": {
            "type": "string",
            "description": "Brach automatically checkout",
            "default": "",
            "hidden": {
                "value": "",
                "path": "git/repository"
            }
        }
    }
},
export type XOnyxiaParams = {
    /**
     * This is where you can reference values from the onyxia context so that they
     * are dynamically injected by the Onyxia launcher.
     *
     * Examples:
     * "overwriteDefaultWith": "user.email"
     * "overwriteDefaultWith": "{{project.id}}-{{k8s.randomSubdomain}}.{{k8s.domain}}"
     */
    overwriteDefaultWith?: string;
    hidden?: boolean;
    readonly?: boolean;
    useRegionSliderConfig?: string;
};

export type XOnyxiaContext = {
    user: {
        idep: string;
        name: string;
        email: string;
        password: string;
        ip: string;
        darkMode: boolean;
    };
    project: {
        id: string;
        password: string;
        basic: string;
    };
    git: {
        name: string;
        email: string;
        credentials_cache_duration: number;
        token: string | undefined;
    };
    vault: {
        VAULT_ADDR: string;
        VAULT_TOKEN: string;
        VAULT_MOUNT: string;
        VAULT_TOP_DIR: string;
    };
    kaggleApiToken: string | undefined;
    s3: {
        AWS_ACCESS_KEY_ID: string;
        AWS_SECRET_ACCESS_KEY: string;
        AWS_SESSION_TOKEN: string;
        AWS_DEFAULT_REGION: string;
        AWS_S3_ENDPOINT: string;
        AWS_BUCKET_NAME: string;
        port: number;
    };
    region: {
        defaultIpProtection: boolean | undefined;
        defaultNetworkPolicy: boolean | undefined;
        allowedURIPattern: string;
        kafka:
            | {
                  url: string;
                  topicName: string;
              }
            | undefined;
        tolerations: unknown[] | undefined;
        from: unknown[] | undefined;
        nodeSelector: Record<string, unknown> | undefined;
        startupProbe: Record<string, unknown> | undefined;
        sliders: Record<
            string,
            {
                sliderMin: number;
                sliderMax: number;
                sliderStep: number;
                sliderUnit: string;
            }
        >;
        resources:
            | {
                  cpuRequest?: `${number}${string}`;
                  cpuLimit?: `${number}${string}`;
                  memoryRequest?: `${number}${string}`;
                  memoryLimit?: `${number}${string}`;
                  disk?: `${number}${string}`;
                  gpu?: `${number}`;
              }
            | undefined;
    };
    k8s: {
        domain: string;
        ingressClassName: string | undefined;
        ingress: boolean | undefined;
        route: boolean | undefined;
        istio:
            | {
                  enabled: boolean;
                  gateways: string[];
              }
            | undefined;
        randomSubdomain: string;
        initScriptUrl: string;
    };
    proxyInjection:
        | {
              httpProxyUrl: string | undefined;
              httpsProxyUrl: string | undefined;
              noProxy: string | undefined;
          }
        | undefined;
    packageRepositoryInjection:
        | {
              cranProxyUrl: string | undefined;
              condaProxyUrl: string | undefined;
              packageManagerUrl: string | undefined;
              pypiProxyUrl: string | undefined;
          }
        | undefined;
    certificateAuthorityInjection:
        | {
              cacerts: string | undefined;
              pathToCaBundle: string | undefined;
          }
        | undefined;
};
"hostname": {
  "type": "string",
  "form": true,
  "title": "Hostname",
  "x-onyxia": {
    "overwriteDefaultWith": "{{project.id}}-{{k8s.randomSubdomain}}.{{k8s.domain}}"
  }
}
onyxia/values.yaml
onyxia:

  web:
    ...
    
  api:
    ...
    regions:
      [
        {
          "id": "paris",
          "name": "Kubernetes DG Insee",
          "services": {
            "defaultConfiguration": {
              "sliders": {
                "cpu": {
                  "sliderMin": 100,
                  "sliderMax": 80000,
                  "sliderStep": 100,
                  "sliderUnit": "m"
                },
                "memory": {
                  "sliderMin": 1,
                  "sliderMax": 400,
                  "sliderStep": 1,
                  "sliderUnit": "Gi"
                },
                "gpu": {
                  "sliderMin": 1,
                  "sliderMax": 4,
                  "sliderStep": 1,
                  "sliderUnit": ""
                },
                "disk": {
                  "sliderMin": 1,
                  "sliderMax": 100,
                  "sliderStep": 1,
                  "sliderUnit": "Gi"
                }
              },
              "resources": {
                "cpuRequest": "100m",
                "cpuLimit": "40000m",
                "memoryRequest": "1Gi",
                "memoryLimit": "200Gi",
                "disk": "10Gi",
                "gpu": "1"
              }
            }
          }
        }
      ]
values.schema.json
{
  "$schema": "http://json-schema.org/schema#",
  "type": "object",
  "properties": {
    "resources": {
      "description": "Your service will have at least the requested resources and never more than its limits. No limit for a resource and you can consume everything left on the host machine.",
      "type": "object",
      "properties": {
        "limits": {
          "description": "max resources",
          "type": "object",
          "properties": {
            "nvidia.com/gpu": {
              "description": "GPU to allocate to this instance. This is also requested",
              "type": "string",
              "default": "0", // Will be overwritten by "1"
              "render": "slider",
              "sliderMin": 0, // Will be overwritten by 1
              "sliderMax": 3, // Will be overwritten by 4
              "sliderStep": 1, // Will be overwritten by 1
              "sliderUnit": "", // Will be overwritten by ""
              "x-onyxia": {
                "overwriteDefaultWith": "region.resources.gpu",
                "useRegionSliderConfig": "gpu" // 👀
              }
            },
            "cpu": {
              "description": "The maximum amount of cpu",
              "title": "CPU",
              "type": "string",
              "default": "30000m",
              "render": "slider",
              "sliderMin": 50,
              "sliderMax": 40000,
              "sliderStep": 50,
              "sliderUnit": "m",
              "sliderExtremity": "up",
              "sliderExtremitySemantic": "Maximum",
              "sliderRangeId": "cpu",
              "x-onyxia": {
                "overwriteDefaultWith": "region.resources.cpuLimit",
                "useRegionSliderConfig": "cpu"
              }
            },
            "memory": {
              "description": "The maximum amount of memory",
              "title": "Memory",
              "type": "string",
              "default": "50Gi",
              "render": "slider",
              "sliderMin": 1,
              "sliderMax": 200,
              "sliderStep": 1,
              "sliderUnit": "Gi",
              "sliderExtremity": "up",
              "sliderExtremitySemantic": "Maximum",
              "sliderRangeId": "memory",
              "x-onyxia": {
                "overwriteDefaultWith": "region.resources.memoryLimit",
                "useRegionSliderConfig": "memory"
              }
            }
          }
        }
      }
    }
  }
}

Defining region scoped resources limit

this other instance
helm-charts-sill
Helm
Kubernetes objects
the service configuration step
onyxia-web
onyxia-web
Here
mustache
github Action setup
this one
https://sill-demo.etalab.gouv.fr/catalog
https://helm.sh/docs/topics/charts/#the-chart-file-structure
The onyxia user profile

Install

Your Onyxia instance, today

Oneliner

TLDR. Here is how you can get an Onyxia instance running in a matter of seconds.

helm repo add onyxia https://inseefrlab.github.io/onyxia

cat << EOF > ./onyxia-values.yaml
ingress:
  enabled: true
  hosts:
    - host: datalab.my-domain.net
EOF

helm install onyxia onyxia/onyxia -f onyxia-values.yaml

With this, you will obtain an instance operating in a degraded mode, which lacks features such as authentication, S3 explorer, secret management, etc. However, you will still have the capability to launch services from the catalog.

Step by step installation guide

In this section, we will set up Onyxia from the ground up, along with all the associated technologies. This includes MinIO for S3, Keycloak for OIDC, and Vault for managing secrets.

Provision a Kubernetes cluster

First you'll need a Kubernetes cluster. If you have one already you can skip this section.

Hashicorp maintains great tutorials for Kubernetes clusters on , or .

Pick one of the three and follow the guide.

You can stop after the .

Ingress controller

Deploy an ingress controller on your cluster:

DNS

Let's assume you own the domain name my-domain.net, for the rest of the guide you should replace my-domain.net by a domain you actually own.

Now you need to get the external address of your cluster, run the command

and write down the External IP assigned to the LoadBalancer.

Depending on the cloud provider you are using it can be an IPv4, an IPv6 or a domain. On AWS for example, it will be a domain like xxx.elb.eu-west-1.amazonaws.com.

If you see <pending>, wait a few seconds and try again.

Once you have the address, create the following DNS records:

If the address you got was an IPv4 (x.x.x.x), create a A record instead of a CNAME.

If the address you got was ans IPv6 (y:y:y:y:y:y:y:y), create a AAAA record.

https://onyxia.my-domain.net will be the URL for your instance of Onyxia. The URL of the services created by Onyxia are going to look like: https://<something>.lab.my-domain.net

SSL

In this section we will obtain a TLS certificate issued by using the commend line tool then get our ingress controller to use it.

If you are already familiar with certbot you're probably used to run it on a remote host via SSH. In this case you are expected to run it on your own machine, we'll use the DNS chalenge instead of the HTTP chalenge.

Now we want to create a Kubernetes secret containing our newly obtained certificate:

Lastly, we want to tell our ingress controller to use this TLS certificate, to do so run:

This command will open your configured text editor, go to line 56 and add:

If you are on a Mac or Window computer you can install then enable Kubernetes.

Port Forwarding

You'll need to . It's done from the administration panel of your domestic internet Box. If you're on a corporate network, no luck for you I'm afraid.

DNS

Let's assume you own the domain name my-domain.net, for the rest of the guide you should replace my-domain.net by a domain you actually own.

In this section we assume that:

  • You have a Kubernetes cluster and kubectl configured

  • onyxia.my-domain.net and *.lab.my-domain.net are pointing to your cluster's external address. my-domain.net being a domain that you own. You can customise "onyxia" and "lab" to your liking, for example you could chose datalab.my-domain.net and *.kub.my-domain.net.

You can now access https://onyxia.my-domain.net and start services. Congratulations! 🥳

At the moment there is no authentication process, everyone can access our platform and and start services.

Let's setup Keycloak to enable users to create account and login to our Onyxia.

For deploying our Keycloak we use .

You can now login to the administration console of https://auth.lab.my-domain.net and login using the credentials you have defined with KEYCLOAK_USER and KEYCLOAK_PASSWORD.

  1. Create a realm called "datalab" (or something else), go to Realm settings

    1. On the tab General

      1. User Profile Enabled: On

Now you want to ensure that the username chosen by your users complies with Onyxia requirement (only alphanumerical characters) and define a list of email domain allowed to register to your service.

Go to Realm Settings (on the left panel) -> Tab User Profile (this tab shows up only if User Profile is enabled in the General tab and you can enable user profile only if you have started Keycloak with -Dkeycloak.profile=preview) -> JSON Editor.

Now you can edit the file as suggested in the following DIFF snippet. Be mindful that in this example we only allow emails @gmail.com and @hotmail.com to register you want to edit that.

Now our Keycloak server is fully configured we just need to update our Onyxia deployment to let it know about it.

Update the onyxia-values.yaml file that you created previously, don't forget to replace all the occurence of my-domain.net by your actual domain.

Don't forget as well to remplace the terms of services of the by your own terms of services. CORS should be enabled on those .md links (Access-Control-Allow-Origin: *).

Now that you have updated onyxia-values.yaml restart onyxia-web with the new configuration.

Now your users should be able to create account, log-in, and start services on their own Kubernetes namespace.

Onyxia-web use to get token and empowered user with storage features. We support any S3 storage compatible with this API. In this context, we are using , which is compatible with the Amazon S3 storage service and we demonstrate how to integrate it with Keycloak.

Before configuring MinIO, let's create a new client for Keycloak (from the previous existing "datalab" realm).

Create a client called "minio".

  1. Client ID: minio

  2. Client Protocol: openid-connect

  3. Root URL: https://minio.lab.my-domain.net/

Complete the content of client "minio" with the following values.

  1. Access Type: confidential

  2. Valid Redirect URIs (two values are required): https://minio.lab.my-domain.net/* and https://minio-console.lab.my-domain.net/*

  3. Web origins: *

Save the content, a new tab called Credentials must be appear. Navigate to Credentials tab and copy the secret value for the next section.

Navigate to Mappers tab and create a protocol Mapper.

  1. Name: policy

  2. Mapper Type: Hardcoded claim

Complete the content of Mapper "policy" with the following values.

  1. Token Claim Name: policy

  2. Claim value: stsonly

  3. Add to ID token: on

We recommand you to follow for this installation and you must activate OIDC authentification. We will use the official Helm in this tutorial. All Helm configuration values can be found within this .

Replace COPY_SECRET_FROM_KEYCLOAK_MINIO_CLIENT by the secret value defined into the "minio" Keycloak client (see previous section).

MinIO is now deployed and is accessible on the console url.

By default, there are 16 MinIO containers running. If this number is too large for your Kubernetes cluster, you can limit it by configuring the 'replicas' key.

Before configuring the onyxia region to create tokens we should go back to Keycloak and create a new client to enable onyxia-web to request token for MinIO. This client is a little bit more complexe than other if you want to manage durations (here 7 days) and this client should have a claim name policy and with a value of stsonly according to our last deployment of MinIO.

From "datalab" realm, create a client called "onyxia-minio"

  1. Client ID: onyxia-minio

  2. Client Protocol: openid-connect

  3. Root URL: https://onyxia.my-domain.net/

Complete the content of client "onyxia-minio" with the following values.

  1. Access Type: public

  2. Valid Redirect URIs: https://onyxia.my-domain.net/*

  3. Web origins: *

Save the content and navigate to Mappers tab and create two protocol Mappers.

Create the first Mapper called "policy".

  1. Token Name: policy

  2. Mapper Type: Hardcoded claim

  3. Token Claim Name: policy

Create the second Mapper called "audience-minio".

  1. Token Name: audience-minio

  2. Mapper Type: Audience

  3. _Included Custom Audience _: minio

S3 storage is configured inside a region in Onyxia api. You have some options to configure this storage and let inform Onyxia web all needed informations how to generate those tokens : keycloak parameters to access storage API, duration of STS tokens, bucket name with a standard prefix and a claim in the user JWT token to generate a unique identifiant for this bucket name, whether Onyxia-web should try to to create this bucket silently or not. There is also options for projects. You should look all options for the version of your need on

Onyxia-web use vault as a storage for two kinds of secrets : 1. secrets or information generate by Onyxia to store differents values (ui preferences for example) 2. user secrets Vault must be configured with JWT or OIDC authentification methods.

As vault need to be initialized with a master key, It can't be directly configured with all parameters such as oidc or access policies and roles. So first step we create a vault with dev mode (do not use this in production and do your initialization with any of the recommanded configuration : shamir, gcp, another vault)

Create a client called "vault"

  1. Root URL: https://vault.lab.my-domain.net/

  2. Valid redirect URIs: https://vault.lab.my-domain.net/*

  3. Web origins: *

TODO;

Technical stack

Technologies at play in Onyxia-web

To find your way in Onyxia, the best approach is to start by getting a surface-level understanding of the libraries that are leveraged in the project.

We are fully committed on keeping everything type safe. If you are a seasoned developer but not fully comfortable with TypeScript yet a good way to get you quickly up to speed is to go through the of the official website.

We also heavily rely on . It's a collection of utilities that help write cleaner TypeScript code. It is crutial to understand at least , , and to be able to contribute on the codebase.

We try, whenever we see an opportunity for it, to publish as standalone NPM module chunks of the code we write for Onyxia-web. It help keep the complexity in check. We use TS-CI as a starter for everything we publish on NPM.

Get your internet box routable IP and create the following DNS records:

https://onyxia.my-domain.net will be the URL for your instance of Onyxia.

The URL of the services created by Onyxia are going to look like: https://xxx.lab.my-domain.net

SSL

In this section we will obtain a TLS certificate issued by LetsEncrypt using the certbot commend line tool.

Now we want to create a Kubernetes secret containing our newly obtained certificate:

Ingress controller

We'll install ingress-nginx in our cluster but any other ingress controller will do.

You have an ingress controller configured with a default TLS certificate for *.lab.my-domain.net and onyxia.my-domain.net.

On the tab login

  1. User registration: On

  2. Forgot password: On

  3. Remember me: On

  • On the tab email, we give an example with **** AWS SES, if you don't have a SMTP server at hand you can skip this by going to Authentication (on the left panel) -> Tab Required Actions -> Uncheck "set as default action" Verify Email. Be aware that with email verification disable, anyone will be able to sign up to your service.

    1. From: noreply@lab.my-domain.net

    2. Host: email-smtp.us-east-2.amazonaws.com

    3. Port: 465

    4. Authentication: enabled

    5. Username: **************

    6. Password: ***************************************

    7. When clicking "save" you'll be asked for a test email, you have to provide one that correspond to a pre-existing user or you will get a silent error and the credentials won't be saved.

  • On the tab Themes

    1. Login theme: onyxia-web (you can also select the login theme on a per client basis)

    2. Email theme: onyxia-web

  • On the tab Localization

    1. Internationalization: Enabled

    2. Supported locales: <Select the languages you wish to support>

  • Create a client called "onyxia"

    1. Root URL: https://onyxia.my-domain.net/

    2. Valid redirect URIs: https://onyxia.my-domain.net/*

    3. Web origins: *

    4. Login theme: onyxia-web

  • In Authentication (on the left panel) -> Tab Required Actions enable and set as default action Therms and Conditions.

  • Add to access token: on

  • Add to userinfo: on

  • Advanced Settings 1. Access Token Lifespan : 7 days 2. Client Session Idle : 7 days 3. Client Session Max: 7 days

    Claim value: stsonly

  • Add to ID token: on

  • Add to access token: on

  • Add to userinfo: on

  • Add to ID token: on

  • Add to access token: on

  • kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.2.0/deploy/static/provider/aws/deploy.yaml
    kubectl get services -n ingress-nginx
    onyxia.my-domain.net CNAME xxx.elb.eu-west-1.amazonaws.com. 
    *.lab.my-domain.net  CNAME xxx.elb.eu-west-1.amazonaws.com. 
    brew install certbot #On Mac, lookup how to install certbot for your OS
    
    #Because we need a wildcard certificate we have to complete the DNS callange.  
    sudo certbot certonly --manual --preferred-challenges dns
    
    # When asked for the domains you wish to optains a certificate for enter:
    #   onyxia.my-domain.net *.lab.my-domain.net
    DOMAIN=my-domain.net
    sudo kubectl create secret tls onyxia-tls \
        -n ingress-nginx \
        --key /etc/letsencrypt/live/onyxia.$DOMAIN/privkey.pem \
        --cert /etc/letsencrypt/live/onyxia.$DOMAIN/fullchain.pem
    kubectl edit deployment ingress-nginx-controller -n ingress-nginx
          - --default-ssl-certificate=ingress-nginx/onyxia-tls
    helm repo add onyxia https://inseefrlab.github.io/onyxia
    
    DOMAIN=my-domain.net
    
    cat << EOF > ./onyxia-values.yaml
    ingress:
      enabled: true
      annotations:
        kubernetes.io/ingress.class: nginx
      hosts:
        - host: onyxia.$DOMAIN
    api:
      regions: 
        [
           {
              "id":"demo",
              "name":"Demo",
              "description":"This is a demo region, feel free to try Onyxia !",
              "services":{
                 "type":"KUBERNETES",
                 "singleNamespace":true,
                 "namespacePrefix":"user-",
                 "usernamePrefix":"oidc-",
                 "groupNamespacePrefix":"projet-",
                 "groupPrefix":"oidc-",
                 "authenticationMode":"serviceAccount",
                 "expose":{
                    "domain":"lab.$DOMAIN"
                 },
                 "monitoring":{
                    "URLPattern":"todo"
                 },
                 "initScript":"https://inseefrlab.github.io/onyxia/onyxia-init.sh"
              }
           }
        ]
    EOF
    
    helm install onyxia onyxia/onyxia -f onyxia-values.yaml
    helm repo add codecentric https://codecentric.github.io/helm-charts
    
    DOMAIN=my-domain.net
    POSTGRESQL_PASSWORD=xxxxx #Replace by a strong password, you will never need it.
    # Credentials for logging to https://auth.lab.$DOMAIN/auth
    KEYCLOAK_USER=admin
    KEYCLOAK_PASSWORD=yyyyyy 
    
    cat << EOF > ./keycloak-values.yaml
    image:
      # We use the legacy variant of the image until codecentric update it's helm chart
      tag: "19.0.3-legacy"
    replicas: 1
    extraInitContainers: |
      - name: realm-ext-provider
        image: curlimages/curl
        imagePullPolicy: IfNotPresent
        command:
          - sh
        args:
          - -c
          - |
            # There is a custom theme published alongside every onyxia-web release
            # The version of the Keycloak theme and the version of onyxia-web don't need 
            # to match but you should update the theme from time to time.  
            # https://github.com/InseeFrLab/onyxia-web/releases
            curl -L -f -S -o /extensions/onyxia.jar https://github.com/InseeFrLab/onyxia-web/releases/download/v2.29.4
    /keycloak-theme.jar
        volumeMounts:
          - name: extensions
            mountPath: /extensions
    extraVolumeMounts: |
      - name: extensions
        mountPath: /opt/jboss/keycloak/standalone/deployments
    extraVolumes: |
      - name: extensions
        emptyDir: {}
    extraEnv: |
      - name: KEYCLOAK_USER
        value: $KEYCLOAK_USER
      - name: KEYCLOAK_PASSWORD
        value: $KEYCLOAK_PASSWORD
      - name: JGROUPS_DISCOVERY_PROTOCOL
        value: kubernetes.KUBE_PING
      - name: KUBERNETES_NAMESPACE
        valueFrom:
         fieldRef:
           apiVersion: v1
           fieldPath: metadata.namespace
      - name: KEYCLOAK_STATISTICS
        value: "true"
      - name: CACHE_OWNERS_COUNT
        value: "2"
      - name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
        value: "2"
      - name: PROXY_ADDRESS_FORWARDING
        value: "true"
      - name: JAVA_OPTS
        value: >-
          -Dkeycloak.profile=preview -XX:+UseContainerSupport -XX:MaxRAMPercentage=50.0 -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true 
    ingress:
      enabled: true
      servicePort: http
      annotations:
        kubernetes.io/ingress.class: nginx
        ## Resolve HTTP 502 error using ingress-nginx:
        ## See https://www.ibm.com/support/pages/502-error-ingress-keycloak-response
        nginx.ingress.kubernetes.io/proxy-buffer-size: 128k
      rules:
        - host: "auth.lab.$DOMAIN"
          paths:
            - path: /
              pathType: Prefix
      tls:
        - hosts:
            - auth.lab.$DOMAIN
    postgresql:
      postgresqlPassword: $POSTGRESQL_PASSWORD
    EOF
    
    helm install keycloak codecentric/keycloak -f keycloak-values.yaml
    {
      "attributes": [
        {
          "name": "username",
          "displayName": "${username}",
          "validations": {
            "length": {
              "min": 3,
              "max": 255
            },
    +       "pattern": {
    +         "error-message": "${alphanumericalCharsOnly}",
    +         "pattern": "^[a-zA-Z0-9]*$"
    +       },
            "username-prohibited-characters": {}
          }
        },
        {
          "name": "email",
          "displayName": "${email}",
          "validations": {
            "email": {},
    +       "pattern": {
    +         "pattern": "^[^@]+@([^.]+\\.)*((gmail\\.com)|(hotmail\\.com))$"
    +       },
            "length": {
              "max": 255
            }
          }
        },
        {
          "name": "firstName",
          "displayName": "${firstName}",
          "required": {
            "roles": [
              "user"
            ]
          },
          "permissions": {
            "view": [
              "admin",
              "user"
            ],
            "edit": [
              "admin",
              "user"
            ]
          },
          "validations": {
            "length": {
              "max": 255
            },
            "person-name-prohibited-characters": {}
          }
        },
        {
          "name": "lastName",
          "displayName": "${lastName}",
          "required": {
            "roles": [
              "user"
            ]
          },
          "permissions": {
            "view": [
              "admin",
              "user"
            ],
            "edit": [
              "admin",
              "user"
            ]
          },
          "validations": {
            "length": {
              "max": 255
            },
            "person-name-prohibited-characters": {}
          }
        }
      ]
    }
    +serviceAccount:
    +  clusterAdmin: true
     ingress:
       enabled: true
       annotations:
         kubernetes.io/ingress.class: nginx
       hosts:
         - host: onyxia.my-domain.net
     web:
    +  env:
    +    TERMS_OF_SERVICES: |
    +      { 
    +        "en": "https://www.sspcloud.fr/tos_en.md", 
    +        "fr": "https://www.sspcloud.fr/tos_fr.md" 
    +      }
     api:
       env:
    +    authentication.mode: openidconnect
    +    oidc.issuer-uri: "https://auth.lab.my-domain.net/auth/realms/datalab"
    +    oidc.clientID: "onyxia"
    +    oidc.audience: "onyxia"
       regions:
         [
            {
               "id":"demo",
               "name":"Demo",
               "description":"This is a demo region, feel free to try Onyxia !",
               "services":{
                  "type":"KUBERNETES",
    -             "singleNamespace": true,
    +             "singleNamespace": false,
                  "namespacePrefix":"user-",
                  "usernamePrefix":"oidc-",
                  "groupNamespacePrefix":"projet-",
                  "groupPrefix":"oidc-",
                  "authenticationMode":"serviceAccount",
                  "expose":{
                     "domain":"lab.my-domain.net"
                  },
                  "monitoring":{
                     "URLPattern":"todo"
                  },
                  "initScript":"https://inseefrlab.github.io/onyxia/onyxia-init.sh"
               }
            }
         ]
    helm upgrade onyxia inseefrlab/onyxia -f onyxia-values.yaml
    helm repo add minio https://charts.min.io/
     
    DOMAIN=my-domain.net
    
    cat << EOF > ./minio-values.yaml
    ## replicas: 16
    ingress:
      enabled: true
      annotations:
        kubernetes.io/ingress.class: nginx
      path: /
      hosts:
        - minio.lab.$DOMAIN
      tls:
        - hosts:
            - minio.lab.$DOMAIN
    consoleIngress:
      enabled: true
      annotations:
        kubernetes.io/ingress.class: nginx
      paths: /
      hosts:
        - minio-console.lab.$DOMAIN
      tls:
        - hosts:
            - minio-console.lab.$DOMAIN
    environment:
      MINIO_BROWSER_REDIRECT_URL: https://minio-console.lab.$DOMAIN
    oidc:
      enabled: true
      configUrl: "https://auth.lab.$DOMAIN/auth/realms/datalab/.well-known/openid-configuration"
      clientId: "minio"
      claimName: "policy"
      scopes: "openid,profile,email"
      redirectUri: "https://minio-console.lab.$DOMAIN/oauth_callback"
      claimPrefix: ""
      comment: ""
      clientSecret: COPY_SECRET_FROM_KEYCLOAK_MINIO_CLIENT
    policies:
      - name: stsonly
        statements:
          - resources:
              - 'arn:aws:s3:::oidc-${jwt:preferred_username}'
              - 'arn:aws:s3:::oidc-${jwt:preferred_username}/*'
            actions:
              - "s3:*"
    EOF
    
    helm install minio minio/minio -f minio-values.yaml
    serviceAccount:
      clusterAdmin: true
     ingress:
       enabled: true
       annotations:
         kubernetes.io/ingress.class: nginx
       hosts:
         - host: onyxia.my-domain.net
     web:
      env:
        KEYCLOAK_REALM: datalab
        KEYCLOAK_URL: https://auth.lab.my-domain.net/auth
        TERMS_OF_SERVICES: |
          { "en": "https://www.sspcloud.fr/tos_en.md", "fr": "https://www.sspcloud.fr/tos_fr.md" }
     api:
       env:
        authentication.mode: openidconnect
        keycloak.realm: datalab
        keycloak.auth-server-url: https://auth.lab.my-domain.net/auth
       regions:
         [
            {
               "id":"demo",
               "name":"Demo",
               "description":"This is a demo region, feel free to try Onyxia !",
               "services":{
                  "type":"KUBERNETES",
                  "singleNamespace": false,
                  "namespacePrefix":"user-",
                  "usernamePrefix":"oidc-",
                  "groupNamespacePrefix":"projet-",
                  "groupPrefix":"oidc-",
                  "authenticationMode":"admin",
                  "expose":{
                     "domain":"lab.my-domain.net"
                  },
                  "monitoring":{
                     "URLPattern":"todo"
                  },
                  "cloudshell":{
                     "catalogId":"inseefrlab-helm-charts-datascience",
                     "packageName":"cloudshell"
                  },
                  "initScript":"https://inseefrlab.github.io/onyxia/onyxia-init.sh"
               },
               "data":{
                  "S3":{
    -                "URL":"todo",
    +                "type": "minio",
    +                "URL": "https://minio.lab.my-domain.net",
    +                "region": "us-east-1",
    +                "bucketPrefix": "oidc-",
    +                "groupBucketPrefix": "projet-",
    +                "bucketClaim": "preferred_username",
    +                "defaultDurationSeconds": 86400,
    +                "keycloakParams":
    +                {
    +                      "URL": "https://auth.lab.my-domain.net/auth",
    +                      "realm": "datalab",
    +                      "clientId": "onyxia-minio",
    +                },
    +                "acceptBucketCreation": true,
                     "monitoring":{
                        "URLPattern":"minio"
                     }
                  }
               },
               "auth":{
                  "type":"openidconnect"
               },
               "location":{
                  "lat":48.8164,
                  "long":2.3174,
                  "name":"Montrouge (France)"
               }
            }
         ]
    helm upgrade onyxia inseefrlab/onyxia -f onyxia-values.yaml
    helm repo add hashicorp https://helm.releases.hashicorp.com
     
    DOMAIN=my-domain.net
    
    cat << EOF > ./vault-values.yaml
    server:
      dev:
        enabled: true
        # Set VAULT_DEV_ROOT_TOKEN_ID value
        devRootToken: "root"
      ingress:
        enabled: true
        annotations:
          kubernetes.io/ingress.class: nginx
        hosts:
          - host: "vault.lab.$DOMAIN"
        tls:
          - hosts:
              - vault.lab.$DOMAIN
    EOF
    
    helm install vault hashicorp/vault -f vault-values.yaml

    The following command is for AWS.

    For GCP use this command.

    For Azure use this command.

    You can customise "onyxia" and "lab" to your liking, for example you could chose datalab.my-domain.net and *.kub.my-domain.net.

    The obtained certificate needs to be renewed every three month.

    To avoid the burden of having to remember to re-run the certbot command periodically you can setup cert-manager and configure a DNS01 challenge provider on your cluster but that's out of scope for Onyxia.

    You may need to delegate your DNS Servers to one of the supported DNS service provider.

    Docker desktop isn't available on Linux, you can use Kind instead.

    Installing Onyxia using helm

    As of today the default service catalog will only work with ingress-nginx.

    This will be addressed in the near future.

    Through out this guide we make as if everything was instantaneous. In reality if you are testing on a small cluster you will need to wait several minutes after hitting helm install for the services to be ready.

    Use kubectl get pods to see if your pods are up and ready.

    (Optional) Make sure that your cluster is ready for Onyxia

    To make sure that your Kubernetes cluster is correctly configured let's deploy a test web app on it before deploying Onyxia.

    DOMAIN=my-domain.net
    
    cat << EOF > ./test-spa-values.yaml
    ingress:
      enabled: true
      annotations:
        kubernetes.io/ingress.class: nginx
      hosts:
        - host: test-spa.lab.$DOMAIN
    EOF
    
    helm repo add etalab https://etalab.github.io/helm-charts
    helm install test-spa etalab/keycloakify-demo-app -f test-spa-values.yaml
    echo "Navigate to https://test-spa.lab.$DOMAIN, see the Hello World"
    helm uninstall test-spa

    You have the ability to customize the user interface (UI) of Onyxia through the provision of specific environment variables to the UI. For details on the available options, please consult the 'UI Customization' section of this file.

    If you are unsure about how to supply these variables, refer to the later section of this guide where we discuss how to provide the KEYCLOAK_* parameters. You'll then be able to add your UI-related parameters alongside them.

    Enabling user authentication

    Notes if you already have a Keycloak server

    If you already have a Keycloak server it is up to you to pick from this guide what is rellevent to you.

    Main takeway is that you probably want to load the Onyxia custom Keycloak theme and enable -Dkeycloak.profile=preview in order to be able to enforce that usernames are well formatted and define an accept list of email domains allowed to create account on your Onyxia instance.

    You probably want to enable Terms and Conditions as required actions.

    That out of the way, note that you can configure onyxia-web to integrate with your existing Keycloak server, you just need to set some dedicated environment variable in the values.yaml of the onyxia helm chart. Example:

    S3 Storage

    Create a Keycloak client for Accessing Keycloak

    Install MinIO

    Create a Keycloak Client for Onyxia/MinIO

    Update Onyxia

    Vault

    terraforming
    AWS
    GCP
    Azure
    configure kubectl section
    LetsEncrypt
    certbot
    Docker desktop
    forward the TCP ports 80 and 443 to your local machine
    codecentric's helm chart
    sspcloud
    AWS Security Token Service API
    MinIO
    MinIO documentation
    link
    github
    Refer to the legacy documentation.
    Enable Kubernetes in Docker desktop
    onyxia.my-domain.net A <YOUR_IP>
    *.lab.my-domain.net  A <YOUR_IP>
    brew install certbot #On Mac, lookup how to install certbot for your OS
    
    # Because we need a wildcard certificate we have to complete the DNS callange.  
    sudo certbot certonly --manual --preferred-challenges dns
    
    # When asked for the domains you wish to optains a certificate for enter:
    #   onyxia.my-domain.net *.lab.my-domain.net
    kubectl create namespace ingress-nginx
    DOMAIN=my-domain.net
    sudo kubectl create secret tls onyxia-tls \
        -n ingress-nginx \
        --key /etc/letsencrypt/live/onyxia.$DOMAIN/privkey.pem \
        --cert /etc/letsencrypt/live/onyxia.$DOMAIN/fullchain.pem
    cat << EOF > ./ingress-nginx-values.yaml
    controller:
      extraArgs:
        default-ssl-certificate: "ingress-nginx/onyxia-tls"
    EOF
    
    helm install ingress-nginx ingress-nginx \
        --repo https://kubernetes.github.io/ingress-nginx \
        --namespace ingress-nginx \
        -f ./ingress-nginx-values.yaml

    If you have DDNS domain you can create CNAME instead example:

    You can customise "onyxia" and "lab" to your liking, for example you could chose datalab.my-domain.net and *.kub.my-domain.net.

    The obtained certificate needs to be renewed every three month.

    To avoid the burden of having to remember to re-run the certbot command periodically you can setup and configure a on your cluster but that's out of scope for Onyxia.

    You may need to delegate your DNS Servers to one of the supported .

    Anything contained in the src/ui directory.

    The UI toolkit used in the project, you can find the setup of onyxia-UI in onyxia-web here: src/ui/theme.tsx.

    Onyxia-UI is fully compatible with MUI.

    Onyxia-UI offers a library of reusable components but you can also use MUI components in the project, their aspect will automatically be adapted to blend in with the theme.

    We currently offers builtin support for four color palettes:

    • France: datalab.sspcloud.fr?THEME_ID=france

    • Ultraviolet: datalab.sspcloud.fr?THEME_ID=ultraviolet

    • Verdant: datalab.sspcloud?THEME_ID=verdant

    • Onyxia (default):

    You can also provide your own palette.

    The fonts are loaded in the public/index.html. It's important to keep it that way for Keycloakify.

    To release a new version of Onyxia-UI. You just need to bump the package.json's version and push. The CI will automate publish a new version on NPM.

    If you want to test some changes made to onyxia-ui in onyxia-web before releasing a new version of onyxia-ui to NPM you can link locally onyxia-ui in onyxia-web.

    Now you can make changes in ~/github/onyxia/ui/and see the live updates.

    If you want to install/update some dependencies, you must remove the node_modules, do you updates, then link again.

    The library we use for styling.

    Rules of thumbs when it comes to styling:

    • Every component should accept an optional classNameprop it should always overwrite the internal styles.

    • A component should not size or position itself. It should always be the responsibility of the parent component to do it. In other words, you should never have height, width, top, left, right, bottom or margin in of your components.

    • You should never have a color or a dimension hardcoded elsewhere than in . Use theme.spacing() (, , ) and .

    Onyxia is mostly used on desktop computer screens. It's not worth the effort to create a fully flege responsive design for the UI. screen-scaler enables us to design for a sigle canonical screen size. The library take charge of scaling/shrinking the image. depending on the real size of the screen. It also asks to rotate the screen when the app is rendered in protrait mode.

    It enables us to test the graphical components in isolation. See sources.

    To launch Storybook locally run the following command:

    We need to be able to do:

    Then, somehow, access OIDC_URL in the code like process.env["OIDC_URL"].

    In theory it shouldn't be possible, onyxia-web is an SPA, it is just static JS/CSS/HTML. If we want to bundle values in the code, we should have to recompile. But this is where cra-envs comes into play.

    It enables to run onyxia-web again a specific infrastructure while keeping the app docker image generic.

    Checkout the helm chart:

    • All the accepted environment variables are defined here: .env. They are all prefixed with REACT_APP_ to be compatible with create-react-app. Default values are defined in this file.

    • Only in development (yarn start) .env.local is also loaded and have priority over .env

    • Then, in the code the variable can be accessed .

    It's a collection general purpose react hooks. Let's document the few use cases you absolutely need to understand:

    For the sake of performance we enforce that every component be wrapped into React.memo(). It makes that a component only re-render if one of their prop has changed.

    However if you use inline functions or useCallback as callbacks props your components will re-render every time anyway:

    We always use useConstCallback for callback props. And useCallbackFactory for callback prop in lists.

    It is very handy to be able to get the height and the width of components dynamically. It prevents from having to hardcode dimension when we don’t need to. For that we use useDomRect``

    It's a build tool that enables to implement the login and register pages that users see when they are redirected to Keycloak for authentication.

    If the app is being run on Keycloak the kcContext isn't undefined and it means shat we should render the login/register pages.

    If you want to test, uncomment this line and run yarn start. You can also test the login pages in a local keycloak container by running yarn keycloak. All the instructions will be printed on the console.

    The keycloak-theme.jar file is automatically build and uploaded as a GitHub release asset by the CI.

    The library we use for routing. It's like react-router but type safe.

    For internalization and translation.

    The project is a non-ejected create-react-app using typescript template (you can find here the template repo that was used as a base for this project).

    We use react-app-rewired instead of the default react-scripts to be able to use custom Webpack plugins without having to eject the App. The custom webpack plugins that we use are defined here /config-overrides.json. Currently we only one we use is circular-dependency-plugins.

    Anything contained in the src/core directory.

    The framework used to implement strict separation of concern betwen the UI and the Core and high modularity of the code.

    EVT is an event management library (like RxJS is).

    A lot of the things we do is powered under the hood by EVT. You don't need to know EVT to work on onyxia-web however, in order to demystify the parts of the codes that involve it, here are the key ideas to take away:

    • If we need to perform particular actions when a value gets changed, we useStatefullEvt.

    • We use Ctxto detaches event handlers when we no longer need them. (See line 108 on this playground)

    • In React, we use the useEvt hook to work with DOM events.

    Modules marked by 🐔 are our own.

    Typescript

    You can skip anything related to class we don't do OOP in the project.

    tsafe 🐔

    TS-CI 🐔

    For working on what the end user 👁

    What's new section
    tsafe
    assert
    id
    Equals
    symToStr
    cd ~/github
    git clone https//github.com/InseeFrLab/onyxia
    cd onyxia/web
    yarn install
    
    cd ~/github/onyxia #This is just a suggestion, clone wherever you see fit.
    git clone https://github.com/InseeFrLab/onyxia-ui ui
    cd ui
    yarn install
    yarn build
    yarn link-in-web
    npx tsc -w
    
    # Open a new terminal
    cd ~/github/onyxia/web
    yarn start
    
    yarn storybook
    docker run --env OIDC_URL="https://url-of-our-keycloak.fr/auth" InseeFrLab/onyxia-web
      web:
        replicaCount: 2
        env:
          MINIO_URL: https://minio.lab.sspcloud.fr
          VAULT_URL: https://vault.lab.sspcloud.fr
          OIDC_URL: https://auth.lab.sspcloud.fr/auth
          OIDC_REALM: sspcloud
          TITLE: SSP Cloud

    Onyxia-UI 🐔

    integration

    🎨 Palettes

    🔡 Fonts

    🔡 Linking onyxia-ui in onyxia-web

    tss-react 🐔

    screen-scaler 🐔

    Storybook

    cra-envs 🐔

    Please try not to access the environment variable to liberally through out the code. In principle they should only be accessed . We try to keep things as much as possible.

    powerhooks 🐔

    Avoiding useless re-render of Components

    Measuring Components

    Keycloakify 🐔

    type-routes

    i18nifty 🐔

    create-react-app

    We plane to move to Vite when will support it.

    For working on 🧠 of the App

    redux-clean-architecture 🐔

    EVT 🐔

     web:
      env:
        # Available env are documented here: https://github.com/InseeFrLab/onyxia-web/blob/main/.env
        KEYCLOAK_URL: https://auth.lab.my-domain.net/auth
        KEYCLOAK_CLIENT_ID: onyxia
        KEYCLOAK_REALM: datalab
        JWT_EMAIL_CLAIM: email
        JWT_FAMILY_NAME_CLAIM: family_name
        JWT_FIRST_NAME_CLAIM: given_name
        JWT_USERNAME_CLAIM: preferred_username
        JWT_LOCALE_CLAIM: locale
    onyxia.my-domain.net CNAME jhon-doe-home.ddns.net.
    *.lab.my-domain.net  CNAME jhon-doe-home.ddnc.net.
    cert-manager
    DNS01 challenge provider
    DNS service provider
    The hello world SPA deployed
    datalab.sspcloud.fr?THEME_ID=onyxia
    the root styles
    the theme configuration
    ex1
    ex2
    ex3
    theme.colors.useCases.xxx
    like this
    MUI
    here
    pure
    Keycloakify
    GitHub - InseeFrLab/helm-charts-interactive-services: Collection of helm charts to deploy services from Onyxia's data science catalogGitHub
    GitHub - InseeFrLab/helm-charts-interactive-services: Collection of helm charts to deploy services from Onyxia's data science catalogGitHub
    Logo
    Logo
    https://mango-dune-07a8b7110.1.azurestaticapps.net/?repo=InseeFrLab/onyxia-apimango-dune-07a8b7110.1.azurestaticapps.net
    GitHub - InseeFrLab/helm-charts-automationGitHub
    Logo
    GitHub - InseeFrLab/helm-charts-automationGitHub
    Logo
    Provision an AKS cluster in Azure | Terraform | HashiCorp DeveloperProvision an AKS cluster in Azure | Terraform | HashiCorp Developer
    Logo
    GitHub - InseeFrLab/helm-charts-databases: A collection of helm charts to deploy BDD inside Onyxia DatalabGitHub
    GitHub - InseeFrLab/helm-charts-databases: A collection of helm charts to deploy BDD inside Onyxia DatalabGitHub
    Provision an EKS cluster (AWS) | Terraform | HashiCorp DeveloperProvision an EKS cluster (AWS) | Terraform | HashiCorp Developer
    Logo
    Logo
    tsafe — Leverage the more advanced TypeScript featureswww.tsafe.dev
    Logo
    GitHub - garronej/ts-ci: 🚀 A project starter for module publisherGitHub
    Logo
    Logo
    Provision a GKE cluster (Google Cloud) | Terraform | HashiCorp DeveloperProvision a GKE cluster (Google Cloud) | Terraform | HashiCorp Developer
    Logo
    Setting up a new story
    GitHub - keycloakify/keycloakify: 🔏 Keycloak theming for the modern webGitHub
    Create React App is deprecated.create-react-app.dev
    Logo
    GitHub - garronej/tss-react: ✨ Dynamic CSS-in-TS solution, based on EmotionGitHub
    Logo
    GitHub - InseeFrLab/onyxia-ui: 🌓 Onyxia UI toolkitGitHub
    GitHub - garronej/cra-envs: ⚙️ Bundle env var in CRA at launch time!GitHub
    Logo
    GitHub - garronej/screen-scaler: 🖥️ One-size-fit-all web design: Screen-size agnostic development environment.GitHub
    Logo
    GitHub - garronej/powerhooks: A collection of powerfull react hooksGitHub
    Storybook: Frontend workshop for UI developmentStorybook
    Logo
    Logo
    GitHub - garronej/clean-architecture: 📐 A clean architecture frameworkGitHub
    Logo
    Logo
    Logo
    GitHub - zilch/type-route: The flexible, type safe routing library.GitHub
    EVT - EventEmitter's typesafe replacementwww.evt.land
    Logo
    Playground to explain the usefulness of useConstCallback
    Logo
    i18niftywww.i18nifty.dev
    Logo