# Build the dsci-lab

This file:
* Build of `dsci-lab_22.04_ss24*`, 2024-03-02
* Basis: Xubuntu 22.04.4 LTS (Jammy Jellyfish), 64 bit

previous versions:
* dsci-lab WS 2023: <http://www.jbusse.de/2023_ws_dsci-lab/>

Important: You are *not* requested to build the dsci-lab by yourself. Instead you are advised to download the fully configured dsci-lab as a ready-to-use virtual machine, see {doc}`t_download`.

However, if you are a lecturer or a deeply interested student you might want to learn how we set up the dsci-lab. Here are the steps.



## Install VM Virtual Box 6.1

* {ref}`install-virtualbox`

## Create the virtual XUbuntu machine

Download an ISO image:
* e. g. from <https://cdimage.ubuntu.com/xubuntu/releases/jammy/release/>
* file xubuntu-22.04.4-desktop-amd64.iso : <https://cdimage.ubuntu.com/xubuntu/releases/jammy/release/xubuntu-22.04.4-desktop-amd64.iso> , 	2024-02-16 18:48, 3.0G: right click on file, "save as", save to disk (e.g. `/home/user/v/`)

Start Oracle VM Virtual Box Manager
 
* Maschine > Neu > Name: e.g. *dsci-lab_22.04_ss24*
* Ort: z.B. `/home/jb/VirtualBox VMs`
* Typ: Linux, Version: Ubuntu (64-bit)
* Experten-Modus
* Speichergröße: minimal 4096 MB, besser z.B. 16000 MB ;-) )
* RAM Grafikkarte: 128 MB
* Platte: Festplatte erzeugen > Erzeugen
* Dateityp der Festplatte: VDI
* Art der Speicherung: dynamisch alloziert
* Dateiname und Größe: hier mindestens 100 GB angeben > Erzeugen

Start machine *dsci-lab_22.04_ss24*. A window will pop up: "Medium für Start auswählen"
* Medium hinzufügen > e.g. `/home/jb/v/xubuntu-22.04.3-desktop-amd64.iso` on your disc > Auswählen
* Starten > Try or install XUbuntu

VirtualBox will start the ISO image:

* Select language, e.g. "Deutsch" >  "Install XUbuntu"
* Tastaturbelegung: German > German
* Aktualisierungen herunterladen
* Installationsart: Festplatte löschen und installieren (this will clean only your newly allocated virtual hard disc, NOT the disk of your host system;-) ); > Installieren
* Wo befinden Sie sich? > Berlin
* "Who are You?":
   * Name: install
   * Name des Rechners: dsci-lab-ss24
   * Benutzername: install
   * password: 1nstall (the first char is the number "1" one instead of the letter "i")

A basic version of xubuntu will be installed.

"Die Installation ist abgeschlossen. Sie müssen jetzt den Rechner neu starten, um das System zu benutzen" > Jetzt neu starten
* Virtualbox will become black and "freeze". Close the Window: "X"-button > die virtuelle Maschine ausschalten
* Start the new machine again.
* Log in with user *install*, password *1nstall*

Whisker-Menue > Einstellungen > Sprachen
* Unvollständige Sprachunterstützung: Sprachunterstützung nachinstallieren? > Ja; (on request enter password, i.e. "dscidsci")
* Sprache für Menues und Fenster (je nach Geschmack): das dsci-lab verwendet English ; "Ziehen Sie die Sprachen in die gewünschte Reihenfolge";  systemweit anwenden

## Update Basic System

"Aktualisierungsverwaltung" will start automatically and ask you to update the system: "Aktualisierte Anwendungen wurden seit der Veröffentlichung von Ubuntu 22.04 herausgegeben" > "jetzt installieren". 

Alternatively you may want to start Aktualisierungsverwaltung manually:
* [Whisker-Menu](https://wiki.ubuntuusers.de/Whisker_Menu/) > Einstellungen > Aktualisierungsverwaltung

To update your installation, regularly start "Aktualisierungsverwaltung". As an alternative to using "Aktualisierungsverwaltung" you can keep the system current with `apt`:

```sh
sudo apt update; sudo apt upgrade
```
Try it! Your system already should be up to date.

Restart the machine afterwards. Login again.

* "Update standard folders to current language?" > im dsci-lab wollen wir englische Ordnernamen, also "update names"

We will use the command line hereafter where possible. Get a new terminal by typing `Strg-Alt-t`. 



## VirtualBox Guest Extensions

To install new kernel modules - i.e. Oracle VirtualBox Guest Extensions - we need (the gnu C compiler) `gcc`:

```sh
sudo apt install gcc make perl
```

Oracle Virtual Box > Geräte > Gasterweiterungen einlegen: A window pops up, showing the directory `/media/data/VBox_GAs_6.1.42./` > rightclick on background, "Open Terminal here",  a new terminal opens. Type in:

```sh
sudo ./VBoxLinuxAdditions.run
```

Some new kernel modules will be built. To replace the running kernel modules with the new ones you have to reboot the VM:

```sh
sudo reboot
```

Login again. You now should be able to resize the VM window. Try it!

Activate bidirectional Copy & Paste from Windows-Host to VB etc. 

```
Geräte > Gemeinsame Zwischenablage > bidirektional
```

After updating virtualbox on the host you might want to update the guest extensions on the guest. Simply repeat the steps above. Donn't forget to reboot your guest engine.

## Firefox


Xubuntu 22 comes with a Snap-Version of firefox. This causes some ugly configuration issues. We remove this (so called "rapid") version of firefox and echange it with a `.deb` Version:

* <https://www.omgubuntu.co.uk/2022/04/how-to-install-firefox-deb-apt-ubuntu-22-04>

Test Firefox: Whisker-Menu "Internetnavigator" launces Firefox. Navigate to <http://jbusse.de/dsci-lab/dsci-lab-build.html>. Alternatively manually type in (and don't forget to add the ampersand "&" at the end of the line):

```sh
firefox http://jbusse.de/dsci-lab/r_build.html &
```
	
If your WLAN works properly, Firefox will open the page you are actually reading here in your new linux machine. This allows you to copy & paste the following commands to your terminal. Bookmark this page in Firefox by hitting CTRL-D.


## Firefox ESR

Install Firefox ESR: <https://ubuntuhandbook.org/index.php/2022/03/install-firefox-esr-ubuntu/>

```
sudo add-apt-repository ppa:mozillateam/ppa
sudo apt update
sudo apt install firefox-esr
```



### Firefox Privacy

Start firefox. Hit the "ALT"-button to show the menu.

Edit > Settings (Bearbeiten > Einstellungen):
* Home > New Windows and Tabs (Startseite und neue Fenster): point default to `https://www.startpage.com/`
  * Inhalte des Firefox-Startbildschirms: privacy! ... but do not deactivate these widges (don't close your eyes), but go to
* Datenschutz und Sicherheit > Benuzerdefiniert > Cookies und Website-Daten
  * Cookies und Website-Daten beim Beenden von Firefox löschen: Check!
  * Hauptpasswort verwenden: Enter a save password!
  * Chronik: Firefox wird eine Chronik > nach benutzerdefinierten Einstellungen anlegen
    * "Die Chronik löschen, wenn Firefox geschlossen wird": Check!
  * Datenerhebung durch Firefox und deren Verwendung
    * (de-)select whatever you want

Extras > Add-Ons und Themes > Erweiterungen
  * Ghostery – datenschutzorientierter Werbeblocker
    * optionally configure Ghostery: "ausführen in privaten Fenstern erlauben"


<!-- View markdown with Firefox

(1) install Add-On: Extras > Add-Ons und Themes > Erweiterungen
  * Markdown Viewer Webext von Kulero, Cimbali

(2) add MIME type: <https://github.com/KeithLRobertson/markdown-viewer#support-for-local-files-on-linux>: 
Firefox on Linux may not know how to handle markdown files by default (see #2). There are a number of possible workarounds for this (see this SuperUser question for example). [...] Another workaround (which might cover other OSs as well), is to edit Firefox’s private mime types. These mime types are stored in a file indicated by helpers.private_mime_types_file, by default it is `~/.mime.types`:

```sh
echo 'type=text/plain exts=md,mkd,mkdn,mdwn,mdown,markdown, desc="Markdown document"' >> ~/.mime.types
```

Then restart firefox. Firefox now should render markdown files as html.

-->


### personalize this file 

To document subsequent personal installations of your VM you might wish to update this very file 
you are reading currently.

* In Firefox push the `.md`-Button in the upper right corner of the page
and save the file to `Downloads/dsci-lab-build.md`. 
* Alternatively you might want to download the markdown-source of this file locally:

```sh
cd ~/Downloads
wget http://jbusse.de/dsci-lab/_sources/r_build.md
cd
```

Edit the file `dsci-lab-build.md` e.g. with Mousepad. Alternatively you want to use your favourite programming editor. As of 2023 we recommend *Microsoft Visual Studio Code* (aka VS Code) <https://code.visualstudio.com/docs/python/python-tutorial>:

```sh
snap install code --classic
```

Only if you are over 50 years old ;-) you still want to stick to `vim` or `emacs`:

```sh
sudo apt install vim
sudo apt install emacs
```


> From File > Preferences > Settings, search for telemetry, and set the Telemetry: Telemetry Level setting to off. This will silence all telemetry events including crash reporting from VS Code. You will need to restart VS Code for the setting change to take effect. <https://code.visualstudio.com/docs/supporting/FAQ#_how-to-disable-crash-reporting>

Optionally also get MyST-Marksown Syntax extension for VS Code: <https://marketplace.visualstudio.com/items?itemName=ExecutableBookProject.myst-highlight>



## More basic Packages

At this stage you have a clean, brand new virtual XUbuntu machine, with VirtualBox Guest extensions enabled.

To get our dsci-lab version you have to install some more packages.  You can do so by simply copying the following commands into a terminal (open a new terminal e.g. by typing `Strg-Alt-t`):


### freeplane

Mindmap (incl. Java):

```sh
sudo apt install freeplane htop
```

### LaTeX


<https://jupyterbook.org/en/stable/advanced/pdf.html> recommends using the texlive distribution.

We have found that in combination with jupyterbook (see below) the following packes are actually required, with a total size about 2 GB (!):

```sh
sudo apt install imagemagick\
     	 	 texlive-latex-recommended texlive-latex-extra \
                 texlive-fonts-recommended texlive-fonts-extra \
                 texlive-xetex latexmk texlive-lang-german
sudo apt install xindy biber texlive-science
```

Test the integration of LaTeX installation and jupyterbook:

```sh
cd c
jb build test --builder pdflatex
atril test/_build/latex/book.pdf &
cd
```


# Add new non-privileged user

```sh
sudo adduser dsci 
sudo addgroup vboxsf
sudo adduser dsci vboxsf
```

Ausloggen als (Admin-) User Xubuntu, wieder einloggen als user dsci.

# Login as user dsci




## (Mini-)Conda

https://docs.conda.io/projects/conda/en/stable/user-guide/install/linux.html

Miniconda Installer:
<https://docs.conda.io/projects/miniconda/en/latest/> > *Quick command line install*:

```
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
```

> ==> For changes to take effect, close and re-open your current shell. <==

Why close and open? In an earlier step you have installed `conda`. Conda puts an extra virtual environment layer over the standard Python installation, so we can work with multiple Python configurations in parallel. To learn more about virtual environments:

* <https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-conda.html>
* <https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html>

Your termial command line now should start with `(base)`, which is the name of your current virtual environment:

```
(base) dsci@dsci-lab-ss2022:~$ 
```

Keep `conda` current:

```sh
conda update --all
```


(Note:Instead of installing miniconda you instead want to install `anaconda` (<https://www.anaconda.com/products/individual>). Anaconda is much more complete than miniconda, but IMHO fo "fat".  In our dsci-lab we prefer a lightweight system. This allows you to look more easily "under the hood", to understand what's going on, and to maintain the whole system - the dependencies in our setup are complex enough anyhow.)


### Jupyter Notebook

in WS 2023:

> As an existing Python user, you may wish to install Jupyter using Python’s package manager, pip, instead of Anaconda. (<https://test-jupyter.readthedocs.io/en/latest/install.html)

```sh
pip install jupyter
```


in WS 2022 (legacy): installation of jupyter with conda
```sh
sudo apt install jupyter
```

<!--
NEW since Ubuntu 22: Firefox is installed via snap. To start juypter we have to set additional environment variables:

```sh
export JUPYTER_DATA_DIR=/home/$USER/Public
export JUPYTER_RUNTIME_DIR=/home/$USER/Public
```

To set these variables permanently add them to the file `.profile`. This file will be executed the next tite you start the display manager. To execute `.profile` in a running display manager (i.e. this bash-session) simply source it:

```sh
source .profile
```
-->

Start Jupyter. Jupyter will launch firefox. Stop Jupyter again:

```sh
jupyter notebook &
jupyter notebook stop
```

### Install python packages into the virtual conda environment `base`

As said: Conda is minimalistic. Tus we have to install some modules by ourselfes.

Some important ones are:

```sh
pip install pandas numpy matplotlib scikit-learn seaborn xgboost  lxml markdown python-slugify rdflib owlrl markdownify
```


Notes:

* We do install these packages into the virtual conda environment `base`. If we decide to create another virtual conda environment, it will be empty again, and we have to populate it with libraries again. This is the reason (a) why we prefer lightweight environments, and (b) why we want to learn how to install libraries by ourselfes.
* Caveat: Install Conda *not with sudo*, but instead with the role of a normal user. Every user and every virtual conda environment are completely independent from each other. There is no system-wide installation.


### Jupyterbook

Install Jupyterbook (c.f. <https://pypi.org/project/jupyter-book/>)

```sh
pip install jupyter-book
```

<!--
Generate Jupyterbook config file:

```sh
jupyter notebook --generate-config
```

Edit the file `~/.jupyter/jupyter_notebook_config.py`:
* Search for the line `# c.NotebookApp.use_redirect_file = True`
* uncomment it, and change `True` to `False`. You should get

> c.NotebookApp.use_redirect_file = False
-->

https://jupytext.readthedocs.io/en/latest/install.html

```sh
pip install jupytext --upgrade
```


Test the jupyterbook installation: Build the book according to <https://jupyterbook.org/start/build.html>

```sh
mkdir -p ~/c
cd ~/c
jb create test
```

build html:

```sh
jb build test
firefox-esr test/_build/html/index.html &
```

build pdf via LaTeX:

```sh
jb build test --builder pdflatex
atril test/_build/latex/book.pdf &
```




### spaCy (not used)

<https://spacy.io/usage> :

* Linux, X86, conda, CPU
* NO virtual env
* Trained pipelines: English, German

```sh
conda install -c conda-forge spacy
python -m spacy download en_core_web_sm
python -m spacy download de_core_news_sm
```


## PyCharm (not used)

Warning: PyCharm is HUGE, we do not use it. (Rather try Visual Studio Code). However, if you want to play with PyCharm:

<https://www.jetbrains.com/help/pycharm/installation-guide.html>:

* RAM: 4 GB (min), 8 GB (recommended)
* Disk space: 2.5 GB and another 1 GB for caches (min), SSD drive with at least 5 GB of free space (recommended)
* how to install: Standalone installation > Linux > Install using snap packages > Community Edition (same as <https://snapcraft.io/install/pycharm-community/ubuntu>)

```sh
sudo snap install pycharm-community --classic
```

get started with PyCharm: <https://www.jetbrains.com/help/pycharm/quick-start-guide.html>

Run `pycharm-community` in the Terminal.

<!--
https://www.jetbrains.com/help/pycharm/migrating-from-text-editors.html#emacs_external_tool
-->

TBD: initially configure PyCharm
* point to our conda virtual environment, including python 3.9 interpreter


## Export OVA

Maschine aktualisieren, Gasterweiterungen installieren, Shared Folder deaktivieren, Medium auswerfen, Maschine 'runterfahren, USB auf 1.0 umstellen, dann *Export OVA*:
* /home/jb/v/dsci-lab_22.04_ss24.ova
* "alle Netzwerkadressen mit einbeziehen"
* Hersteller-URL: jbusse.de/dsci-lab

SHA und Größe der Datei feststellen:

```
ls -ls v/dsci-lab_22.04_ss24.ova
# 11558000 -rw------- 1 jb jb 11835387904 Sep 30 09:20 v/dsci-lab_22.04_ss24.ova

sha256sum v/dsci-lab_22.04_ss24.ova
# XXXXXf810530ff34b1a52d6b79f96cb313b8935ccfebd8dcbf56e7868e232cf5fb0a4  v/dsci-lab_22.04_ss24.ova
```

Auf den Server hochladen, auch dort `sha256sum` berechnen, kontrollieren, erst dann in {doc}`t_download` eintragen.




## Admin


Aktualisieren:

```sh
sudo apt update; sudo apt upgrade
sudo snap refresh
```

## next steps

<http://jbusse.de/dsci-lab/h_shared-folders.html>

<http://jbusse.de/dsci-lab/h_customizing.html>