Duplicate to your account
18min
|
|
Updated 22 Feb 2023

Rucio Step-by-step

Running Rucio Demo Environment and Rucio UI on Ubuntu

Prerequisite - Install Docker

Docker installed and running: https://docs.docker.com/desktop/install/linux-install/ 

The following commands were used when following the steps to install on a fresh Ubuntu instance: https://docs.docker.com/desktop/install/ubuntu/ 

Terminal
|
sudo apt-get update 
sudo apt-get install ca-certificates curl gnupg lsb-release 
sudo apt install docker-compose 
sudo mkdir -p /etc/apt/keyrings 
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg 
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null 
sudo apt-get update 
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin 
sudo service docker start 
sudo docker run hello-world 
sudo curl -k -O -L https://desktop.docker.com/linux/main/amd64/docker-desktop-4.12.0-amd64.deb
sudo apt-get update 
sudo apt-get install ./docker-desktop-4.12.0-amd64.deb 
systemctl --user start docker-desktop 
docker-compose version 
docker version


Step 1 - Rucio Server

Install Rucio demo environment by following the steps outlined in https://app.archbee.com/doc/LUZSWyVR7NPwpyzgBD7S6/-M-Y16M3mlUPcM9wQnZFr 

Clone Rucio

Terminal
|
git clone https://github.com/rucio/rucio
cd rucio


(Optional) Update Docker Compose File

Note that if you plan to access rucio outside of localhost you need to update the extra_hosts and ports within the rucio service in the ./etc/docker/dev/docker-compose.yml file:

Original

Terminal
|
extra_hosts:
      - "ruciodb:127.0.0.1"
      - "graphite:127.0.0.1"
      - "influxdb:127.0.0.1"
      - "elasticsearch:127.0.0.1"
      - "activemq:127.0.0.1"
ports:
    - "127.0.0.1:8443:443"
    - "127.0.0.1:5432:5432"
    - "127.0.0.1:8080:80"
    - "127.0.0.1:8086:8086"
    - "127.0.0.1:9200:9200"
    - "127.0.0.1:9300:9300"
    - "127.0.0.1:61613:61613"


Updated

Terminal
|
extra_hosts:
      - "ruciodb:0.0.0.0"
      - "graphite:0.0.0.0"
      - "influxdb:0.0.0.0"
      - "elasticsearch:0.0.0.0"
      - "activemq:0.0.0.0"
ports:
    - "0.0.0.0:8443:443"
    - "0.0.0.0:5432:5432"
    - "0.0.0.0:8080:80"
    - "0.0.0.0:8086:8086"
    - "0.0.0.0:9200:9200"
    - "0.0.0.0:9300:9300"
    - "0.0.0.0:61613:61613"


Jump into the docker container

Terminal
|
docker-compose --file ./etc/docker/dev/docker-compose.yml up -d
docker exec -it dev_rucio_1 /bin/bash



The rest of the commands are to be executed within the Docker container

Initialize the database

Terminal
|
tools/run_tests_docker.sh -i


Update the ca-certificates package

The image has python3 linked in /usr/bin/python but yum requires python2.

One workaround is to change the first line of /usr/bin/yum and /usr/libexec/urlgrabber-ext-down to use /usr/bin/python2:

  • vi /usr/bin/yum
    • change "python" to "python2" in the first line
  • vi /usr/libexec/urlgrabber-ext-down
    • now change "python" to "python2" in the first line
Terminal
|
yum update ca-certificates


(Optional) Generate New Cert for Custom IP

If you will be accessing the Rucio server outside localhost, you will need to update the cert to include your IP or domain. Replace PUBLIC_IP_HERE with your IP or add a DNS entry for a custom domain name.

Terminal
|
openssl req -new -key /etc/grid-security/hostkey.pem -nodes  -subj \
  "/CN=rucio" > /etc/grid-security/hostcert_rucio.csr

openssl x509 -req -days 9999 -CAcreateserial -extfile \
  <(printf "subjectAltName=DNS:rucio,DNS:localhost,IP:PUBLIC_IP_HERE") \
  -in /etc/grid-security/hostcert_rucio.csr \
  -CA /opt/rucio/etc/certs/rucio_ca.pem \
  -CAkey /opt/rucio/etc/certs/rucio_ca.key.pem \
  -out /etc/grid-security/hostcert.pem \
  -passin pass:123456


Configure the Storj RSE

Terminal
|
# replace YOUR_BUCKET_NAME with your storj bucket
rucio-admin rse add STORJ
rucio-admin rse add-protocol --hostname gateway.storjshare.io --scheme https --port 443 --prefix YOUR_BUCKET_NAME --impl rucio.rse.protocols.gfal.NoRename --domain-json '{"wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}, "lan": {"read": 1, "write": 1, "delete": 1}}' STORJ
rucio-admin rse set-attribute --rse STORJ --key sign_url --value s3
rucio-admin rse set-attribute --rse STORJ --key skip_upload_stat --value True
rucio-admin rse set-attribute --rse STORJ --key verify_checksum --value False
rucio-admin rse set-attribute --rse STORJ --key strict_copy --value True


Create rse-accounts.cfg

Terminal
|
# get your rse_id
rucio-admin rse info STORJ

# use your rse_id and s3 credentials below
cat >> etc/rse-accounts.cfg <<EOL
{
    "YOUR_RSE_ID": {
        "access_key": "YOUR_ACCESS_KEY",
        "secret_key": "YOUR_SECRET_KEY",
        "signature_version": "s3v4",
        "region": "us-east-1"
    }
}
EOL


Upload and download a file

Terminal
|
rucio upload --rse STORJ --scope archive tools/test.file.1000
rucio download archive:test.file.1000


(Optional) Restart Rucio Container

If you updated certificates to include a custom domain, you may need to restart the server. You can do this by restarting the containers associated with the rucio server:

Terminal
|
docker restart dev_rucio_1
docker restart dev_influxdb_1
docker restart dev_ruciodb_1
docker restart dev_graphite_1
docker restart dev_elasticsearch_1
docker restart dev_activemq_1


Step 2 - Rucio WebUI

Run Rucio WebUI by following the steps in https://hub.docker.com/r/rucio/rucio-ui using the information about your running Rucio instance above

A WebUI instance with the minimal configuration can be started like this:

Note that the paths below beginning with /root/rucio reference the path to the rucio project that was git cloned in Rucio Step-by-step above. The /rucio folder will likely be somewhere else on your machine, so make sure to update those references before executing the docker run command.

Terminal
|
docker run \
  --name=rucio-webui \
  -v /root/rucio/etc/certs/rucio_ca.pem:/etc/grid-security/ca.pem \
  -v /root/rucio/etc/certs/hostcert_rucio.pem:/etc/grid-security/hostcert.pem \
  -v /root/rucio/etc/certs/hostcert_rucio.key.pem:/etc/grid-security/hostkey.pem \
  -e RUCIO_CFG_DATABASE_DEFAULT=postgresql://rucio:secret@localhost/rucio \
  -e RUCIO_CFG_DATABASE_SCHEMA=dev \
  -p 443:443  --network="host" \
  -e RUCIO_PROXY="localhost:8443" \
  -e RUCIO_AUTH_PROXY="localhost:8443" \
  rucio/rucio-ui


Now you should be able to access the Rucio WebUI at https://localhost:443 or https://<your-public-ip> (the app is listening on port 443, so as long as you use https, you shouldn't need the port number) and login with the demo credentials:

  • Username: ddmlab
  • Password: secret
Document image

Document image


Step 3 - Rucio Client

If you are launching the container from the same host that Rucio server is running on:

Shell
|
docker run \
   -e RUCIO_CFG_RUCIO_HOST=https://localhost:8443 \
   -e RUCIO_CFG_AUTH_HOST=https://localhost:8443 \
   -e RUCIO_CFG_AUTH_TYPE=userpass \
   -e RUCIO_CFG_USERNAME=ddmlab \
   -e RUCIO_CFG_PASSWORD=secret \
   -e RUCIO_CFG_ACCOUNT=root \
   -e RUCIO_CFG_CA_CERT=/opt/rucio/etc/ca.pem \
   -v /root/rucio/etc/certs/rucio_ca.pem:/opt/rucio/etc/ca.pem \
   --name=rucio-client  --network="host" \
   -it --rm rucio/rucio-clients


If you are launching the container from a different host:

Terminal
|
docker run \
   -e RUCIO_CFG_RUCIO_HOST=https://PUBLIC_IP_HERE:8443 \
   -e RUCIO_CFG_AUTH_HOST=https://PUBLIC_IP_HERE:8443 \
   -e RUCIO_CFG_AUTH_TYPE=userpass \
   -e RUCIO_CFG_USERNAME=ddmlab \
   -e RUCIO_CFG_PASSWORD=secret \
   -e RUCIO_CFG_ACCOUNT=root \
   -e RUCIO_CFG_CA_CERT=/opt/rucio/etc/ca.pem \
   -v /root/rucio/etc/certs/rucio_ca.pem:/opt/rucio/etc/ca.pem \
   --name=rucio-client  --network="host" \
   -it --rm rucio/rucio-clients




Docs powered by archbee 
This content is not produced, endorsed nor maintained by Archbee.
If there's been an abuse, please report it here
TABLE OF CONTENTS
Running Rucio Demo Environment and Rucio UI on Ubuntu
Prerequisite - Install Docker
Step 1 - Rucio Server
Step 2 - Rucio WebUI
Step 3 - Rucio Client