2023年7月26日水曜日

ホームサーバーの環境移行(4)

気が付いたら5か月経っていた。早くね。
2月にやったことをちゃんとまとめていなかった報いか。

現状

  1. samba
  2. gogs
  3. MariaDB
  4. webmin

今回の対象

  • nextcolud

nextcloudの引っ越し

ここを参考にする
https://docs.nextcloud.com/server/latest/admin_manual/maintenance/index.html
https://hub.docker.com/_/nextcloud
https://blog.seigo2016.com/blog/h-blxsnew_s

事前準備

ユーザー作成

  • ユーザー:nextcloud_docker 1004
  • グループ:nextcloud-rtls-docker 200999

サブ UID/サブ GIDの設定

$ cat /etc/subuid
nextcloud_docker:200000:65536

$ cat /etc/subgid
nextcloud_docker:200000:65536

インストール

nextcloud_docker@blackcore:~$ dockerd-rootless-setuptool.sh install
[INFO] systemd not detected, dockerd-rootless.sh needs to be started manually:

PATH=/usr/bin:/sbin:/usr/sbin:$PATH dockerd-rootless.sh 

[INFO] CLI context "rootless" already exists
[INFO] Use CLI context "rootless"
Current context is now "rootless"

[INFO] Make sure the following environment variables are set (or add them to ~/.bashrc):

# WARNING: systemd not found. You have to remove XDG_RUNTIME_DIR manually on every logout.
export XDG_RUNTIME_DIR=/home/nextcloud_docker/.docker/run
export PATH=/usr/bin:$PATH
Some applications may require the following environment variable too:
export DOCKER_HOST=unix:///home/nextcloud_docker/.docker/run/docker.sock
nextcloud_docker@blackcore:~$ cat .config/systemd/user/docker.service 
[Unit]
Description=Docker Application Container Engine (Rootless)
Documentation=https://docs.docker.com/go/rootless/

[Service]
Environment=PATH=/home/nextcloud_docker/bin:/sbin:/usr/sbin:/home/nextcloud_docker/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ExecStart=/bin/dockerd-rootless.sh
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
Type=notify
NotifyAccess=all
KillMode=mixed

[Install]
WantedBy=default.target

設定

loginctl enable-linger nextcloud_docker
nextcloud_docker@blackcore:~$ XDG_RUNTIME_DIR=/run/user/$(id -u nextcloud_docker) systemctl --user daemon-reload
nextcloud_docker@blackcore:~$ XDG_RUNTIME_DIR=/run/user/$(id -u nextcloud_docker) systemctl --user start docker
nextcloud_docker@blackcore:~$ XDG_RUNTIME_DIR=/run/user/$(id -u nextcloud_docker) systemctl --user status docker
● docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/home/nextcloud_docker/.config/systemd/user/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-02-22 06:07:22 JST; 3s ago
       Docs: https://docs.docker.com/go/rootless/
   Main PID: 912102 (rootlesskit)
      Tasks: 54
     Memory: 45.7M
        CPU: 205ms
     CGroup: /user.slice/user-1004.slice/user@1004.service/app.slice/docker.service
             ├─912102 rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /bin/dockerd-rootless.sh
             ├─912113 /proc/self/exe --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /bin/dockerd-rootless.sh
             ├─912132 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 912113 tap0
             ├─912140 dockerd
             └─912166 containerd --config /run/user/1004/docker/containerd/containerd.toml --log-level info

 2月 22 06:07:22 blackcore dockerd-rootless.sh[912140]: time="2023-02-22T06:07:22.095812655+09:00" level=warning msg="WARNING: No io.max (wbps) support"
 2月 22 06:07:22 blackcore dockerd-rootless.sh[912140]: time="2023-02-22T06:07:22.095815109+09:00" level=warning msg="WARNING: No io.max (riops) support"
 2月 22 06:07:22 blackcore dockerd-rootless.sh[912140]: time="2023-02-22T06:07:22.095817634+09:00" level=warning msg="WARNING: No io.max (wiops) support"
 2月 22 06:07:22 blackcore dockerd-rootless.sh[912140]: time="2023-02-22T06:07:22.095820449+09:00" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
 2月 22 06:07:22 blackcore dockerd-rootless.sh[912140]: time="2023-02-22T06:07:22.095823155+09:00" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
 2月 22 06:07:22 blackcore dockerd-rootless.sh[912140]: time="2023-02-22T06:07:22.095831981+09:00" level=info msg="Docker daemon" commit=bc3805a graphdriver=overlay2 version=23.0.1
 2月 22 06:07:22 blackcore dockerd-rootless.sh[912140]: time="2023-02-22T06:07:22.095857399+09:00" level=info msg="Daemon has completed initialization"
 2月 22 06:07:22 blackcore dockerd-rootless.sh[912140]: time="2023-02-22T06:07:22.104160647+09:00" level=info msg="[core] [Server #10] Server created" module=grpc
 2月 22 06:07:22 blackcore systemd[892130]: Started Docker Application Container Engine (Rootless).
 2月 22 06:07:22 blackcore dockerd-rootless.sh[912140]: time="2023-02-22T06:07:22.110857925+09:00" level=info msg="API listen on /run/user/1004/docker.sock"

XDG_RUNTIME_DIR=/run/user/$(id -u nextcloud_docker) systemctl --user enable docker

動作確認

hello-worldが動くかでテスト

nextcloud_docker@blackcore:~$ docker -H unix:///run/user/1004/docker.sock run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete 
Digest: sha256:6e8b6f026e0b9c419ea0fd02d3905dd0952ad1feea67543f525c73a0a790fefb
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

nextcloud用のdocker-composeを作成

nextcloud_docker@blackcore:/mnt/backuparea/nextcloud$ cat db.env
MYSQL_ROOT_PASSWORD=****************
MYSQL_PASSWORD=****************
MYSQL_DATABASE=nextcloud
MYSQL_USER=nextcloud

nextcloud_docker@blackcore:/mnt/backuparea/nextcloud$ cat docker-compose.yml 
version: '3'

services:
  db:
    image: mariadb:10.5
    command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
    restart: always
    volumes:
      - db:/var/lib/mysql
    environment:
      - MARIADB_AUTO_UPGRADE=1
      - MARIADB_DISABLE_UPGRADE_BACKUP=1
    env_file:
      - db.env
    ports:
      - 23306:3306

  redis:
    image: redis:alpine
    restart: always

  app:
    image: nextcloud:fpm-alpine
    restart: always
    volumes:
      - nextcloud:/var/www/html
    environment:
      - MYSQL_HOST=db
      - REDIS_HOST=redis
    env_file:
      - db.env
    depends_on:
      - db
      - redis

  web:
    image: nginx
    restart: always
    ports:
      - 28080:80
    volumes:
      - nextcloud:/var/www/html:ro
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - app

  cron:
    image: nextcloud:fpm-alpine
    restart: always
    volumes:
      - nextcloud:/var/www/html
    entrypoint: /cron.sh
    depends_on:
      - db
      - redis

volumes:
  db:
  nextcloud:

あとは

nextcloud_docker@blackcore:/mnt/backuparea/nextcloud$ cat nginx.conf 
worker_processes auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    # Prevent nginx HTTP Server Detection
    server_tokens   off;

    keepalive_timeout  65;

    #gzip  on;

    upstream php-handler {
        server app:9000;
    }

    server {
        listen 80;

        # HSTS settings
        # WARNING: Only add the preload option once you read about
        # the consequences in https://hstspreload.org/. This option
        # will add the domain to a hardcoded list that is shipped
        # in all major browsers and getting removed from this list
        # could take several months.
        #add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;" always;

        # set max upload size
        client_max_body_size 512M;
        fastcgi_buffers 64 4K;

        # Enable gzip but do not remove ETag headers
        gzip on;
        gzip_vary on;
        gzip_comp_level 4;
        gzip_min_length 256;
        gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
        gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

        # Pagespeed is not supported by Nextcloud, so if your server is built
        # with the `ngx_pagespeed` module, uncomment this line to disable it.
        #pagespeed off;

        # HTTP response headers borrowed from Nextcloud `.htaccess`
        add_header Referrer-Policy                      "no-referrer"   always;
        add_header X-Content-Type-Options               "nosniff"       always;
        add_header X-Download-Options                   "noopen"        always;
        add_header X-Frame-Options                      "SAMEORIGIN"    always;
        add_header X-Permitted-Cross-Domain-Policies    "none"          always;
        add_header X-Robots-Tag                         "none"          always;
        add_header X-XSS-Protection                     "1; mode=block" always;

        # Remove X-Powered-By, which is an information leak
        fastcgi_hide_header X-Powered-By;

        # Path to the root of your installation
        root /var/www/html;

        # Specify how to handle directories -- specifying `/index.php$request_uri`
        # here as the fallback means that Nginx always exhibits the desired behaviour
        # when a client requests a path that corresponds to a directory that exists
        # on the server. In particular, if that directory contains an index.php file,
        # that file is correctly served; if it doesn't, then the request is passed to
        # the front-end controller. This consistent behaviour means that we don't need
        # to specify custom rules for certain paths (e.g. images and other assets,
        # `/updater`, `/ocm-provider`, `/ocs-provider`), and thus
        # `try_files $uri $uri/ /index.php$request_uri`
        # always provides the desired behaviour.
        index index.php index.html /index.php$request_uri;

        # Rule borrowed from `.htaccess` to handle Microsoft DAV clients
        location = / {
            if ( $http_user_agent ~ ^DavClnt ) {
                return 302 /remote.php/webdav/$is_args$args;
            }
        }

        location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
        }

        # Make a regex exception for `/.well-known` so that clients can still
        # access it despite the existence of the regex rule
        # `location ~ /(\.|autotest|...)` which would otherwise handle requests
        # for `/.well-known`.
        location ^~ /.well-known {
            # The rules in this block are an adaptation of the rules
            # in `.htaccess` that concern `/.well-known`.

            location = /.well-known/carddav { return 301 /remote.php/dav/; }
            location = /.well-known/caldav  { return 301 /remote.php/dav/; }

            location /.well-known/acme-challenge    { try_files $uri $uri/ =404; }
            location /.well-known/pki-validation    { try_files $uri $uri/ =404; }

            # Let Nextcloud's API for `/.well-known` URIs handle all other
            # requests by passing them to the front-end controller.
            return 301 /index.php$request_uri;
        }

        # Rules borrowed from `.htaccess` to hide certain paths from clients
        location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/)  { return 404; }
        location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console)                { return 404; }

        # Ensure this block, which passes PHP files to the PHP process, is above the blocks
        # which handle static assets (as seen below). If this block is not declared first,
        # then Nginx will encounter an infinite rewriting loop when it prepends `/index.php`
        # to the URI, resulting in a HTTP 500 error response.
        location ~ \.php(?:$|/) {
            # Required for legacy support
            rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy) /index.php$request_uri;

            fastcgi_split_path_info ^(.+?\.php)(/.*)$;
            set $path_info $fastcgi_path_info;

            try_files $fastcgi_script_name =404;

            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $path_info;
            #fastcgi_param HTTPS on;

            fastcgi_param modHeadersAvailable true;         # Avoid sending the security headers twice
            fastcgi_param front_controller_active true;     # Enable pretty urls
            fastcgi_pass php-handler;

            fastcgi_intercept_errors on;
            fastcgi_request_buffering off;
        }

        location ~ \.(?:css|js|svg|gif)$ {
            try_files $uri /index.php$request_uri;
            expires 6M;         # Cache-Control policy borrowed from `.htaccess`
            access_log off;     # Optional: Don't log access to assets
        }

        location ~ \.woff2?$ {
            try_files $uri /index.php$request_uri;
            expires 7d;         # Cache-Control policy borrowed from `.htaccess`
            access_log off;     # Optional: Don't log access to assets
        }

        # Rule borrowed from `.htaccess`
        location /remote {
            return 301 /remote.php$request_uri;
        }

        location / {
            try_files $uri $uri/ /index.php$request_uri;
        }
    }
}

実行してみる

nextcloud_docker@blackcore:/mnt/backuparea/nextcloud$ docker-compose -H unix:///run/user/1004/docker.sock up
Starting nextcloud_db_1    ... done
Starting nextcloud_redis_1 ... done
Starting nextcloud_cron_1  ... done
Starting nextcloud_app_1   ... done
Starting nextcloud_web_1   ... done
Attaching to nextcloud_db_1, nextcloud_redis_1, nextcloud_app_1, nextcloud_cron_1, nextcloud_web_1
db_1     | 2023-03-16 20:30:42+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.19+maria~ubu2004 started.

動いた!


バックアップデータを復元してみる

https://docs.nextcloud.com/server/latest/admin_manual/maintenance/index.html
https://docs.nextcloud.com/server/latest/admin_manual/maintenance/migrating.html#

dbデータ

  • import
mysql -h localhost --port=23306 -u root -p nextcloud < nextcloud-sqlbkp.bak 
  • oc_storagesの変更

dataファイル

  • コピー
sudo cp -r nextcloud/ /mnt/backuparea/nextcloud/data/
sudo chown 200081 -R /mnt/backuparea/nextcloud/data/
sudo chgrp 200081 -R /mnt/backuparea/nextcloud/data/
  • パーミッション
adeno@blackcore:/mnt/backuparea/nextcloud$ ls -l data/nextcloud/data
合計 135988
drwxr-xr-x  7 200081 extcloud-rtls-docker      4096  2月 17  2019 admin
(略)
  • 備忘録
    最終手段でシンボリックリンクで対応したと思ったけど、何だっけ??

微調整

セキュリティ&セットアップ警告

HTTPSの対応

Traefikを試してみる?

https://coders-shelf.com/traefik-intro/
https://qiita.com/adwin/items/ccc34ef5f4c88d8fa02c#おまけ-https-化も楽勝

cat traefik.yml 
------------------------------------------------------------
api:
  insecure: true # WebUI にアクセスできるように設定
  dashboard: true

entryPoints:
  http:
    address: ":80"

  https:
    address: ":20443"

providers:
  docker:
#    network: sample_traefik
    exposedByDefault: false

cat docker-compose.yml 
------------------------------------------------------------
ersion: '3'

services:
  db:
    image: mariadb:10.5
    command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
    restart: always
    volumes:
      - ./data/db:/var/lib/mysql
    environment:
      - MARIADB_AUTO_UPGRADE=1
      - MARIADB_DISABLE_UPGRADE_BACKUP=1
    env_file:
      - db.env
    ports:
      - 23306:3306

  redis:
    image: redis:alpine
    restart: always

  app:
    image: nextcloud:fpm-alpine
    restart: always
    volumes:
      - ./data/nextcloud:/var/www/html
    environment:
      - MYSQL_HOST=db
      - REDIS_HOST=redis
    env_file:
      - db.env
    depends_on:
      - db
      - redis

  web:
    image: nginx
    restart: always
    ports:
      - 28080:80
    volumes:
      - ./data/nextcloud:/var/www/html:ro
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - app
    labels:
      - traefik.enable=true
      - traefik.http.routers.servicename.rule=Host(`blackcore.local`)
      - traefik.http.routers.servicename.entrypoints=https
      - traefik.http.routers.servicename.tls=true

  cron:
    image: nextcloud:fpm-alpine
    restart: always
    volumes:
      - ./data/nextcloud:/var/www/html
    entrypoint: /cron.sh
    depends_on:
      - db
      - redis

  traefik:
    image: traefik
    ports:
      - 28081:8080
      - 28082:80
      - 20443:20443
    volumes:
      - /run/user/1004/docker.sock:/var/run/docker.sock:ro
      - ./traefik.yml:/etc/traefik/traefik.yml

volumes:
  db:
  nextcloud:

httpsでのアクセスで、404 not foundとなってしまう。

nginxでhttpsを

そもそもリバースプロキシではなくて、nginxでhttps対応ができればやりたいことはできる。
nginx.confに以下を追加

cat nginx.conf
---
	# SSL configuration
	#
	listen 443 ssl default_server;
	listen [::]:443 ssl default_server;
 	#ssl_certificate /etc/nginx/server.crt;
	#ssl_certificate_key /etc/nginx/server.key;
	ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
	ssl_ciphers HIGH:!aNULL:!MD5;
	ssl_certificate     /etc/letsencrypt/live/*****.jp/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/*****.jp/privkey.pem;

長くなってきたし、時間も空いてしまってよくわからなくなってきたので、1回整理したい。

2023年2月16日木曜日

ホームサーバーの環境移行(3)

仮想化の検討

今後の可用性も考えて、サーバーに載せているサービスの仮想化やコンテナ化を考えてみる。
対象はなんだろう

  • samba
  • gogs
  • nextcloud
  • mydns/グローバルIP監視
  • ログインログアウト監視(Slack連携)
  • MariaDB
  • webmin

gogs

まずはgogs

現状

よかったメモしておいて
https://continue-to-challenge.blogspot.com/search?q=gogs

adeno@blackcube:/home/git$ systemctl status gogs
● gogs.service - Gogs (Go Git Service)
   Loaded: loaded (/etc/systemd/system/gogs.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2023-01-23 04:00:09 JST; 1h 47min ago
 Main PID: 2548 (gogs)
    Tasks: 8 (limit: 4915)
   CGroup: /system.slice/gogs.service
           └─2548 /home/git/gogs/gogs web

 1月 23 05:43:38 blackcube gogs[2548]: [Macaron] 2023-01-23 05:43:38: Started GET /admin/config for 192.168.1.37
 1月 23 05:43:38 blackcube gogs[2548]: [Macaron] 2023-01-23 05:43:38: Completed GET /admin/config 200 OK in 21.32905ms
 1月 23 05:43:38 blackcube gogs[2548]: [Macaron] 2023-01-23 05:43:38: Started GET /assets/font-awesome-4.6.3/fonts/fontawesome-webfont.woff2?
 1月 23 05:43:38 blackcube gogs[2548]: [Macaron] [Static] Serving /assets/font-awesome-4.6.3/fonts/fontawesome-webfont.woff2
 1月 23 05:43:38 blackcube gogs[2548]: [Macaron] 2023-01-23 05:43:38: Started GET /img/favicon.png for 192.168.1.37
 1月 23 05:43:38 blackcube gogs[2548]: [Macaron] [Static] Serving /img/favicon.png
 1月 23 05:43:38 blackcube gogs[2548]: [Macaron] 2023-01-23 05:43:38: Completed GET /img/favicon.png 200 OK in 2.466123ms
 1月 23 05:43:38 blackcube gogs[2548]: [Macaron] 2023-01-23 05:43:38: Completed GET /assets/font-awesome-4.6.3/fonts/fontawesome-webfont.woff
 1月 23 05:43:40 blackcube gogs[2548]: [Macaron] 2023-01-23 05:43:40: Started GET /admin/repos for 192.168.1.37
 1月 23 05:43:40 blackcube gogs[2548]: [Macaron] 2023-01-23 05:43:40: Completed GET /admin/repos 200 OK in 107.448955ms

たぶん、ここを参考にservice化したのだろう
https://github.com/gogs/gogs/blob/main/scripts/systemd/gogs.service

adeno@blackcube:/home/git$ cat /etc/systemd/system/gogs.service
[Unit]
Description=Gogs (Go Git Service)
After=syslog.target
After=network.target
After=mysqld.service

[Service]
# Modify these two values and uncomment them if you have
# repos with lots of files and get an HTTP error 500 because
# of that
###
#LimitMEMLOCK=infinity
#LimitNOFILE=65535
Type=simple
User=git
Group=git
WorkingDirectory=/home/git/gogs
ExecStart=/home/git/gogs/gogs web
Restart=always
Environment=USER=git HOME=/home/git

[Install]
WantedBy=multi-user.target

dockerでやってみる

https://github.com/gogs/gogs/tree/main/docker

sudo docker pull gogs/gogs
mkdir -p /mnt/workarea/gogs
sudo docker run --name=gogs -p 10022:22 -p 3000:3000 -v /mnt/workarea/gogs:/data gogs/gogs

久しぶりのdockerで使い方忘れてる
あと、gogsのデータの引っ越しはどうやるんだっけ?

adeno@blackcore:~$ sudo docker ps -a
[sudo] adeno のパスワード:          
CONTAINER ID   IMAGE       COMMAND                  CREATED      STATUS                  PORTS     NAMES
5c8dba490d7e   gogs/gogs   "/app/gogs/docker/st…"   9 days ago   Exited (0) 8 days ago             gogs

データの移行

https://github.com/gogs/gogs/discussions/6876

./gogs backup

で、書き出す。gogs-backup-20230123060827.zipが生成された。
/mnt/workarea/gogsに保存すると、docker内では/dataからアクセスできる。

adeno@blackcore:~$ sudo docker exec -it gogs /bin/bash
bash-5.1# ls
data    docker  gogs    log
bash-5.1# ./gogs -v
Gogs version 0.13.0+dev
bash-5.1# ls data/
gogs-backup-20230123060827.zip  gogs.db                         sessions
bash-5.1# 
bash-5.1# ./gogs restore --from="data/gogs-backup-20230123060827.zip" 
2023/01/31 16:28:35 [ INFO] Restoring backup from: data/gogs-backup-20230123060827.zip
2023/01/31 16:28:38 [FATAL] [gogs.io/gogs/gogs.go:40 main()] Failed to start application: init configuration: user configured to run Gogs is "git", but the current user is "root"
bash-5.1# 

カレントユーザーがrootになっているので、ユーザーgitで実行する

docker-compose

なんか難しそうなので、docker-composeを使ってみる

sudo apt install docker-compose

version: '3'
services:
  gogs:
    image: gogs/gogs:latest
    container_name: gogs
    restart: always
    ports:
      - 3000:3000
    volumes:
      - ./data:/data
    links:
      - mariadb:db

  mariadb:
    image: mariadb:latest
    restart: always
    ports:
      - 13306:3306
    environment:
      - MARIADB_ROOT_PASSWORD=************
      - MARIADB_DATABASE=gogs
      - MARIADB_USER=gogs
      - MARIADB_PASSWORD=************

    volumes:
      - ./mariadb/data:/var/lib/mysql
      - ./mariadb/my.cnf:/etc/mysql/conf.d/my.cnf
      - ./mariadb/sql:/docker-entrypoint-initdb.d

sudo docker-compose up -d
sudo docker-compose ps
sudo docker-compose stop

https://qiita.com/wasanx25/items/d47caf37b79e855af95f
https://mebee.info/2020/08/05/post-15924/

データの引っ越し

  • データベース
  • gogs-repositories
  • config
データベース
mysqldump -u git -p gogs_git > gogs.sql.bak
mysql -u gogs -p gogs --port=13306 < /home/adeno/gogs.sql.bak 
gogs-repositories

data/gogs/data/gogs-repositoriesにコピー

config
[repository]
ROOT = /app/gogs/data/gogs-repositories

結局

gogs backup

を使わなかった。

初期設定でのデーターベース設定は以下を参考にした。
https://mebee.info/2020/08/05/post-15924/

ホスト名をgogs_mariadb_1にする

adeno@blackcore:/mnt/backuparea/gogs$ sudo  docker-compose ps
     Name                   Command                  State                      Ports               
----------------------------------------------------------------------------------------------------
gogs             /app/gogs/docker/start.sh  ...   Up (healthy)   22/tcp, 0.0.0.0:3000-              
                                                                 >3000/tcp,:::3000->3000/tcp        
gogs_mariadb_1   docker-entrypoint.sh mariadbd    Up             0.0.0.0:13306->3306/tcp,:::13306-  
                                                                 >3306/tcp                          

rootで実行される

気になる。
rootで実行するし、作成されるファイルも所有者はroot
でもコンテナ内はgitになっている。

adeno@blackcore:/mnt/backuparea/gogs$ ls -l data/gogs/data/
合計 12
drwxr-xr-x 6 root root 4096  2月  6 12:34 gogs
drwxr-xr-x 7 root root 4096  2月  6 12:34 gogs-repositories
drwx------ 3 root root 4096  2月  6 12:34 sessions
sudo docker exec -it gogs /bin/bash
0f3d4c05ae00:/app/gogs# ls -l data/
total 12
drwxrwxr-x    6 git      git           4096 Feb  5 15:59 gogs
drwxr-xr-x    7 git      git           4096 Feb  5 15:59 gogs-repositories
drwx------    4 git      git           4096 Feb  6 03:35 sessions

https://qiita.com/yitakura731/items/36a2ba117ccbc8792aa7

気になる。

なにか方法があるのか

  • rootless
  • rootless + SELinux
  • Podman

https://e-penguiner.com/rootless-docker-for-nonroot/
https://matsuand.github.io/docs.docker.jp.onthefly/engine/security/rootless/
https://matsuand.github.io/docs.docker.jp.onthefly/engine/security/userns-remap/
https://docs.docker.jp/desktop/install/linux-install.html#linux-install-file-sharing

docker rootlessを試してみる

いろいろ試行錯誤したのを一旦整理

まずは普通のインストール

https://docs.docker.com/engine/install/ubuntu/

以下の手順のみLinux Mintでは異なるので注意

3.Use the following command to set up the repository:

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
$ lsb_release -cs
vanessa

UbuntuのOSバージョンを表示させるにはUBUNTU_CODENAMEが必要

$ cat /etc/os-release 
NAME="Linux Mint"
VERSION="21 (Vanessa)"
ID=linuxmint
ID_LIKE="ubuntu debian"
PRETTY_NAME="Linux Mint 21"
VERSION_ID="21"
HOME_URL="https://www.linuxmint.com/"
SUPPORT_URL="https://forums.linuxmint.com/"
BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
PRIVACY_POLICY_URL="https://www.linuxmint.com/"
VERSION_CODENAME=vanessa
UBUNTU_CODENAME=jammy

なので

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  jammy stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

としちゃう。

動作確認

これで、普通モードの動作確認ができる

sudo docker run hello-world

rootlessモード

ユーザーを作る

  • ユーザー:gogs_docker 1003
  • グループ:gogs-rtls-docker 10099

サブ UID/サブ GIDの設定

$ cat /etc/subuid
gogs_docker:100000:65536

$ cat /etc/subgid
gogs_docker:100000:65536

インストール

https://matsuand.github.io/docs.docker.jp.onthefly/engine/security/rootless/

gogs_docker@blackcore:~$ dockerd-rootless-setuptool.sh install
[INFO] systemd not detected, dockerd-rootless.sh needs to be started manually:

PATH=/home/gogs_docker/bin:/sbin:/usr/sbin:$PATH dockerd-rootless.sh 

[INFO] Creating CLI context "rootless"
Successfully created context "rootless"
[INFO] Use CLI context "rootless"
Current context is now "rootless"
Warning: DOCKER_HOST environment variable overrides the active context. To use "rootless", either set the global --context flag, or unset DOCKER_HOST environment variable.

[INFO] Make sure the following environment variables are set (or add them to ~/.bashrc):

# WARNING: systemd not found. You have to remove XDG_RUNTIME_DIR manually on every logout.
export XDG_RUNTIME_DIR=/home/gogs_docker/.docker/run
export PATH=/home/gogs_docker/bin:$PATH
Some applications may require the following environment variable too:
export DOCKER_HOST=unix:///home/gogs_docker/.docker/run/docker.sock

.bashrcに書くのを忘れない

export XDG_RUNTIME_DIR=/home/gogs_docker/.docker/run
export PATH=/home/gogs_docker/bin:$PATH

動作確認

gogs_docker@blackcore:~$ systemctl --user start docker
Failed to connect to bus: そのようなファイルやディレクトリはありません

gogs_docker@blackcore:~$ systemctl --user status
Failed to connect to bus: そのようなファイルやディレクトリはありません

あれ?

XDG_RUNTIME_DIR=/run/user/$(id -u gogs_docker) systemctl --user status
● blackcore
    State: degraded
     Jobs: 0 queued
   Failed: 2 units
    Since: Sun 2023-02-12 01:52:28 JST; 9h ago
   CGroup: /user.slice/user-1003.slice/user@1003.service
<<略>>

XDG_RUNTIME_DIR=/run/user/$(id -u gogs_docker) systemctl --user start docker
Failed to start docker.service: Unit docker.service not found.

ほう。docker.serviceがないのか・・・。

docker.serviceを手動で作る

仕方がないので
.config/systemd/user/docker.service
を手動で作成した。

[Unit]
Description=Docker Application Container Engine (Rootless)
Documentation=https://docs.docker.com/go/rootless/

[Service]
Environment=PATH=/home/gogs_docker/bin:/sbin:/usr/sbin:/home/gogs_docker/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ExecStart=/home/gogs_docker/bin/dockerd-rootless.sh 
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
Type=notify
NotifyAccess=all
KillMode=mixed

[Install]
WantedBy=default.target

今度こそ
まずはsudoできるユーザーで確認

adeno@blackcore:~$ sudo -u gogs_docker XDG_RUNTIME_DIR=/run/user/$(id -u gogs_docker) systemctl --user status
[sudo] adeno のパスワード:          
● blackcore
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Sun 2023-02-12 12:56:38 JST; 8h ago
   CGroup: /user.slice/user-1003.slice/user@1003.service
           ├─session.slice 
           │ └─pipewire.service 
           │   └─1188 /usr/bin/pipewire
           ├─app.slice 
           │ ├─docker.service 
           │ │ ├─16301 rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin>
           │ │ ├─16310 /proc/self/exe --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=buil>
           │ │ ├─16328 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 16310 tap0
           │ │ ├─16336 dockerd
           │ │ └─16363 containerd --config /run/user/1003/docker/containerd/containerd.toml --log-level info
           │ └─dbus.service 
           │   └─1234 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
           └─init.scope 
             ├─1140 /lib/systemd/systemd --user
             └─1149 (sd-pam)

sudo -u gogs_docker XDG_RUNTIME_DIR=/run/user/$(id -u gogs_docker) systemctl --user start docker

よし。
次に、dockerを実行したい一般ユーザーで確認

XDG_RUNTIME_DIR=/run/user/$(id -u gogs_docker) systemctl --user status
● blackcore
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Sun 2023-02-12 12:56:38 JST; 8h ago
   CGroup: /user.slice/user-1003.slice/user@1003.service
           ├─session.slice 
           │ └─pipewire.service 
           │   └─1188 /usr/bin/pipewire
           ├─app.slice 
           │ ├─docker.service 
           │ │ ├─16301 rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin>
           │ │ ├─16310 /proc/self/exe --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=buil>
           │ │ ├─16328 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 16310 tap0
           │ │ ├─16336 dockerd
           │ │ └─16363 containerd --config /run/user/1003/docker/containerd/containerd.toml --log-level info
           │ └─dbus.service 
           │   └─1234 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
           └─init.scope 
             ├─1140 /lib/systemd/systemd --user
             └─1149 (sd-pam)

良いね。状態取れた。
サンプルを実行してみる

$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

よし。

gogs_docker@blackcore:/mnt/backuparea/gogs_rootless$ docker-compose up
gogs_rootless_mariadb_1 is up-to-date
Starting gogs ... done
Attaching to gogs_rootless_mariadb_1, gogs
<<略>>

OK gogs動いた!

自動起動

 systemctl --user enable docker
 sudo loginctl enable-linger $(whoami)
  • メモ
XDG_RUNTIME_DIR=/run/user/$(id -u gogs_docker) systemctl --user enable docker
Created symlink /home/gogs_docker/.config/systemd/user/default.target.wants/docker.service → /home/gogs_docker/.config/systemd/user/docker.service.

が、再起動後にPSで実行中のコンテナが見れなくなってしまった。

gogs_docker@blackcore:/mnt/backuparea/gogs_rootless$ docker ps -a
Cannot connect to the Docker daemon at unix:///home/gogs_docker/.docker/run/docker.sock. Is the docker daemon running?

sockの場所を明示すると動いた

docker -H unix:///run/user/1003/docker.sock ps
CONTAINER ID   IMAGE              COMMAND                   CREATED       STATUS                 PORTS                                               NAMES
c6a2687906f9   gogs/gogs:latest   "/app/gogs/docker/st…"   3 hours ago   Up 3 hours (healthy)   22/tcp, 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp   gogs
2b74ddf55d44   mariadb:latest     "docker-entrypoint.s…"   3 hours ago   Up 3 hours             0.0.0.0:13306->3306/tcp, :::13306->3306/tcp         gogs_rootless_mariadb_1
$ docker-compose ps
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 718, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 33, in <module>
    sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')())
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 200, in perform_command
    project = project_from_options('.', options)
  File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 60, in project_from_options
    return get_project(
  File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 152, in get_project
    client = get_client(
  File "/usr/lib/python3/dist-packages/compose/cli/docker_client.py", line 41, in get_client
    client = docker_client(
  File "/usr/lib/python3/dist-packages/compose/cli/docker_client.py", line 170, in docker_client
    client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 153, in apport_excepthook
    with os.fdopen(os.open(pr_filename,
FileNotFoundError: [Errno 2] No such file or directory: '/var/crash/_usr_bin_docker-compose.1003.crash'

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 718, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 33, in <module>
    sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')())
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 200, in perform_command
    project = project_from_options('.', options)
  File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 60, in project_from_options
    return get_project(
  File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 152, in get_project
    client = get_client(
  File "/usr/lib/python3/dist-packages/compose/cli/docker_client.py", line 41, in get_client
    client = docker_client(
  File "/usr/lib/python3/dist-packages/compose/cli/docker_client.py", line 170, in docker_client
    client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

こちらも同様

docker-compose -H unix:///run/user/1003/docker.sock ps
         Name                        Command                  State                            Ports                      
--------------------------------------------------------------------------------------------------------------------------
gogs                      /app/gogs/docker/start.sh  ...   Up (healthy)   22/tcp, 0.0.0.0:3000->3000/tcp,:::3000->3000/tcp
gogs_rootless_mariadb_1   docker-entrypoint.sh mariadbd    Up             0.0.0.0:13306->3306/tcp,:::13306->3306/tcp      

.bashrcに書いた

export XDG_RUNTIME_DIR=/home/gogs_docker/.docker/run
が余計だったのかな・・・。

これをコメントアウトして試してみる

これで

export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock
docker-compose ps
         Name                        Command                  State                            Ports                      
--------------------------------------------------------------------------------------------------------------------------
gogs                      /app/gogs/docker/start.sh  ...   Up (healthy)   22/tcp, 0.0.0.0:3000->3000/tcp,:::3000->3000/tcp
gogs_rootless_mariadb_1   docker-entrypoint.sh mariadbd    Up             0.0.0.0:13306->3306/tcp,:::13306->3306/tcp      

今度こそ大丈夫そう。

改めてデータの引っ越しをする。

データの引っ越し

  • データベース
  • gogs-repositories
  • config
データベース
mysqldump -u git -p gogs_git > gogs.sql.bak
mysql -u gogs -p gogs --port=13306 < /home/adeno/gogs.sql.bak 
gogs-repositories

data/gogs/data/gogs-repositoriesにコピー
データの所有者を100999(コンテナ内のgit(10000)相当)にしておく

config
[repository]
ROOT = /app/gogs/data/gogs-repositories

ホスト名をgogs_mariadb_1にする

これでOKOK

更新ができない

なぜか、もともとの/home/git/gogs-repositoriesを参照しようとする。
困ったので、最終手段でシンボリックリンクを張った。

gogs_docker@blackcore:/mnt/backuparea/gogs$ docker exec -it gogs /bin/bash

c6a2687906f9:/home/git/gogs-repositories# mkdir -p /home/git/gogs/gogs
6a2687906f9:/home/git/gogs-repositories# cd /home/git/gogs/gogs/
c6a2687906f9:/home/git/gogs/gogs# ln -s /app/gogs/data/gogs-repositories/gogs-repositories gogs-repositories
c6a2687906f9:/home/git/gogs/gogs# ls -l
total 0
lrwxrwxrwx    1 root     root            50 Feb 15 16:18 gogs-repositories -> /app/gogs/data/gogs-repositories/gogs-repositories

2023年1月22日日曜日

ホームサーバーの環境移行(2) (GPE-2500T/RTL8125Bが切断される)

有線LANにする

結局、データの転送に思いの外、時間がかかるので、有線LANに変更することにした。
せっかくなので、2.5Gにしたいと思う。

あまり予算もないので、PlanexのGPE-2500TとFX2G-05EMを選んだ。
RTL8125Bというチップを使っており、Linuxでの実績もありそうだ。


 

有線LANが切れる

メインPCの方は安定して動いているけど、ホームサーバーの方はしばらくすると通信ができない状態になってしまう。
その時のシステムログはこんな感じ。

[  940.182815] ------------[ cut here ]------------
[  940.182826] NETDEV WATCHDOG: enp1s0 (r8169): transmit queue 0 timed out
[  940.182875] WARNING: CPU: 6 PID: 0 at net/sched/sch_generic.c:477 dev_watchdog+0x277/0x280
[  940.182890] Modules linked in: ccm rfcomm cmac algif_hash algif_skcipher af_alg ip6t_REJECT nf_reject_ipv6 xt_hl ip6_tables ip6t_rt ipt_REJECT nf_reject_ipv4 xt_LOG nf_log_syslog xt_multiport nft_limit bnep xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nft_counter nf_tables nfnetlink zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) intel_rapl_msr intel_rapl_common snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg edac_mce_amd snd_intel_sdw_acpi kvm_amd snd_hda_codec snd_hda_core snd_hwdep kvm snd_pcm iwlmvm snd_seq_midi snd_seq_midi_event btusb nls_iso8859_1 mac80211 rapl input_leds joydev snd_rawmidi btrtl libarc4 btbcm snd_seq btintel bluetooth iwlwifi snd_seq_device wmi_bmof k10temp snd_timer ecdh_generic cfg80211 snd ecc ccp soundcore mac_hid sch_fq_codel nct6775 hwmon_vid msr parport_pc ppdev lp parport ramoops pstore_blk reed_solomon
[  940.183202]  pstore_zone efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear dm_mirror dm_region_hash dm_log raid1 amdgpu hid_generic iommu_v2 gpu_sched i2c_algo_bit drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt usbhid fb_sys_fops hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cec aesni_intel r8169 gpio_amdpt xhci_pci crypto_simd ahci rc_core i2c_piix4 nvme cryptd drm nvme_core libahci xhci_pci_renesas realtek wmi video gpio_generic
[  940.183390] CPU: 6 PID: 0 Comm: swapper/6 Tainted: P           O      5.15.0-58-generic #64-Ubuntu
[  940.183395] Hardware name: To Be Filled By O.E.M. A520M-ITX/ac/A520M-ITX/ac, BIOS P2.20 12/27/2022
[  940.183399] RIP: 0010:dev_watchdog+0x277/0x280
[  940.183405] Code: eb 97 48 8b 5d d0 c6 05 67 17 69 01 01 48 89 df e8 ce 64 f9 ff 44 89 e1 48 89 de 48 c7 c7 50 62 ed b8 48 89 c2 e8 ef d3 19 00 <0f> 0b eb 80 e9 de 3d 23 00 0f 1f 44 00 00 55 48 89 e5 41 57 41 56
[  940.183410] RSP: 0018:ffffa1a0c0314e70 EFLAGS: 00010282
[  940.183417] RAX: 0000000000000000 RBX: ffff8d1ddbd18000 RCX: 0000000000000000
[  940.183421] RDX: ffff8d24de3ac240 RSI: ffff8d24de3a0580 RDI: 0000000000000300
[  940.183425] RBP: ffffa1a0c0314ea8 R08: 0000000000000003 R09: fffffffffffd7cd0
[  940.183429] R10: 0000000000ffff0a R11: 0000000000000001 R12: 0000000000000000
[  940.183433] R13: ffff8d1ddb1f1e80 R14: 0000000000000001 R15: ffff8d1ddbd184c0
[  940.183436] FS:  0000000000000000(0000) GS:ffff8d24de380000(0000) knlGS:0000000000000000
[  940.183441] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  940.183446] CR2: 000055733bccb020 CR3: 0000000110410000 CR4: 0000000000750ee0
[  940.183450] PKRU: 55555554
[  940.183453] Call Trace:
[  940.183457]  <IRQ>
[  940.183462]  ? pfifo_fast_enqueue+0x160/0x160
[  940.183471]  call_timer_fn+0x2c/0x120
[  940.183479]  __run_timers.part.0+0x1e3/0x270
[  940.183485]  ? ktime_get+0x46/0xc0
[  940.183493]  ? native_x2apic_icr_read+0x20/0x20
[  940.183501]  ? lapic_next_event+0x20/0x30
[  940.183508]  ? clockevents_program_event+0xad/0x130
[  940.183517]  run_timer_softirq+0x2a/0x60
[  940.183522]  __do_softirq+0xd9/0x2e7
[  940.183530]  irq_exit_rcu+0x94/0xc0
[  940.183539]  sysvec_apic_timer_interrupt+0x80/0x90
[  940.183547]  </IRQ>
[  940.183549]  <TASK>
[  940.183552]  asm_sysvec_apic_timer_interrupt+0x1b/0x20
[  940.183558] RIP: 0010:native_safe_halt+0xb/0x10
[  940.183566] Code: 2c ff 5b 41 5c 41 5d 5d c3 cc cc cc cc 4c 89 ee 48 c7 c7 80 43 65 b9 e8 23 91 8d ff eb ca cc eb 07 0f 00 2d d9 e0 45 00 fb f4 <c3> cc cc cc cc eb 07 0f 00 2d c9 e0 45 00 f4 c3 cc cc cc cc cc 0f
[  940.183570] RSP: 0018:ffffa1a0c010be78 EFLAGS: 00000202
[  940.183577] RAX: ffffffffb85afc40 RBX: ffff8d1dc0373280 RCX: 7fffff251d8fda07
[  940.183582] RDX: 00000000000235a1 RSI: 0000000000000006 RDI: 00000000000235a2
[  940.183586] RBP: ffffa1a0c010be80 R08: 000000cd42eda501 R09: 0000000000000000
[  940.183590] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
[  940.183593] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  940.183598]  ? __cpuidle_text_start+0x8/0x8
[  940.183606]  ? default_idle+0xe/0x20
[  940.183611]  arch_cpu_idle+0x15/0x20
[  940.183620]  default_idle_call+0x3e/0xd0
[  940.183625]  cpuidle_idle_call+0x179/0x1e0
[  940.183633]  do_idle+0x83/0xf0
[  940.183640]  cpu_startup_entry+0x20/0x30
[  940.183644]  start_secondary+0x12a/0x180
[  940.183649]  secondary_startup_64_no_verify+0xc2/0xcb
[  940.183658]  </TASK>
[  940.183661] ---[ end trace 32949fbdb853d046 ]---
[ 1106.094315] r8169 0000:01:00.0 enp1s0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).
[ 1106.095552] r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
[ 1106.096672] r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
[ 1106.097791] r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
[ 1106.098915] r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
[ 1106.100034] r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
[ 1106.101153] r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
[ 1106.121214] r8169 0000:01:00.0 enp1s0: rtl_mac_ocp_e00e_cond == 1 (loop: 10, delay: 1000).

以降、[rtl_chipcmd_cond][rtl_ephyar_cond][rtl_mac_ocp_e00e_cond]の繰り返し。

  • インターフェースのLANコネクタのACTは点滅を繰り返している
  • 再起動(reboot)では復旧しない
  • シャットダウンで復旧する
  • BIOSは最新
  • Xログインしていても、していなくても発生する
  • ping打ち続けても発生する

メインPCとの比較

項目 メインPC サーバー
uname -a 5.15.0-57-generic #63-Ubuntu 5.15.0-58-generic #64-Ubuntu
lsb_release Linux Mint 21 (vanessa) Linux Mint 21.1 (vera)
ドライバ r8169 ※ r8169 ※

※マザーボードに搭載されているLANと同じドライバを使用しているみたい。

[    0.896998] r8169 0000:01:00.0 eth0: RTL8125B, **:**:**:**:**:**, XID 641, IRQ 39
[    0.897002] r8169 0000:01:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]

[    0.912515] r8169 0000:04:00.0 eth1: RTL8168h/8111h, **:**:**:**:**:**, XID 541, IRQ 48
[    0.912518] r8169 0000:04:00.0 eth1: jumbo features [frames: 9194 bytes, tx checksumming: ko]|

OSバージョンの違いがあるくらいか・・・。

ドライバを最新にしてみる

https://www.realtek.com/ja/component/zoo/category/network-interface-controllers-10-100-1000m-gigabit-ethernet-pci-express-software

r8169からr8125に変わった。

[    0.882838] r8125: loading out-of-tree module taints kernel.
[    0.882928] r8125: module verification failed: signature and/or required key missing - tainting kernel
[    0.883378] r8125 2.5Gigabit Ethernet driver 9.011.00-NAPI loaded
[    0.902733] r8125: This product is covered by one or more of the following patents: US6,570,884, US6,115,776, and US6,327,625.
[    0.904748] r8125  Copyright (C) 2022 Realtek NIC software team <nicfae@realtek.com> 
[    2.653915] r8125 0000:01:00.0 enp1s0: renamed from eth0
[   10.077752] r8125: enp1s0: link up

しばらく様子見
使っていたら

enp1s0: cmd = 0xff, should be 0x07

となって通信断してしまった。
たぶんapt upgradeで大きの通信をしたタイミングのようだ。

Linux Mint 21 (vanessa)

メインPCと同じ Mint 21をライブUSBにて起動して試してみる。
スピードテストやapt upgradeでは切れないようだ。
1時間様子見でも切れない。

Linux Mint 21.1 (vera)

この状態でまた21.1を起動してみる
スピードテストやapt upgradeでは切れないようだ。
1時間様子見でも切れない。

iperf(ライブUSB)

サーバー <ー メインPC
adeno@drakorange:~$ iperf -c 192.168.1.34 -t 30 
------------------------------------------------------------
Client connecting to 192.168.1.34, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.29 port 53526 connected with 192.168.1.34 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-30.0242 sec  8.22 GBytes  2.35 Gbits/sec
サーバー ー> メインPC
iperf -c 192.168.1.34 -R -t 30 
------------------------------------------------------------
Client connecting to 192.168.1.34, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.29 port 34464 connected with 192.168.1.34 port 5001 (reverse)
[ ID] Interval       Transfer     Bandwidth
[ *1] 0.0000-45.7757 sec   108 MBytes  19.8 Mbits/sec

サーバーPCがリブートしたOrz

iperf(通常起動)

サーバー <ー メインPC
adeno@drakorange:~$ iperf -c 192.168.1.34 -t 30 
------------------------------------------------------------
Client connecting to 192.168.1.34, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.29 port 32992 connected with 192.168.1.34 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-30.0178 sec  8.22 GBytes  2.35 Gbits/sec

サーバー ー> メインPC

adeno@drakorange:~$ iperf -c 192.168.1.34 -R -t 30 
------------------------------------------------------------
Client connecting to 192.168.1.34, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.29 port 51360 connected with 192.168.1.34 port 5001 (reverse)
[ ID] Interval       Transfer     Bandwidth
[ *1] 0.0000-30.0122 sec  8.22 GBytes  2.35 Gbits/sec

が、上記10回くらい繰り返したり、パラメータを変えたりしていると

iperf -c 192.168.1.34 -R -t 60 -b 200M
enp1s0: cmd = 0xff, should be 0x07

が発生した。
メインPCの方は大丈夫。

Linux Mint 21 (vanessa)

再度、21で試してみる

iperf -c 192.168.1.34 -t 30
iperf -c 192.168.1.34 -R -t 30 

を10セット

iperf -c 192.168.1.34 -R -t 60 -b 200M

大丈夫だ。
いやだめだ。

NETDEV WATCHDOG: enp1s0 (r8169): transmit queue 0 timed out

でもその後も通信できている
いやだめだ。

もしかして、個体差?

個体を入れ替えてみる

メインPCとサーバーのGPE-2500Tを入れ替えてみた
するとすぐに切断された状態になってしまった。
もしかして、個体差?

まとめると

今までの検証内容をまとめると

ハード GPE-2500T OS ドライバ 長期ping スピードテスト apt upgrade iperf 総合
メインPC A Linux Mint 21 (vanessa) r8169
サーバー B Linux Mint 21.1 (vera) r8169 × ×
サーバー B Linux Mint 21.1 (vera) r8125 × ×
サーバー B Linux Mint 21.1 (vera) r8169 × ×
サーバー B Linux Mint 21 (vanessa) r8169 ×
メインPC B Linux Mint 21 (vanessa) r8169 ? ? × × ×
サーバー A Linux Mint 21 (vanessa) r8169

交換

仕方がないのでもう1つGPE-2500Tを入手した。
これで収束すると良いのだけど。
あと、サポセンに連絡したら返品とか交換とかしてもらえるのかな・・・。

sambaの転送速度

晴れて、有線LANになったことで、転送速度は
14MB/s(112Mbps)→190MB/s(1,520Mbps)となった。
HDDでソフトウェアRAIDということもあり、ワイヤースピートに迫ることはなかったけど、満足感のある結果となった。