rix – rix的笔记

如果正常情况下，官方的配置应该没有太大问题。但我的环境有些特殊，由于我的邮箱服务器搭建在内网的 unraid 中，而我的访问接口需要在公网上，所以，在中间不得不添加转发的机制。目前我的大致访问流程是这样的：

公网 nginx 代理(公网)->zerotier（网关）->内网 nginx 代理->mailu（1.9）

上述所有的部分除了 zerotier，都是通过 docker 部署的。因为内网所有的服务都是通过内网的 nginx 反向代理的，因此在公网的 nginx 部分配置 tcp 代理是可以的。但如此一来，也导致了内网的 nginx 拿到的 ip 都是 zerotier 网关的 ip,或者拿到的公网的 nginx 代理的 docker 网关的 ip。对于后续部署 fail2ban 部分带来很大的麻烦。如果都是 http/https 服务，或许还好弄些，但我的是 nginx 还使用 tcp 代理了邮箱的各个端口，以便转发给内网部分。

为了给公网上的服务器添加上 ip 屏蔽的功能，只好进行了下面的改造

公网(docker 版本的 nginx+fail2ban 定制) -> zerotier（不动）-> 内网(docker nginx 小修改+fail2ban 定制+mailu 代码修改)

公网 nginx 上的改造

公网上的 nginx 也是在 docker 中的，为了方便进行 fail2ban 的管理，我又增加了一套 docker 版本的 fail2ban。 nginx 的改造如下:

stream {
server {
listen 443 reuseport;
proxy_pass 10.10.10.10:443;
proxy_protocol on; 
ssl_preread on;
}
server {
listen 587 reuseport;
proxy_pass 10.10.10.10:587;
proxy_protocol on; 
}
  #...其他端口类似
}

proxy-protocol 这个字段添加后，在 header 部分会多出一个 “Proxy-Protocol-Addr”字段，这个字段保存的是客户端的真实 ip，在后续的部分，需要用这个字段来识别客户端的真实 ip，并且用于 fail2ban 的屏蔽。

内网改造

nginx(代理) 上的改造

由于外网使用了 stream 来代理，并且开启了 proxy_protocol, 因此内网也需要对应的开启。这样一来，内网就没办法通过普通的 https 直接访问了。如果要访问，可以在内网部分再添加个 stream 代理

server {
listen 443 ssl http2 proxy_protocol;
server_name you_subadmin;
real_ip_header proxy_protocol;
real_ip_recursive on;
location / {
proxy_set_header   Host             $http_host;
proxy_set_header   X-Real-IP        $proxy_protocol_addr;
proxy_set_header   X-Forwarded-For  $proxy_protocol_addr;
proxy_set_header   X-Forwarded-Proto $scheme;
proxy_pass https://mailu:8443;
proxy_buffering off;
tcp_nodelay on;
keepalive_timeout 55;
}
}

上面的部分，可以让 mailu 部分的 front 获取到正确的外网地址

mailu 的 front 部分的 nginx 的改造

# Basic configuration
user nginx;
worker_processes auto;
error_log /dev/stderr notice;
pid /var/run/nginx.pid;
load_module "modules/ngx_mail_module.so";

events {
    worker_connections 1024;
}

http {
    # Standard HTTP configuration with slight hardening
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    sendfile on;
    keepalive_timeout 65;
    server_tokens off;
    absolute_redirect off;
    resolver {{ RESOLVER }} valid=30s;

    {% if REAL_IP_HEADER %}
    #real_ip_header {{ REAL_IP_HEADER }};
    {% endif %}
    {% if REAL_IP_FROM %}{% for from_ip in REAL_IP_FROM.split(',') %}
    set_real_ip_from {{ from_ip }};
    {% endfor %}{% endif %}
    real_ip_header proxy_protocol;
    proxy_set_header   X-Real-IP        $proxy_protocol_addr;
    proxy_set_header X-Forwarded-For $proxy_protocol_addr;
    real_ip_recursive on;
    # Header maps
    map $http_x_forwarded_proto $proxy_x_forwarded_proto {
      default $http_x_forwarded_proto;
      ''      $scheme;
    }
    map $uri $expires {
      default off;
      ~*\.(ico|css|js|gif|jpeg|jpg|png|woff2?|ttf|otf|svg|tiff|eot|webp)$ 97d;
    }

    map $request_uri $loggable {
      /health 0;
      /auth/email 0;
      default 1;
    }
    log_format main '$proxy_protocol_addr $remote_addr - $remote_user [$time_local] '
    '"$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent" "$gzip_ratio"';

    #access_log /var/log/nginx/access.log main;
    access_log /dev/stdout main;# if=$loggable;

    # compression
    gzip on;
    gzip_static on;
    gzip_types text/plain text/css application/xml application/javascript
    gzip_min_length 1024;
    # TODO: figure out how to server pre-compressed assets from admin container

    {% if KUBERNETES_INGRESS != 'true' and TLS_FLAVOR in [ 'letsencrypt', 'cert' ] %}
    # Enable the proxy for certbot if the flavor is letsencrypt and not on kubernetes
    #
    server {
      # Listen over HTTP
      listen 80;
      listen [::]:80;
      {% if TLS_FLAVOR == 'letsencrypt' %}
      location ^~ /.well-known/acme-challenge/ {
          proxy_pass http://127.0.0.1:8008;
      }
      {% endif %}
      # redirect to https
      location / {
        return 301 https://$host$request_uri;
      }

    }
    {% endif %}

    # Main HTTP server
    server {
      # Favicon stuff
      root /static;
      # Variables for proxifying
      set $admin {{ ADMIN_ADDRESS }};
      set $antispam {{ ANTISPAM_WEBUI_ADDRESS }};
      {% if WEBMAIL_ADDRESS %}
      set $webmail {{ WEBMAIL_ADDRESS }};
      {% endif %}
      {% if WEBDAV_ADDRESS %}
      set $webdav {{ WEBDAV_ADDRESS }};
      {% endif %}
      client_max_body_size {{ MESSAGE_SIZE_LIMIT|int + 8388608 }};

      # Listen on HTTP only in kubernetes or behind reverse proxy
      {% if KUBERNETES_INGRESS == 'true' or TLS_FLAVOR in [ 'mail-letsencrypt', 'notls', 'mail' ] %}
      listen 80;
      listen [::]:80;

      {% endif %}

      # Only enable HTTPS if TLS is enabled with no error and not on kubernetes
      {% if KUBERNETES_INGRESS != 'true' and TLS and not TLS_ERROR %}
        listen 443 ssl http2;
      listen [::]:443 ssl http2;
        real_ip_header proxy_protocol;
        proxy_set_header   X-Real-IP        $proxy_protocol_addr;
        real_ip_recursive on;
      include /etc/nginx/tls.conf;
      ssl_stapling on;
      ssl_stapling_verify on;
      ssl_session_cache shared:SSLHTTP:50m;
      add_header Strict-Transport-Security 'max-age=31536000';

      {% if not TLS_FLAVOR in [ 'mail', 'mail-letsencrypt' ] %}
      if ($proxy_x_forwarded_proto = http) {
        return 301 https://$host$request_uri;
      }
      {% endif %}
      {% endif %}

      # Remove headers to prevent duplication and information disclosure
      proxy_hide_header X-XSS-Protection;
      proxy_hide_header X-Powered-By;

      add_header X-Frame-Options 'SAMEORIGIN';
      add_header X-Content-Type-Options 'nosniff';
      add_header X-Permitted-Cross-Domain-Policies 'none';
      add_header X-XSS-Protection '1; mode=block';
      add_header Referrer-Policy 'same-origin';

      {% if TLS_FLAVOR == 'mail-letsencrypt' %}
      location ^~ /.well-known/acme-challenge/ {
          proxy_pass http://127.0.0.1:8008;
      }
      {% endif %}

      # If TLS is failing, prevent access to anything except certbot
      {% if KUBERNETES_INGRESS != 'true' and TLS_ERROR and not (TLS_FLAVOR in [ 'mail-letsencrypt', 'mail' ]) %}
      location / {
        return 403;
      }
      {% else %}
      include /overrides/*.conf;

      # Actual logic
      {% if ADMIN == 'true' or WEBMAIL != 'none' %}
      location ~ ^/(sso|static)/ {
        include /etc/nginx/proxy.conf;
          proxy_set_header   X-Real-IP        $proxy_protocol_addr;
          proxy_set_header X-Forwarded-For $proxy_protocol_addr;
        proxy_pass http://$admin;
      }
      {% endif %}

      {% if WEB_WEBMAIL != '/' and WEBROOT_REDIRECT != 'none' %}
      location / {
        expires $expires;
      {% if WEBROOT_REDIRECT %}
        try_files $uri {{ WEBROOT_REDIRECT }};
      {% else %}
        try_files $uri =404;
      {% endif %}
      }
      {% endif %}

      {% if WEBMAIL != 'none' %}
      location {{ WEB_WEBMAIL }} {
        {% if WEB_WEBMAIL != '/' %}
        rewrite ^({{ WEB_WEBMAIL }})$ $1/ permanent;
        rewrite ^{{ WEB_WEBMAIL }}/(.*) /$1 break;
        {% endif %}
        include /etc/nginx/proxy.conf;
        auth_request /internal/auth/user;
        error_page 403 @webmail_login;
          proxy_set_header   X-Real-IP        $proxy_protocol_addr;
          proxy_set_header X-Forwarded-For $proxy_protocol_addr;
        proxy_pass http://$webmail;
      }

      {% if WEB_WEBMAIL == '/' %}
      location /sso.php {
      {% else %}
      location {{ WEB_WEBMAIL }}/sso.php {
      {% endif %}
        {% if WEB_WEBMAIL != '/' %}
        rewrite ^({{ WEB_WEBMAIL }})$ $1/ permanent;
        rewrite ^{{ WEB_WEBMAIL }}/(.*) /$1 break;
        {% endif %}
        include /etc/nginx/proxy.conf;
          proxy_set_header   X-Real-IP        $proxy_protocol_addr;
        auth_request /internal/auth/user;
        auth_request_set $user $upstream_http_x_user;
        auth_request_set $token $upstream_http_x_user_token;
        proxy_set_header X-Remote-User $user;
        proxy_set_header X-Remote-User-Token $token;
        error_page 403 @webmail_login;
        proxy_pass http://$webmail;
      }

      location @webmail_login {
        return 302 /sso/login;
      }
      {% endif %}
      {% if ADMIN == 'true' %}
       location {{ WEB_ADMIN }} {
         include /etc/nginx/proxy.conf;
         proxy_pass http://$admin;
           proxy_set_header   X-Real-IP        $proxy_protocol_addr;
         expires $expires;
       }

      location {{ WEB_ADMIN }}/antispam {
        rewrite ^{{ WEB_ADMIN }}/antispam/(.*) /$1 break;
        auth_request /internal/auth/admin;
        proxy_set_header X-Real-IP "";
        proxy_set_header X-Forwarded-For "";
        proxy_pass http://$antispam;
      }
      {% endif %}

      {% if WEBDAV != 'none' %}
      location /webdav {
        rewrite ^/webdav/(.*) /$1 break;
        auth_request /internal/auth/basic;
        auth_request_set $user $upstream_http_x_user;
        include /etc/nginx/proxy.conf;
        proxy_set_header X-Remote-User $user;
        proxy_set_header X-Script-Name /webdav;
        proxy_pass http://$webdav;
      }

      location ~ ^/.well-known/(carddav|caldav) {
        return 301 /webdav/;
      }
      {% endif %}
      {% endif %}

      location /internal {
        internal;

        #proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Real-IP $proxy_protocol_addr;
        proxy_set_header Authorization $http_authorization;
        proxy_pass_header Authorization;
        proxy_pass http://$admin;
        proxy_pass_request_body off;
        proxy_set_header Content-Length "";
      }

      location /health {
        return 204;
      }
    }

    # Forwarding authentication server
    server {
      # Variables for proxifying
      set $admin {{ ADMIN_ADDRESS }};

      listen 127.0.0.1:8000;

      location / {
        proxy_pass http://$admin/internal$request_uri;
      }
    }
}

mail {
    server_name {{ HOSTNAMES.split(",")[0] }};
    auth_http http://127.0.0.1:8000/auth/email;
    proxy_pass_error_message on;
    resolver {{ RESOLVER }} valid=30s;
    error_log /dev/stderr debug;
    #error_log /var/log/nginx/mail_error.log info;
    {% if TLS and not TLS_ERROR %}
    include /etc/nginx/tls.conf;
    ssl_session_cache shared:SSLMAIL:50m;
    {% endif %}
    #proxy_protocol on;

    # Advertise real capabilites of backends (postfix/dovecot)
    smtp_capabilities PIPELINING SIZE {{ MESSAGE_SIZE_LIMIT }} ETRN ENHANCEDSTATUSCODES 8BITMIME DSN;
    pop3_capabilities TOP UIDL RESP-CODES PIPELINING AUTH-RESP-CODE USER;
    imap_capabilities IMAP4 IMAP4rev1 UIDPLUS SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+;

    # Default SMTP server for the webmail (no encryption, but authentication)
    server {
      listen 10025;
      protocol smtp;
      smtp_auth plain;
      auth_http_header Auth-Port 10025;
    }

    # Default IMAP server for the webmail (no encryption, but authentication)
    server {
      listen 10143;
      protocol imap;
      smtp_auth plain;
      auth_http_header Auth-Port 10143;
    }

    # SMTP is always enabled, to avoid losing emails when TLS is failing
    server {
      listen 25 proxy_protocol;
      listen [::]:25 proxy_protocol;
      {% if TLS and not TLS_ERROR %}
      {% if TLS_FLAVOR in ['letsencrypt','mail-letsencrypt'] %}
      ssl_certificate /certs/letsencrypt/live/mailu/fullchain.pem;
      ssl_certificate /certs/letsencrypt/live/mailu-ecdsa/fullchain.pem;
      {% endif %}
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
      ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA;
      ssl_prefer_server_ciphers on;
      starttls on;
      {% endif %}
      protocol smtp;
      smtp_auth none;
      auth_http_header Auth-Port 25;
    }

    # All other protocols are disabled if TLS is failing
    {% if not TLS_ERROR %}
    server {
      listen 143 proxy_protocol;
      listen [::]:143 proxy_protocol;
      {% if TLS %}
      starttls only;
      {% endif %}
      protocol imap;
      imap_auth plain;
      auth_http_header Auth-Port 143;
    }

    server {
      listen 110 proxy_protocol;
      listen [::]:110 proxy_protocol;
      {% if TLS %}
      starttls only;
      {% endif %}
      protocol pop3;
      pop3_auth plain;
      auth_http_header Auth-Port 110;
    }

    server {
      listen 587 proxy_protocol;
      listen [::]:587 proxy_protocol;
      {% if TLS %}
      starttls only;
      {% endif %}
      protocol smtp;
      smtp_auth plain login;
      auth_http_header Auth-Port 587;
    }

    {% if TLS %}
    server {
      listen 465 ssl proxy_protocol;
      listen [::]:465 ssl proxy_protocol;
      protocol smtp;
      smtp_auth plain login;
      auth_http_header Auth-Port 465;
    }

    server {
      listen 993 ssl proxy_protocol;
      listen [::]:993 ssl proxy_protocol;
      protocol imap;
      imap_auth plain;
      auth_http_header Auth-Port 993;
    }

    server {
      listen 995 ssl proxy_protocol;
      listen [::]:995 ssl proxy_protocol;
      protocol pop3;
      pop3_auth plain;
      auth_http_header Auth-Port 995;
    }
    {% endif %}
    {% endif %}
}

主要修改部分就是给 mail 的 server 对外部分添加上 proxy_protocol，因为外网的 nginx 的 stream 添加了，因此这部分也需要添加上，不然服务器会转发失败的。

mailu 的授权主要在 admin 部分，因此我们只要修改这部分就可以了。这部分默认读取的 IP 是从 Client-Ip 字段中来的，现在真实的 IP 保存在”Proxy-Protocol-Addr”中，在 nginx.py 和 auth.py 中搜索 Client-Ip，并在合适的位置添加上下面的内容:

if headers.get("Proxy-Protocol-Addr"):
        client_ip = flask.request.headers["Proxy-Protocol-Addr"]

但 admin 的日志中显示的还是错误的地址，为了配合 fail2ban 的使用，我在所有授权失败的地方添加上了这样的错误日志的输出:

app.logger.error(f'Auth Fail user ip {client_ip!r} Auth not supported 403')

主要是添加 fail2ban 可识别的正则表达式的格式。admin 部分默认的日志输出都是在 stderr 或者 stdout 的，我也修改了 start.py 一下，让日志保存到/var/log 中

start_command="".join([
    "gunicorn --threads ", str(os.cpu_count()),
    " -b :80 ",
    "--access-logfile /var/log/access.log " if (log.root.level<=log.INFO) else "",
    "--error-logfile /var/log/error.log ",
    "--preload ",
    "'mailu:create_app()'"])

os.system(start_command)

fail2ban 部分

内网和外网的 fail2ban 部分基本一致，除了外网的 fail2ban 只进行屏蔽和解封的操作，剩余的判定和遥控都是在内网进行的。

filter.d 部分

内外网的 fail2ban 都需要

#filename rix-mailu-admin.local
[INCLUDES]
before = common.conf
[Definition]
failregex = .* Auth (Fail|None) .* ip '<HOST>' .* (403)$
ignoreregex =
datepattern = \[%%Y-%%m-%%d %%H:%%M:%%S,%%f\]

jail.d 部分

[rix-mailu-admin]
enabled = true
chain = DOCKER-USER
filter = rix-mailu-admin
maxretry = 5
findtime = 30m
bantime = 30d
#下面这两行外网不需要，内网需要
action = rix_remote[name=rix-mailu-admin,port="443,4431,80,25,110,143,465,587,993,995",domain="your_email_domain"]
logpath = /remotelogs/mailu/error.log

action.d 部分（只内网需要）

[Definition]
actionstart= /config/rix_remote_cmd start fail2ban-<name>
actionstop = /config/rix_remote_cmd stop fail2ban-<name>
actioncheck = /config/rix_remote_cmd check fail2ban-<name>
actionban = /config/rix_remote_cmd ban <name> <domain> <ip> <port> 
actionunban = /config/rix_remote_cmd unban <name> <domain> <ip> <port> 
[Init]
chain = INPUT
name = default
protocol = tcp

上面的配置中，rix_remote_cmd 是我写的一个小脚本，由于执行远程服务器上的命令的。内容如下：

#!/usr/bin/env bash
OP=${1:-help}
LOG=/config/log/fail2ban/rix_remote.log
URL="a"
function curl_post()
{
    curl -X POST -d "$1" -H "Content-Type: application/json" $URL 
}
function help()
{
    echo "help" 
}
function start()
{
    echo "start" $* >> $LOG
}
function stop()
{
    echo "stop" $* >> $LOG
}
function check()
{
    echo "check" $* >> $LOG
}
function ban()
{
    curl_post '{"op": "ban", "name":"'$1'", "ip":"'${3}'", "port": "'${4}'", "domain": "'${2}'"}' >> $LOG
}
function unban()
{
    curl_post '{"op": "unban", "name":"'$1'", "ip":"'${3}'", "port": "'${4}'", "domain": "'${2}'"}' $LOG
}
echo "all_cm" $* >> $LOG
shift
$OP $*

我用现有内网中的 node.js 的 http 服务器，搭建了一个简单的执行 ssh 命令的服务。核心内容如下

async function fail2ban(req)
{
    logger.info(`fail2ban req=${JSON.stringify(req)}`);
    let data = JSON.parse(req.data);
    let {name, ip, domain} = data;
    let ssh_config = {}
    let ssh = new NodeSSH();
    try {
        await ssh.connect(ssh_config);
        let func = {
            ban: async ()=> {
                let exec_ret = await ssh.exec(`docker exec -t fail2ban fail2ban-client set ${data.name} banip ${data.ip}`, []);
                logger.info(`fail2ban ban return ${exec_ret}`);
            },
            unban: async()=> {
                let exec_ret = await ssh.exec(`docker exec -t fail2ban fail2ban-client set ${data.name} unbanip ${data.ip}`, []);
                logger.info(`fail2ban ban unreturn ${exec_ret}`);
            },
            list: async()=> {
                let exec_ret = await ssh.exec(`docker exec -t fail2ban fail2ban-client get ${data.name} banip --with-time`, []);
            }
        };
        if(func[req.op]) {
            await func[req.op]();
        } else {
            logger.info(`fail2ban unknow op=${req.op}`);
        }
    } catch(e) {
        logger.info(`fail2ban error`,e);
    } finally {
        if(ssh.connection) {
            ssh.connection.end();
        }
    }
}

如此，实现了内网 mailu 的外网访问，内网识别和屏蔽恶意 IP 的功能

作者： rix

Android改直流供电自动开机自动开无线调试

mailu 配置 fail2ban