如果正常情况下,官方的配置应该没有太大问题。但我的环境有些特殊,由于我的邮箱服务器搭建在内网的 unraid 中,而我的访问接口需要在公网上,所以,在中间不得不添加转发的机制。目前我的大致访问流程是这样的:
公网 nginx 代理(公网)->zerotier(网关)->内网 nginx 代理->mailu(1.9)
上述所有的部分除了 zerotier,都是通过 docker 部署的。 因为内网所有的服务都是通过内网的 nginx 反向代理的,因此在公网的 nginx 部分配置 tcp 代理是可以的。 但如此一来,也导致了内网的 nginx 拿到的 ip 都是 zerotier 网关的 ip,或者拿到的公网的 nginx 代理的 docker 网关的 ip。对于后续部署 fail2ban 部分带来很大的麻烦。如果都是 http/https 服务,或许还好弄些,但我的是 nginx 还使用 tcp 代理了邮箱的各个端口,以便转发给内网部分。
为了给公网上的服务器添加上 ip 屏蔽的功能,只好进行了下面的改造
公网(docker 版本的 nginx+fail2ban 定制) -> zerotier(不动)-> 内网(docker nginx 小修改+fail2ban 定制+mailu 代码修改)
公网 nginx 上的改造
公网上的 nginx 也是在 docker 中的,为了方便进行 fail2ban 的管理,我又增加了一套 docker 版本的 fail2ban。 nginx 的改造如下:
stream { server { listen 443 reuseport; proxy_pass 10.10.10.10:443; proxy_protocol on; ssl_preread on; } server { listen 587 reuseport; proxy_pass 10.10.10.10:587; proxy_protocol on; } #...其他端口类似 }
proxy-protocol 这个字段添加后,在 header 部分会多出一个 “Proxy-Protocol-Addr”字段,这个字段保存的是客户端的真实 ip, 在后续的部分,需要用这个字段来识别客户端的真实 ip,并且用于 fail2ban 的屏蔽。
内网改造
nginx(代理) 上的改造
由于外网使用了 stream 来代理,并且开启了 proxy_protocol, 因此内网也需要对应的开启。这样一来, 内网就没办法通过普通的 https 直接访问了。如果要访问,可以在内网部分再添加个 stream 代理
server { listen 443 ssl http2 proxy_protocol; server_name you_subadmin; real_ip_header proxy_protocol; real_ip_recursive on; location / { proxy_set_header Host $http_host; proxy_set_header X-Real-IP $proxy_protocol_addr; proxy_set_header X-Forwarded-For $proxy_protocol_addr; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass https://mailu:8443; proxy_buffering off; tcp_nodelay on; keepalive_timeout 55; } }
上面的部分,可以让 mailu 部分的 front 获取到正确的外网地址
mailu 的 front 部分的 nginx 的改造
# Basic configuration user nginx; worker_processes auto; error_log /dev/stderr notice; pid /var/run/nginx.pid; load_module "modules/ngx_mail_module.so"; events { worker_connections 1024; } http { # Standard HTTP configuration with slight hardening include /etc/nginx/mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; server_tokens off; absolute_redirect off; resolver {{ RESOLVER }} valid=30s; {% if REAL_IP_HEADER %} #real_ip_header {{ REAL_IP_HEADER }}; {% endif %} {% if REAL_IP_FROM %}{% for from_ip in REAL_IP_FROM.split(',') %} set_real_ip_from {{ from_ip }}; {% endfor %}{% endif %} real_ip_header proxy_protocol; proxy_set_header X-Real-IP $proxy_protocol_addr; proxy_set_header X-Forwarded-For $proxy_protocol_addr; real_ip_recursive on; # Header maps map $http_x_forwarded_proto $proxy_x_forwarded_proto { default $http_x_forwarded_proto; '' $scheme; } map $uri $expires { default off; ~*\.(ico|css|js|gif|jpeg|jpg|png|woff2?|ttf|otf|svg|tiff|eot|webp)$ 97d; } map $request_uri $loggable { /health 0; /auth/email 0; default 1; } log_format main '$proxy_protocol_addr $remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" "$gzip_ratio"'; #access_log /var/log/nginx/access.log main; access_log /dev/stdout main;# if=$loggable; # compression gzip on; gzip_static on; gzip_types text/plain text/css application/xml application/javascript gzip_min_length 1024; # TODO: figure out how to server pre-compressed assets from admin container {% if KUBERNETES_INGRESS != 'true' and TLS_FLAVOR in [ 'letsencrypt', 'cert' ] %} # Enable the proxy for certbot if the flavor is letsencrypt and not on kubernetes # server { # Listen over HTTP listen 80; listen [::]:80; {% if TLS_FLAVOR == 'letsencrypt' %} location ^~ /.well-known/acme-challenge/ { proxy_pass http://127.0.0.1:8008; } {% endif %} # redirect to https location / { return 301 https://$host$request_uri; } } {% endif %} # Main HTTP server server { # Favicon stuff root /static; # Variables for proxifying set $admin {{ ADMIN_ADDRESS }}; set $antispam {{ ANTISPAM_WEBUI_ADDRESS }}; {% if WEBMAIL_ADDRESS %} set $webmail {{ WEBMAIL_ADDRESS }}; {% endif %} {% if WEBDAV_ADDRESS %} set $webdav {{ WEBDAV_ADDRESS }}; {% endif %} client_max_body_size {{ MESSAGE_SIZE_LIMIT|int + 8388608 }}; # Listen on HTTP only in kubernetes or behind reverse proxy {% if KUBERNETES_INGRESS == 'true' or TLS_FLAVOR in [ 'mail-letsencrypt', 'notls', 'mail' ] %} listen 80; listen [::]:80; {% endif %} # Only enable HTTPS if TLS is enabled with no error and not on kubernetes {% if KUBERNETES_INGRESS != 'true' and TLS and not TLS_ERROR %} listen 443 ssl http2; listen [::]:443 ssl http2; real_ip_header proxy_protocol; proxy_set_header X-Real-IP $proxy_protocol_addr; real_ip_recursive on; include /etc/nginx/tls.conf; ssl_stapling on; ssl_stapling_verify on; ssl_session_cache shared:SSLHTTP:50m; add_header Strict-Transport-Security 'max-age=31536000'; {% if not TLS_FLAVOR in [ 'mail', 'mail-letsencrypt' ] %} if ($proxy_x_forwarded_proto = http) { return 301 https://$host$request_uri; } {% endif %} {% endif %} # Remove headers to prevent duplication and information disclosure proxy_hide_header X-XSS-Protection; proxy_hide_header X-Powered-By; add_header X-Frame-Options 'SAMEORIGIN'; add_header X-Content-Type-Options 'nosniff'; add_header X-Permitted-Cross-Domain-Policies 'none'; add_header X-XSS-Protection '1; mode=block'; add_header Referrer-Policy 'same-origin'; {% if TLS_FLAVOR == 'mail-letsencrypt' %} location ^~ /.well-known/acme-challenge/ { proxy_pass http://127.0.0.1:8008; } {% endif %} # If TLS is failing, prevent access to anything except certbot {% if KUBERNETES_INGRESS != 'true' and TLS_ERROR and not (TLS_FLAVOR in [ 'mail-letsencrypt', 'mail' ]) %} location / { return 403; } {% else %} include /overrides/*.conf; # Actual logic {% if ADMIN == 'true' or WEBMAIL != 'none' %} location ~ ^/(sso|static)/ { include /etc/nginx/proxy.conf; proxy_set_header X-Real-IP $proxy_protocol_addr; proxy_set_header X-Forwarded-For $proxy_protocol_addr; proxy_pass http://$admin; } {% endif %} {% if WEB_WEBMAIL != '/' and WEBROOT_REDIRECT != 'none' %} location / { expires $expires; {% if WEBROOT_REDIRECT %} try_files $uri {{ WEBROOT_REDIRECT }}; {% else %} try_files $uri =404; {% endif %} } {% endif %} {% if WEBMAIL != 'none' %} location {{ WEB_WEBMAIL }} { {% if WEB_WEBMAIL != '/' %} rewrite ^({{ WEB_WEBMAIL }})$ $1/ permanent; rewrite ^{{ WEB_WEBMAIL }}/(.*) /$1 break; {% endif %} include /etc/nginx/proxy.conf; auth_request /internal/auth/user; error_page 403 @webmail_login; proxy_set_header X-Real-IP $proxy_protocol_addr; proxy_set_header X-Forwarded-For $proxy_protocol_addr; proxy_pass http://$webmail; } {% if WEB_WEBMAIL == '/' %} location /sso.php { {% else %} location {{ WEB_WEBMAIL }}/sso.php { {% endif %} {% if WEB_WEBMAIL != '/' %} rewrite ^({{ WEB_WEBMAIL }})$ $1/ permanent; rewrite ^{{ WEB_WEBMAIL }}/(.*) /$1 break; {% endif %} include /etc/nginx/proxy.conf; proxy_set_header X-Real-IP $proxy_protocol_addr; auth_request /internal/auth/user; auth_request_set $user $upstream_http_x_user; auth_request_set $token $upstream_http_x_user_token; proxy_set_header X-Remote-User $user; proxy_set_header X-Remote-User-Token $token; error_page 403 @webmail_login; proxy_pass http://$webmail; } location @webmail_login { return 302 /sso/login; } {% endif %} {% if ADMIN == 'true' %} location {{ WEB_ADMIN }} { include /etc/nginx/proxy.conf; proxy_pass http://$admin; proxy_set_header X-Real-IP $proxy_protocol_addr; expires $expires; } location {{ WEB_ADMIN }}/antispam { rewrite ^{{ WEB_ADMIN }}/antispam/(.*) /$1 break; auth_request /internal/auth/admin; proxy_set_header X-Real-IP ""; proxy_set_header X-Forwarded-For ""; proxy_pass http://$antispam; } {% endif %} {% if WEBDAV != 'none' %} location /webdav { rewrite ^/webdav/(.*) /$1 break; auth_request /internal/auth/basic; auth_request_set $user $upstream_http_x_user; include /etc/nginx/proxy.conf; proxy_set_header X-Remote-User $user; proxy_set_header X-Script-Name /webdav; proxy_pass http://$webdav; } location ~ ^/.well-known/(carddav|caldav) { return 301 /webdav/; } {% endif %} {% endif %} location /internal { internal; #proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Real-IP $proxy_protocol_addr; proxy_set_header Authorization $http_authorization; proxy_pass_header Authorization; proxy_pass http://$admin; proxy_pass_request_body off; proxy_set_header Content-Length ""; } location /health { return 204; } } # Forwarding authentication server server { # Variables for proxifying set $admin {{ ADMIN_ADDRESS }}; listen 127.0.0.1:8000; location / { proxy_pass http://$admin/internal$request_uri; } } } mail { server_name {{ HOSTNAMES.split(",")[0] }}; auth_http http://127.0.0.1:8000/auth/email; proxy_pass_error_message on; resolver {{ RESOLVER }} valid=30s; error_log /dev/stderr debug; #error_log /var/log/nginx/mail_error.log info; {% if TLS and not TLS_ERROR %} include /etc/nginx/tls.conf; ssl_session_cache shared:SSLMAIL:50m; {% endif %} #proxy_protocol on; # Advertise real capabilites of backends (postfix/dovecot) smtp_capabilities PIPELINING SIZE {{ MESSAGE_SIZE_LIMIT }} ETRN ENHANCEDSTATUSCODES 8BITMIME DSN; pop3_capabilities TOP UIDL RESP-CODES PIPELINING AUTH-RESP-CODE USER; imap_capabilities IMAP4 IMAP4rev1 UIDPLUS SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+; # Default SMTP server for the webmail (no encryption, but authentication) server { listen 10025; protocol smtp; smtp_auth plain; auth_http_header Auth-Port 10025; } # Default IMAP server for the webmail (no encryption, but authentication) server { listen 10143; protocol imap; smtp_auth plain; auth_http_header Auth-Port 10143; } # SMTP is always enabled, to avoid losing emails when TLS is failing server { listen 25 proxy_protocol; listen [::]:25 proxy_protocol; {% if TLS and not TLS_ERROR %} {% if TLS_FLAVOR in ['letsencrypt','mail-letsencrypt'] %} ssl_certificate /certs/letsencrypt/live/mailu/fullchain.pem; ssl_certificate /certs/letsencrypt/live/mailu-ecdsa/fullchain.pem; {% endif %} ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA; ssl_prefer_server_ciphers on; starttls on; {% endif %} protocol smtp; smtp_auth none; auth_http_header Auth-Port 25; } # All other protocols are disabled if TLS is failing {% if not TLS_ERROR %} server { listen 143 proxy_protocol; listen [::]:143 proxy_protocol; {% if TLS %} starttls only; {% endif %} protocol imap; imap_auth plain; auth_http_header Auth-Port 143; } server { listen 110 proxy_protocol; listen [::]:110 proxy_protocol; {% if TLS %} starttls only; {% endif %} protocol pop3; pop3_auth plain; auth_http_header Auth-Port 110; } server { listen 587 proxy_protocol; listen [::]:587 proxy_protocol; {% if TLS %} starttls only; {% endif %} protocol smtp; smtp_auth plain login; auth_http_header Auth-Port 587; } {% if TLS %} server { listen 465 ssl proxy_protocol; listen [::]:465 ssl proxy_protocol; protocol smtp; smtp_auth plain login; auth_http_header Auth-Port 465; } server { listen 993 ssl proxy_protocol; listen [::]:993 ssl proxy_protocol; protocol imap; imap_auth plain; auth_http_header Auth-Port 993; } server { listen 995 ssl proxy_protocol; listen [::]:995 ssl proxy_protocol; protocol pop3; pop3_auth plain; auth_http_header Auth-Port 995; } {% endif %} {% endif %} }
主要修改部分就是给 mail 的 server 对外部分添加上 proxy_protocol, 因为外网的 nginx 的 stream 添加了,因此这部分也需要添加上,不然服务器会转发失败的。
mailu 的授权主要在 admin 部分,因此我们只要修改这部分就可以了。这部分默认读取的 IP 是从 Client-Ip 字段中来的,现在真实的 IP 保存在”Proxy-Protocol-Addr”中,在 nginx.py 和 auth.py 中搜索 Client-Ip, 并在合适的位置添加上下面的内容:
if headers.get("Proxy-Protocol-Addr"): client_ip = flask.request.headers["Proxy-Protocol-Addr"]
但 admin 的日志中显示的还是错误的地址,为了配合 fail2ban 的使用,我在所有授权失败的地方添加上了 这样的错误日志的输出:
app.logger.error(f'Auth Fail user ip {client_ip!r} Auth not supported 403')
主要是添加 fail2ban 可识别的正则表达式的格式。admin 部分默认的日志输出都是在 stderr 或者 stdout 的, 我也修改了 start.py 一下,让日志保存到/var/log 中
start_command="".join([ "gunicorn --threads ", str(os.cpu_count()), " -b :80 ", "--access-logfile /var/log/access.log " if (log.root.level<=log.INFO) else "", "--error-logfile /var/log/error.log ", "--preload ", "'mailu:create_app()'"]) os.system(start_command)
fail2ban 部分
内网和外网的 fail2ban 部分基本一致,除了外网的 fail2ban 只进行屏蔽和解封的操作,剩余的判定和遥控都是在 内网进行的。
filter.d 部分
内外网的 fail2ban 都需要
#filename rix-mailu-admin.local [INCLUDES] before = common.conf [Definition] failregex = .* Auth (Fail|None) .* ip '<HOST>' .* (403)$ ignoreregex = datepattern = \[%%Y-%%m-%%d %%H:%%M:%%S,%%f\]
jail.d 部分
[rix-mailu-admin] enabled = true chain = DOCKER-USER filter = rix-mailu-admin maxretry = 5 findtime = 30m bantime = 30d #下面这两行外网不需要,内网需要 action = rix_remote[name=rix-mailu-admin,port="443,4431,80,25,110,143,465,587,993,995",domain="your_email_domain"] logpath = /remotelogs/mailu/error.log
action.d 部分(只内网需要)
[Definition] actionstart= /config/rix_remote_cmd start fail2ban-<name> actionstop = /config/rix_remote_cmd stop fail2ban-<name> actioncheck = /config/rix_remote_cmd check fail2ban-<name> actionban = /config/rix_remote_cmd ban <name> <domain> <ip> <port> actionunban = /config/rix_remote_cmd unban <name> <domain> <ip> <port> [Init] chain = INPUT name = default protocol = tcp
上面的配置中,rix_remote_cmd 是我写的一个小脚本,由于执行远程服务器上的命令的。内容如下:
#!/usr/bin/env bash OP=${1:-help} LOG=/config/log/fail2ban/rix_remote.log URL="a" function curl_post() { curl -X POST -d "$1" -H "Content-Type: application/json" $URL } function help() { echo "help" } function start() { echo "start" $* >> $LOG } function stop() { echo "stop" $* >> $LOG } function check() { echo "check" $* >> $LOG } function ban() { curl_post '{"op": "ban", "name":"'$1'", "ip":"'${3}'", "port": "'${4}'", "domain": "'${2}'"}' >> $LOG } function unban() { curl_post '{"op": "unban", "name":"'$1'", "ip":"'${3}'", "port": "'${4}'", "domain": "'${2}'"}' $LOG } echo "all_cm" $* >> $LOG shift $OP $*
我用现有内网中的 node.js 的 http 服务器,搭建了一个简单的执行 ssh 命令的服务。核心内容如下
async function fail2ban(req) { logger.info(`fail2ban req=${JSON.stringify(req)}`); let data = JSON.parse(req.data); let {name, ip, domain} = data; let ssh_config = {} let ssh = new NodeSSH(); try { await ssh.connect(ssh_config); let func = { ban: async ()=> { let exec_ret = await ssh.exec(`docker exec -t fail2ban fail2ban-client set ${data.name} banip ${data.ip}`, []); logger.info(`fail2ban ban return ${exec_ret}`); }, unban: async()=> { let exec_ret = await ssh.exec(`docker exec -t fail2ban fail2ban-client set ${data.name} unbanip ${data.ip}`, []); logger.info(`fail2ban ban unreturn ${exec_ret}`); }, list: async()=> { let exec_ret = await ssh.exec(`docker exec -t fail2ban fail2ban-client get ${data.name} banip --with-time`, []); } }; if(func[req.op]) { await func[req.op](); } else { logger.info(`fail2ban unknow op=${req.op}`); } } catch(e) { logger.info(`fail2ban error`,e); } finally { if(ssh.connection) { ssh.connection.end(); } } }
如此,实现了内网 mailu 的外网访问,内网识别和屏蔽恶意 IP 的功能