January – Mooray, updates!

Sorry for the lack of december update news.
 
 

Important change for SAL users (see “support development” on the right sidebar)

You can now set WATCHDOG_EXTERNAL_CHECKS=y in mailcow.conf to enable an open relay check. The check is run about every minute.
In the future, you will be able to shut down Postfix whenever watchdog-mailcow detects an open relay.
Your source IP must match your mailcow IP, will only work with unmodified mailcows.
 
 

Important changes for all moos

  • App passwords! They work for IMAP and SMTP connections, not yet for SOGo – but we are working on it. Login as user to find them. You can also restrict access via ACL.
  • Do not reject .doc per se, but reject when any document has a macro assigned to it.
  • SOGo can be built using a subscription:
      sogo-mailcow:
        build:
          context: ./data/Dockerfiles/sogo
          dockerfile: Dockerfile
          args:
            - SOGO_DEBIAN_REPOSITORY=https://user:pass@packages.inverse.ca/SOGo/release/4/debian/
    
  • Sieve and Rspamd presets were improved. Create presets in data/web/inc/presets/rspamd/. Headline can be a lang string. Please feel free to add more useful presets!
  • Mail forwards and rejects were improved. Rejects are now signed, forwards are only ARC_SIGNED and remain SPF and DKIM valid.

A BIG THANK YOU to all supporters! Thank you so much for keeping mailcow alive. 🙂
Another BIG THANK YOU goes out to all contributors!
 
 

Changes (please also see the commit history)

[API] Added DKIM get route to api docs
[API] Added docs for new status api
[API] Added new status route to get some system infos
[API] Fixed api docs not being displayed correctly
[API] Make Solr API return data if Solr is enabled
[API] Update API docs with app password routes
[Rspamd] ARC remains active for forwards. Result: fully signed and trusted forwards and signed rejects in sieve.
[Rspamd] block all Office documents with macros
[CI] Added automated testing using drone (#3278)
[ClamAV] Whitelist JS in PDF – too many false-positives
[Web] Disable refresh button, while refreshing (#3199)
[Dovecot] Add map for app passwds
[Dovecot] Change LUA path
[Dovecot] Delete ham/spam hash if previously learned; Change LUA script pathes
[Dovecot] Drop logs
[Dovecot] Enable editheaders plugin in sieve for all users
[Dovecot] Fix app passwds: allow multiple pass hashes by using LUA construct
[Dovecot] Fix lua error when trying to escape empty domains
[Dovecot] Really strange race condition when reading an untouched LUA file on slower systems
[Dovecot] Remove CONTROL from shared namespace – thanks to @Keessaus
[Dovecot] Set BCC in quarantine notify
[Git] Ignore auto generated Dovecot LUA
[Git] Ignore whitelist.ign2
[IMPORTANT] If you run Ubuntu 16.04, upgrade your kernel to linux-generic-hwe-16.04
[Nginx] Catch case-insensitive /sogo$ request and redirect to /SOGo
[PHP-FPM] Remove useless flag for gd
[Postfix] Add bl.suomispam.net
[Postfix] Client rcpt rate limit set to 50
[Postfix] Set CA path for smtpd
[Postfix] Update Postscreen whitelist
[Rspamd] Add mailcow_networks map
[Rspamd] Allow empty envfrom for system mails, add only Dovecot to sign_networks and sign by header when sign_networks fires.
[Rspamd] allow_hdrfrom_mismatch true, auth_only false (sieve)
[Rspamd] Decrease weight of missed charset
[Rspamd] Do not normalise domains to eSLD for ARC
[Rspamd] Lower map watch interval
[Rspamd] Ratelimit for bounces reduced, max_rcpt for ratelimit increased
[Rspamd] SA trivial converter (wip)
[Rspamd] Set rspamd as trusted host, rspamd is not spoofing
[Rspamd] Split deprecated metrics.conf to actions.conf and groups.conf
[SOGo] Fix for whitespaces in mysql return; Order aliases
[SOGo] Make view more readable
[SOGo] Read build args
[SSL] fix bug with pruning old certificates (#3272)
[Update] Split metrics to actions and groups, warn if metrics is different from repo
[Web] Use main_name in the “Yubico OTP Authentifizierung” modal and in the mailbox edit modal.
[Watchdog] Add external check for open relay, requires SAL
[Watchdog] Fix ipv6 config check
[Watchdog] Retry to get current ACME log status, if empty (may fix watchdog mails on very busy servers – eg while running a backup)
[Watchdog] Revert acme-mailcow threshold to 1
[Watchdog] smtp-cli 3.10 (yay) and a new check for IPv6 configuration problems
[Web] Add “add” button to header of table
[Web] Add missing lang strings for edit
[Web] Add more password generator links
[Web] Add more map types soon; Do not expose private key via API if hidden in vars (fixes #3231)
[Web] Add more sieve presets
[Web] Add new preset for Rspamd settings map: Only allow specific senders to send to a mailbox
[Web] Allow to set BCC for quarantine
[Web] Allow to use data/web/css/build/0081-custom-mailcow.css for ignored overrides
[Web] Better mobileconfig handling
[Web] Complain about non-email email fields
[Web] Deleted hashes previously learned
[Web] Do not show Solr and Clam status when disabled, thanks to Tina
[Web] Feature: Allow app passwords for imap/smtp, allow to set acl permission for app passwords (domain admin [when logged in as user] and user)
[Web] Finally fix solr and clam status…
[Web] Fix global maps
[Web] Fix lang.en.json
[Web] Fix policy map selection for dane
[Web] Fix quarantine for sneaky dots, also fixes #3263
[Web] Fix Solr status and sort containers
[Web] Fix some major errors in app passwds but disable app passwds due to a show stopper… todo: fix asap
[Web] Fix some transport verifications
[Web] Fix transport validation for hostnames
[Web] Generate longer passwords for app passwords
[Web] Generate longer passwords for app passwords (edit was missing)
[Web] Get all app passwd ids for a single user by using get/app-passwd/all/user@domain
[Web] Hide app passwords from logs
[Web] hide echoed var
[Web] Make mobile usage less annoying; anchors for maps; sidebar for maps
[Web] Minor style fix and re-enable app passwds
[Web] Remove “add domain” from table when not admin, fixes #3267
[Web] Remove tracking for custom-mailcow css
[Web] Revert dropup to dropdown
[Web] Revert some style changes, mobile view should be fixes/better with bootstrap 4
[Web, Rspamd] Add bad language map, add map to mailcow UI
[Web] Show hint when SOGo admin login is enabed, fix sieve preset in API
[Web] Small adjustments to presets
[Web] Update languages
[Web] Various fixes for app passwd functions

Update your Kernel, if you are on Ubuntu 16.04

You may encounter errors with Dovecot or ClamAV (and probably other containers besides mailcow), if you run Ubuntu 16.04 with its default kernel 4.4 and Docker from the official Docker repository.

Please install the HWE kernel from the Ubuntu repository and reboot:

apt-get update
apt-get install --install-recommends -y linux-generic-hwe-16.04
reboot

MariaDB 10.3 rollout

We just included MariaDB 10.3. If you run ./update.sh, you will encounter some errors in mysql-mailcow, that will be fixed by the upgrade process triggered by php-fpm-mailcow.
A SQL backup is recommended.

EN: If you have a mailcow support subscription, feel free to use the ticket system for assistance or help after a failed update.
DE: Falls ein mailcow Supportpaket gebucht wurde, helfen wir gerne beim Update der mailcow Installation oder bei etwaigen Problemen.

Mootember updates

Sorry for the delay, mailcow was still worked on. Time is running too fast. 🙁

Instead of writing down all commits you can already find on GitHub, I will only mention important changes or fixes dating back from the previous post until today.

Edit: Forgot to say thanks to @irgendwr for fixing XSS!

  • The GAL (Global Address List) is now enabled by default. Various actions like availability in calendars depends on it.
  • Images are prefetched on ./update.sh (`–prefetch` will only prefetch images and exit)
  • Added SOGO_EXPIRE_SESSION variable to mailcow.conf to define when a session in SOGo times out
  • Added a whitelist map for IPs in Rspamd (`data/conf/rspamd/custom/ip_wl.map`, CIDR)
  • SAL was introduced – an **optional** license with some benefits in the future. Think about basic monitoring etc. 🙂
  • Various XSS fixes by @patschi and @irgendwr – thanks!
  • Some services were finally ported to Py3 – thank you @zkryakgul! Also thank you for exposing your policyd password in your commit, we will take care of it.
  • Added a bad word list, that triggers only, when received from a fishy tld (yes, that’s a new map, too)
  • You can now allow a mailbox to send from an external domain or only a defined set of mail addresses (edit a mailbox to find this feature)
  • @christianbur forces me to update the images more regulary, thank you.
  • Various fixes and changes in mailcow UI
  • Solr is now exposed to 127.0.0.1:18389 by default, you can setup a reverse proxy to browse its fancy UI – do not expose it to the internet!
  • I broke the anonymize headers, again. @iiegn and @patschi pointed it out, thanks!

Update your HTTP to HTTPS redirection and mailcow to prevent acme-mailcow HTTP verification failures

Hi,

Please see the updated reverse proxy docs and adjust your site.conf (only if you are not using a reverse proxy!).

An older draft excluded autoconfig from the redirection. Older setups may still have two server blocks at the top of that file. This is obsolete and needs to be changed! This is what it looked like before, it is wrong!

Correct and new: https://mailcow.github.io/mailcow-dockerized-docs/u_e-80_to_443/
Wrong and old: https://github.com/mailcow/mailcow-dockerized-docs/blob/cdf1a436ca904186d78e391bab71f2747af2a1af/docs/u_e-80_to_443.md

Please also update mailcow or at least acme-mailcow (make sure you use at least v1.60).

June updates

For commercially used mailcows, please consider buying a support subscription or help to keep mailcow alive by donating. 🙂

ONCE AGAIN A BIG THANKS TO ALL CONTRIBUTORS!

Important changes

  • Thanks to @c-rosenberg we integrated olefy into mailcow for macro scanning. It is skipped for authenticated users.
  • Quarantine can now be configured to remove old items.
  • *.autoconfig is back as part of default SANs in the certificate acquired by “acme-mailcow”.
    IMPORTANT: The certificate request will fail, if you copied the site config from our previous HTTP -> HTTPS redirect docs! Please update your site config (see here), make sure to remove the autoconfig block (see here).

Summary

[ACME] Autoconfig is back (re-added to SAN list by default for all mail domains)
[ACME] Better HTTP verification
[ClamAV] Fix missing exit code var, fixes #2746
[ClamAV] Update to 0.101.2
[Dovecot] Fix cleanup of old quarantine items, fixes #2721
[Dovecot] Rename sieve_after to global_sieve_after and create a global_sieve_before file
[Dovecot] Run cronjob at 4:15 AM to remove max aged quarantine items (default: 365 days)
[Dovecot] Fix permissions of console
[Helper] Omits the verbose option for gzip in the backup script
[Netfilter] Keep musl-dev, update pip
[Netfilter] Remove unused files after installation
[Netfilter] Set some f2boptions to int
[Olefy] A new container is born, thanks to @c-rosenberg
[PHP-FPM] Add exif module
[PHP-FPM] Add Q_MAX_AGE key and set to 365 days if missing
[PHP-FPM] Update PHP and some modules
[Postfix] Add UA header check, not enabled by default
[Postfix] Create resource maps
[Postfix] Remove authed user from header
[Postfix] Remove duplicate proxy read maps, add resource maps
[Rspamd] Added comment to composite
[Rspamd] Add OLEFY_MACRO symbol
[Rspamd] Add oletools via olefy, big thanks to @c-rosenberg
[Rspamd] Increase OLEFY_MACRO score
[Rspamd] Less aggressive bayes
[Rspamd] Remove authenticated user from auth results header
[Rspamd] Sign ARC inbonud, thanks to @Kraeutergarten
[Update] Add missing WATCHDOG_NOTIFY_BAN update option
[Update] Check if file is tracked before running git rm
[Update] Hide error when running git rm on old worker passwd file
[Update] Remove controller passwd file from index
[update.sh] Rename enable_ipv6 option
[Watchdog, Config] Added WATCHDOG_NOTIFY_BAN to disable IP ban notifications
[Watchdog] Fix broken mail with more than one rcpt in some cases
[Watchdog] Fix for fix for wrong mails
[Watchdog] Minor text changes and send whois report of banned IP
[Watchdog] Send mail when IP was banned
[Web] Allow to set max_age for quarantine items
[Web] Fix BCC error message
[Web] Fix lang strings for sieve pathes and sieve flow
[Web] Fix SPF link
[Web] Generate readable passwords
[Web] increased db version
[Web] Show resource alias

Moo-y? May updates!

For commercially used mailcows, please consider buying a support subscription or help to keep mailcow alive by donating. 🙂

BIGGEST THANKS TO ALL CONTRIBUTORS! I think you guys fixed more bugs than I could add… seriously, you guys did a great job, thanks!

Important changes

  • Quota is no more recalculated in a daily cronjob. If you need to recalc the quota, run “docker-compose exec dovecot-mailcow doveadm quota recalc -A”
  • White and blacklists (IP) are read every minute. You don’t need to restart netfilter-mailcow after adding blacklist records anymore, but you will need to wait about a minute for new entries to apply.
  • A lot of scripts are now compatible with Py3, thank you guys!!!

Summary

[ACME] acme-tiny with python3 (thanks to @christianbur)
[ACME] Add 0 byte check for cert.pem
[ACME] Allow to skip all names but MAILCOW_HOSTNAME
[ACME] Changed the threshold for certificate renewal
[ACME] Register error when no hostname could be validated
[ClamAV] Increase watchdog clamd-mailcow thresholds
[Compose] Remove oom check for compatibility
[Config] Clarification about mailcow_hostname
[DockerAPI] Python3 (big thanks to @christianbur, again)
[Dovecot] Added domain alias handling to quarantine function; Add recipients row to quarantine overview (thanks to @franz.reiter)
[Dovecot] Auto-generate shared namespace
[Dovecot] (Dirty/Workaround) Fix memory leak when quarantine sender has non-ascii chars in mail address
[Dovecot] Fix processing imapsync custom parameters (big thanks to @hunter-nl)
[Dovecot] Removed nightly quota recalc job (too intensive on larger systems)
[Dovecot] Remove shared namespace
[Dovecot] Revert to previous imapsync cron script
[Dovecot] Set default_vsz_limit = 1024 M
[Dovecot] Trim watchdog logs (thanks to @HorayNarea for fixing this commit)
[Dovecot] Update Dovecot to v2.3.6 and Pigeonhole to v0.5.6
[Dovecot] Update imapsync script to 1.937 and added noreleasecheck parameter
[GitHub Templates] Added funding template (thanks to @ntimo)
[Git] Ignore shared namespace file
[Helper] Update Nextcloud helper – todo: fix upgrade…
[Netfilter] Reworked by @Kraeutergarten – thanks! –
[Nextcloud] Always install under subdomain, minor changes to site config and install script
[Postfix] Do not allow DSN for postscreen
[Rspamd] Add SIEVE_HOST map and skip spoof check for these IPs
[rspamd] Allow to easily use custom rspamd lua plugins
[Rspamd] Auto-generate SIEVE_HOST map and add dnsutils
[Rspamd] Change spoofed mail handling
[Rspamd] Do not apply SPOOFED_UNAUTH on ARC_ALLOW
[Dovecot] Set sieve_redirect_envelope_from to rcpt
[Rspamd] Fix spoofing detection
[Rspamd] Improve spoofing detection
[Rspamd] meta_exporter: return false if not matched
[Rspamd] Much higher scores for DMARC failures
[Rspamd] Set to to_ip to_ip_from rate buckets to 100 / 1s
[Rspamd] Update to 1.9.2, minor entrypoint changes
[Solr] Make entrypoint executable
[Update] Increase docker-compose timeout
[Update] Prefetch images, big thanks to everyone in #2637!
[Watchdog] Change error message for acme-mailcow
[Watchdog] Send mail when starting
[Web] Add function to read F2B data via API
[Web] Add only existing domains in table to the filter and removes additional ajax request (thanks to @Kraeutergarten)
[Web] Allow aliases as send-as
[Web] Allow to rename “alias” to “Alias”
[Web] API reads JSON body, big thanks to @feldsam
[Web] Change session timeout handling
[Rspamd] Add missing spamassassin.conf
[Web] Disable refresh button on reload, re-enable after table init
[Web] Fix class for full mailbox
[Web] Fix null ua in debug.js – fixes #2615
[Web] Fix showing domain with disabled sender check (thanks to @foutrelis)
[Web] Fix some breakpoints
[Web] Form cache for user passwd change modal disabled
[Web] Handle mobileconfig display names with special characters (thanks to @emericklaw)
[Web] Minor fixes
[Web] More minor css fixes
[Web] Show error when connection to redis fails (instead of 5xx)
[Web] Updated php-mime-mail-parser library to 5.1 to fix webui html preview (thanks to @Howaner)
[Web] Various UI fixes
[Web] Write API logs when format is data binary

Mid-April updates

For commercially used mailcows, please consider buying a support subscription or help to keep mailcow alive by donating. 🙂

I really, really miss working on mailcow full time. Only because of all amazing contributors, mailcow is still growing and gaining new features. Thanks!!!

Greetings to tinc.gmbh!

Important changes

  • Thanks to @mhofer117 we can now add “ALLOW_ADMIN_EMAIL_LOGIN=y” to mailcow.conf to allow an administrator to login as a mailbox user to SOGo!
  • New setups will now use ~/Maildir as default mail path.
  • A spamassassin.conf for Rspamd was missing, so Peer Heinleins SA rules were not enabled – damn! Thanks, Peer!
  • We added a name for mailcows Docker bridge: br-mailcow

Summary

[ACME] Allow to skip http verification
[ACME] Set mode 600 for key files
[ACME] Write redis key on non-empty exit code
[ClamAV] Increase watchdog clamd-mailcow thresholds
[Compose] Add named volume sogo-web-vol-1 for static web content
[Compose] Add SKIP_HTTP_VERIFICATION defaulting to n
[Compose] IMPORTANT: Added name for mailcow Docker bridge
[Compose] Update Dovecot, PHP-FPM, Solr, watchdog, Rspamd, ClamAV, SOGo and ACME images
[Config] Add MAILDIR_SUB, “Maildir” for new setups by default
[Config] Add SKIP_HTTP_VERIFICATION
[Dovcot] Cleanup random user maildirs
[Dovecot] Read MAILDIR_SUB for mail_home
[Dovecot] Remove auth cache
[Dovecot] Update Dovecot to v2.3.5.1
[Helper] Do not delete updater for Nextcloud
[Helper] Fix nc script, fixes #2484
[Helper] Remove custom_apps from Nextcloud
[Nextcloud] Always install under subdomain, minor changes to site config and install script
[Nginx] Fix site when ALLOW_ADMIN_EMAIL_LOGIN=y and reverse proxy is used, fixes #2489
[PHP-FPM] Fix SQL upgrade script
[Rspamd] Add missing spamassassin.conf
[Rspamd] Improve spoofing detection
[Rspamd] Update to 1.9 stable repository
[SOGo] Adjust sync parameters, revert if you run into problems!
[SOGo] Remove unnamed volume and rsync web content to named volume
[SOGo] Revert to previous settings
[Solr] Bootstrap cannot be omitted and must occur before mounting the data directory
[Solr] Change default configset before bootstrapping
[Solr] Keep EdgeNGramFilterFactory out of query
[Solr] Make entrypoint executable
[Update] Add MAILDIR_SUB= for updated mailcows
[Update, Config] Set mode 600 for mailcow.conf
[Update] Fix MAILDIR_SUB
[Update] Make the update check in update.sh readonly
[Update] Remove obsolete check/replace command
[Watchdog] Check for ACME failures
[Watchdog] Send mail when starting
[Web] Add ACL for unlimited quota (default 0)
[Web] Allow logout with broken session
[Web] Change session timeout handling
[Web] Continue when a check in add_alias fails
[Web, Dovecot] Allow empty/unlimited quota
[Web] Fix class for full mailbox
[Web] Fix js when adding resource
[Web] Fix “null” output in mailbox table when comments are missing
[Web] Fix rejected mails not being quarantized properly if they are tagged
[Web] Fix slow UI by switching QR provider and only generating qr image on demand
[Web] Fix totp qr code, fixes #2490
[Web] Quarantine – Enhanced JS + Show btn fix event binding
[Web] Reload view and memcached when changing a resource
[Web] Show unlimited quota in user view
[Web] Try to set aria hidden to false when a modal opens
[Web] Update bootstrap slider

Almost-spring-updates

Updated on March 10, 2019: Moooore!

For commercially used mailcows, please consider buying a support subscription or help to keep mailcow alive by donating. 🙂

Many under-the-hood changes since Foobruary updates.

Important changes

  • Updated on March 10, 2019: Due to changes in Solr, you should trigger a rescan (not a full reindex!) for all users: docker-compose exec dovecot-mailcow doveadm fts rescan -A – yes, again – sorry!
  • TLSv1.2 is now the min. required protocol for mandatory encryption in Postfix.
    This affects per-user TLS encryption (when a user enforces TLS) or any TLS policy created with “encrypt” policy or higher.
    We only make an exception for authenticated connections over port 587 and 465, where we accept TLSv1 and higher.
    Dovecot remains at TLSv1+ for IMAP, while we require TLSv1.2+ for LMTP connections now.
  • “vacation-seconds” can now be used in sieve filters
  • The IPv6 NAT check in watchdog-mailcow is less CPU intensive
  • A Postfix transport destination “*” now excludes hosts matching /localhost$/
  • Rspamd settings map is checked for changes instead of always re-applying it in Rspamd – saves CPU time

Summary

Added on March 10

[Web] Change core to dovecot-fts
[Dovecot] Use dovecot-fts core
[Solr] Use fixed, recommended schema but add EdgeNGramFilterFactory
[Compose] Update Rspamd, Postfix, Dovecot and Solr images
[Dovecot] v2.3.5 (PH 0.5.5)
[Dovecot] Change Solr cronjob to fit dovecot-fts
[Postfix] Fix sasl_passwd query from alias domain, fixes #2410
[Rspamd] Remove buggy last-modified check


[ClamAV] Create directory before handling whitelist
[ClamAV] More checks and permission fixes
[Compose] Update ClamAV, watchdog, SOGo and Rspamd images
[Dovecot] Add flags and notify to sieve_extensions
[Dovecot] Enable sieve vacation seconds not just for global scripts
[Dovecot] Fix very stupid error in quarantine_notify.py – thanks to @DevTek314
[Dovecot] Remove vacation-seconds from global-only
[Postfix] Fix mandatory encryption protocols and always require at least TLS 1.2 for LMTP
[Postfix] Mandatory encryption protocol is now min. TLS 1.2
[Postfix] Mandatory protocol for authenticated clients over 587/tcp and 465/tcp is now TLSv1.0+
[Postfix] Force route localhost$ over local:
[Postfix] Remove sasl requiring policies from port 25
[Rspamd] Add fuzzy worker with worker-fuzzy.inc
[Rspamd] Drop rspamd.conf.local
[Rspamd] Make upstream an object
[Rspamd] Mime from and rcpt can now be checked by from_mime and rcpt_mime instead of “header { XY }”
[Rspamd] Reduce SOGO_CONTACT score to -99
[Rspamd] Use almost-stable unstable 🙂
[Rspamd] Check if filterconf table was changed and return Last-Modified accordingly
[Update] Add /opt/bin to PATH, fixes #2391
[Watchdog] Do not hammer API too much when running Ipv6 NAT check
[Watchdog] Run IPv6 NAT check more often (300s sleep instead of 3600s)
[Watchdog] Minor fixes, print last log lines on error
[Watchdog] Use ipv6nat-mailcow instead of ipv6nat
[Web] Fix bootstrap pathes
[Web] Fix transport_check over port 465, fixes #2386
[Web] Strip < and > from start/end of full name
[Web] Update bootstrap to 3.4.1, fixes #2381