- Zimbra Logger
- Configuring Disk Space Notifications
- Monitoring Servers
- Configuring Denial of Service Filter Parameters
- Working with Mail Queues
- Monitoring Mailbox Quotas
- Viewing MobileSync Statistics
- Monitoring Authentication Failures
- Viewing Log Files
- Reading a Message Header
- Fixing Corrupted Mailbox Index
- SNMP Monitoring and Configuration
- Checking MariaDB
- Checking for {product-name} Software Updates
- Updating Zimbra Connector for Microsoft Outlook
- Notifications and Alerts Sent by {product-name}
The {product-name} ({product-abbrev}) includes the following to help you monitor the Zimbra servers, usage, and mail flow:
-
Zimbra Logger package to capture and display server statistics and server status, and to create nightly reports
-
Mailbox quota monitoring
-
MTA mail queue monitoring
-
Log files
Also, selected error messages generate SNMP traps, which can be monitored using an SNMP tool.
|
Note
|
Checking the overall health of the system as a whole is beyond the scope of this document. |
The Logger includes tools for syslog aggregation and reporting. Installing the Logger is optional, but if you do not install it, server statistics and server status information are not captured.
In environments with more than one {product-name} server, Logger is enabled on one mailbox server only. This server is designated as the monitor host. The {product-name} monitor host is responsible for checking the status of all the other {product-name} servers and presenting this information on the Zimbra administration console. Real-time service status, MTA, spam, virus traffic and performance statistics can be displayed. The Logger creates a daily report about mail activity, such as the number of messages, average delivery delay, and errors generated.
|
Note
|
In a multi-server installation, you must set up the syslog configuration files on each server to enable Logger to display the server statistics on the Administration Console, and you must enable the Logger host. If you did not configure this when you installed {product-name}, do so now. |
Enable server statistics to show both system- wide and server specific data about the inbound message volume, inbound message count, anti-spam/anti-virus activity and disk usage for messages processed in the last 48 hours, 30 days, 60 days, and the last year.
-
On each server, as root, type
/opt/zimbra/libexec/zmsyslogsetup. This updates the syslog configuration to enable gathering server statistics. -
On the logger monitor host, you must configure syslog to accept syslog messages from remote machines. See https://wiki.zimbra.com/wiki/Configuring-Logger-Host for details.
|
Note
|
These steps are not necessary for a single-node installation. |
- Admin Console:
-
Home → Monitor
The Server Status page lists all servers and services, their status, and when the server status was last checked. The servers include the MTA, LDAP, and mailbox server. The services include MTA, LDAP, Mailbox, SNMP, Anti-Spam, Anti-Virus, Spell checker, and Logger.
To start a server if it is not running, use the zmcontrol CLI command. You
can stop and start services from the Administration Console.
- Admin Console:
-
Home → Configure → Servers → server
Server services are enabled or disabled from the Servers → server page. Select Services in the Navigation pane and select to enable or disable services.
If the Logger package is installed on a Zimbra mailbox server, Server Statistics shows bar graphs of the message count, message volume, anti-spam, and anti-virus activity. The information is displayed for the last 48 hours, and 30 days, 60 days, and 365 days.
When Server Statistics is selected in the Navigation pane, consolidated statistics for all mailbox servers is displayed. Selecting a specific server in the expanded view shows statistics for that server only. Server specific information also includes disk usage, session information, and mailbox quota details.
The following display system-wide information:
-
Message Count — counts message transactions. A transaction is defined as either the SMTP receipt of a message per person (by Postfix) or a LMTP delivery of it (by mailboxd) per person. For example, if a message is sent to three people, six transactions are displayed. Three for SMTP to Postfix and three for LMTP to mailboxd. The message count is increased by six.
-
Message Volume — displays the aggregate size in bytes of transactions sentand received per hour and per day. Graphs show the total inbound data by volume in bytes.
-
Anti-Spam/Anti-Virus Activity — displays the number of messages that werechecked for spam or viruses and the number of messages that were tagged as spam or deemed to contain a virus. The AS/AV count is increased by one per message scanned. One message sent to three people counts as only one message processed by AS/AV.
The Message Count and the Anti-spam/Anti-virus Activity graphs display a different message count because:
-
Outbound messages may not go through the Amavisd filter, as the system architecture might not require outbound messages to be checked.
-
Messages are received and checked by Amavisd for spam and viruses before being delivered to all recipients in the message. The message count shows the number of recipients who received messages.
-
Server-specific statistics also include the following details:
-
Disk — for a selected server displays the disk used and the disk space available. The information is displayed for the last hour, day, month, and year.
-
Session — displays information about the active Web client, administrator and IMAP sessions. You can see how many active sessions are opened, who is logged on, when the session was created and the last time the session was accessed.
-
Mailbox Quota — displays information about each account sorted by mailbox size in descending order. See Monitoring Mailbox Quotas.
The Logger generates a report about mail activity daily at 11:30 p.m. and sends it to the administrator’s email address.
You can configure the number of accounts to include in the report. The default is 25 sender and 25 recipient accounts.
-
Changing the number of recipients to add to the report:
zmlocalconfig -e zimbra_mtareport_max_recipients=<number>
-
Changing the number of senders to add to the report:
zmlocalconfig -e zimbra_mtareport_max_senders=<number>
You should regularly review your disk capacity and when disks are getting full, take preventative measures to maintain service. A warning alert email notification is sent to the administrator account when disk space is low. The default is to send a warning alert when the threshold reaches 85% and a critical alert when the threshold reaches 95%.
You can change these values. Use zmlocalconfig to configure the
disk warning thresholds.
-
Warning alerts
zmdisklog_warn_threshold
-
Critical alert:
zmdisklog_critical_threshold
When starting services with zmcontrol, if the threshold is exceeded a
warning is displayed before the services are started. You should clean up
your disk to free up space.
The {product-name} server collects many performance related statistics that can help you diagnose problems and load issues.
- Admin Console:
-
Home → Monitor → Advanced Statistics
The Advanced Statistics page includes advanced graphing options that lets you generate various charts based on statistical information for the CPU, IO, mailboxd, MTA queue, MariaDB and other components.
To chart the graphics in Advanced Statistics, select one of these groups and then select from the list of specific counters for the type of information to display.
The information covers a wide array of data:
-
cpu.csv — CPU utilization. This group contains counters to keep track ofCPU usage (iowait, idle, system, user, time etc.). CPU information can be tracked both at the server level and the process level.
-
df.csv — Captures disk usage. Disk utilization is tracked for each diskpartition.
-
fd.csv — file descriptor count. Keeps track of system file descriptor usageover time. This is primarily used to track down "out-of-file descriptor" errors.
-
mailboxd.csv — {product-name} server and JVM statistics. Mailboxdstores almost all of its statistics here. Interesting numbers to keep track of are heap_used, heap_free, imap_conn, soap_sessions, pop_conn, db_conn_count.
-
mtaqueue.csv — Postfix queue. This measures the mail queue size innumber of messages and the size in bytes.
-
proc.csv — Process statistics for Zimbra processes. For example mailboxd/java, MariaDB, OpenLDAP, etc.)
-
soap.csv — SOAP request processing time.
-
threads.csv — JVM thread counts. Counts the number of threads with acommon name prefix.
-
vm.csv — Linux VM statistics (from the vmstat command).
-
io-x.csv and io.csv — store data from the
iostat(1)command (io-x.csvwithiostat -x).
The denial-of-service filter (DoSFilter) limits exposure to requests flooding over HTTP/HTTPS. The DoSFilter throttles clients sending a large number of requests over a short period of time.
DosFilter is only applied to HTTP and HTTPS requests, in other words, it does not affect requests for any other protocols like POP3, IMAP or SMTP. You can modify the configuration to accommodate your specific environmental needs. DoSFilter is enabled by default on {product-abbrev}. Disabling the DoSFilter is not recommended. For information on preventing multiple failed login attempts see Password Policy
Sometimes Zimbra Connector for Outlook (ZCO), mobile ActiveSync clients, or
running some zmprov commands trigger the DoSFilter. When this happens, the
Zimbra mailbox service is unavailable. You can review the following logs
to see if the DoSFilter was applied.
-
/opt/zimbra/log/sync.log.
sync.log entry showing the DoSFilter2021-01-15 15:52:20,426 WARN [qtp1635701107-91:https://x.x.x.x/
Microsoft-Server-ActiveSync?User=zsupport2&DeviceId=Appl5ddddd3NR&DeviceType=iPhone&Cmd=FolderSync][[email protected];mid=64;ip=10.1.2.3;Cmd=FolderSync;DeviceID=Appl5K0113UN3NR;Version=12.1;] sync - Service exception com.zimbra.common.service.ServiceException: error while proxying request to target server: HTTP/1.1 503 Service Unavailable
ExceptionId:qtp1635701107-91:https://10.10.0.54:443/Microsoft-Server-ActiveSync?User=zsupport2&DeviceId=Appl5K0113UN3NR&DeviceType=iPhone&Cmd=FolderSync:1358286740426:c5ca7f36bb0a038f Code:service.PROXY_ERROR Arg:(url, STR,"http://mail.domain.com:80/service/soap/SyncRequest"-
/opt/zimbra/log/zmmailboxd.out
zmmailboxd.out entry showing the DoSFilter2021-01-15 15:57:32.537:WARN:oejs.DoSFilter:DOS ALERT:ip=127.0.1.1,session=null,user=nullThe following attributes are used with zmprov to configure the DoSFilter.
These attributes can be configured as global settings and as server
settings. If these attributes are set in the server, the server settings
override the global settings.
You can modify these settings, but the default configuration is recommended.
| Attribute | Description |
|---|---|
DoSFilter Delay |
The delay given to all requests over the rate limit before they are considered. The default is -1.
zmprov mcf zimbraHttpDosFilterDelayMillis {x} |
DoSFilter Maximum Requests Per Second |
The maximum number of requests from a connection per second. Requests in excess of this are throttled. The default is 30 and the minimum is 1. zmprov mcf zimbraHttpDosFilterMaxRequestsPerSec {x} |
DoSFilter IP Addresses Whitelist |
IP addresses to ignore when applying the DosFilter. This attribute does not have a default value, however the following loopback IPs are whitelisted by default.
The IP addresses should be comma separated. zmprov mcf zimbraHttpThrottleSafeIPs {addresses} |
A mailbox server restart is required after modifying these attributes. Type:
zmmailboxdctl restart-
{product-abbrev} Member Servers — {product-abbrev} servers under the control of a single masterLDAP server are automatically whitelisted by IP address. These hosts are discovered using a GetAllServersRequest. Type as
zmprov gas. -
External Provisioning Hosts/SOAP API — External provisioning hosts can be added to the IP whitelist to ensure that the DoSFilter does not block some requests. For example, a mailbox reindex might make several calls per second that can trigger the DoSFilter.
When the Zimbra MTA receives mail, it routes the mail through a series of queues to manage delivery; incoming, active, deferred, held, and corrupt.
The incoming message queue holds the new mail that has been received. Each message is identified with a unique file name. Messages are moved to the active queue when there is room. If there are no problems, message move through this queue very quickly.
The active message queue holds messages that are ready to be sent. The MTA sets a limit to the number of messages that can be in the active queue at any one time. From here, messages are moved to and from the anti-virus and anti-spam filters before being delivered to another queue.
Messages that cannot be delivered are placed in the deferred queue. The reasons for the delivery failures are documented in a file in the deferred queue. This queue is scanned frequently to resend the message. If the message cannot be sent after the set number of delivery attempts, the message fails. The message is bounced back to the original sender. The default for the bounce queue lifetime is five days.
The held message queue keeps mail that could not be processed. Messages stay in this queue until the administrator moves them. No periodic delivery attempts are made for messages in the held queue.
The corrupt queue stores damaged unreadable messages.
-
The MTA server’s bounce queue lifetime is set for five days. To change the default queue lifetime setting
zmlocalconfig -e bounce_queue_lifetime={#} -
To permanently have messages bounced back to the sender, instead of being sent to the deferred queue first
zmlocalconfig -e zimbraLmtpPermanentFailureWhenOverQuota=TRUE
Before the bounce queue lifetime sends the message back to the sender, senders can be notified that the message they sent is in the deferred queue and has not been delivered.
Configure the following attributes to send a warning message to the sender.
-
Configure the time after which the sender receives the message headers of email that is still queued.
zmlocalconfig -c postfix_delay_warning_time=0h
-
Configure the recipient of postmaster notifications with the message headers of mail that the MTA did not deliver.
zmlocalconfig -c postfix_bounce_notice_recipient=postmaster
-
Configure the list of error classes that are reported to the postmaster.
zmlocalconfig -c postfix_notify_classes=resource,software
|
Note
|
See Postfix documentation for details on the impact of changes to these Postfix attributes. |
You can monitor the mail queues for delivery problems from the Administration Console.
- Admin Console:
-
Home → Monitor → Mail Queues
If you are having problems with mail delivery, you can view the mail queues from the Mail Queues page in the Administration Console to see if you can fix the mail delivery problem. When you open mail queues, the content of the deferred, incoming, active, hold, and corrupt queues at that point in time can be viewed. You can view the number of messages and where they are coming from and going to.
For each queue, the Summary pane shows a summary of messages by receiver domain, origin IP, sender domain, receiver address, sender address, and for the deferred queue, by error type. You can select any of the summaries to see detailed envelope information by message in the Messages pane.
The Messages pane displays individual message envelope information for search filters selected from the Summary pane.
The following mailbox queue functions can be performed for all the messages in a queue:
-
Hold to select a set of messages that you want to hold. Incoming, active,deferred, and corrupt messages can be moved to the Held queue. Messages stay in this queue until the administrator moves them.
-
Release to remove all message from the Held queue. Messages are moved to the Deferred queue.
-
Requeue all messages in the queue being viewed. Requeuing messages can be used to send messages that were deferred because of a configuration problem that has been fixed. Messages are re-evaluated and earlier penalties are forgotten.
-
Delete all messages in the queue being viewed.
The Zimbra MTA, Postfix queue file IDs are reused. If you requeue or delete a message, note the message envelope information, not the queue ID. It is possible that when you refresh the mail queues, the queue ID could be used on a different message.
Mailbox quotas apply to email messages, attachments, calendar appointments, and tasks in a user’s account. When an account quota is reached, all mail messages are rejected. Users must delete mail from their account to get below their quota limit - this includes emptying their Trash, or you can increase their quota.
You can check mailbox quotas for individual accounts from Server Statistics on the Administration Console. Mailbox Quota gives you an instant view of the Mailbox Size and Quota Used information for each account.
- Admin Console:
-
Home → Monitor → Server Statistics
-
Select the server for which you want to view statistics.
-
In the Navigation pane, select Mailbox Quota. The Mailbox Quota page displays with the following information:
-
Quota column shows the mailbox quota allocated to the account. Quotas are configured either in the COS or by account.
-
Mailbox Size column shows the disk space used.
-
Quota Used column shows what percentage of quota is used.
-
From a COS or Account, you can configure a quota threshold that, when reached, sends a message alerting users that they are about to reach their mailbox quota.
- Admin Console:
-
Home → Configure → Class of Service → COS → Advanced
Home → Manage → Accounts → account → Advanced
-
Scroll down to the Quota section.
-
Modify the quota settings.
-
Click Save.
The MobileSync Statistics page in the Monitor section in the admin console displays the number of currently connected ActiveSync devices that are on the {product-name} system.
To protect against dictionary-based and distributed attacks, you can
configure the zmauditwatch. The script attempts to detect more advanced
attacks by looking at where the authentication failures are coming from and
how frequently they are happening for all accounts on a Zimbra mailbox
server and sends an email alert to the administrator’s mailbox.
The types of authentication failures checked include:
-
IP/Account hash check — The default is to send an email alert if 10authenticating failures from an IP/account combination occur within a 60 second window.
-
Account check — The default is to send an email alert if 15 authentication failures from any IP address occur within a 60 second window. This check attempts to detect a distributed hijack based attack on a single account.
-
IP check — The default is to send an email alert if 20 authentication failures to any account occur within a 60 second window. This check attempts to detect a single host based attack across multiple accounts.
-
Total authentication failure check — The default is to send an email alert if1000 auth failures from any IP address to any account occurs within 60 seconds. The default should be modified to be 1% of the active accounts on the mailbox server.
The default values that trigger an email alert are changed in the following
zmlocalconfig parameters:
-
IP/Account value, change
zimbra_swatch_ipacct_threshold -
Account check, change
zimbra_swatch_acct_threshold -
IP check, change
zimbra_swatch_ip_threshold -
Total authentication failure check, change
zimbra_swatch_total_threshold
Configure zimbra_swatch_notice_user with the email address that should
receive the alerts.
{product-name} logs its activities and errors to a combination of system logs through the syslog daemon as well as Zimbra specific logs on the local file system. The logs described below are the primary logs that are used for analysis and troubleshooting.
Local logs containing {product-name} activity are in the /opt/zimbra/log
directory.
-
audit.log — This log contains authentication activity of users and administrators and login failures. In addition, it logs admin activity to be able to track configuration changes.
-
clamd.log — This log contains activity from the anti-virus application clamd.
-
freshclam.log — This log contains log information related to the updating of the clamd virus definitions.
-
mailbox.log — This log is a mailboxd log4j server log containing the logs from the mailbox server. This includes the mailbox store, LMTP server, IMAP and POP servers, and Index server.
-
myslow.log — This slow query log consists of all SQL statements from the mailbox server that took more then
long_query_timeseconds to execute.Notelong_query_timeis defined in/opt/zimbra/conf/my.cnf. -
spamtrain.log — This log contains output from
zmtrainsaduring regularly scheduled executions from the cron. -
sync.log — This log contains information about {product-name} mobilesync operations.
Other logs include:
-
/opt/zimbra/jetty/logs/ — This is where Jetty-specific activity is logged.
-
/opt/zimbra/db/data/<hostname>.err — This is the message store database error log.
-
/opt/zimbra/logger/db/data/<hostname>.err — This is the Logger database error log.
{product-name} activity logged to System syslog
-
/var/log/zimbra.log — The Zimbra syslog details the activities of the ZimbraMTA (Postfix, amavisd, anti-spam, anti-virus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to
zimbra.log.
{product-name} modifies the systems syslog daemon to capture data from the mail and
local syslog facility to /var/log/zimbra.log. This allows syslogd to
capture data from several {product-name} components including
Postfix, Amavis, ClamAV, mailboxd, zmconfigd, and logger. The SNMP module
uses the data from the log file to generate traps for critical errors. The
zmlogger daemon also collects a subset of the data in this file to provide
statistics on the utilization of {product-name} via the
Administration Console.
By default, mailboxd is configured to log its output to
/opt/zimbra/log/mailbox.log. You can enable mailboxd to take advantage
of a centralized syslogd infrastructure by enabling the following either
globally or by server:
zmprov mcf zimbraLogToSysLog TRUEThe {product-name} server uses log4j, a Java logging package as the
log manager. By default, the {product-name} server has log4j
configured to log to the local file system. You can configure log4j to
direct output to another location. Go to the
Log4j website for information about
using log4j.
{product-abbrev} does not check the log4j changes. To remove all account loggers and
reloads in /opt/zimbra/conf/log4j.properties, use the zmprov
resetAllLoggers command.
The default logging level is set to include logs that are generated for INFO, WARNING, ERROR and FATAL. When problems start to occur, you can turn on the DEBUG or TRACE log levels.
To change the logging levels, edit the log4j properties,
log4j.properties, log4j.logger.zimbra.
When enabling DEBUG, you can specify a specific category to debug. For
example, to see debug details for POP activity, you would type
logger.zimbra.pop=DEBUG.
The following categories are predefined in log4j:
|
Account operations |
|
ACL operations |
|
Backup and restore |
|
Inmemory cache operations |
|
Calendar operations |
|
DAV operations |
|
Database connection tracing |
|
Server extension loading |
|
Mail filtering |
|
GAL operations |
|
IMAP protocol operations |
|
Index operations |
|
Filesystem operations |
|
LDAP operations |
|
LMTP operations (incoming mail) |
|
General mailbox operations |
|
Miscellaneous |
|
Changes to mailbox state |
|
POP protocol operations |
|
Redo log operations |
|
Security events |
|
User session tracking |
|
SMTP operations (outgoing mail) |
|
SOAP protocol |
|
SQL tracing |
|
Mail store disk operations |
|
Sync client operations |
|
Startup/shutdown and other system messages |
|
Wiki operations |
|
Zimlet operations |
|
Note
|
Changes to the log level take affect immediately. |
| Level | Local? | Syslog | SNMP Trap | When Used |
|---|---|---|---|---|
FATAL |
Y |
Y |
Y |
Designates very severe error events that the application to abort or impact a large number of users. For example, being unable to contact the MariaDB database. |
ERROR |
Y |
Y |
N |
Designates error events that might still allow the application to continue running or impact a single user. For example, a single mailbox having a corrupt index or being unable to delete a message from a mailbox. |
WARN |
Y |
N |
N |
Designates potentially harmful situations but are usually recoverable or can be ignored. For example, user log in failed. |
INFO |
Y |
N |
N |
Designates information messages that highlight the progress of the application, basic transaction-level logging. For example, server start-ups, mailbox creation/deletion, account creation. |
DEBUG |
Y |
N |
N |
Events that would generally be useful to help a customer debug problems. |
(*) A few non-critical messages such, as service startup messages, will generate traps.
Protocol trace is available in the following logging categories:
zimbra.smtp zimbra.lmtp zimbra.soap zimbra.imap zimbra.imap-client zimbra.pop zimbra.pop-client
The mailbox.log file contains every action taken on the mailbox server,
including authentication sessions, LMTP, POP3, and IMAP servers, and Index
server. Review the mailbox.log to find information about the health of
your server and to help identify problems.
mailbox.log records valid and invalid login attempts, account activity
such as opening email, deleting items, creating items, indexing of new
mail, server activities including start and stop. The progress of an
activity on the mail server is logged as INFO. If the expected results of
the activity fails and errors occurs, an exception is written to the log.
You can set up logging options for a single account in order to trace
account activity for one user without filling up mailbox.log with log
messages for unrelated accounts. See Command-Line
Utilities, the zmprov miscellaneous section.
Log pattern
by default log entries in mailbox.log have the following Log4j pattern:
%d %-5p [%t] [%z] %c{1} - %m%n
This pattern consists of 6 blocks of data:
-
Date and time (e.g.:
2021-01-22 19:23:07,100) -
Log level (e.g.
INFO) -
Thread name (e.g.
[qtp1043351526-547:https:https://localhost:7071/service/admin/soap/DeleteAccountRequest],[Index-9], etc.) -
{product-name} context
-
Component name (e.g.
soap,mailbox,mbxmgr, etc.) -
Log message. Note: the log message section may span multiple lines. When a log message contains an exception, the stack trace will always start on a new line below the error message.
You can read more about Log4j patterns in Log4j PatternLayout documentation.
Thread name in mailbox.log
Thread names in mailbox.log are prefixed to identify internal components. Most threads have one of the following naming convention: "{thread prefix}-{thread number}" or "{thread prefix}-{thread number}:{url}".
The following {thread prefix} values are currently used for thread names in {product-name}: btpool, pool, LmtpServer, ImapServer,ImapSSLServer, Pop3Server, Pop3SSLServer, ScheduledTask, Timer, AnonymousIoService, CloudRoutingReaderThread, GC, SocketAcceptor, Thread, qtp.
Threads with prefix qtp are created by Jetty QueuedThreadPool and have the following naming convention: "qtp{hash code}-{thread number}:{url}" where {hash code}` is the hash code value of the instance of QueuedThreadPool that owns the thread (see Object::hashCode in Java platform documentation).
{thread number} in thread names is an integer that monotonically increases within each thread factory. Thread numbers are reset when mailboxd process is stopped or restarted.
Log records reported by threads that serve SOAP requests will usually contain URL of the request being served in {url} part of thread name, as in the following example:
Due to a known bug in {product-name} {url} part of the thread name may contain duplicate protocol identifier, as in the following example:
[qtp1043351526-547:https:https://localhost:7071/service/admin/soap/DeleteAccountRequest]
{product-name} Context in mailbox.log
[%z] section in the log pattern describes {product-name} context and consists of key-value pairs in the format key=value, separated by semi-colons (;). In cases where a value contains a semi-colon, the semi-colon is replaced with a double semi-colon (;;). E.g., browser UserAgent strings often include semi-colons, such as this one "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36". In mailbox.log, this UserAgent string will appear as following:
The following key value pairs are currently supported and may be recorded in log entries in any order and any combination:
-
ip— IP of the TCP/IP client making a request -
oip— originating IP address. When a request is made through NGINX proxy, this value will contain the IP address of the client application, whileipvalue will contain the IP address of the proxy server -
cid— connection id of a server that is monotonically increasing - useful for tracking individual connections -
id— ID of the target account -
name— name of the target account (email address) -
aid— ID of the authenticated account. Only present if target account is different then authenticated account -
aname— name of the authenticated account. Only present if target account is different then authenticated account -
mid— ID of requested mailbox. Only present if request is dealing with a mailbox -
ua— name of the client application (i.e. User Agent) -
via— list of IP addresses and user-agents of the request’s proxy chain -
soapId— ID assigned to a SOAP request to track proxied hops for a particular request -
msgid— value of Message-ID header of the message being operated on -
ds— name of the Data Source being operated on -
port— server port to which the client connected -
oport— originating port number of request -
oproto— originating protocol of request. This can be passed by internal components that make SOAP requests on behalf of a user (e.g. MTA)
The example below is a record showing that
on October 25, 2021, 28 minutes after midnight, a POP3 client with IP address 222.173.186.17 has contacted the {product-name} server and that the request was proxied through a local proxy server with IP 10.1.1.136.
The following example shows a record of a failed IMAP STATUS request sent by [email protected] using AquaMail mobile app. The user’s device has IP address 72.83.144.255 (as reported in oip field). The request came to IMAP server via {product-name} nginx proxy, which has IP address 10.4.4.138 (as reported in ip and via fields).
The following example shows a record of LMTP server delivering a message. The IP address in this log message most likely belongs to {product-name} MTA running on local network.
The next example shows a record of MailboxPurge thread purging message with ID 462 from the mailbox of [email protected]. This log message does not have ip, oip, port or via fields, because it originates from an internal process rather than from an external request.
If an error occurs during the progress of an activity, a handler exception is added to the end of the log record to notify you that an event occurred during the execution of the process that disrupted the normal flow. This signals that some type of error was detected.
007-06-25 00:00:10,379 INFO [btpool0-1064] [[email protected];mid=228;ip=10.2.3.4;ua=zimbra Desktop/0.38;] SoapEngine - handler exceptionSometimes a stack trace is displayed after the exceptions notification. A stack trace reports the threads and monitors in Zimbra’s mailboxd service. This information aids in debugging, because the trace shows where the error occurred. The last few entries in the stack often indicate the origin of the problem. When the caused by descriptor is included in the log line, this is the root of the error. In the example below, the error was caused by 501, bad address syntax.
com.example.cs.mailbox.MailServiceException: Invalid address: Jon R
at com.example.cs.mailbox.MailServiceException.internal_SEND_FAILURE (MailServiceException.java:412)
at com.example.cs.mailbox.MailServiceException.SEND_ABORTED_ADDRESS_FAILURE MailServiceException.java:416)
...
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Caused by: com.example.cs.mailbox.MailSender$SafeSendFailedException: 501 Bad address syntax; chained exception is: com.sun.mail.smtp.SMTPAddressFailedException: 501 Bad address syntax
at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1196)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:584)
at javax.mail.Transport.send0(Transport.java:169)
at javax.mail.Transport.send(Transport.java:98)
at com.example.cs.mailbox.MailSender.sendMessage(MailSender.java:409)
at com.example.cs.mailbox.MailSender.sendMimeMessage(MailSender.java:262)
... 30 moreThe mailbox.log files rotate daily. The mailbox log files are saved in
/opt/zimbra/log. Previous mailbox.log file names include the date the
file was made. The log without a date is the current log file. You can
back up and remove these files.
To review the mailbox.log for errors, search for the email address or the
service that is experiencing the problem. Also, search for WARN or ERROR
log levels, read the text of the message. When you find the error, review
the records, tracing the events that happened before the problem was
recorded.
When your system crashes, locate the startup message and then look for errors before the startup message date. This example shows an out-of-memory error on June 17, 2021.
2021-06-25 01:56:18,725 INFO [main] [] soap - Servlet SoapServlet starting upLook for errors before the startup message.
2021-06-17 20:11:34,194 FATAL [btpool0-3335] [[email protected];[email protected];mid=142;ip=10.3.4.5;ua=zimbraConnectorForBES/5.0.207;] system - handler exception java.lang.OutOfMemoryError: PermGen spaceLocate the "LmtpServer" service. This example includes a stack trace report with a caused by explanation that the recipient address was rejected as the address must be a fully-qualified address.
2021-06-25 10:47:43,008 INFO [LmtpServer-250]
[[email protected];mid=30;msgid=<[email protected]>;] lmtp - rejecting message [email protected]: exception occurred
com.zimbra.cs.mailbox.MailServiceException: redirect to too failed
at com.zimbra.cs.mailbox.MailServiceException.internal_SEND_FAILURE (MailServiceException.java:412)
at com.zimbra.cs.mailbox.MailServiceException.SEND_FAILURE(MailServiceException.java:424)
at com.zimbra.cs.filter.zimbraMailAdapter.executeActions(zimbraMailAdapter.java:286)
at org.apache.jsieve.SieveFactory.evaluate(SieveFactory.java:151)
at com.zimbra.cs.filter.RuleManager.applyRules(RuleManager.java:177)
at com.zimbra.cs.lmtpserver.zimbraLmtpBackend.deliverMessageToLocalMailboxes(zimbraLmtpBackend.java:325)
at com.zimbra.cs.lmtpserver.zimbraLmtpBackend.deliver(zimbraLmtpBackend.java:140)
at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHandler.java:441)
at com.zimbra.cs.lmtpserver.LmtpHandler.processCommand(LmtpHandler.java:205)
at com.zimbra.cs.tcpserver.ProtocolHandler.processConnection(ProtocolHandler.java:231)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(ProtocolHandler.java:198)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
Caused by:
com.zimbra.cs.mailbox.MailSender$SafeSendFailedException: 504 <too>: Recipient address rejected: need fully-qualified address ;
chained exception is: com.sun.mail.smtp.SMTPAddressFailedException: 504 <too>: Recipient address rejected: need fully-qualified address
at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1196)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:584)
at javax.mail.Transport.send0(Transport.java:169)
at javax.mail.Transport.send(Transport.java:120)
at com.zimbra.cs.filter.zimbraMailAdapter.executeActions(zimbraMailAdapter.java:281)
... 10 moremailbox.log logs any successful or unsuccessful login attempts from IMAP,
POP3 or ZWC. When you are looking for a login error, start by looking for
"Auth." This example shows that someone from IP address 10.4.5.6 was
trying to log in as admin on the {product-short} {web-client}, using Firefox in a
Windows OS. Permission was denied because it was not an admin account.
2021-06-25 09:16:11,483 INFO [btpool0-251] [ip=10.4.5.6;ua=zimbraWebClient - FFX.X (Win);] SoapEngine - handler exception
com.zimbra.common.service.ServiceException: permission denied: not an admin account
at com.zimbra.common.service.ServiceException.PERM_DENIED(ServiceException.java:205)
at com.zimbra.cs.service.admin.Auth.handle(Auth.java:103)When you are looking for a log because of an IMAP or POP issue, look for "ImapServer/Pop3Server." This example shows a fatal IMAP server error occurred while trying to connect [email protected].
mailbox.log.2021-06-19:2021-06-19 15:33:56,832 FATAL [ImapServer-2444] [[email protected];ip=127.0.0.1;] system - Fatal error occurred while handling connectionEach email message includes a header that shows the path of an email from its origin to destination. This information is used to trace a message’s route when there is a problem with the message. The Zimbra email message header can be viewed from the {product-short} {web-client} Message view. Right-click on a message and select Show Original.
The following lines are in the message header:
-
Date — The date and time the message was sent. When you specify time, you can specify range by adding start and stop time to search for messages.
-
From — The name of the sender and the email address
-
To — The name of the recipient and the email address. Indicates primary recipients.
-
Message-ID — Unique number used for tracing mail routing
-
In-Reply-To — Message ID of the message that is a reply to. Used to link related messages together.
-
Received: from — The name and IP address the message was sent from. The header displays Received: from information from the MTA to the LMTP and from the local host.
Mail messages and attachments are automatically indexed before messages are deposited in a mailbox. Each mailbox has an index file associated with it. This index file is required to retrieve search results from the mailbox.
If a mailbox’s index file becomes corrupt or is accidentally deleted, you can re-index the messages in the mailbox from the Administration Console.
Text searches on an account might or might not fail with errors when the index is corrupt. You cannot count on a user reporting a failed text search to identify that the index is corrupt. You must monitor the index log for messages about corrupt indexes. If the server detects a corrupt index, a message is logged to the Zimbra mailbox.log at the WARN logging level. The message starts with Possibly corrupt index. When this message is displayed, the administrator must correct the problem. In many cases correcting the problem might mean reindexing the mailbox.
Reindexing a mailbox’s content can take some time, depending on the number of messages in the mailbox. Users can still access their mailbox while reindexing is running, but because searches cannot return results for messages that are not indexed, searches may not find all results.
Run a sanity check on a specific mailbox index using the zmprov
verifyIndex command.
zmprov verifyIndex <[email protected]>If problems are detected, a failure status is returned and a repair can be performed on the index.
Use the reIndexMailbox command to repair and reindex a corrupt index.
zmprov reIndexMailbox <[email protected]> startThis returns a status of started.
You will probably want to implement server monitoring software in order to monitor system logs, CPU and disk usage, and other runtime information.
{product-name} uses swatch to watch the syslog output to generate SNMP traps.
{product-name} includes an installer package with SNMP monitoring. This package should be run on every server ({product-name}, OpenLDAP, and Postfix) that is part of the {product-name} configuration.
The only SNMP configuration is the destination host to which traps should be sent.
The MariaDB database is automatically checked weekly to verify the health of the database. This check takes about an hour. If any errors are found, a report is sent to the administrator’s account. The report name that runs the MariaDB check is zmbintegrityreport, and the crontab is automatically configured to run this report once a week.
|
Note
|
When the MariaDB database is checked, running this report can consume a significant amount of I/O. This should not present a problem, but if you find that running this report does affect your operation, you can change the frequency with which zmbintegrityreport is run. See {product-abbrev} Crontab Jobs. |
When {product-name} is installed, the {product-name} software update utility is automatically configured to check for the latest {product-name} version once a day and if there is an update, to send notification to the address that is configured in the Administration Console’s Server Updates.
The dates and times {product-name} checked for updates is saved to the Updates tab and an email notification is sent out until you update the {product-abbrev} version. If you do not want to receive an email notification of updates, disable Send notification email when updates are available.
You can configure the following:
-
Server that checks for updates — Available servers are listed and only one server is configured. The selected server checks for updates and the result of the update response from www.zimbra.com is stored in LDAP.
-
Check for updates every x — The default is to check once a day. You can change the frequency interval to check every x hours, minutes, or seconds. A cron job is configured to check for new updates. If the frequency interval is less than 2 hours, the crontab file must be modified.
-
Updates URL — This address is the URL that the server connects to when checking for updates. When a {product-name} server checks for updates, it transmits its version, platform, and build number to Zimbra. Normally, this URL is not changed.
-
To be notified of updates, check the Send notification email when updates are available and enter the send to and send from addresses. The default address is the administrator’s address.
-
A generic email is created. The subject and content of the email can be changed.
-
When a server polls the URL specified, the response is displayed.
The Zimbra Connector for Microsoft Outlook (ZCO) msi file is available from
the Zimbra Utilities Downloads page on the Administration Console. When a
newer version of ZCO is released before a new version of {product-abbrev}, you can
upload the newer ZCO msi file to the {product-abbrev} server from the Administration
Console. The file is uploaded to the
/opt/zimbra/jetty/webapps/zimbra/downloads folder.
- Admin Console:
-
Home → Tools and Migration → Client Upload
-
Download the new ZCO file to a computer that you can access from Client Upload in the Administration Console
-
Click Browse to locate the ZCO file to upload.
-
Restart {product-abbrev}:
zmcontrol restart
or run
/opt/zimbra/libexec/zmupdatedownload
-
The downloads/index.html file is updated with the latest ZCO client
version. This new file can be downloaded from the ZCO link on the
Administration Console Home → Tools and Migration → Download page.
|
Note
|
If you do not restart the server, the ZCO download link on the Zimbra Utilities Download page does not select the newer version to download. |
This notification is sent when service are stopped or restarted.
Subject: Service <service_name> started on <zimbra_host> Service status change: <zimbra_host> <service> changed from stopped to running
Subject: Service <service_name> stopped on <zimbra_host> Service status change: <zimbra_host> <service> changed from running to stopped
A warning alert email notification is sent to the admin account when disk space is low. The default is to send a warning alert when the threshold reaches 85% and a critical alert when the threshold reaches 95%
Subject: Disk <volume> at ##% on <zimbra_host> Disk warning: <zimbra_host> <volume> on device <device_name> at ##%
A script is executed to see if mysqld process is running to detect cases where corruption is likely to be caused. An email is generated if it finds more than 1 mysqld process running.
Subject: ZCS: Duplicate mysqld processes detected! PID:$pid PPID:$ppid PGRP:$pgrp CMD: $cmdline More then $maxcnt mysqld processes are running Parent processes include: $procs This should be investigated immediately as it may lead to database corruption
A report runs on the first of each month and warns of certificates expiring with the next 30 days.
Subject: ZCS: SSL Certificates approaching expiration! The Administration Console and CLI Certificate Tools guide provides instructions on how to replace you self-signed or commercial certificate. https://wiki.zimbra.com/index.php?title=Administration_Console_and_CLI_Certificate_Tools SSL Certificate expiration checked with $0 on <zimbra_host>.
When the logger package is installed, a daily mail report is automatically scheduled in the crontab. The report is sent daily to the administrator’s mailbox.
Subject: Daily mail report for <day> <daily report data>
The MariaDB database can be checked by running the zmdbintegrityreport automatically scheduled in the crontab to run on a weekly basis. A report is sent to the administrator’s mailbox.
Subject: Database Integrity check report for <zimbra_host> Generating report can't run $cmd: $! Database errors found. $cmd --password=XXXXXXXX <cmd output> No errors found command failed $!
When configuring the type of backups that should be run, you can set up to receive notification about the results of a backup session.
Subject: ZCS BackupReport:SUCCESS Server: <server> Type: incremental Status: completed Started: Fri, 2021/07/13 01:00:05.488 PDT Ended: Fri, 2021/07/13 01:10:09.842 PDT Redo log sequence range: 2 .. 2 Number of accounts: 500