SMTP blocking / log timestamps

Contribute source code here. Anything not accompanied by source code will be removed.

SMTP blocking / log timestamps

Postby trialnerror » Wed Mar 21, 2018 7:05 am

As discussed in this thread http://www.alarmdecoder.com/forums/viewtopic.php?f=3&t=755, if a SMTP service is down for some reason, SMTP notifiers can block recording of events in the AD log, throw off their timestamps, and also block other notification vectors. As a quick and inelegant solution to the problem, I modified the EmailNotification class in the /opt/alarmdecoder-webapp/ad2web/notifications/types.py module to spin off a separate thread to handle the sending whenever an SMTP notification is needed. This prevents blocked SMTP from delaying the processing of other notifications (but does nothing to solve the same problem if other notification vectors block).

You can adjust the variables retryinterval_sec (socket timeout on a send attempt) and finaltimeout_sec (how long to keep attempting to send) as you see fit. If a send finally times out, it will post the error in the App Log.

I'm not submitting this as a pull request to the developers because it doesn't address the issue in a particularly clean way, or for all notifiers. After posting I'm going to alter my sig to appropriately disclaim "I am not a programmer".

Code: Select all
class EmailNotification(BaseNotification):
    def __init__(self, obj):
        BaseNotification.__init__(self, obj)

        self.source = obj.get_setting('source')
        self.destination = obj.get_setting('destination')
        self.subject = obj.get_setting('subject')
        self.server = obj.get_setting('server')
        self.port = obj.get_setting('port', default=25)
        self.tls = obj.get_setting('tls', default=False)
        self.ssl = obj.get_setting('ssl', default=False)
        self.authentication_required = obj.get_setting('authentication_required', default=False)
        self.username = obj.get_setting('username')
        self.password = obj.get_setting('password')
        self.suppress_timestamp = obj.get_setting('suppress_timestamp',default=False)

    def send(self, type, text):
        message_timestamp = time.ctime(time.time())
        message_timestamp = " [" + datetime.datetime.now().strftime("%I:%M:%S%p %Z %a %m/%d/%y") + "]"  #bwm

        if self.suppress_timestamp == False:
            text = text + "\r\n\r\nMessage sent at " + message_timestamp + "."

        if check_time_restriction(self.starttime, self.endtime):
            msg = MIMEText(text)

            if self.suppress_timestamp == False:
                self.subject = self.subject + " (" + message_timestamp + ")"

            msg['Subject'] = self.subject
            msg['From'] = self.source
            recipients = re.split('\s*;\s*|\s*,\s*', self.destination)
            msg['To'] = ', '.join(recipients)

            sself=self
            t=threading.Thread(target=self.sendthread,args=(sself,current_app._get_current_object(),recipients,msg,message_timestamp,))
            t.start()


    def sendthread(self, sself, caller_app, recipients, msg, message_timestamp):
        s = None
        retryinterval_sec = 60*1                    #set as you see fit
        finaltimeout_sec = retryinterval_sec * 5    #set as you see fit
        timeout_exception = False
        other_exception = False

        numretries = int(finaltimeout_sec/retryinterval_sec)

        for x in range(numretries):
            timeout_exception = False
            other_exception = False

            # Since AD won't catch any exceptions in this separate thread, put everything
            # important in the try loop and catch so a warning can be sent to the AD log file.
            try:
                if sself.ssl:
                    s = smtplib.SMTP_SSL(sself.server, sself.port, timeout=retryinterval_sec)

                else:
                    s = smtplib.SMTP(sself.server, sself.port, timeout=retryinterval_sec)

                if sself.tls and not sself.ssl:
                    s.starttls()

                if sself.authentication_required:
                    s.login(str(sself.username), str(sself.password))

                s.sendmail(sself.source, recipients, msg.as_string())

            except socket.timeout:
                errinfo = sys.exc_info()
                timeout_exception = True

            except:
                errinfo = sys.exc_info()
                other_exception = True

            if (other_exception is True) or (timeout_exception is False):
                break

        if (timeout_exception is True) or (other_exception is True):
            caller_app.logger.error('Notification Email send failed: '
                                    + msg['Subject'] + ' '
                                    + message_timestamp + ' ('
                                    + str(errinfo[0]) + '; '
                                    + str(errinfo[1]) + ')')
        s.quit()
trialnerror
Junior Nut
Junior Nut
 
Posts: 38
Joined: Wed Jan 03, 2018 11:10 am

Return to Code Contributions

Who is online

Users browsing this forum: No registered users and 8 guests