SMTP blocking and innacurate log timestamps fix/hack
Posted: Wed Mar 21, 2018 7:20 am
As discussed in this thread http://www.alarmdecoder.com/forums/viewtopic.php?f=3&t=755, if a SMTP service is down for some reason, SMTP notifiers can temporarily block recording of events in the AD log, throw off their timestamps, and also block other notification vectors. As a quick and inelegant solution to the problem, I modified the EmailNotification class in the /opt/alarmdecoder-webapp/ad2web/notifications/types.py module to spin off a separate thread to handle the sending whenever an SMTP notification is needed. This prevents blocked SMTP from delaying the processing of other notifications (but does nothing to solve the same problem if other notification vectors block).
You can adjust the variables retryinterval_sec (socket timeout on a send attempt) and finaltimeout_sec (how long to keep attempting to send) as you see fit. If a send finally times out, or otherwise fails, it will post the error and some related info in the App Log. As noted in the other thread, I've tested this with over 250 notifications spun up in threads this way without any apparent adverse impact on AD performance (I did not release the 250 threads for sending because I wasn't sure how my ISP would take that many nearly simultaneous connection requests from one IP). Testing with 3-4 blocked notifications (using iptables to block/unblock) and releasing to send before the retry window closed works fine.
I'm not submitting this as a pull request to the developers because it doesn't address the issue in a particularly clean way, or for all notifiers. Also, this disclaimer: "I am not a programmer".
You can adjust the variables retryinterval_sec (socket timeout on a send attempt) and finaltimeout_sec (how long to keep attempting to send) as you see fit. If a send finally times out, or otherwise fails, it will post the error and some related info in the App Log. As noted in the other thread, I've tested this with over 250 notifications spun up in threads this way without any apparent adverse impact on AD performance (I did not release the 250 threads for sending because I wasn't sure how my ISP would take that many nearly simultaneous connection requests from one IP). Testing with 3-4 blocked notifications (using iptables to block/unblock) and releasing to send before the retry window closed works fine.
I'm not submitting this as a pull request to the developers because it doesn't address the issue in a particularly clean way, or for all notifiers. Also, this disclaimer: "I am not a programmer".
- Code: Select all
class EmailNotification(BaseNotification):
def __init__(self, obj):
BaseNotification.__init__(self, obj)
self.source = obj.get_setting('source')
self.destination = obj.get_setting('destination')
self.subject = obj.get_setting('subject')
self.server = obj.get_setting('server')
self.port = obj.get_setting('port', default=25)
self.tls = obj.get_setting('tls', default=False)
self.ssl = obj.get_setting('ssl', default=False)
self.authentication_required = obj.get_setting('authentication_required', default=False)
self.username = obj.get_setting('username')
self.password = obj.get_setting('password')
self.suppress_timestamp = obj.get_setting('suppress_timestamp',default=False)
def send(self, type, text):
message_timestamp = time.ctime(time.time())
if self.suppress_timestamp == False:
text = text + "\r\n\r\nMessage sent at " + message_timestamp + "."
if check_time_restriction(self.starttime, self.endtime):
msg = MIMEText(text)
if self.suppress_timestamp == False:
self.subject = self.subject + " (" + message_timestamp + ")"
msg['Subject'] = self.subject
msg['From'] = self.source
recipients = re.split('\s*;\s*|\s*,\s*', self.destination)
msg['To'] = ', '.join(recipients)
sself=self
t=threading.Thread(target=self.sendthread,args=(sself,current_app._get_current_object(),recipients,msg,message_timestamp,))
t.start()
def sendthread(self, sself, caller_app, recipients, msg, message_timestamp):
s = None
retryinterval_sec = 60*1 #set as you see fit
finaltimeout_sec = retryinterval_sec * 5 #set as you see fit
timeout_exception = False
other_exception = False
numretries = int(finaltimeout_sec/retryinterval_sec)
for x in range(numretries):
timeout_exception = False
other_exception = False
# Since AD won't catch any exceptions in this separate thread, put everything
# important in the try loop and catch so a warning can be sent to the AD log file.
try:
if sself.ssl:
s = smtplib.SMTP_SSL(sself.server, sself.port, timeout=retryinterval_sec)
else:
s = smtplib.SMTP(sself.server, sself.port, timeout=retryinterval_sec)
if sself.tls and not sself.ssl:
s.starttls()
if sself.authentication_required:
s.login(str(sself.username), str(sself.password))
s.sendmail(sself.source, recipients, msg.as_string())
except socket.timeout:
errinfo = sys.exc_info()
timeout_exception = True
except:
errinfo = sys.exc_info()
other_exception = True
if (other_exception is True) or (timeout_exception is False):
break
if (timeout_exception is True) or (other_exception is True):
caller_app.logger.error('Notification Email send failed: '
+ msg['Subject'] + ' '
+ message_timestamp + ' ('
+ str(errinfo[0]) + '; '
+ str(errinfo[1]) + ')')
s.quit()