I tried setting the timout to 10 sec and then spawning a separate thread if that connect failed and ran into oddities where the socket connect succeeded but then seemed to time out in later operations on the same send. Not sure there is a sweet spot between waiting long enough to ensure connection works and not so long as to throw timestamps significantly off if you have a lot of hung notifications. Timestamping the event outside of the blocking notification/send loop would fix log timestamps, but events still wouldn't appear in the log file until blocking notifications before them completed. An "ideal" approach might be to spin up a separate thread for each notification mechanism and que notifications for that mechanism. A specific notifier thread would process things in its queue FIFO without being blocked by other notifiers that may have problems (googling came across Queue.queue() to explore communication between threads).
My first approach has been to write some code for the SMTP sender that immediately puts the SMTP notification in its own thread. This fixes the (my) immediate problem for SMTP blocking and log entries. I've tested by spinning up over 250 notification threads while blocking and that didn't seem to phase AD operations in any way. In real life I don't ever expect to get that many pending notifications queued up. I'll post the code over in the code contributions section (http://www.alarmdecoder.com/forums/viewtopic.php?f=6&t=767), but this is enough of a hack - only fixing the SMTP notifier - that I don't think it belongs in a pull request.
