MHC Outage

Unplanned Connectivity Outage: B50

Mon, 06 Feb 2012 09:12:00 -0800

9:00: We have brought a temporary server up to allow internet connectivity. We are working to repair the existing server.

8:45: we noticed that DHCP addressing was not being handed out to clients.

Current affected services:

- Internet browsing

- Wi-Fi access on Public / LAN access

Current unaffected services:

- Phones

- Static services

Unplanned Outage: Sammamish Internet

Mon, 30 Jan 2012 15:06:47 -0800

1/29/2012 6:30am: We are aware of an internet issue affecting the Sammamish location. We are working with our service providers to troubleshoot this issue.

Unplanned Outage: West Seattle Internet

Sun, 29 Jan 2012 20:14:00 -0800

10:09 pm: This issue has been resolved.

1/29/2012 8:14pm: We are aware of an issue with West Seattle’s internet. We are working with our service provider to resolve this issue.

Unplanned Outage: Central Servers

Thu, 26 Jan 2012 11:03:00 -0800

12:00pm: This issue has been resolved.

At approximately 11:30am, we will be performing needed maintenance on a central server. This maintenance is being performed during business hours due to its time-sensitive nature.

Between 11:30am and 12:00pm, the following services may be temporarily unavailable:

Phones (your phone will likely restart)
Wireless LAN (you may need to turn your Airport off, then on again)
Communication between sites (connectivity will automatically be restored and tested)
2x shared applications (please save your work and exit the program by 11:30AM)

Thank you for your understanding and patience. This post will be updated as we work through this.

Unplanned Outage: Rainier Valley Internet

Fri, 20 Jan 2012 11:41:00 -0800

1/22/2012 8:45am: This issue has been resolved.

1/19/2012 1:45pm: The Rainier Valley church is currently experiencing an internet outage. We are working with church staff and vendors to resolve this issue.

Power Related Internet Outage

Fri, 20 Jan 2012 00:26:40 -0800

8:35 AM : The Sammamish church primary Internet connection has gone down. Likely cause is weather related power outage.

12:36 AM (Jan 20th) : Site is operational

Testing New Features

Thu, 19 Jan 2012 23:59:51 -0800

Added a support tab to the sidebar of this page. Filling out the form currently does not work.

Unplanned site outage

Thu, 19 Jan 2012 12:42:00 -0800

12:40 PM : unplanned outage at our OC location. Currently investigating.

12:43 PM : service restored.

Power Related Internet Outage

Wed, 18 Jan 2012 21:00:00 -0800

8:50 PM : Shoreline church primary and secondary Internet connections are down. Likely cause is weather related power outage.

12:53 AM : Shoreline church primary and secondary Internet connections have been restored.

Shared Applications Restart

Tue, 17 Jan 2012 15:44:38 -0800

Our team has restarted the server hosting 2x and applications using 2x.

Outage Site Updates

Thu, 05 Jan 2012 10:21:18 -0800

We have changed our Twitter account to post information much faster than before. Updates will be posted immediately (assuming Twitter is up). Twitter updates will now include information from our server monitors when email goes down and up.

Unplanned Email Outage

Wed, 04 Jan 2012 16:00:00 -0800

4pm - We are currently working with our email provider to resolve an unplanned server outage.

Affected services:

- Webmail

- ActiveSync programs (Outlook, Mail.app, etc)

We will update this as we have additional details.

5:30pm - service restored. All systems functional

Voicemail Delivery Delays

Mon, 19 Dec 2011 14:59:00 -0800

12/22 1:00pm: This issue has been resolved. Past voicemails will begin being delivered to users. It may take some time for all voicemails to be delivered.

12/22 9:30am: We are working with our vendor to repair this issue. During this time, voicemail service may be completely unavailable. Voicemails will be unable to be recorded and will not be queued for later delivery. We apologize for the inconvenience.

12/20 9:00am: We are aware of issues causing delays in voicemail deliveries and are working toward a solution. We are working with our vendors to work towards resolution.

Voicemails are being recorded and will be delivered at a later time.

Unplanned Email and Web Outage

Wed, 07 Dec 2011 20:49:00 -0800

7:00pm: A power outage is affecting our web and email servers.

Technicians with our provider are working to resolve the issue.

9:30pm: power has been restored. Still working cause and determining losses.

Followup: Issue was traced to power loss at one of our sites. Power has been restored and everything is working.

Planned Email Service Maintenance

Thu, 17 Nov 2011 14:59:39 -0800

On 18 November 2011 from 8pm to 9pm PDT our email environment will be restarted and reconfigured to ensure that spam and email are properly routed to our new email server and that everything has properly been moved over to our new server. During this time, all incoming email will be cached and outgoing email may not be able to be sent. Once the maintenance has finished, cached email will be delivered and you will be able to send email.

We will update this post with updates as needed.

Email Connection Problems

Fri, 11 Nov 2011 18:42:00 -0800

On November 9th our mail provider started the first of several steps to migrate our Exchange server from Exchange 2007 to 2010.

Despite testing, there were some connection problems with some clients. The problems stem from client connection problems that we are working to fix. When email clients first start, they look for configuration settings and download them from the email server. Some email programs work better than others at this. This is why there have been some connection problems.

If you are experiencing problems connecting to your email:

1. Use the VPN to connect. The client is already pre-installed on Mac computers. It’s called Network Connect and the URL to use is https://vpn.marshillchurch.org. Login with Mars Hill user name and password.

2. Use webmail. You can access webmail at: https://mail.marshillchurch.org.

Affected clients (these clients most likely won’t be able to properly connect to email unless on VPN):

Outlook for Mac 2011
Entourage 2008
Android phones

Mostly unaffected clients (in most cases, users with Outlook for Mac connection problems are able to get mail using the following clients):

Webmail
Outlook for Windows
Windows Phone 7
Most mail.app clients
Most iPhone users

We will be updating this post as we work to resolve these issues. The latest ETA from our mail provider is that this will be fixed by 14 November 2011. We are regularly checking in with them as things change and will update this site accordingly.

Update 12 Nov 2011: All users in the staff group will be moved from the Exchange 2007 server to the Exchange 2010 server over the weekend. Please continue to use webmail if there are any problems. Our mail provider also found that part of the problem was a problem with a URL redirect and should have this fixed by 14 November 2011 as well.

Update 13 Nov 2011 Number 1: Working on speeding up the mailbox move. More details as we have them.

Update 13 Nov 2011 Number 2: Most users have been moved over to Exchange 2010. This should be finished by 14 November.

Update 13 Nov 2011 Number 3: Currently, some Outlook for Mac clients work, and some don’t. We are researching specific settings changes to make affected clients work. In the meantime, please continue to use WebMail.

Update 14 November 2011: We have completed moving staff and volunteer mailboxes from our old to new servers. Our mail provider is still working on adjusting some security settings which should be done by 15 November. Please continue to use webmail / VPN to connect to email. If your mobile device is not working, remove and readd the account. Send an email to helpdesk@marshill.com if this does not fix the problem.

Update 15 November 2011: All mailboxes have been moved over to the new server. Our mail provider is still adjusting security service settings. All email programs should work, but some mobile devices may not be able to get email still because of this. Double check the server addresses that you have in your mail clients and change them if needed. Here is what to do:

Outlook for Mac: Open Outlook for Mac and click on Tools > Accounts. Click on the account to change and click on Advanced. Make sure the settings look like this screenshot: http://cl.ly/023f331Y0I232R3R263F
Mail.app: Open mail.app and click on Mail > Preferences > Accounts. Make sure the settings look like this screenshot: http://cl.ly/3B1M0v2y3U0M162A0y0u

Outlook for Windows does not need to be updated. When making any changes, please only change the server address. If you have questions, please email helpdesk@marshill.com and refer back here for more updates.

Update 16 November 2011: All clients should be in a working state. If mobile devices are not working, remove and readd the accounts. Send an email to helpdesk if there are any problems. There will be one more update tomorrow to resolve a few outstanding issues.

Update 18 November 2011: All systems go.

Unplanned Outage: Ballard Internet

Fri, 07 Oct 2011 12:39:00 -0700

4:15pm: This issue has been resolved.

1:30pm: We received word from our service provider that this issue is a larger network outage. They are working on resolving the issue, but don’t have any timeframe on a resolution at this time.

12:10pm: We are currently experiencing an outage at our Ballard campus. We are working with our service provider to resolve this issue.

Unplanned Outage: Albuquerque Internet

Thu, 06 Oct 2011 13:54:00 -0700

10/7 8:45am: This issue has been resolved.

1:35pm: We’re experiencing issues with the internet connection between Albuquerque and Seattle. While normal internet access is available, services such as phones, direct access to servers at other sites, and email may not be available. We have opened a ticket with our service provider to get resolution.

Unplanned Outage: 2x Application Access

Mon, 03 Oct 2011 11:09:01 -0700

11:08am: This issue has been resolved.

10am: We have discovered an issue with application access via 2x. We are currently working on resolving this issue.

Planned Maintenance on Helpdesk Server

Fri, 30 Sep 2011 17:29:00 -0700

From 5:30pm to 6pm our team will be performing maintenance on the server. Please do not send emails to the helpdesk email address during this time. If you sent an email to helpdesk and did not get a confirmation that it was submitted to the queue today, please resend it.

5:30pm: started uploading update to server

5:31pm: uploading 5.3.45496 update

5:34pm: update complete

5:42pm: uploading 5.3.45497 update

5:47pm: update complete

5:50pm: system back online

All systems normal