TaskClone Service Downtime
Update 12/16 9:00am
We've now addressed the speed issues for both Evernote and OneNote and processing times should be much improved. Evernote Business should be working normally. There were two remaining issues observed with OneNote which have now been isolated and being resolved.
- For OneNote Business, Microsoft has outdated documentation published for how to renew subscriptions for notifications so our attempts to continue to receive notifications of OneNote changes are failing. We discovered this issue is impacting many businesses and believe we've found an alternative method which we intend to apply tomorrow.
- The OneNote API is adding extra characters to some Trigger Tags causing them not to recognized by our service. We're still isolating the circumstances when this occurs so it can be resolved quickly. We expect both of these issues to be resolved this week, but isolating these things can be tricky if it varies by OneNote app, Operating system and tag placement.
Update 12/10 12:00pm PST
After a false start yesterday, we've finally been able to address the speed issues on the Evernote side of things and processing times are even better than expected in many cases. We are seeing 10-30 seconds in most cases, but we may see it settle in to closer to 1 min as we add OneNote, TaskCam and others to the same process. We will be applying the same solutions to OneNote and TaskCam over the next day or so. We are still aware of some remaining issues for Evernote Business, which will now be easier to address without the delay of the speed issues.
Update 12/4 1:30pm PST
Although it has been a long time since our last update, we've been hard at work to resolve the issues. The memory issues has been resolved and the service has been stable, but slow for several days. It took time to verify this which is why there was no message sooner. To address the memory issues required we purchase and install additional diagnostic software to isolate and then address all code potentially causing an issue. An additional benefit of this diagnostic software is the ability to further optimize our code for speed and stability after this initial crisis is over.
As mentioned previously, we added a layer of redundancy hosted by Microsoft to ensure no notifications were lost in the event of a server failure or other downtime. Unfortunately, that layer has not functioned properly, causing significant slowness when under our regular load and generating huge storage logs/bills for us. We've had to hire a separate expert in these Microsoft technologies and no have identified a resolution that will be implemented over the next few days to ensure proper testing. We expect by Monday, December 9th services will be back to their snappy normal speeds.
As has been the case since this began, our support team has shortened our normal response times to provide updates and address questions.
Update 11/27 - 11:30am PST
In the process of restoring services, we have identified a memory issue which varies in impact based on usage. This is causing intermittent interruption of processes which can impact all services in what appears a random basis. We are working hard to identify the source of this memory issue and restore consistent service. Until that time, we are getting reports from some that things are fine, some report delay and others report no service. This is extremely frustrating and we are taking it with the seriousness it deserves. We will post further as we seek a resolution and take necessary steps to ensure this never happens again.
Update 11/20 - 11:30pm PST
While basic service to clone individual tasks are restored, cloning notes and a few other features are still experiencing difficulty. We hope to update here soon.
Update 11/20 - 1:20pm PST
While we are still experiencing an issue the service that processes notifications, we've implemented a workaround to ensure service is restored to our clients.
We will still need to implement a permanent solution over the next days. We will continue to add updates here.
Notice 11/20 - 9:00am PST
We regrettably have to report that the the new server environment that is processing notifications from Evernote, OneNote, TaskCam is down. We are therefore not getting notifications to process. We are working with vendor.
For OneNote, you can manually initiate cloning from the Sources section of our dashboard as our underlying service is working fine. We never faced such an issue with Evernote, so don't have a manual process yet available for Evernote.
We will update here when there is more information.