Sunday, September 4, 2016

Migration Complete!

...and now it's time for victory beers.
It is presently 18:45 CDT, and I am happy to be able to issue a general "all clear" on the outage issues that started yesterday and the migration/downtime that happened today: as of right now, normal functionality has been restored and most users should see immediate resolution to the 504/502 and other timeout issues that have been occurring since yesterday afternoon.

Hit the jump for some additional details and thoughts.

It took some doing, but during the last eight and a half hours, I managed to:

  • Refactor the deployment and init code for the Manager to include a "run mode" controller in the init_script file
  • Put the old production deployment into "downtime" mode
  • Move all user data to a new Google Cloud Platform VPS
  • Deploy a new instance of the Manager on the new VPS
  • Manually migrate/copy all the DNS configs from Linode's nameservers to Google's

At this point, the DNS update seems mostly to have propagated and three users have logged into the new instance and started managing campaigns, so I think the maintenance work is officially a done deal, for now.

(Though I will be monitoring it through the night and tomorrow: the second you think you've got this sysadmin stuff handled, that's the exact moment when it starts your hair on fire).

Also, The new provision seems to perform much better than the old one. And, speaking of the old provision, according to, issues from yesterday's DDOS attack persist:
Update - Some users may be experiencing delayed DNS resolution in Atlanta due to filtering that has been put in place. We are currently working to reconfigure where necessary and restore normal DNS resolution. Sep 4, 20:13 UTC
As of Sun Sep  4 19:43:16 EDT 2016 the old production instance of is running in downtime mode, so until DNS fully propagates (i.e. while some name requests are processed through Linode's nameservers rather than Google's), some users may be directed to the static "maintenance" page. This should go away pretty soon.

I still have a lot of applications and resources living on my VPS in the Atlanta Linode data center and I will probably start migrating the "mission critical" ones over to Google this week.

Nevertheless, seems like it's in a good place now, so I'm going to go ahead and call it a wrap.

Thanks again for using the Manager!

No comments:

Post a Comment