Migration update - moving to Ireland

Dear friends of wikiloops,

Todays post is a follow-up to the February news, in which I informed about the need to migrate the wikiloops project to a more robust server setup after another heavy dDos-attack took the project down in late February.

First of all, let me thank all of those of you who sent funds to support the server migration!
I can not nearly express my gratefulness for that support, to witness how many people share my view of "wikiloops is valuable, let's keep it around" gives a lot of motivation,
and I'm relieved to know I'll be able to fund the additional cost of the new servers for some time to come.

Thanks to all who contributed.

Let me give you a quick insight in what I have been able to achieve so far:
- at the time of writing (and probably you are reading this blog post), you are still using the unchanged wikiloops network as it has existed for the past years - with our main servers hosted in Germany.
- at the same time, there is now a completely independent installation of wikiloops on another server in Ireland, which belongs to the Amazon cloud network.
That new installation will eventually replace the current setup, and it includes its own copies of any data, images and music hosted on wikiloops.
That basically means we'd be able to switch to that new server quite quickly anytime we needed, without loosing more than a few days of recent data*, so, a lot of my intended safety mission has been achieved by now.

* as there are two instances of wikiloops right now, the one that is being used "runs ahead" of the backup one as stuff is constantly added (think_ likes, comments, tracks...).
I'm currently trying to keep the backup server no more than one week behind the live server, data-wise, so, that not-yet-backed-up data is what may be lost in the worst case.

While some aspects of migrating wikiloops were astoundingly quick, I had to notice that transferring over 1.000.000 files to S3-servers takes quite a while.
I also took the opportunity to cleanup some of the dead-weed left after developing on our old server for over ten years, and I managed to finally setup a working Git version control system for the new instance, which is a really big improvement for the entire project.
Those familiar with git will probably find it hard to imagine I worked without it for ten years, well, so do I by now.

What is still in the process of testing and adjusting are any functionalities of wikiloops where new data is added.
The whole process of handling additional music files, encoding them, tagging the mp3s, creating the waveform representation image and so on needs to be modified to work in a cloud server scenario, and that turns out to be quite a bit of work.

Also, any background tasks like email sending, album generation & publication and stuff like payments handling and pdf-generation will still need to be tested -
some will work straight away, some will need adjustments, there's still some work ahead.

I guess its needles to say I'm happy to do this properly while you guys have a working wikiloops to play with, having a community of upload-happy users on your back while solving code problems that keep them from uploading is something I'm familiar with (no criticism!) from the past. I'll put it this way: I can work without that pressure quite OK ;)

Once I'm done with testing and adopting, there will come the day where we'll switch over to the new system for good, and I'll give notice before that happens. It's a bit frustrating I had to interrupt the app coding project that I have been working on for wikiloops before the February-incident, and I'm hoping to be able to get back to that quite soon.

I must admit that moving to that new server is once again widening the technical options-at-hand, which were somewhat limited on the prior servers. That does let me reconsider some ideas which had to be dropped due to our servers limitations... who knows what good will come from finally making the move. I'm just stating that to once more thank those who support my moving forward with wikiloops.

So, here I end for today and return to debugging,
have a nice weekend & many happy jams on wikiloops

Richard

p.s.
For the IT folks around, here's a little shortlist of stuff I've been digging into... just for your enjoyment. To those who aren't aware, I am a complete DIY-coder, all I know is how to learn stuff from the internet.
Credit goes out to the zillions of free-information-sharers in IT, on StackOverflow and beyond, that spirit of sharing knowledge echoes in what we do on wikiloops.
So, I've been dealing with
Git (what a relief, once its running) & gitHooks (that took me long to establish),
S3 (used the php-library before), CloudFront (well, OK),
AWS linux setup (previously totally dreaded),
certbot (shizzle folks, it could be easier),
EBS volume partitioning and other server maintenance tasks which I was unfamiliar with.

Currently re-routing all static file storage to S3 ...
The postponed app project is a flutter app, btw, which might be found on gitHub (invitation required, it's not public).

:x!