Home »Forum»Developers blog»Engine room report feburary 2022

Engine room report feburary 2022

Dear friends of wikiloops,

thanks for stopping by and checking out this post, in which I'll attempt to cover what has been going on in the wikiloops engine room these days.

I guess most of you will have noticed that we had some server outage on Thursday 24th of February, lasting into the following Friday.

Maybe some of you saw the "currently unavailable" message during that period, or later the rather crude "we are in maintenance mode"-page which I quickly came up with, to at least signal out that someone was trying to fix the problems.

The armed conflict between Russia and Ukraine started the very same day, and I don't know about your feelings, but I couldn't help wondering if there might be some connection between the two events.
To tell you the truth - I'm not sure I have a straight answer to this question, but I'll share what I have, and what my current thoughts are.

When I learned that there was a problem on wikiloops (a classic Kaiser-family situation: my mom calls and tells my wife to tell me wikiloops is down), there were a couple of things I suspected as probable causes - hardware failure, some data-center-wide connection failure, a bot attack of some kind... the kind of things that we have seen in the past.

So, digging into the matter, I do some routine checks to see which parts are still working, and which ones are not.
Turns out:
Both server and database are operating quite smoothly,
but anything trying to connect via the wikiloops domain name had only a 3% chance of getting a proper response - anyone else sees the dreadful 503-Error screen.
What is this? Me wonders - clearly no hardware crash there, that rather looked like some sort of bot attack thing to me.

Now, without going into all the tech-details, one thing about bots is that all you'll ever know about them is their traces in the servers log-files, which basically reveal little more than "someone tried to access page X at this time, and received/didn't receive a response".
The potential responses are labeled by types, you'll know some of those, like the well known code "404" for "page not found".
The crucial problem on day one of the incident was: I couldn't access these log files straight away to see what was going on, but had to wait half a day to get the daily log extract supplied from the hosting company, so, there I sat with little insight what could be happening.

So I went troubleshooting, and since there was not much to break in that situation, I followed the very basic idea of "if I block some countries, and if there is an attack coming from a certain region, I might get to the point where wikiloops comes back up for the remaining unblocked regions".
And yes, I frankly admit that there are some countries that are a bit more notorious for rude bot attacks than others, so I blocked these first.
Long story shortened: I went on until anything but Germany was blocked, but to my surprise none of that made a difference. I even blocked Germany for a moment and saw the block coming up on my own screen, i played around, nothing worked.
So, back to nagging the support folks and waiting for the logs...

I'll try to cut this story, so you hopefully have a moment left for my follow-up thoughts:
After quite some mail exchange with our hosting company, it turns out what we witnessed was a mix of a really heavy bot attack, plus our hosting companies efforts to fire-fight the issue.
What the support guys did was basically throttle general access to wikiloops, as zillions of Chinese bots tried to call up wikiloops pages.

After I could make it clear to them that their defensive move was just killing our platform they slowly lifted their throttling on the main .com-domain, letting the bot requests be blocked by the regional block I had set up one level lower - that's how you guys got a working wikiloops back some time on Friday, if you weren't trying to access the German or french versions of wikiloops, and if you happened to live somewhere outside of Asia.

The actual attack wore off during the course of Saturday, and as the logs show, traffic piked at around 12 times of what we usually have on wikiloops.
And we do have a lot of bots crawling wikiloops on any day, I'd estimate that bots make up 90% of wikiloops traffic on a normal day, so to see that figure multiply that much is quite scary, and certainly beyond what our current server setup is built for.

I did announce an optimistic "we are back in full service" some time on Saturday,
and luckily wikiloops has been operating quite smoothly since.
So, is that a happy end?
Not quiet, now that I have had some time to think this thru (watch out, turning point of this news post approaching...)

I could have put this in the "shizzle happens" section, and we might as well say let's sit and wait if something like this happens again.

What struck me was this feeling of total dependency of those server support folks, who at will shut down wikiloops "in protection of their other customers".

I'm aware that the internet is a rough place, and digital attacks of all kinds happen on any day,
but the moment I sit there and block a whole continent + some select countries from accessing wikiloops,
I can't help but notice that I am just leaving the path of what wikiloops was intended to be:
An international meeting place for musicians, where one could cross borders and make friends just by making music.

As I see things today, our current hosting company is probably not willing to let us re-open for any country (or they will come back and ask me to raise fences again as soon as something goes wrong), so my choice comes down to:
Let go of the "global jam" ideal, or get a different server setup that can provide the "global access" plus the needed defense systems to stay online when under attack.

Personal note:
I've done this here for over ten years now, and it means something to me, you know?
Some time this weekend I received a birthday notification of one of wikiloops Russian members, and knowing Russia is on the block list since this weekend really made me feel bad.
Alex, if you ever read this - a late happy birthday to you!
I'll say the following as a flaming personal statement:
Folks, we are here to look a little beyond our local horizons,
and wikiloops is there to let us experience something different to the stereotypes we may have adopted. Let's never forget how many funky bassists live in any country before thinking some nation was "totally evil" - end of my personal peace-pledge for the day, sorry, had to say it.

Back to the engine room news:
I have spent the weekend "evacuating" our current server and moving wikiloops into the almighty amazon cloud, with the aim to upgrade our infrastructure to a point where wikiloops can continue its global mission,
independent of some German hosting companies ideas of how to limit traffic.

I'll be really honest and add that the location of our current data center on the border between Germany and Poland doesn't really feel like a safe place to be in the current political situation,
so when I called my server moves "evacuation" earlier, then that's the reason why.
One benefit of amazons cloud is regional independence, I might end up replicating servers to anywhere if need be.

Now, the moving process for a project the size of wikiloops is nothing one could do in a day,
but I am happy to say that I managed to set up a replicated instance of wikiloops on some Irish aws-server over the weekend,
so even if things should go down the very bad way some time in the future, I'm positive I'll be able to relaunch quite quickly.

You may imagine it was quite a relief off my shoulders when importing the data tables holding more than 1.000.000 comments and similar treasures to a safe environment...
and I guess I'll continue getting that server ready so we are set to hop over.

I'll make sure you are informed before that happens,
because that change-over might take a day or two.

What I would like to leave you with before I'll end are two impressions:

First, please note that someone is taking care of this project, and that the will to keep it in its best shape is strong, even if the technological hurdles to achieve that have not become less steep in the past years.
I feel wikiloops is worth the effort, and if I say wikiloops, then to me that is our international community of people willing to collaborate. This is worth saving, so that's what I'll try to do.
Second, I'd like you to know that I've had moments during the past weekend where I felt like "Richard, you need to get all the help you can possibly get to pull this off!", and we actually had a small family council first to make sure the time and cash investment needed to move on with wikiloops can be covered.
My decision to upgrade the infrastructure will come with a hefty price tag attached, so if anyone feels like supporting the "global jam"-missions future, I'd really appreciate if you'd send some support via the wikiloops donations page.

All that being said,
let me thank you for reading thru and for being part of this community

stay safe and healthy out there & keep up the positive spirit in these odd times we are witnessing

all the best from Germany, a thousand miles west of Kiev
Richard

wikiloops founder

Comments

mpointon
A really interesting read, Richard. Speaking as a tech man myself, I know the rabbit holes you can go down trying to track down these things.

You make a very interesting point about the global nature of Wikiloops versus the possible need to suppress those making dissenting noises or disruptive actions. A tough call and I understand your dilemma as you are very much about peace and collaboration, whatever the inherent differences. But, personally, I feel there will always be people who want to ruin everyone else's fun. In the real world, they're asked to leave the party. In the online world, you block them. There is no shame in that activity.

Without timely access to IP information you cannot enact such [temporary] measures though. I could bang on about the superb anti-DDoS features of the big boys such as AWS but, as you rightly point out, they come with a big price tag.

Sadly, due to pretty obvious events going on right now, it's more important than ever that global harmony is promoted, no matter how small the gesture. The world is in genuinely real trouble at the moment and now, more than ever, we need projects such as Wikiloops to show we can rise above our differences. And be at least an island of light from the gathering darkness outside. There is an enormous range of musical skills on here and the fact that players here neither laugh at the less experienced or angrily envy the more experienced is a lesson the world could do with learning right now.
+3
metaled
good job great story as i prob have some german in me with a last name like schrock i have also sang for musicians on both sides of the show i know i can be maybe to blunt but i share the message you spoke of on stereo types idk if you know about autisim spectrum disorder but we generally don't see stereo types at all thank you for running this great site it has helped many people make great music including me+0
FrankieJ
https://arstechnica.com/information-technology/2022/03/unending-data-floods-and-complete-resource-exhaustion-ddoses-get-meaner/+0
TeeGee
Thank you so much Richard, donation has been made - your work is appreciated <3+0
SoulFingers
Working for a big tech company myself I know what you went through during the last few days Richard. AWS is powerful but it takes lots of initial setup effort if you wanna have everything automated. Donation made.+0
wjl
Thanks for all the good work Richard - and all the best to the whole Kaiser family :)+0
Krasimir
What i can say thank you for prompt action.:)+0
titi
faced with this access problem, I must say that my first thought reflex was the same as yours... Thank you for your information.+0
rootshell
sorry about the outage, but love this tech stuff and sleuthing <3 great work Richard!+0
Filo974
I am really happy that you took the time and obtained the means to make this technical decision in the face of this problem experienced for my part, and for the well-being of all loopers. Nothing should reach a site like Wikiloops since we are in this place only for shared music (in principle)...
My best thoughts to you and yours. Donation made. :)
+0
Posted on
User Avatar

Founder of
Posts: 2564
Joined: 30.12.10

Let's stay connected:

wikiloops online jamsessions are brought to you with friendly support by:
allfeel from United States

"Just a Basement hobbyist but the hours of entertainment wikiloops has brought me and my guitar rig are worth so much more than my (father of 2 little ones) butt can give! Thanks Wikiloops keep up the great work (all of you musicians as well!)."

wikiloops uses Cookies and processes data in compliance with the GDPR,
as stated in our data privacy policy ..