Project Chaos – Part 1

Project Chaos Part 1

Intro

This article will be the first one of an ongoing story, I’ll try to document my journey on this project through posts, probably shorter than the usual ones.

So, I’ve got this group or tabletop role player that I know for 15 years through different instanciation of the same internet community. It’s the group of people I know for the longest time (for almost half of my life until now) and as all online community it’s shaped by the tools we use (and the tools we use are shaped by our community).

There were the bulletin board like forums (phpBB then fluxBB), both of them managed by the techies (not me) and a third iteration based on drupal. And facebook. Because before 2007 we used the bulletin board forums as a way to keep in touch, to plan parties or to help each other.

Since the facebook happens, the forum mostly turned to be a game-lore database, all the social thing slowly moved over there. And then the forum started to slowly die.

Also, Shadowrun, the game we used to play, gone through some editorial crisis, the new edition has been heavily criticized and some personal dramas did technically killed this forum and group of friends.

They’re still in touch through Facebook, Google cloud sharing services and stuff like that. But since I won’t get there I’ve been a little bit ostracized (not that I really mind, I love my loneliness) but this community was not a community anymore. We ended up with half a dozen of people doing all the community services, while all the others are just feasting on it.

Typical of communities of that age I think. Some of us moved far away, other are having babies, but still, internet tools were supposed to help us to stay in touch.

For the last two years, we did however launch a lot of meetups in different places. And I fighted the uses of google docs to the profit of free alternatives such as etherpad and ethercalc (more than enough for our use case of writing down recipes and errands to run to feed 30 people) with some success.

And then came the sharing. We’re sharing a lot of music and playlists. Especially when you want to run a game, you’re always looking for some atmosphere, so we talk a lot about music (I mean, a 6hours long game is 6 hour of music non stop, you need to find some). The thing is, they want some of my music and, since I won’t use spotify nor google, we’re stuc with sneakernet. Reminds me high school where we exchanged tapes, but with USB hard drive of more than one terabyte.

So, they wanted to share, including what politicians would call illegal files. And, in what sounds like a surprise – but interesting – move, they want to do it themselves (OK, they want me to get involved and helps them) but they want to self-host their stuff.

10 years of advocating free software. 5 of advocating for getting out of social centralized web. 2 after setting up a social network for our characters. 1 after showed them how powerful free and decentralized software can be, they asked me to helps them build a community sharing server.

At last.

Next step

So, we’re currently writing down what we want and need, using etherpad and calcs. There will be a lot of learning implied, and some code to write. But at least, we’re going to do it together.

And yes, they actually asked me to show them how to administrate a server, even if it implies running a shell.

So this will be my log for this journey. I hope we’ll reach the destination. Theres adventures to come, but I have faith in their motivation.

Searx

Welcome to searx

You might have noticed some change on my seeks node since it’s not a seeks node anymore, but instead it’s a searx node.

Searx is a project started by asciimoo after Taziden gave a talk at Camp zer0 about going forward with seeks and opening it up to a wider base of developper.

The idea is that seeks ‑ currently written in hardcore C++ ‑ is a prototype and an exploratory project about search and decentralization of search, and that we can now build almost from scratch a search engine which will implement the concept behind seeks but in a more developper friendly way, for instance in python.

We already had a lot of discussion with people hanging on #seeks@irc.freenode.net about this and, technically, there’s two tool to develop. An easily extensible metasearch engine which will feed a DHT of result shared with different nodes.

And then asciimoo wrote searx, a meta search engine, easily extensible. Now, we "just" have to connect it to a DHT. But I’ll save that for later.

So, how did I installed it? I’ve fought a little bit with uwsgi and nginx, but now it works. Here’s how:

Setup

Getting the code, the dependencies and everything else

Create a searx user for it’s a good practice (don’t run things as root) and do some git cloning and virtualise you’re environment. Oh, before I forgot, I’m running a debian stable and I try to keep the distribution clean (so no pip install outside of virtualenv)

cd /usr/local git clone https://github.com/asciimoo/searx.git chown searx:searx -R /usr/local/searx cd /usr/local/searx virtualenv searx-ve . searx-ve/bin/activate/

Now, you have a running virtual environnement in ”/usr/local/searx/searx-ve” and the code in the parent directory. You need to install some dependencies, so launch that command and go get a cup of coffee.

pip install -r requirements.txt

Now, the code is alive. You can test it by running the flask instance:

python searx/webapp.py

And you can proxy requests to ”http://localhost:8888” from your favorite webserver. It works.

Uwsgi

Since it’s not daemonized, and you’ve got only one worker, I wanted to have something more maintainable. So I needed something like uwsgi (or gunicorn, or whatever) to run the apps right from nginx.

Since debian splitted uwsgi config in a lot of modules, don’t forget to install python module (I was stuck with that a lot). So, let’s install uwsgi and required dependencies.

apt-get install uwsgi uwsgi-plugin-python

Next step is to create an app. In debian, uwsgi has the same apps-{available,enabled} file structure than on nginx or apache. Here’ my config file for searx:

vim /etc/uwsgi/apps-available/searx.ini  [uwsgi] # Who will run the code uid = searx gid = searx  # Number of workers workers = 4  # The right granted on the created socket chmod-socket = 666  # Plugin to use and interpretor config single-interpreter = true master = true plugin = python  # Application base folder base = /usr/local/searx  # Module to import module = searx.webapp  # Virtualenv and python path virtualenv = /usr/local/searx/searx-ve/ pythonpath = /usr/local/searx/ chdir = /usr/local/searx/searx/  # The variable holding flask application callable = app

Once that’s done, symlink this file in apps-enabled and start uwsgi.

cd /etc/uwsgi/apps-enabled ln -s ../apps-available/searx.ini /etc/init.d/uwsgi start

By default, the socket used by uwsgi will be in ”/run/uwsgi/ap/searx/socket”. This is where nginx will chat with uwsgi.

Nginx

Hard part is done, if you already have nginx installed, just use yet another vhost.

vim /etc/nginx/sites-available/searx  server {     listen 80;     server_name searx.example.com;     root /usr/local/searx      location / {             include uwsgi_params;             uwsgi_pass unix:/run/uwsgi/app/searx/socket;     } }

Then activate the newly created sites and restart nginx.

ln -s /etc/nginx/sites-{enabled,available}/searx /etc/init.d/nginx restart

And go visit searx.example.com or whatever your FQDN is) on port 80, it would works.

I suggest you to install some SSL? but it’s beyond the scope of this tutorial.