How to bypass those annoying filtered internet accesses

Today I had to login on one of those FON access points where http/https are ok but the rest was not.
Here is my recipe to quickly and painlessly bypass those.
The only prerequisite : to have a public server or vps ready on the internet to help you.

1. make your ssh daemon wait on port 443 on your public server.

The reason is that 443 is the https port which is normally open even on stalinian firewalls (note: it happened to me once that the firewall detected that the traffic was not a genuine https one but it should be really rare)
File /etc/ssh/sshd_config

[...]
Port 443
[...]

2. client side, setup a specific ssh config

prepare a configuration for your server in ~/.ssh/config

[...]
Host myserver
HostName server.domain.tld
   User myuser
   Port 443
   DynamicForward localhost:3141
[...]

It says : create a shortcut called "myserver" towards the machine "server.domain.tld", login with user "myuser", on the https port and, most importantly, create a local socks server on port 3141 that will tunnel the traffic through it.

3. install tsocks

Install tsocks on your client. It is a tool that redirects all network traffic from your applications towards a specific socks server.

# on gentoo (be sure the flag tordns is not active)
emerge -av net-proxy/tsocks

4. configure tsocks

File : /etc/socks/tsocks.conf

                                                                                                     

server = 127.0.0.1
server_port = 3141

It just tells tsocks to use your local socks proxy you have setup previously

5. start your tunnel

Login to your server from one terminal and leave it there, it should create a socks proxy.

                                                                                                     
ssh myserver

6. enable tsocks on a session

let tsocks hijack the session to redirect the sockets creation toward the tunnel

                                                                                                      
. tsocks on

Note: the . (or source) is really important here
You can check if the hijacking has been done correctly with the command "tsocks show", it should answer LD_PRELOAD="/lib/libtsocks.so"

7. enjoy the "almost" full internet access

any application on the same shell started after the ". tsocks on" should connect through your tunnel with no port/protocol restriction whatsoever.

The Go Language is faster than the computer language benchmarks game thinks it is

Note : yes this is anecdotal but my point is that a complete test could be interesting.
I noticed that the computer language benchmarks compiled Go only with the default toolchain compiler
I was surprised at how slow the Go language was so I tried it myself quickly on my sandybridge laptop
For example on this simple nbody test example.

# fetch nbody.c and nbody.go
$ gccgo -pipe -Wall -O3 -fomit-frame-pointer -march=native -mfpmath=sse -msse3 -o nbody.go_gcc nbody.go
$ go build nbody.go; mv nbody nbody.go_gc
$ gcc -pipe -Wall -O3 -fomit-frame-pointer -march=native -mfpmath=sse -msse3 nbody.c -o nbody.gcc_run -lm

# so the reference 
$ time ./nbody.gcc_run 50000000 
-0.169075164
-0.169059907
./nbody.gcc_run 50000000  8.13s user 0.00s system 99% cpu 8.135 total

# The toolkit version
$ time ./nbody.go_gc 50000000
-0.169075164
-0.169059907
./nbody.go_gc 50000000  14.27s user 0.00s system 99% cpu 14.285 total

# The fair comparison : same compiler same options as C
$ time ./nbody.go_gcc 50000000
-0.169075164
-0.169059907
./nbody.go_gcc 50000000  8.31s user 0.01s system 99% cpu 8.324 total

Lang	C	Go compiler	Go Gcc
Results	8.135	14.285	8.324
My normalized results to 1.5 for C	1.5	2.63	1.53
Benchmark Game normalized results	1.5	2.3	?

So Go should be between C and Clean, smoking scala, clojure, java and lisp.
Edit: Took the wrong timing values in the table, but it doesn't change the ratios

Setup a fancy development environment ... on a linux system console only

My original goal was to see if I could work as a software developer on a linux system console only for energy efficiency.
This is a normal powertop state on an idle desktop under X:

Unload X, unload a couple of daemons, dim your screen and you can get to that :

But then you miss a lot of stuff, let see how far we can get to compensate those losses

Setup the low level : Fixup your keyboard bindings (facultative)

I use the US international keyboard with dead keys, to enable that (or any other favorite keymap you want):

% sudo vi /etc/conf.d/keymaps
# change keymap to us-acentos
% rc-update add keymaps default
% /etc/init.d/keymaps start

Setup the low level : Framebuffer

For the kernel configuration to run your console on a framebuffer, you will need to dig on google the tutorials as it is really hardware dependent. For me it worked out great with KMS under intel graphics.
Once you have that, force the resolution you are on to the applications by doing this :

% emerge sys-apps/fbset # or equivalent under your distribution
% sudo fbset -s > /etc/fb.modes

Mine looks like this to stay on the mode KMS actually set at boot time:

% cat /etc/fb.modes                                                                                                            
mode "1920x1080"
    geometry 1920 1080 1920 1080 32
    timings 0 0 0 0 0 0 0
    accel true
    rgba 8/16,8/8,8/0,0/0
endmode

For the gentoo people, you need to activate the framebuffer support on your applications:

% sudo vi /etc/portage/make.conf # or /etc/make.conf
# and add directfb fbcon to the USE variable
% emerge --keep-going --ask --update --newuse --deep --oneshot -t world
# recompile everything that has changed

You gonna need some extra rights to access the console: add yourself in the console, tty and video groups

% useradd -G console myuser
% useradd -G tty myuser
% useradd -G video myuser

Another annoyance to sort out, tty group has not enough rights on the ttys (if you are on a shared system, check if it is secure to do that). You need to patch udev rules for that, here is my custom addition :

% cat /etc/udev/rules.d/99-tty.rule
SUBSYSTEM=="tty", KERNEL=="tty[0-9]*", GROUP="tty", MODE="0660"

Load a simple app and test your framebuffer :

% emerge media-gfx/fbida
% fbi myimage.png # it should show the given image, it is a simple image viewer

We can enhance a little bit the default console with:

% emerge -av media-fonts/terminus-font # be sure you have the pcf flag set
% vi /etc/conf.d/consolefont
# for example change the font to consolefont="ter-v16b"
% rc-update add consolefont default
% /etc/init.d/consolefont start

Setup the low level : mouse support !

Another permission annoyance to solve first:
You need to add those rights in udev so your normal user can grab the mouse.

% cat /etc/udev/rules.d/99-mouse.rules                                                                                         
SUBSYSTEM=="input", KERNEL=="mouse*|mice|event*", MODE="0660", GROUP="console"

We need gpm:

% sudo vi /etc/portage/make.conf # or /etc/make.conf
# and add gpm to the USE variable
% emerge --keep-going --ask --update --newuse --deep --oneshot -t world
# recompile everything that has changed
% rc-update add gpm default
% /etc/init.d/gpm start

Now a "cursor" should appear on your console and you should be able to copy/paste text from it by left click drag to select and right click to paste.

Setup a decent console emulator

I love my powerlines in vim and zsh. But the standard linux console supports something like 500 characters and no truetype font support !
So we need a decent console with utf-8 support.
The only one I found so far is fbterm. Let's install this.

% emerge app-i18n/fbterm
% fbterm # should work

We need to hack our zshrc/bashrc to force TERM to "fbterm" if we are on the console:
In my .zshrc I put :

if [ $TERM = "linux" ]; then
    export TERM="fbterm"
fi

It should have created a .fbtermrc template for you.
Now you can set it up so it gets your powerline fonts for example

% vi ~/.fbtermrc
font-names=Monaco for Powerline
font-size=12

Note: be sure you use my version of powerline for zsh with this patch in
Now vim should be magnificient too...

Setup a web browser

After a good vim, a developer needs a web browser...
After trying 3 or 4 different ones, I still feel that links2 in directfb mode looks the best
Note : you cannot pile framebuffer applications on the same console so I recommend to switch to another virtual console CTRL-ALT-F2 F3 etc ... so you can keep fbterm on the main one.

% emerge -av www-client/links
% links -g -drivers directfb http://news.ycombinator.com/

To improve the rendering you can tweak the subpixel rendering in Display optimization LCD, RGB (should be the most common, if it doesn't look good, try the other one)

Look at images and pdfs

we already saw fbi from the fbida package to look at images
with fbgs from the same package you can read pdfs

Playing a movie

mplayer is working fine for that it automagically detected everything for me.
Note I still have some tearing.

% mplayer mymovie.mkv

Taking screenshots

% emerge media-gfx/fbgrab

I use it with a timer :

% fbgrab -s 30 /tmp/youpi.png &
% # hurry to prepare your screenshot

Playing doom2 !

% emerge games-fps/prboom
# roam on the internet to find Doom2.wad
% prboom -iwad Doom2.wad

Conclusion

Feel free to contact me if you find other goodies on a simple framebuffer/linux console.

Quick hack to plot data on a map in python

I wanted to share a cool code mash up I did to show some stats about Err usage in the world.

The result is here, in blue roughly after a non reliable homebrew datamining from the apache logs:

in blue the "fidelity" of an installation

in red the "development" activity

I applied this tip

As dependencies you gonna need :

matplotlib

GeoIP

Basemap

PyQT4 (but you can change the backend to your favorite toolkit)

Here is the full snippet :

#! /usr/bin/python
import matplotlib
matplotlib.use('Qt4Agg')
from datetime import datetime
import os
DIR = 'data/'

dirList=os.listdir(DIR)

unzip = lambda l:tuple(zip(*l))

all_dates = {}
for fname in dirList:
    if fname.startswith('gootz-access'):
        counts = {}
        d = datetime.strptime(fname.split('.')[1], '%Y-%m-%d')
        with open(DIR+fname) as f:
            data = f.readlines()
            for line in data:
                if line.find('err/version') != -1:
                    addr = line.split('-')[0].strip()
                    if counts.has_key(addr):
                        counts[addr] += 1
                    else:
                        counts[addr] = 1
        all_dates[d] = counts

sorted_dates = sorted(all_dates.keys())

import GeoIP
gi = GeoIP.open('GeoLiteCity.dat',GeoIP.GEOIP_STANDARD)

insts_pos = {}
devs_pos = {}

for date in sorted_dates:
    for ip, count in all_dates[date].iteritems():
        record = gi.record_by_name(ip)
        if record and record['latitude']:
            lon = record['longitude']
            lat = record['latitude']

            toinc = devs_pos if count > 5 else insts_pos
            if toinc.has_key((lon,lat)):
                toinc[(lon,lat)] += 1
            else:
                toinc[(lon,lat)] = 1

from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import numpy as np
# lon_0 is central longitude of robinson projection.
# resolution = 'c' means use crude resolution coastlines.
m = Basemap(projection='robin',lon_0=0,resolution='c')
#set a background colour
m.drawmapboundary(fill_color='#85A6D9')
# draw coastlines, country boundaries, fill continents.
m.fillcontinents(color='white',lake_color='#85A6D9')
m.drawcoastlines(color='#6D5F47', linewidth=.4)
m.drawcountries(color='#6D5F47', linewidth=.4)
# draw lat/lon grid lines every 30 degrees.
m.drawmeridians(np.arange(-180, 180, 30), color='#bbbbbb')
m.drawparallels(np.arange(-90, 90, 30), color='#bbbbbb')

inst_lngs = [entry[0][0] for entry in insts_pos.iteritems()]
inst_lats = [entry[0][1] for entry in insts_pos.iteritems()]
inst_count = [entry[1] for entry in insts_pos.iteritems()]
inst_x,inst_y = m(inst_lngs,inst_lats)

s_inst_count = [p * p for p in inst_count]
m.scatter(
    inst_x,
    inst_y,
    s=s_inst_count, #size
    c='blue', #color
    marker='o', #symbol
    alpha=0.25, #transparency
    zorder = 2, #plotting order
    )
for population, xpt, ypt in zip(inst_count, inst_x, inst_y):
    label_txt = int(round(population, 0)) #round to 0 dp and display as integer
    plt.text(
        xpt,
        ypt,
        label_txt,
        color = 'blue',
        size='small',
        horizontalalignment='center',
        verticalalignment='center',
        zorder = 3,
        )

devs_lngs = [entry[0][0] for entry in devs_pos.iteritems()]
devs_lats = [entry[0][1] for entry in devs_pos.iteritems()]
devs_count = [entry[1] for entry in devs_pos.iteritems()]
devs_x,devs_y = m(devs_lngs,devs_lats)

s_devs_count = [p * p for p in devs_count]
m.scatter(
    devs_x,
    devs_y,
    s=s_devs_count, #size
    c='red', #color
    marker='o', #symbol
    alpha=0.25, #transparency
    zorder = 4, #plotting order
    )
for population, xpt, ypt in zip(devs_count, devs_x, devs_y):
    label_txt = int(round(population, 0)) #round to 0 dp and display as integer
    plt.text(
        xpt,
        ypt,
        label_txt,
        color = 'red',
        size='small',
        horizontalalignment='center',
        verticalalignment='center',
        zorder = 5,
        )


#add a title and display the map on screen
plt.title('From where Err is used.')
plt.show()

powerline for zsh

I discovered some time ago vim-powerline a cool semi-graphical status line for vim

Jérémy Romey adapted it for zsh.

You can see here for example if you have any pending files to commit on git in the current git repo of your working directory and the current branch you are on.

Note : this is my slightly adapted version with the international format for the time.

Here is a quick tutorial to install this version.

Safety first ... backup your .zshrc, you never know if you gonna brake something

cd                                                            
cp .zshrc .zshrc.bak

Install and make sure you use a powerline patched font in your terminal.
For example I like Envy R Monaco

Now, simply grab my for of "oh-my-zsh" and copy the config template.

git clone git://github.com/gbin/oh-my-zsh.git ~/.oh-my-zsh
cp ~/.oh-my-zsh/templates/zshrc.zsh-template ~/.zshrc

Change the theme to "powerline", add your aliases in your ~/.zshrc, and some plugins. Here is mine.

# Path to your oh-my-zsh configuration.
ZSH=$HOME/.oh-my-zsh

# Set name of the theme to load.
ZSH_THEME="powerline"

alias ls='ls --color=auto'
alias ll='ls -l --color=auto'
alias la='ls -A --color=auto'
alias l='ls -CF --color=auto'
alias diff='colordiff'

alias -s gz='tar -xzvf'
alias -s bz2='tar -xjvf'
alias -s zip='unzip'
alias -s txt=$EDITOR
alias -s html=$BROWSER

# Set to this to use case-sensitive completion
# CASE_SENSITIVE="true"

# Comment this out to disable weekly auto-update checks
# DISABLE_AUTO_UPDATE="true"

# Uncomment following line if you want to disable colors in ls
# DISABLE_LS_COLORS="true"

# Uncomment following line if you want to disable autosetting terminal title.
# DISABLE_AUTO_TITLE="true"

# Uncomment following line if you want red dots to be displayed while waiting for completion
# COMPLETION_WAITING_DOTS="true"

# Which plugins would you like to load? (plugins can be found in ~/.oh-my-zsh/plugins/*)
# Custom plugins may be added to ~/.oh-my-zsh/custom/plugins/
# Example format: plugins=(rails git textmate ruby lighthouse)
plugins=(git github pip python)

source $ZSH/oh-my-zsh.sh

Open a new terminal and boom, it should work. Awesome no ?

Setup a plugin development environment for Err

Err makes your life as easy as possible to extend it. Here I present you a quick walk-through to get you started.

First, clone err from git. I assume you are somewhere in your work/projects directory

.
⚫  gbin@sal work % git clone git://github.com/gbin/err.git
Cloning into 'err'...
remote: Counting objects: 1742, done.
remote: Compressing objects: 100% (638/638), done.
remote: Total 1742 (delta 1177), reused 1626 (delta 1061)
Receiving objects: 100% (1742/1742), 897.87 KiB | 552 KiB/s, done.
Resolving deltas: 100% (1177/1177), done.
⚫  gbin@sal work % 

Cool now, just copy the config template and prepare a data directory:

.
⚫  gbin@sal work % cd err 
⚫  gbin@sal err % cp errbot/config-template.py config.py
⚫  gbin@sal err % mkdir data

Then change the config to point the data directory and the log to this newly created directory in your config.py :

.
BOT_LOG_FILE = './data/err.log' # or an absolute path
BOT_DATA_DIR = './data/'        # or an absolute path

You know what ? you can already start your bot instance in development mode :

.
⚫  gbin@sal err % ./scripts/err.py -T
INFO:Config check passed...
INFO:Activate internal commands
INFO:Activating all the plugins...
INFO:Activate plugin: Webserver
INFO:Activating Webserver with min_err_version = 1.6.4 and max_version = 1.6.4
INFO:Webserver is not configured. Forbid activation
INFO:Activate plugin: VersionChecker
INFO:Activating VersionChecker with min_err_version = 1.6.4 and max_version = 1.6.4
INFO:Activate plugin: ChatRoom
INFO:Activating ChatRoom with min_err_version = 1.6.4 and max_version = 1.6.4
INFO:
INFO:Notifying connection to all the plugins...
INFO:Callback_connect
INFO:Plugin activation done.
Talk to  me >>

Note : do a ctrl-C or ctrl-D to quit

If you want to get even fancier, do a -G instead of -T (provided you have installed pyside), if not the bot will tell you how to do that.

.
⚫  gbin@sal err % ./scripts/err.py -G

So now, let's get a template of a plugin and tell err to take it into consideration

go to your work / projects directory and for example clone the helloworld project

.
⚫  gbin@sal err % cd ..
⚫  gbin@sal work % git clone git://github.com/gbin/err-helloworld.git
Cloning into 'err-helloworld'...
remote: Counting objects: 7, done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 7 (delta 1), reused 7 (delta 1)
Receiving objects: 100% (7/7), done.
Resolving deltas: 100% (1/1), done.

go back to your err directory and change the config.py to point to this template

.
BOT_EXTRA_PLUGIN_DIR = '../err-helloworld' # or an absolute path

From there you see that the plugin is loaded and works:

.
INFO:Config check passed...
INFO:Activate internal commands
INFO:Activating all the plugins...
INFO:Activate plugin: Webserver
INFO:Activating Webserver with min_err_version = 1.6.4 and max_version = 1.6.4
INFO:Webserver is not configured. Forbid activation
INFO:Activate plugin: VersionChecker
INFO:Activating VersionChecker with min_err_version = 1.6.4 and max_version = 1.6.4
INFO:Activate plugin: ChatRoom
INFO:Activating ChatRoom with min_err_version = 1.6.4 and max_version = 1.6.4
INFO:Activate plugin: HelloWorld
INFO:Activating HelloWorld with min_err_version = None and max_version = None
INFO:
INFO:Notifying connection to all the plugins...
INFO:Callback_connect
INFO:Plugin activation done.
Talk to  me >>!status
INFO:received command = status matching [status] with parameters []
Yes I am alive... 
With those plugins (L=Loaded, E=Error, B=Blacklisted/Unloaded), C=Needs to be configured) :
    [L] ChatRoom
    [L] HelloWorld
    [L] VersionChecker
    [C] Webserver
   Load 0.07, 0.24, 0.28
   GC 0->301 1->5 2->4
Talk to  me >>!hello
INFO:received command = hello matching [hello] with parameters []
Hello World !
Talk to  me >>!help hello
INFO:received command = help matching [help] with parameters [hello]
this command says hello

So now, you just have to rename everything to your plugin name, and get started !

For deeper information, go to the plugin development docs.

Note: You can run err.py under debugger without any problem. I personally use pycharm and it works really well. I might make another tutorial for it in the future.

Enjoy !