xL2TPd on Arch Linux and Ubuntu.

I have rent a VPS for the first time from a provider called MiniVPS and built a L2TP tunnel between my Arch Linux and the Ubuntu VPS.
Unfortunately, available distros were CentOS, Fedora and Ubuntu, and I had to go with Ubuntu since it had the latest kernel among them.
However, I’m not sure whether it is a openvz limitation or not, there were no l2tp kernel modules available which I believe improve l2tp
performance.

So far, my experience with minivps is not too good, but not bad.
Speedtest results on average were like below and bandwidth differs
a lot from time to time.
I measured the speed while on lt2p tunneland also ssh dynamic port forwarding.

Ping response: 250ms
Download: 1Mbps ~ 15Mbps
Upload: 1Mbps

Other than bandwidth, I am very satisfied. OS reinstallation finishes in less than 10 minutes,
and you can change distributions among fedora, ubuntu, and centos anytime.
Serial console is always available in case of some accident.

Here is a note on how to set up l2tp without ipsec.

0. Networking.

# Enable ip forwarding in the kernel
echo 1 > /proc/sys/net/ipv4/ip_forward
# To make it permanent
vim /etc/sysctl.d/40-ip-forward.conf and add
net.ipv4.ip_forward=1# If you need to use internet via VPS, enable NAPT by running…
sudo iptables -t nat -A POSTROUTING -j MASQUERADE

# You also need to open 1701 udp port if you are behind firewall.

1. Install xl2tpd from the repository.


# For Ubuntu
sudo apt-get install xl2pd
## installed version was xl2tpd-1.3.6, it was not compiled with kernel support.
# For Arch
pacman -S xl2tpd
## installed version was xl2tpd-1.3.6-1

2. Server side setting.(Ubuntu)
Edit Main configuration of xL2TPd
vim /etc/xl2tpd/xl2tpd.conf


[global]
access control = yes
auth file = /etc/ppp/chap-secrets
debug avp = no
debug network = no
debug packet = no
debug state = no
debug tunnel = no
[lns “name of your preference”]
require chap = yes
ppp debug = no
pppoptfile = /etc/ppp/options.l2tpd
require pap = no
assign ip = yes
hostname = hostname # Can be any, i just put hostname of the server machine
ip range = 10.0.0.10 – 10.0.0.20 # IP address range for a tunnel
local ip = 10.0.0.1 # IP address for a tunnel interface (ppp0)
challenge = no
lac = 1.2.3.4 # IP address of client (likely to be global address)
require authentication = no

Edit /etc/ppp/chap-secrets and add username and password for chap authentication.
make sure to chmod 600 /etc/ppp/chap-secrets


# client server secret IP addresses
username * password *

Create /etc/ppp/options.l2tpd for a ppp control


ipcp-accept-local
ipcp-accept-remote
mtu 1410
mru 1410
ms-dns 8.8.8.8
require-mschap-v2
asyncmap 0
auth
crtscts
lock
hide-password
modem
debug
name username # username for chap authentication
proxyarp
lcp-echo-interval 10
lcp-echo-failure 100
connect-delay 5000

3. Client Side (Arch Linux)
Main configuration of xl2tp
vim /etc/xl2tpd/xl2tpd.conf

[global]
access control = no
auth file = /etc/ppp/chap-secrets
debug avp = no
debug network = no
debug packet = no
debug state = no
debug tunnel = no
[lac “lns name you specified on the server”]
lns = “lns ip address (likely to be global address)”
redial = yes
redial timeout = 5
require chap = yes
require authentication = yes
ppp debug = no
pppoptfile = /etc/ppp/options.l2tpd
require pap = no
autodial = yes
name = “username for chap authentication”

Edit /etc/ppp/chap-secrets and add username and password for chap authentication.
make sure to chmod 600 /etc/ppp/chap-secrets


# client server secret IP addresses
username * password *

Create /etc/ppp/options.l2tpd for a ppp control
vim /etc/ppp/options.l2tpd

ipcp-accept-local
ipcp-accept-remote
refuse-eap
require-mschap-v2
noccp
noauth
idle 1800
mtu 1410
mru 1410
defaultroute
usepeerdns
debug
lock
connect-delay 5000
name "username for chap auth"
password "passworf for chap auth"

4. Start the daemon. On both the server and the client, run a command below on both a client and a server, and change routing to your liking

sudo xl2tpd -D
# For split tunneling
ip route add "desired destination ip address" via "ppp peer ip address" dev ppp0
# To route all traffic.
sudo ip route add "Global IP address of your server/32" via 192.168.0.1 dev enp0s25
sudo ip route del default via 192.168.0.1 dev enp0s25
sudo ip route add default via 10.0.0.1 dev ppp0
# Do not forget to re-add your default default gateway after terminating the session.

5. That’s all. This is a pure L2TP VPN tunnel, so there is no encryption involved.
People needing security should consider running ipsec over l2tp, or using openvpn.
If you are paranoid on performance like me, probably this is one of the best solution.

Working with iptables

1. Flushing current iptables
iptables -F or iptables –flush

2. Stating default action for a chain
iptables -P “chain name” “action”
or
iptables –policy “chain name” “action”

For a complete filtering, specify INPUT, OUTPUT, FORWARD for a chaing name.
FOWARD filters NIC to another NIC packets.
e.g.
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP

3. Allow all packets based on interface.
e.g. Allow all packets from loopback interface.
# for incming
iptables -A INPUT -i lo -j ACCEPT
# for outgoing
iptables -A OUTPUT -o lo -j ACCEPT

4. IP address based filtering, you can also combine with interface filtering above.
e.g.
# For incoming
iptables -A INPUT -s “10.0.0.1/24” -j DROP
# For outgoing
iptables -A OUTPUT -d “10.0.0.1/24” -j DROP

5. Port based filtering(-p udp to filter udp). You can also combine with ip address and interface based filtering.
# For incoming
iptables -A OUTPUT -p tcp –dport 22 -j ACCEPT
# For outgoing.
iptables -A INPUT -p tcp –sport 22 -j ACCEPT

6. NAT
Make sure to allow incoming packets to the forwarded port or ip address.
# Destination NAT (Port only, ip address is unaltered )
iptables -t nat -A PREROUTING -p tcp –dport 9999 -j DNAT –to 22
# Source NAT (Port only, ip address is unchanged )
iptables -t nat -A POSTROUTING -p tcp –sport 9991 -j SNAT –to 122

# Destination NAT (IP address only, port unaltered )
iptables -t nat -A PREROUTING -d “10.0.0.1/24” -j DNAT –to “192.168.0.1”
# Source NAT (Port only, ip address is unchanged )
iptables -t nat -A POSTROUTING -d “10.0.0.1/24” -j SNAT –to “192.168.0.1”

# Destination NAT (IP address and Port)
iptables -t nat -A PREROUTING -d “10.0.0.1/24” -p tcp –dport 22-j DNAT –to “192.168.0.1:12222”
# Source NAT (Port only, ip address is unchanged )
iptables -t nat -A POSTROUTING -d “10.0.0.1/24” -p tcp –sport 80 -j SNAT –to “192.168.0.1:18888”

7. Modifying tables
# Specifying position to insert to
iptables -I INPUT “rule number” “some policy”
# Replace some rule
iptables -R OUTPUT “rule number” “some polixy”
# Delete some rule
iptables -D INPUT “rule number”

8. Permitting established connections
iptables -A INPUT -m conntrack –ctstate ESTABLISHED -j ACCEPT

If you want to permit related connections as well,
iptables -A INPUT -m conntrack –ctstate ESTABLISHED,RELATED -j ACCEPT

8. Saving the rule
For CentOS 7.2
/usr/libexec/iptables/iptables.init save

Using ftplib in Python 3

This script download files with timestamp less than argv[1] days old.

#!/usr/bin/python3

from ftplib import FTP
import datetime
from sys import argv

url = ("url")
user = ("anonymous")
password = ("anonymous")

today = datetime.date.today()

ftp = FTP(url)
fs = ftp.login(user, password)
print (fs)
fs = ftp.makepasv()
print (fs)
# Change directory
fs = ftp.cwd("/pub/")
print (fs)

# List directory and storing in varibale "lines".
lines = []
fs = ftp.retrlines('LIST', lines.append)
print (lines)

# Excluding directories from varibale "lines".
list = []
for line in lines:
    if str("d") not in line[0]:
        sline = line.split()
        list.append(sline)
print (list)

# Download only files less than argv[1] days old.
for file in list:
    for daysago in range(int(argv[1])):
        date = today - datetime.timedelta(days=daysago)
        ftpdate = date.strftime("['%b', '%d']")
        if str(ftpdate) == str(file[5:7]):
            ftp.retrbinary("RETR " + file[8], open(file[8], 'wb').write)

Web Scraping with urllib in Python 3

Here is an example of logging in to some website, and get some content.

#!/usr/bin/python3
# Importing modules for handling http and cookie
import http.cookiejar, urllib.request

# Storing cookies in cj variable
cj = http.cookiejar.CookieJar()

# Defining a handler for later http operations with cookies(cj).
op = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))

# Logging in
url = ('https://127.0.0.1/index.php?')
val = {'user' : 'username', 'password' : 'password'}
data = urllib.parse.urlencode(val)
asciidata = data.encode('ascii')
res = opener.open(url, asciidata)

# Saving a file
f = open("content.jpg", "wb")
res = op.open('https://127.0.0.1/index.php/apps/contents.jpg')
f.write(res.read())
f.close()

Web scraping with Python 3 and Selenium

Notes on how to use python module selenium.
Selenium is very useful for automating web browsing tasks.
It is very intuitive and I personally find a lot easier than
phantomjs. If you have X or Xvfb running on your machine,
and know small about javascript, I totally recommend Selenium.

1. Installing the module

Get the source from https://pypi.python.org/pypi/selenium and run
python setup.py install
# If you want to use virtual display install xorg-server-xephyr and pyvirtualdisplay module.

2. Importing the module

#!/usr/bin/python3
from selenium import webdriver
# If you want to use virtual display
from pyvirtualdisplay import Display
xephyr=Display(visible=0, size=(1600, 900)).start()

3. Choosing chrome as a browser to use.

br = webdriver.Chrome()

4. Open a website

br.get('url')

5. Fill a form, and submit it.

# Find the form by name and select it.
el = br.find_element_by_name('inputId')
# Find the form by xpath and select it.
# You can find xpaths of some contents by "inspect element" and right-click and choose "copy xpath" on Chromium.
br.find_element_by_xpath('//*[@id="box"]/ul/li[2]/a/img').click()
# Send words
el.send_keys('words')
# submit it
el.submit()
# or click a button
br.find_element_by_name("certain_name").click()

6. Switching between iframes. A lot of websites these days have frames. When you get an error with find_element, it is likely that wrong frame is chosen.

br.switch_to_default_content()
br.switch_to_frame(1)

7. A lot of websites open a new tab on click, so you have to switch to an active tab. The number is the placement of tab counting from left starting from zero. The example chooses a second tab from left

br.switch_to_window(br.window_handles[1])

8. Finishing the script.

br.quit()
exit()

9. Extras


For clicking certain position in order to click on Flash contents and etc.
# Install pyuserinput module for general mouse and keyboard control. You will need python port of xlib
pip3 install PyUserInput
# Import module
from pymouse import PyMouse
# Clicking
m = PyMouse()
# Get current cursor position
m.position()
# Click x y position.
m.click(x,y)

This isn’t all. I am planning to include some other functions of selenium as I will find out.