Getting the grasp of grep, awk and sed

I’ve been looking at my server logs and been wanting to check which ip’s are hitting my server, both good and bad.

I looked through my logs and one of things I noticed in my maillog was authentication failures.
Dec 3 06:56:08 madbull postfix/smtpd[24006]: warning: unknown[x.x.x.x]: SASL LOGIN authentication failed: UGFzc3dvcmQ6

After a little bit of searching and reading I was able to put together this:

grep 'SASL LOGIN authentication failed' /var/log/maillog | awk '{print $7}' | sed 's/\unknown//g;s/\[//g;s/\]//;s/\://g' | uniq -c | sort

I grep for the term ‘SASL LOGIN authentication failed’ in the file /var/log/maillog, then pipe it into awk that prints out the 7th row in the line. From there it is piped into sed which does a search for unknown, [, ] and :. From there it’s piped into uniq and counted so that I see if there are any repeat offenders. Finally it is piped into sort, so that I have the ip’s with the most hits at the bottom.

The grep command is self explanatory, so I’ll jump in and explain what awk does. In the curly braces it is told to print out the 7th row and if you count the numbers of rows that are in the line from the log you will see that Dec is one, 3 is two, 06:56:08 is three and so on. It looks for spaces, so the whole postfix/smtpd[24006]: is one row. So we get unknown[x.x.x.x]: as our new string to pass along.

Turn to sed and get some substitution going :D. The ‘s says that we want to substitute the text that matches our search term after the delimiter /, the \ in that part is there to show the character / (there are other delimiters also). We are looking for the text unknown so we set up the unknown string as the word we are searching for. After the second slash / we can write what we want it to substitute with, for instance sed ‘s/\unknown/known/’ would change unknown to known. The last slash ends the substitution and g is the Global flag. Is says that if it occurs more than one time on a line, substitute that too. So if there where 10 unknown strings, they all would be deleted since i have nothing in between the second and the third slash. The rest of the sed line is just more of the same.

uniq -c put every instance of a matching string together, so that if I had ten instances of 192.168.1.1 in there with uniq it would show up as just one, but with the -c flag it would also count them and give you the line 10 192.168.1.1, and that is a nice way to see if there is a repeat offender knocking at our door.

The sort is just for letting me see the ones with the highest number at the bottom.

Man (the manual command in linux) is your friend when it comes to explaining more of what the different commands does. I’m reading up on them myself so that I can understand better what is going on under the hood.

For some more sed magic: Sed

Virtualization on my Dell XPS 15 – 9530

So I’ve tried virtualization on this machine before and everything ran like it was stuck i syrup. I was trying to setup Kali Linux to do some pentests on my network. I have a router from the isp and was not sure if they where setup with WPS protection and if I was able to actually turn it of.

First I checked the bios to see if there was an option for virtual machines and sure thing, it has an option to turn virtualization on or off. But no more than that. You can’t turn features on or off beyond that.

I found a great site with a great way to check what support you have and also what the different things mean linux-xen-vmware-kvm-intel-vt-amd-v-support

So I ran:

egrep -wo 'vmx|lm|aes' /proc/cpuinfo | sort | uniq\ | sed -e 's/aes/Hardware encryption=Yes (&)/g' \ -e 's/lm/64 bit cpu=Yes (&)/g' -e 's/vmx/Intel hardware virtualization=Yes (&)/g'

and got:

Hardware encryption=Yes (aes)
64 bit cpu=Yes (lm)
Intel hardware virtualization=Yes (vmx)

Score!!

So then I checked for others also, just to see what my system is capable of. I found EPT, VPID, TPR_SHADOW and VNMI. I then moved on the check if my install of Fedora 24 had the right packages installed and if there where support for it in the kernel. A quick dmesg with a little grep showed me that I had no support in the kernel πŸ˜›

I ran this and got nothing..:

dmesg | grep -i kvm

Then I was off to google.com again and searched for how to setup virtualization in Fedora 24. Getting_started_with_virtualization

But I also searched for VirtualBox since I’ve tried that on other machines and I like the interface. I also tried it on this machine once before and it was sloooooooooooow. So before I do anything else I’m going to try and follow the installation for VirtualBox and see if it does what I need. install-virtualbox-with-yum-on-fedora-centos-red-hat-rhel/


cd /etc/yum.repos.d
sudo wget http://download.virtualbox.org/virtualbox/rpm/fedora/virtualbox.repo

sudo dnf update <-- Here I get a question if I want to accept the key for the repo.

sudo dnf install binutils gcc make patch libgomp glibc-headers glibc-devel kernel-headers kernel-devel dkms


sudo dnf install VirtualBox-5.1


sudo /usr/lib/virtualbox/vboxdrv.sh setup <-- This fails..


sudo usermod -a -G vboxusers myuser

The reason why the setup with vboxdrv.sh fails is that this laptop has secure boot. But since I have been building kernel modules on this before for my wifi dongle it is easy to get the driver signed.


/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo -n vboxdrv)

modprobe vboxdrv

As it stands now, it looks like my problem all along was the missing kernel module. All the kvm stuff is for Fedora's own way of doing virtualization. I might test that later also, but VirtualBox works fine for me πŸ™‚

First mesh test

I installed the package batctl on my desktop which have a 2.4 ghz wifi card and on one of my laptops. After running a modprobe for the batman-adv module everything was set for the test to begin.

I followed the quick setup from: Batman-adv quick start guide

I ran:

ip link set mtu 1532 dev wlan0
iwconfig wlan0 mode ad-hoc essid my-mesh-network ap 02:12:34:56:78:9A channel 1

batctl if add wlan0
ip link set up dev wlan0

After that was done I tried checking if I had any neighbors on the network.

batctl n

This gave me the other host that I set up and I tried to run the same command on the other machine and it gave me the first system. Success. They’re at least on the same network since they can see each other. I then tried to setup an ip on the bat0 interface. But I had no luck getting it to ping each other. I will try again and read up some more on the setups.

A deeper dive into mesh

Wifi has different modes it can use where the most used are managed, master and monitor.
From iwconfig man pages:

mode Set the operating mode of the device, which depends on the net‐
work topology. The mode can be Ad-Hoc (network composed of only
one cell and without Access Point), Managed (node connects to a
network composed of many Access Points, with roaming), Master
(the node is the synchronisation master or acts as an Access
Point), Repeater (the node forwards packets between other wire‐
less nodes), Secondary (the node acts as a backup mas‐
ter/repeater), Monitor (the node is not associated with any cell
and passively monitor all packets on the frequency) or Auto.
Example :
iwconfig eth0 mode Managed
iwconfig eth0 mode Ad-Hoc

Managed mode is where your wifi card is used to connect to your access point and surf the web, browse your samba shares and so on. This is what regular users use it for and all wifi cards support this mode.

Master mode is where your wifi card is set up as an access point and shares resources. This can be a gateway or just a hub for a service you want to provide. Not all drivers/wifi cards support it. Some cards flat out don’t support it and others just need an driver update for it to work.

Monitor mode is used for testing purposes. Not all drivers support this but for the most part I’ve never encountered a card that doesn’t support it.

Ad-Hoc mode is where a wifi card is used to connect two systems (laptop, desktop etc.) together without the use of a access point. The problem with this setup is that you only get the routing that is setup for that network. Let’s say you have a Ad-Hoc network of 5 systems where two of them has internet access. If the route is setup so that system 1 is the gateway and system 1 goes down the others loose their path out to the internet. To rectify this problem you must change the route settings to the other system that has internet access. When system 1 is online again the route is still pointing to the other system. Some operating systems also support just one Ad-Hoc connection so there can only be two computers in the network.

This is where Mesh networking comes in to play. It can be seen as an Ad-Hoc system with a lot of masters but none of them have actual control or authoritative control of the network. A mesh network tries to route the packages dynamically by choosing which route it thinks is the best. If one host goes down the network updates and tries to route around that missing host and when it goes online again the route is reestablished. The biggest problem with a mesh network is that is has to do a lot of link checking to be sure that it’s neighbors are online and where they point to so you get some bandwidth use that is not actual user traffic. The bigger the network, the bigger the bandwidth usage and congestion.

In the first post in the mesh section I mentioned Babel and Batman. There is also OSLR, which stand for Optimized Link State Routing. I think I’ll be trying out Batman first and see if I can get a mesh net up at least.

From what I have read and if I understand everything, Batman used to be in userspace and worked on layer three of the OSI model. But now it is in the kernel and runs on layer two.
Copied from the batman-adv wiki:

Most other wireless routing protocol implementations (e.g. the batman daemon) operate on layer 3 which means they exchange routing information by sending UDP packets and bring their routing decision into effect by manipulating the kernel routing table. Batman-adv operates entirely on ISO/OSI Layer 2 - not only the routing information is transported using raw ethernet frames but also the data traffic is handled by batman-adv. It encapsulates and forwards all traffic until it reaches the destination, hence emulating a virtual network switch of all nodes participating. Therefore all nodes appear to be link local and are unaware of the network's topology as well as unaffected by any network changes.
This design bears some interesting characteristics:

network-layer agnostic - you can run whatever you wish on top of batman-adv: IPv4, IPv6, DHCP, IPX ..
nodes can participate in a mesh without having an IP
easy integration of non-mesh (mobile) clients (no manual HNA fiddling required)
roaming of non-mesh clients
optimizing the data flow through the mesh (e.g. interface alternating, multicast, forward error correction, etc)
running protocols relying on broadcast/multicast over the mesh and non-mesh clients (Windows neighborhood, mDNS, streaming, etc

I’ll do an install on both laptops that I’m going to try it on and see where it gets me πŸ˜€

Mesh networking on Fedora 24

I’ve been looking at mesh networks for a while but never done any testing. But at least I have begun reading.

B.A.T.M.A.N-adv is one of the protocols I’ve been looking at and it stands for Better Approach To Mobile Ad-hoc Networking. Clever name and they have a nice quick start guide. Batman-adv quick start guide

Batman-adv is in the fedora repo’s and the configuration tool is called batctl. A snippet from dnf info:

dnf info batctl
Last metadata expiration check: 26 days, 6:34:57 ago on Sat Aug 13 13:47:26 2016.
Available Packages
Name : batctl
Arch : x86_64
Epoch : 0
Version : 2016.0
Release : 2.fc24
Size : 51 k
Repo : fedora
Summary : B.A.T.M.A.N. advanced control and management tool
URL : http://www.open-mesh.org/
License : GPLv2
Description : batctl offers a convenient way to configure the batman-adv kernel
: module as well as displaying debug information such as originator
: tables, translation tables and the debug log. In combination with
: a bat-hosts file batctl allows the use of host names instead of
: MAC addresses.
:
: B.A.T.M.A.N. advanced operates on layer 2. Thus all hosts
: participating in the virtual switched network are transparently
: connected together for all protocols above layer 2. Therefore the
: common diagnosis tools do not work as expected. To overcome these
: problems batctl contains the commands ping, traceroute, tcpdump
: which provide similar functionality to the normal ping(1),
: traceroute(1), tcpdump(1) commands, but modified to layer 2
: behavior or using the B.A.T.M.A.N. advanced protocol.

Babel is another and I’ve just started reading about it. I’m going to google some more and see what I find.
But I’ve found a great link to start at: Babel mesh

This package is also in Fedora 24 repo’s. A snippet from dnf info:

dnf info babeld
Last metadata expiration check: 26 days, 6:47:35 ago on Sat Aug 13 13:47:26 2016.
Available Packages
Name : babeld
Arch : x86_64
Epoch : 0
Version : 1.7.1
Release : 1.fc24
Size : 90 k
Repo : fedora
Summary : Ad-hoc network routing daemon
URL : http://www.pps.univ-paris-diderot.fr/~jch/software/babel/
License : MIT
Description : Babel is a loop-avoiding distance-vector routing protocol roughly
: based on HSDV and AODV, but with provisions for link cost
: estimation and redistribution of routes from other routing
: protocols.

I have a couple of laptops and some desktop/servers that I’m not using for anything particular, so I’ll get fedora on to them and run some tests. I might get other distros later but for testing Fedora is more than good enough, it might be more than enough for a production network also.

When I’ve got some tests done I’m going to try to make a bigger mesh-net here where I live. I’m also looking at making myself some more powerful antennas and I’ve been looking at Andrew Mcneil’s youtube channel and I would really like to build some of his designs. Link to his channel: Andrew

A kernel update and my 8812ua

My laptop is running Fedora 24 and today there was updates for the system and one of the updates was a new kernel. I was looking forward to see if the module for my Realtek usb wifi dongle got updated also. Too check for this I run tree.


[root@Threadstone ~]# tree /var/lib/dkms/8812au/
4.3.14_13455.20150212/ kernel-4.7.2-201.fc24.x86_64-x86_64/
kernel-4.6.7-300.fc24.x86_64-x86_64/
[root@Threadstone ~]# tree /var/lib/dkms/8812au/
/var/lib/dkms/8812au/
β”œβ”€β”€ 4.3.14_13455.20150212
β”‚Β Β  β”œβ”€β”€ 4.6.7-300.fc24.x86_64
β”‚Β Β  β”‚Β Β  └── x86_64
β”‚Β Β  β”‚Β Β  β”œβ”€β”€ log
β”‚Β Β  β”‚Β Β  β”‚Β Β  └── make.log
β”‚Β Β  β”‚Β Β  └── module
β”‚Β Β  β”‚Β Β  └── 8812au.ko
β”‚Β Β  β”œβ”€β”€ 4.7.2-201.fc24.x86_64
β”‚Β Β  β”‚Β Β  └── x86_64
β”‚Β Β  β”‚Β Β  β”œβ”€β”€ log
β”‚Β Β  β”‚Β Β  β”‚Β Β  └── make.log
β”‚Β Β  β”‚Β Β  └── module
β”‚Β Β  β”‚Β Β  └── 8812au.ko
β”‚Β Β  └── source -> /usr/src/8812au-4.3.14_13455.20150212
β”œβ”€β”€ kernel-4.6.7-300.fc24.x86_64-x86_64 -> 4.3.14_13455.20150212/4.6.7-300.fc24.x86_64/x86_64
└── kernel-4.7.2-201.fc24.x86_64-x86_64 -> 4.3.14_13455.20150212/4.7.2-201.fc24.x86_64/x86_64

12 directories, 4 files
[root@Threadstone ~]# uname -a
Linux Threadstone 4.7.2-201.fc24.x86_64 #1 SMP Fri Aug 26 15:58:40 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

As one can see the running kernel is 4.7.2-201 and that is also the newest kernel on Fedora 24. The module 8812au.ko is built against that same kernel as is shown in the tree. So I try to load it.. but get an error.


[root@Threadstone ~]# modprobe 8812au
modprobe: ERROR: could not insert '8812au': Required key not available

So the problem is that the module is not signed against the new kernel. I generated keys for the signing when I first built the module but for some reason I can’t find them. So I’ll build new ones.


[root@Threadstone ~]# mkdir keys
[root@Threadstone ~]# cd keys/
[root@Threadstone keys]# ls
[root@Threadstone keys]# openssl req -new -x509 -newkey rsa:2048 -keyout MOK.priv -outform DER -out MOK.der -nodes -days 36500 -subj "/CN=ardal/"
Generating a 2048 bit RSA private key
...........................................+++
.................................................................................................................................+++
writing new private key to 'MOK.priv'
-----
[root@Threadstone keys]# ls
MOK.der MOK.priv

So now the keys are in a directory that I know where is for the next time I need to sign the module. So then it’s time to sign the module and get it up and running again.


[root@Threadstone keys]# sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo -n 8812au)

[root@Threadstone keys]# mokutil --import MOK.der
input password:
input password again:

From there a reboot is needed since it’s needed to get it into uefi secure boot. After the computer has booted up I check if it’s in the listing when running iwconfig.


[root@Threadstone ~]# iwconfig

wlp0s20u4 unassociated Nickname:""
Mode:Managed Frequency=2.412 GHz Access Point: Not-Associated
Sensitivity:0/0
Retry:off RTS thr:off Fragment thr:off
Encryption key:off
Power Management:off
Link Quality:0 Signal level:0 Noise level:0
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:0 Invalid misc:0 Missed beacon:0

Since I didn’t find the MOK.priv and the MOK.der files that I used last time I couldn’t use those for signing again. Next time I need to do this I have the keys in a known location, so it’s just the signing part that should be necessary.

Updated the site with Lets Encrypt

I like to use secure communication as much as possible since we never know what new threats are out on the web. I don’t think I’ve got the skill set to battle of the best hackers, but stopping at least the script kiddies and such should be possible.

So first I ran a update so that I was sure everything is good to go.

sudo yum update

I then went and install git so that I can download the letsencrypt software.

sudo yum install git

I then downloaded letsencrypt and setup apache for ssl.

sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt

cd /opt/letsencrypt

./letsencrypt-auto --apache -d example.com -d www.example.com

You get some questions and one of them are if you want http and https or redirect to https. I chose redirect since I want the site to always use https.

Letsencrypt puts the files in /etc/httpd/sites-enabled so I moved the to sites-avaliable and made a symlink to enabled instead.

After a restart of httpd the site is up and running with https πŸ™‚

I then ran a test of the server on ssl labs ssl test.
It complained about SSLv3, so I had to fix that.

So I located ssl.conf under /etc/httpd/conf.d and located

SSLProtocol all -SSLv2

and added -SSLv3 to it.

Kurt

Wifi and other stuff

So.. I bought myself a usb wifi adapter with detachable antenna so that I can test different antennas and maybe try to build some of my own. I went on ebay and found a cheap one and thought everything was going to be all good since a lot of wifi chipsets are supported in linux..

Well… not mine, not in the kernel atleast. I was lucky to find a blog post with my chipset: http://dustymabe.com/2016/01/24/802-11ac-on-linux-with-netgear-a6100-rtl8811au-usb-adapter/
It also built the module with dkms which is a great way to create loadable modules. My problem now was that I got an error when trying to load the module. I run Fedora 24 on a Dell XPS 15″ (9530) and using uefi secure-boot, so I need to sign the module.

I went to google and I hit gold again. I found a blog from a guy that had to sign a module for vbox and vmware. I read his post and adapted it to what I needed which was the signing part of the post: http://gorka.eguileor.com/vbox-vmware-in-secureboot-linux/

After a reboot I got a boot screen asking me to go through my new keys. I found the new key and approved it and SUCCESS, the driver get loaded when I insert the usb wifi dongle. Great πŸ˜€ I might do a complete writeup some time later with every step I did since I’m most likely going to do this on another laptop and maybe on my desktop.

Kurt