Archive for the 'Uncategorized' Category
Its long title I know. While doing some other related personal research, I wrote this excerpt:
We can also look at the randomness of a domain name. For example ajksudmapx.com looks to be random for a human. To reverse randomness would be similarly as difficult as reversing a hashing algorithm. We could look at how domain names are spelt. We could look at the distribution of letters, assuming
You can see that I didn’t finish the sentence. I put what I was working on hold and thought I would investigate the possibility of detecting randomly-generated domain names. It is a bit rough and not completely polished, but I would have never gotten around to putting it online otherwise.
Get the PDF >>here<<
June 12 2010 | Uncategorized | No Comments »
Solid state hard drives are the new craze – and for good reason too. They offer a linear access speed regardless of where the data is located on the drive, a improved MTBF (Mean Time Between Failure) and have a lightening fast access/seek time. We are looking at using SSD drives for our new servers and I wondered how reliable they were compared to normal hard drives. For example, would a single SSD drive be more reliable that 2 SATA hard drives in RAID 1? The answer is a definite “no” and by a long shot.
Hourly Reliability
I did a few calculations and plotted two different graphs. The first one is the probabilities of failure within the same hour. It can be assumed that 2 drives dieing in the same hour in a RAID1 array would destory the array.
Daily Reliability
The next graph shows the probability of a drive failing in a 24 hour period.
RAID 1 Failure
It really shows how well RAID1 is against drive failure. 1 SSD versus 2 SATA in RAID1 is no where near as reliable. The reason why there is such a massive difference is that for a RAID1 drive to fail, both drives have to fail within the period of bringing up a replacement drive. In other words, drive 1 and drive 2 need to fail. To calculate this, we bring the probability of a single drive failing to the exponent of the number of drives in the RAID1 array.
RAID 0 Failure
It is interesting to see that the SSD’s in RAID0 have a failure rate less than a single SATA drive. The extra speed gained by SSD’s in RAID0 is quite a small cost in terms of reliability. To calculate the probability of a RAID0 failing it is simply if drive 1 or drive 2 fails.
May 07 2009 | Uncategorized | No Comments »
Background
I bought a 500GB WD MyBook World edition with the intention of using it to backup various servers on my network. I did a bit of research and found out that it ran Linux and also had a pretty decent community following; here and here. It isn’t anywhere near what the NSLU2 had (I was planning to purchase an NSLU2 and found out they were no longer in production) but it was still enough to convince me of my purchase. The plan was to mount the servers file systems locally on the MyBook and then use rsnapshot to take snapshots. The MyBook would sit in my cupboard doing its thing each night backing up files.
After purchasing the box it was pretty quick to enable SSH. I hit a brick wall when I tried to compile the CIFS module on the MyBook. It only had GCC 3.4 and the kernel was compiled with GCC 4.1 which would mean the strings wouldn’t match and wouldn’t load itself in. After many, many frustrating hours setting up a cross compiling ARM toolchain on my laptop, I managed to compile the module and eventially loaded it into the kernel.
Doing it yourself
SSH into the MyBook and issue the following commands.
# wget http://files.doylenet.net/linux/mybook/modules/2.6.17.14/cifs.ko
Make the directory for CIFS and copy it accross
# mkdir /lib/modules/2.6.17.14/kernel/fs/cifs
# cp cifs.ko /lib/modules/2.6.17.14/kernel/fs/cifs/
Now we would normally use depmod to add the module to the modules.dep file and find any dependancies that module requires, but the MyBook doesn’t have it installed and I couldn’t be bothered compiling it, so we can add the line that is required manually. Don’t forget the double “>”’s!!! I cannot stress this enough. If you don’t use >> then the entire file will get overwritten and you will brick your MyBook!
# echo "/lib/modules/2.6.17.14/kernel/fs/cifs/cifs.ko:" >> \
# /lib/modules/2.6.17.14/modules.dep
Now use modprobe to load the module into the kernel
# modprobe cifs
Finally we mount the CIFS share using the mount.cifs program (its part of samba). Add /usr/local/samba/sbin into your PATH if you want to use the “mount -f cifs” style, but the way shown below works fine.
# /usr/local/samba/sbin/mount.cifs //server/share /mnt \
# -o username=someuser,password=somepass
That should be it! Remeber to pass the ro (read only) option if you are using this for backup purposes. And remember, I take NO responsibility if you brick your MyBook. This worked for my 500GB MyBook World. I assume it will work for other models but won’t garuntee anything.
Conclusion
It was painful getting this module to compile so it would load cleanly into the kernel. I had a lot of trouble with buildroot (what the MyBook is based off) but eventially found a version that compiled for me. Ill blog a bit later on getting NFS mounted onto the MyBook as well. I’ve got the module ready but I am having a couple of issues getting nfs-utils to compile.
October 18 2008 | Uncategorized | 25 Comments »
I recently had to replace a failed drive in my Linux server (in fact, the server that this blog is hosted). It is setup as 2 x 200GB PATA hard drives configured in Linux software RAID1.
Once you have identified that the RAID has failed (you will get an email about the event if you have set your server up properly), make sure you have a disk of equal of greater size. I only had a spare 250GB HDD spare, so I used that.
The following commands assume that hdc was the failed drive and that hda is the drive that is still working
# sfdisk -d /dev/hda | sfdisk /dev/hdc
# mdadm /dev/md0 -a /dev/hdc1
# mdadm /dev/md1 -a /dev/hdc2
# mdadm /dev/md2 -a /dev/hdc3
I have 3 partitions, md0 is the boot partition, md1 is swap and md2 is my root partition. Modify your configuration to suit. You can then view the rebuilding by executing
# watch -n .5 ‘cat /proc/mdstat’
You will also want to copy over the boot record so you will be able to boot the server from hdc incase hda fails next. Pretty much every linux uses grub now so I will show how to use that.
# grub
grub> root (hd1,0)
grub> setup (hd1)
grub> quit
And that should be it. That is what I did on my system and it worked fine. That said I don’t take any responsibility for breaking anyones RAID.
October 08 2008 | Uncategorized | No Comments »
I’ve recently completed a small project for Uni; an image based CAPTCHA mechasisim I have called Jaci (Just Another Captcha Implementation). It requires you to drag and drop relevent images onto eachother in order to pass the test. The images are not static and uses Google Image Search for the source of the images.
The main motivation behind the work is a dislike for current CAPTCHA mechasisims. I have good vision and still find the common distorted word captcha frustrating at times. This test is useless for the dislexic and vision impared and creates some accessability problems that will be more and more prevailant as CAPTCHA type mechanisims are intergrated into everyday life. To read more about it, check it out here http://ryandoyle.net/devel/jaci
September 06 2008 | Uncategorized | No Comments »
***UPDATE 11.05.2008***: You will occasionally get errors about temporary directories that are failing to be created properly. To be honest, I don’t know what the cause of this is. It seems that either my setup (more likely) or HAVP is not production ready. If you force reload the page (ctrl + F5) the page should load properly afterwards. Also, after I wrote this tutorial, I realised that there was a HAVP RPM in the Dag repository. If you want to tweak this setup to use the Dag RPM, go ahead. Configuration file locations may vary though, so I cannot guarantee that these instructions will be completely portable.
I have had some trouble with finding a suitable solution for virus scanning using Squid. I tried squidclam which I didn’t have too much success running. There was also a plug-in for DansGuardian that provided virus filtering but I didn’t want the overhead of Dans and only wanted the virus scanning. Most of this guide is taken directly from http://www.opensourcehowto.org/how-to/squid/squid-clamav–havp.html with a few adjustments that were needed for my setup.
This setup assumes that you using CentOS 5 and already have Squid running and installed. If you need to setup Squid, Google is going to be your friend. There are heaps of tutorials around dealing with this. Also, if you run any redirect scripts such as Adzapper, this is fine as we will not be using this functionality of Squid to plug-in, but setting up a cache peer. More on that later.
INSTALLING CLAMAV
If you have not got Dag Wieers repo you will need to add this to the /etc/yum.repos.d/ directory to install ClamAV. Create a file in the yum.repos.d directory, /etc/yum.repos.d/Dag.repo and add the following:
[dag]
name=Dag RPM Repository for Red Hat Enterprise Linux
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag/
gpgcheck=1
gpgkey=http://dag.wieers.com/packages/RPM-GPG-KEY.dag.txt
enabled=1
We will need the development tools to compile HAVP. Install these and ClamAV with the following commands.
yum groupinstall "Development Tools"
yum install clamav clamav-devel
This will take some time. You could probably install the tools that you will need to compile HAVP by hand, but the Development Tools meta-package will alleviate some headaches later on. An updater script will be placed in /etc/cron.daily/freshclam. Modify the frequency of this to your own needs.
Now we will download HAVP and compile it. Head over to http://www.server-side.de/download.htm to get the latest version. 0.87 was the most current at the time of writing.
cd /tmp
wget http://www.server-side.de/download/havp-0.87.tar.gz
tar zxvf havp-0.87.tar.gz
cd havp-0.87
./configure
make
make install
SETTING UP FOR HAVP
This will install most of the files in /usr/local. Before we start messing around with the config file, we need to create a new user account for HAVP to run under and create a few directories/change some permissions.
First we will add the havp user
adduser -c "HAVP proxy virus scanner" -M -s /bin/false havp
Now we need to change the permissions of some directories that HAVP uses
chown -R havp:havp /var/run/havp
chown -R havp:havp /var/log/havp
HAVP requires a directory to store its temporary files. This file system MUST have mandatory locking enabled to function. We can cheat a bit to get around this by creating an image file and then mounting the image on the file-system with locking enabled. For this we will use the command dd.
dd if=/dev/zero of=havp.img count=1 bs=256M
This will create the image, havp.img with a size of 256 MB. Change the size to suit but I feel 256 MB is about enough. Move the image to an appropriate directory, format it and then mount it. This shown below.
mv havp.img /usr/
mke2fs /usr/havp.img (You will get some errors about doing this operation, Just accept them).
mount -o loop,mand /usr/havp.img /usr/tmp/havp
chown -R havp:havp /usr/tmp/havp
We will also want this to mount at boot time so add the following to your /etc/rc.local. I originally added a line in my fstab, but realised this didn’t work when my machine failed to boot properly next time i rebooted it.
mount -o loop,mand /usr/havp.img /var/tmp/havp
CONFIGURE HAVP
Now its time to configure HAVP. Open /usr/local/etc/havp/havp.config in your favorite editor. Everything is commented out. There are a lot of defaults set, but we will take away a lot of the commented out lines just so we know exactly what is going on. Below are all the changes you will want to make.
#REMOVETHISLINE deleteme
USER havp
GROUP havp
DAEMON true
PIDFILE /var/run/havp/havp.pid
SERVERNUMBER 20
MAXSERVERS 100
ACCESSLOG /var/log/havp/access.log
ERRORLOG /var/log/havp/havp.log
LOG_OKS false
LOGLEVEL 0
SCANTEMPFILE /var/tmp/havp/havp-XXXXXX
TEMPDIR /var/tmp
PORT 8080
BIND_ADDRESS 127.0.0.1
SCANIMAGES false (leave this enabled if you have a fast machine)
MAXSCANSIZE 5000000
KEEPBACKBUFFER 200000
ENABLECLAMLIB true
You will only really want to change settings relating to the scan size, buffers and streaming options. Also enable image scanning if you have the grunt.
Save and start HAVP using the init script that it installed
/etc/init.d/havp start
The HAVP init script does not work with chkconfig, so we can just tell it to start upon system boot by entering it in the /etc/rc.local file.
echo "/etc/init.d/havp start" >> /etc/rc.local
Check the logs to make sure that it has started properly. The logs a pretty useful and do give you a good indication to the cause of a problem if you have one. It will most likely be problems with permissions so checking these will be a good start.
CONFIGURE SQUID
Lastly we need to add HAVP as a cache peer of Squid, Open the /etc/squid/squid.conf file and enter the following line.
cache_peer 127.0.0.1 parent 8080 0 no-query no-digest no-netdb-exchange default
TESTING
Now restart Squid. Once Squid as restarted and all calmed down (the AdZapper script initially seems to add quite a delay to Squid responding once it has started), check you can initially access the web. If this is fine, we can now test our virus scanning. A special test signature called “eicar” has been created for the purpose of testing anti-virus programs. Head over to http://www.eicar.org/anti_virus_test_file.htm and try to download one of the files (through HTTP). If all goes well, you should receive a message such as shown below.
If you have got to this point, congratulations. If you are still having trouble, check the Squid logs as well as the HAVP logs. You might want to change the verbosity of the HAVP logs to check exactly what is happening. Remember, this is definitely not bulletproof. Both the fact that ClamAV is not the best engine for detecting viruses as well as the fact that only a certain file size will be scanned.
April 11 2008 | Linux and Uncategorized | 2 Comments »
EDIT: I found an article describing this scenario avaliable on the open SPF website, here
Sender Policy Framework is the RFC the defines an authorisation system for who is allowed to send email from a particular domain. It works by publishing TXT records in the root DNS for that domain that follow a correct syntax. This TXT record defines what servers are allowed to send mail from that domain. You can specify a network range, individual servers, hosts that are also listed as MX servers and so on. Its a good system that is slowly being adapted in an effort to reduce spam.
Another method that relies on the cooperation of ISPs is blocking outgoing port 25 outside of the ISP network. This means that infected zombie PC’s of DSL and cable subscribers wont be able to send mail out of the ISP network as this traffic will be blocked. To send legitimate mail, the user needs to setup their mail client to send mail via the ISPs SMTP server, which will then send it out to the rest of the world.
Therein lies a problem. We are combatting spam by blocking any outgoing traffic from port 25, but at the same time we are defining what servers can send mail from a particular domain. Say you have hundreds of remote users behind different ISPs, what do you do then? Do you manually add ALL of these servers to the allowed list? Unforunately there are many drawbacks of the SMTP protocol. It was drafted back in 1982 for a start, and really hasn’t seen any improvements to fighting spam over the years. The problem is, you cannot just change everything over in a day. It takes years and years to be adopted. Hopefully we can find patches to the current problems and ensure that any new standards have digital signing built in by default.
A few practical solutions to these problems would for the user to use a webmail client or perhaps VPN into the workplace (assuming we are talking a work environement) and use the company’s SMTP server.
Another technology called “Domain Keys” is simmilar to SPF in its operation to verify that the mail has been sent from a domain that is authorised to send. How it works is that the sent mail is digitally signed and the public key is avaliable through a DNS record. One small drawback is that this method requires inspection into the body of the message to get the domain key information, meaning more bandwidth is wasted on spam email.
SPF and DKIM only really addresses domain-wide authentication as to who (which servers) are allowed to send email. Anyone can technically send mail from anyone elses email address. There is no security as to who is allowed to send. There is no authentication stage for the user to send mail. For SMTP servers that are SASL aware and require a user name and password before the mail is relayed, only checks for a valid user. It does not check the sender of the email and make sure that the username and password entered is the same username that is authorised to send from that particular address. PGP (Pretty Good Privacy) and S/MIME can be implemented to sign mail to ensure that it did originate from the actual sender, but this is not widely adopted.
December 13 2007 | Uncategorized | 2 Comments »
Hi
I Thought I should post a quick description of what this site is about. Here I’ll post day to day tips and tricks and just keep a log of the system admin work I do. There are a few things that I’ll document which I have found little complete documentation on. Recently I have had heaps of trouble with NTLM and all the different implementations scattered all over the Internet.
July 05 2007 | Uncategorized | No Comments »