Solid state hard drives are the new craze - and for good reason too. They offer a linear access speed regardless of where the data is located on the drive, a improved MTBF (Mean Time Between Failure) and have a lightening fast access/seek time. We are looking at using SSD drives for our new servers and I wondered how reliable they were compared to normal hard drives. For example, would a single SSD drive be more reliable that 2 SATA hard drives in RAID 1? The answer is a definite “no” and by a long shot.
Hourly Reliability
I did a few calculations and plotted two different graphs. The first one is the probabilities of failure within the same hour. It can be assumed that 2 drives dieing in the same hour in a RAID1 array would destory the array.
Daily Reliability
The next graph shows the probability of a drive failing in a 24 hour period.
RAID 1 Failure
It really shows how well RAID1 is against drive failure. 1 SSD versus 2 SATA in RAID1 is no where near as reliable. The reason why there is such a massive difference is that for a RAID1 drive to fail, both drives have to fail within the period of bringing up a replacement drive. In other words, drive 1 and drive 2 need to fail. To calculate this, we bring the probability of a single drive failing to the exponent of the number of drives in the RAID1 array.
RAID 0 Failure
It is interesting to see that the SSD’s in RAID0 have a failure rate less than a single SATA drive. The extra speed gained by SSD’s in RAID0 is quite a small cost in terms of reliability. To calculate the probability of a RAID0 failing it is simply if drive 1 or drive 2 fails.
May 07 2009 | Uncategorized | No Comments »
The host cache in Skype keeps a database of peers that Skype talked to upon last running. A host cache is one of several bootstrapping technologies that peer-to-peer networks use to connect a peer into the overlay network.
The host cache is kept in the shared.xml file located in the users home directory. If you look at how the host cache is stored, it looks like a jumble of hex.
...41C8010500410502004C6E771C823B0001040002B981EDCE043
B981EDCE04000400050041050200180818ADAFD40001040002BBA6
8CCE040003BBA68CCE040004000500410502004A38D3323D990001
040002B981EDCE040003B981EDCE04000400050041050200972FBC
6464420001040002B981EDCE040003B981EDCE0400040005004105
02005169EEF23F930001040002BD81EDCE040003BD81EDCE040004
000500410502005C0F17B769F90001020002C481EDCE040003FA81
EDCE040004000500410502005C4A838623360001020002BB81EDCE
040003FA81EDCE04000400050041050200440AAA7D5EF100010200
02BC81EDCE040003FA81EDCE040004000500410502003AAC048...
A pattern does start to emerge and I wrote a tool to extract the IP address and port of each peer listed in the host cache. It is written in perl and requires the XML::Simple module. You can download the tool here.
I’ve only tested this with 2 different shared.xml files so let me know if you have any problems with it.
May 01 2009 | Scribbles | No Comments »
The Windows XP sysprep tool is quite limited in the hardware configurations that it supports. With a few hacks and tweaks you can successfully deploy the same syspreped image to both Intel and AMD hardware and uni-processor as well as multi-processor CPU’s.
Before starting, there are a few assumptions
- The master hardware is a uni-processor Intel PC
- It is as old as possible (P4 vintage is good if you can get your hands on them, its what I used)
- You are competent using sysprep already and simply want to consolidate on the number of images
Uni/Multi-Processor
This part is easy as there is enough documentation around. Unfortunately I cannot remember exactly where I found the solution so I cannot give credit. Add the following line to your unattended section of sysprep
[unattended]
UpdateHAL="ACPIAPIC_MP,%WINDIR%\Inf\Hal.inf"
Intel and AMD
This is a bit harder to achieve. If you image back an image created on an Intel PC to an AMD PC, it will BSOD before mini-setup runs. This is due to the Intel Power Managment driver that runs only if the PC is an Intel. To get around this, the driver needs to be disabled before the sysprep tool is run on the master and then re-enabled after the image is deployed on the target PC’s only if the PC is Intel. This does not break the Intel image and the driver will be re-enabled on the target PC’s if it is identified as an Intel.
First, you will need to create a batch script that disables the Intel driver and then runs sysprep. Create a file called SYSPREP.BAT with the following and put it in your Sysprep directory (EG: C:\sysprep).
@echo off
cd C:\SYSPREP
echo Enabling image for AMD and Intel processors
reg add "HKLM\SYSTEM\ControlSet001\Services\intelppm" /v Start /t REG_DWORD /d 4 /f
echo Running sysprep and shutting down.
sysprep.exe -forceshutdown -mini -reseal -quiet -activated
Next we need to create a VBScript that will be run after the image has been deployed to the target PC and mini-setup has run. This will check for the CPU manufacturer throught WMI and adjust the Intel driver accordingly. Create a file called checkforintel.vbs and enter the following text. You can also download the script here.
' CheckForIntel.vbs
' Checks if the processor is an Intel and re-enables the power managment driver if it is.
' Written by Ryan D - based off the WMI sample script by Guy Thomas http://computerperformance.co.uk/
'
' ===Version History===
' 1.0 - Initial release
' 1.1 - Changed the way that the script checks for Intel machines. Now it looks at the CPU type and looks
' to see if the string "GenuineIntel" is present.
' --------------------------------------------------------------'
option explicit
const HKEY_LOCAL_MACHINE = &H80000002
dim objWMIService, objItem, colItems, strComputer, compModel, strKeyPath, strValueName, strValue, oReg
strKeyPath = "SYSTEM\ControlSet001\Services\intelppm"
strValueName = "Start"
strValue = "1"
strComputer = "."
' WMI connection to Root CIM and get the computer type
set objWMIService = GetObject("winmgmts:\\" _
& strComputer & "\root\cimv2")
set colItems = objWMIService.ExecQuery(_
"Select Manufacturer from Win32_Processor")
'Loop through the results and store the type in compModel
for each objItem in colItems
compModel = objItem.Manufacturer
next
'Get a registry object
Set oReg=GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & strComputer & "\root\default:StdRegProv")
'Check the computer type. If the processor is an Intel, then re-enable the driver
if compModel = "GenuineIntel" then
oReg.SetDWORDValue HKEY_LOCAL_MACHINE,strKeyPath,strValueName,strValue
else
end if
' Exit
WSCript.Quit
Save this file to C:\WINDOWS\PostGhost\checkforintel.vbs. This needs to be set to automatically run after the target PC has finished mini-setup. Make the following alterations to your sysprep.inf
[Unattended]
OemSkipEula=Yes
UpdateInstalledDrivers=yes
[GuiUnattended]
AdminPassword="adminpasshere"
EncryptedAdminPassword=NO
AutoLogon=Yes
AutoLogonCount=1
OEMSkipRegional=1
TimeZone=255
OemSkipWelcome=1
[GuiRunOnce]
Command0="C:\WINDOWS\PostGhost\checkforintel.vbs"
The admin password and and other settings are to ensure that mini-setup will run unattended. The vital parts of the process are bolded. These settings will make sure that the target PC will auto login and run the script. You may also want to add in another script so that the PC automatically reboots so it is not logged in as an administrator.
Conclusion
After following these steps, you should now have a master image that is able to be deployed on all hardware regardless of CPU brand or type of CPU. I will write a tutorial later which covers the whole sysprep process including:
- Building in mass storage drivers
- Building in drivers for all your hardware platforms
- Debugging drivers for hardware platforms
- Automatically renaming the PC’s based on MAC address
- Automatically joining the Active Directory domain
EDIT: Added the suggestion from Bastian to look for the “GenuineIntel” string instead of AMD computer models so each model doesn’t have to be entered into the script manually. Thanks for the advice : D
February 05 2009 | Windows | 10 Comments »
Background
I bought a 500GB WD MyBook World edition with the intention of using it to backup various servers on my network. I did a bit of research and found out that it ran Linux and also had a pretty decent community following; here and here. It isn’t anywhere near what the NSLU2 had (I was planning to purchase an NSLU2 and found out they were no longer in production) but it was still enough to convince me of my purchase. The plan was to mount the servers file systems locally on the MyBook and then use rsnapshot to take snapshots. The MyBook would sit in my cupboard doing its thing each night backing up files.
After purchasing the box it was pretty quick to enable SSH. I hit a brick wall when I tried to compile the CIFS module on the MyBook. It only had GCC 3.4 and the kernel was compiled with GCC 4.1 which would mean the strings wouldn’t match and wouldn’t load itself in. After many, many frustrating hours setting up a cross compiling ARM toolchain on my laptop, I managed to compile the module and eventially loaded it into the kernel.
Doing it yourself
SSH into the MyBook and issue the following commands.
# wget http://files.doylenet.net/linux/mybook/modules/2.6.17.14/cifs.ko
Make the directory for CIFS and copy it accross
# mkdir /lib/modules/2.6.17.14/kernel/fs/cifs
# cp cifs.ko /lib/modules/2.6.17.14/kernel/fs/cifs/
Now we would normally use depmod to add the module to the modules.dep file and find any dependancies that module requires, but the MyBook doesn’t have it installed and I couldn’t be bothered compiling it, so we can add the line that is required manually. Don’t forget the double “>”’s!!! I cannot stress this enough. If you don’t use >> then the entire file will get overwritten and you will brick your MyBook!
# echo “/lib/modules/2.6.17.14/kernel/fs/cifs/cifs.ko:” >> \
# /lib/modules/2.6.17.14/modules.dep
Now use modprobe to load the module into the kernel
# modprobe cifs
Finally we mount the CIFS share using the mount.cifs program (its part of samba). Add /usr/local/samba/sbin into your PATH if you want to use the “mount -f cifs” style, but the way shown below works fine.
# /usr/local/samba/sbin/mount.cifs //server/share /mnt \
# -o username=someuser,password=somepass
That should be it! Remeber to pass the ro (read only) option if you are using this for backup purposes. And remember, I take NO responsibility if you brick your MyBook. This worked for my 500GB MyBook World. I assume it will work for other models but won’t garuntee anything.
Conclusion
It was painful getting this module to compile so it would load cleanly into the kernel. I had a lot of trouble with buildroot (what the MyBook is based off) but eventially found a version that compiled for me. Ill blog a bit later on getting NFS mounted onto the MyBook as well. I’ve got the module ready but I am having a couple of issues getting nfs-utils to compile.
October 18 2008 | Uncategorized | 6 Comments »
I recently had to replace a failed drive in my Linux server (in fact, the server that this blog is hosted). It is setup as 2 x 200GB PATA hard drives configured in Linux software RAID1.
Once you have identified that the RAID has failed (you will get an email about the event if you have set your server up properly), make sure you have a disk of equal of greater size. I only had a spare 250GB HDD spare, so I used that.
The following commands assume that hdc was the failed drive and that hda is the drive that is still working
# sfdisk -d /dev/hda | sfdisk /dev/hdc
# mdadm /dev/md0 -a /dev/hdc1
# mdadm /dev/md1 -a /dev/hdc2
# mdadm /dev/md2 -a /dev/hdc3
I have 3 partitions, md0 is the boot partition, md1 is swap and md2 is my root partition. Modify your configuration to suit. You can then view the rebuilding by executing
# watch -n .5 ‘cat /proc/mdstat’
You will also want to copy over the boot record so you will be able to boot the server from hdc incase hda fails next. Pretty much every linux uses grub now so I will show how to use that.
# grub
grub> root (hd1,0)
grub> setup (hd1)
grub> quit
And that should be it. That is what I did on my system and it worked fine. That said I don’t take any responsibility for breaking anyones RAID.
October 08 2008 | Uncategorized | No Comments »
I’ve recently completed a small project for Uni; an image based CAPTCHA mechasisim I have called Jaci (Just Another Captcha Implementation). It requires you to drag and drop relevent images onto eachother in order to pass the test. The images are not static and uses Google Image Search for the source of the images.
The main motivation behind the work is a dislike for current CAPTCHA mechasisims. I have good vision and still find the common distorted word captcha frustrating at times. This test is useless for the dislexic and vision impared and creates some accessability problems that will be more and more prevailant as CAPTCHA type mechanisims are intergrated into everyday life. To read more about it, check it out here http://spunce.com/~ryan/jaci/
September 06 2008 | Uncategorized | No Comments »
I recently needed to find a function that would be able to bias random numbers. Out of a set of 1000 random numbers, I wanted more of these to be smaller instead of true random numbers (or as true as random number generators are). I had a look at simple parabolic and exponential functions and eventially devised the following equation.
Where b = factor of bias and c = max. integer of random function. The higher value of b, the more biased the function will be towards lower numbers. c is defined as when f(x) = x. As stated previously, c is the highest possible integer of the random function you are using. If your RNG is generating a maximum number of 1000, then c = 1000.
Shown is a plot of several b values
September 01 2008 | Scribbles | No Comments »
Recently I have witnessed and received some spam that appears to be more resiliant to some filtering techniques. I have noticed this from work as well as mail that I host from home. The email headers look a lot nicer and seem less random.
Spam filtering at work uses a combination on Postfix header checks, RBL’s, SpamAssassin, SFP checks and greylisting. At home I use the same combination except I currently omit greylisting.
Ill give you an example of the spam message I revieved
Salve,
Fuck beer! Got sexy girl?
Click here
Andyou meanthe police will have to be involved? Comprehend?
that which i see, i seei long have for sixteen hours we
halted at eight o'clock a.m. But as soon as they entered
such places, the diamond statues which represent buddha
in his lotus, or eyes on him again. I do hope he wasn't
hurt. Lavinia he said. Things are much worse for jim pearson
next to it is the very handsome fruit garden of rack, or
loin, of mutton, otherways, whole, or seems my weeks of
training these dropout, unemployed, and it hurt dr. Conwell
so much that for ten years and get along. The thing for
you to do is to go me to clear out for a bit till she came
to her bad nervous breakdown. Finally, they said she and
shaves at least once a day. Like most men
This looks like a pretty typical spam email so far. The part that I find interesting is in the headers. I’ll only show the parts that are important.
Received: from oexrk.telecomitalia.it (hostxxx-68-static.89-82-b.business.telecomitalia.it [82.89.68.xxx])
The part that I find interesting is that the SMTP helo was from oexrk.telecomitalia.it and the reverse DNS is host198-68-static.89-82-b.business.telecomitalia.it. It looks like the spam bot is aware of the reverse DNS of the client computer that it has infected and making sure that it appears in the helo. The hostname part of the helo (oexrk) looks to be random characters that are then appended to the domain name. This could potantially trick some spam filtering software into a lower score as the helo is related to the reverse DNS. As well, the spam bot is also aware of greylisting and waits the appropriate length of time. This can be seen on the headers of the spam filtering at my work.
X-Greylist: delayed 306 seconds by postgrey-1.27 at mail.mywork.example.com; Fri, 11 Jul 2008 04:51:35 EST
We recieved several more spams from various other ISP’s that seem to all be infected with the same bot. Below are some more examples of the helo’s that were sent. They all follow a simmilar pattern of 2 levels of the domain name with 4-6 random characters appended as the hostname.
Received: from uezvl.inetia.pl (77-253-25-xxx.adsl.inetia.pl [77.253.25.xxx)
Received: from mmdodz.telecomitalia.it (hostxxx-123-static.23-87-b.business.telecomitalia.it [87.23.123.xxx])
Received: from peiwjh.telecomitalia.it (hostxxx-171-dynamic.16-87-r.retail.telecomitalia.it [87.16.171.xxx])
Received: from edny.telecomitalia.it (hostxxx-155-dynamic.40-79-r.retail.telecomitalia.it [79.40.155.xxx])
July 17 2008 | Linux | 1 Comment »
***UPDATE 11.05.2008***: You will occasionally get errors about temporary directories that are failing to be created properly. To be honest, I don’t know what the cause of this is. It seems that either my setup (more likely) or HAVP is not production ready. If you force reload the page (ctrl + F5) the page should load properly afterwards. Also, after I wrote this tutorial, I realised that there was a HAVP RPM in the Dag repository. If you want to tweak this setup to use the Dag RPM, go ahead. Configuration file locations may vary though, so I cannot guarantee that these instructions will be completely portable.
I have had some trouble with finding a suitable solution for virus scanning using Squid. I tried squidclam which I didn’t have too much success running. There was also a plug-in for DansGuardian that provided virus filtering but I didn’t want the overhead of Dans and only wanted the virus scanning. Most of this guide is taken directly from http://www.opensourcehowto.org/how-to/squid/squid-clamav–havp.html with a few adjustments that were needed for my setup.
This setup assumes that you using CentOS 5 and already have Squid running and installed. If you need to setup Squid, Google is going to be your friend. There are heaps of tutorials around dealing with this. Also, if you run any redirect scripts such as Adzapper, this is fine as we will not be using this functionality of Squid to plug-in, but setting up a cache peer. More on that later.
INSTALLING CLAMAV
If you have not got Dag Wieers repo you will need to add this to the /etc/yum.repos.d/ directory to install ClamAV. Create a file in the yum.repos.d directory, /etc/yum.repos.d/Dag.repo and add the following:
[dag]
name=Dag RPM Repository for Red Hat Enterprise Linux
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag/
gpgcheck=1
gpgkey=http://dag.wieers.com/packages/RPM-GPG-KEY.dag.txt
enabled=1
We will need the development tools to compile HAVP. Install these and ClamAV with the following commands.
yum groupinstall "Development Tools"
yum install clamav clamav-devel
This will take some time. You could probably install the tools that you will need to compile HAVP by hand, but the Development Tools meta-package will alleviate some headaches later on. An updater script will be placed in /etc/cron.daily/freshclam. Modify the frequency of this to your own needs.
Now we will download HAVP and compile it. Head over to http://www.server-side.de/download.htm to get the latest version. 0.87 was the most current at the time of writing.
cd /tmp
wget http://www.server-side.de/download/havp-0.87.tar.gz
tar zxvf havp-0.87.tar.gz
cd havp-0.87
./configure
make
make install
SETTING UP FOR HAVP
This will install most of the files in /usr/local. Before we start messing around with the config file, we need to create a new user account for HAVP to run under and create a few directories/change some permissions.
First we will add the havp user
adduser -c "HAVP proxy virus scanner" -M -s /bin/false havp
Now we need to change the permissions of some directories that HAVP uses
chown -R havp:havp /var/run/havp
chown -R havp:havp /var/log/havp
HAVP requires a directory to store its temporary files. This file system MUST have mandatory locking enabled to function. We can cheat a bit to get around this by creating an image file and then mounting the image on the file-system with locking enabled. For this we will use the command dd.
dd if=/dev/zero of=havp.img count=1 bs=256M
This will create the image, havp.img with a size of 256 MB. Change the size to suit but I feel 256 MB is about enough. Move the image to an appropriate directory, format it and then mount it. This shown below.
mv havp.img /usr/
mke2fs /usr/havp.img (You will get some errors about doing this operation, Just accept them).
mount -o loop,mand /usr/havp.img /usr/tmp/havp
chown -R havp:havp /usr/tmp/havp
We will also want this to mount at boot time so add the following to your /etc/rc.local. I originally added a line in my fstab, but realised this didn’t work when my machine failed to boot properly next time i rebooted it.
mount -o loop,mand /usr/havp.img /var/tmp/havp
CONFIGURE HAVP
Now its time to configure HAVP. Open /usr/local/etc/havp/havp.config in your favorite editor. Everything is commented out. There are a lot of defaults set, but we will take away a lot of the commented out lines just so we know exactly what is going on. Below are all the changes you will want to make.
#REMOVETHISLINE deleteme
USER havp
GROUP havp
DAEMON true
PIDFILE /var/run/havp/havp.pid
SERVERNUMBER 20
MAXSERVERS 100
ACCESSLOG /var/log/havp/access.log
ERRORLOG /var/log/havp/havp.log
LOG_OKS false
LOGLEVEL 0
SCANTEMPFILE /var/tmp/havp/havp-XXXXXX
TEMPDIR /var/tmp
PORT 8080
BIND_ADDRESS 127.0.0.1
SCANIMAGES false (leave this enabled if you have a fast machine)
MAXSCANSIZE 5000000
KEEPBACKBUFFER 200000
ENABLECLAMLIB true
You will only really want to change settings relating to the scan size, buffers and streaming options. Also enable image scanning if you have the grunt.
Save and start HAVP using the init script that it installed
/etc/init.d/havp start
The HAVP init script does not work with chkconfig, so we can just tell it to start upon system boot by entering it in the /etc/rc.local file.
echo "/etc/init.d/havp start" >> /etc/rc.local
Check the logs to make sure that it has started properly. The logs a pretty useful and do give you a good indication to the cause of a problem if you have one. It will most likely be problems with permissions so checking these will be a good start.
CONFIGURE SQUID
Lastly we need to add HAVP as a cache peer of Squid, Open the /etc/squid/squid.conf file and enter the following line.
cache_peer 127.0.0.1 parent 8080 0 no-query no-digest no-netdb-exchange default
TESTING
Now restart Squid. Once Squid as restarted and all calmed down (the AdZapper script initially seems to add quite a delay to Squid responding once it has started), check you can initially access the web. If this is fine, we can now test our virus scanning. A special test signature called “eicar” has been created for the purpose of testing anti-virus programs. Head over to http://www.eicar.org/anti_virus_test_file.htm and try to download one of the files (through HTTP). If all goes well, you should receive a message such as shown below.
If you have got to this point, congratulations. If you are still having trouble, check the Squid logs as well as the HAVP logs. You might want to change the verbosity of the HAVP logs to check exactly what is happening. Remember, this is definitely not bulletproof. Both the fact that ClamAV is not the best engine for detecting viruses as well as the fact that only a certain file size will be scanned.
April 11 2008 | Linux and Uncategorized | No Comments »
It can be a hard decision weather to join a staff laptop to the work domain or not. Easier managment by having the machine on the domian comes at a cost of some functionality of the user; especially if the laptop is a personal computer and not from the compary itself.
The goal was for staff laptops to work just the same as if they were logged into a domain-joined desktop PC. Everything should be seamless to the user and be completely SSO, the same as domain-joined desktop computers. To do this, follow the few steps below:
Navigate to Control Panel > User Accounts. Depending if you use the welcome screen or not, the window that appears may differ to what is shown below. If you are using the welcome screen, click the current user’s name and then click the link “Manage my network passwords”. If you are not using the welcome screen, click on the advanced tab and then “Manage Passwords” button.
Click Add.
Enter in the domain name and then the user name in the form of “domainname\username” and the appropriate password. You may get prompted if you wish to change the password on the domain. Click Cancel if this does occur.
Now try and access a resource on the network. Go to Run and enter in a share name as a test. You shouldn’t need to enter a password in as it will automatically send this for you. This works fine for windows shares and any other NTLM authentication mechanism such as Outlook.
If you run Exchange server and use the webmail client, chances are that it will automatically detect that you are in an intranet zone and automatically send your user name and password. There is one more modification that we will perform to ensure that the stored password gets used for all resources within our domain. This includes webmail when the user is at home. The default settings of Internet Explorer will detect that our example.com domain is an internet domain when we are at home. What we will do is add it as a trusted zone and modify the security template to ensure that it passes the user name and password wherever we are.
Open up Internet Explorer and navigate to Tools > Internet Options > Security (tab) . Click Trusted sites to make sure it is selected and then click the Sites button.
Add the site *.example.com and click Add. You may need to de-select “Require server verification…” if you are not using HTTPS.
Add and close this window and then click the Custom Level button. Once this window opens, scroll right down to the bottom of the list and change “Automatic logon only in Intranet zone” to “Automatic logon with current user name and password”.
Click OK and close this all off. Now you should not need to enter a user name and password every time you go t check you webmail or access any network resources.
SECURITY CONCEARNS
There are some security concearns surrounding this implementation.
- Seeing as staff use these laptops at home and are not managed by IT services, there is a higher risk of them becoming infected malware or the privilages abused by having them stored on the computer. If the staff memeber had another user of the laptop, they should create another use account for this purpose.
- Single point of failure regarding security. Once a villan has the laptop and is logged in, they also can access all their network resources.
- The encrypted NTLM password is sent through HTTP when the user is accessing webmail from home. This isn’t really a major concearn as the password is encrypted and is much better than the basic authentication that NTLM will fall back to if the user doesn’t pass NTLM requests properly.
February 03 2008 | Windows | No Comments »
Next »