Wednesday, 17 April 2013
Backing up QNAP Server with CentOS Rsnapshot Server
Webmin
The CentOS server has Webmin installed for easy configuration. Access to webmin is trough port 10000.
QNAP shares work around
For backup up the QNAP the internal Rsync server is enabled and a username and password is set. To prevent the necessity of having to add all shares of the QNAP manually, and updating them for each new future share, we added a special share. This trick is simple, create a new share and manually add the path. This path should simply be "/", this way this share will hold all shares on the QNAP, including the Time Machine shares and others.
Rsnapshot Configuration
Next up was configuring the rsnapshot.conf on the CentOS machine. Contents of config file: (used cat /etc/rsnapshot.conf | grep -v -e ^# to get this output btw)
(ALL SPACES MUST BE TABS!!)
config_version 1.2
snapshot_root /backups/
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
cmd_ssh /usr/bin/ssh
cmd_logger /usr/bin/logger
cmd_du /usr/bin/du
interval hourly 6
interval daily 7
interval weekly 4
verbose 2
loglevel 3
logfile /var/log/rsnapshot
lockfile /var/run/rsnapshot.pid
backup / europe/ +rsync_long_args=--exclude=proc/ --exclude=sys/ --exclude=dev/ --exclude=.dbus/ --exclude=media/ --exclude=mnt/
backup rsync://rsnapshot@192.168.0.2/all/ jupiter/ +rsync_long_args=--password-file=/etc/rsnapshot.password --exclude=.timemachine/ --exclude=Jupiter/
Anacron scheduling
To schedule all backups automatically the following was added to the /etc/crontab file:
0 8,12,16,20 * * * root /usr/bin/rsnapshot hourly
50 23 * * * root /usr/bin/rsnapshot daily
30 23 1,8,15,22 * * root /usr/bin/rsnapshot weekly
10 23 1 * * root /usr/bin/rsnapshot monthly
Friday, 9 December 2011
Linux Server Rebuild
Hardware Setup:
A new 19" C2D DFI-ACP machine is used with four SATA drives:
- 250 GB System
- 500 GB Network Share
- 500 GB Backups
- 1 TB Zoneminder Events
Installation:
Ubuntu was installed from a CD without LVM on /dev/sda (250GB) Options chose where:
- Web Server
- File Server
- MySQL
- PostGreSQL
First network settings:
auto eth0
iface eth0 inet static
address 192.168.0.5
netmask 255.255.255.0
gateway 192.168.0.3
sudo ifdown -a
sudo ifup -a
The rest is mainly taken from this guide on www.zoneminder.com
http://www.zoneminder.com/wiki/index.php/Ubuntu_Server_11.04_64-bit_with_ZoneMinder_1.25.0_Preview,_FFmpeg,_libjpeg-turbo,_Webmin,_Cambozola
Update system software
sudo apt-get update
sudo apt-get upgrade
Set Zoneminder Shared Memory
sudo nano /etc/sysctl.conf
#Add in the following lines at the bottom of the file followed by an empty line:
kernel.shmall = 167772160
kernel.shmmax = 167772160
sudo reboot
sudo su
Optimize GCC compiler
export CFLAGS="-march=native -O2 -pipe" && \
export CXXFLAGS="${CFLAGS}" && \
export CPPFLAGS="${CFLAGS}"
Install prerequisites from repositories
aptitude install acpid build-essential linux-headers-`uname -r` automake perl libauthen-pam-perl \
apache2 libpam-runtime libio-pty-perl libmysqlclient-dev php5 php5-cli libapache2-mod-php5 php5-mysql \
libarchive-zip-perl libdate-manip-perl libdevice-serialport-perl libjpeg62 libmime-perl libstdc++6 libwww-perl \
zlib1g zip unzip patch ntp openssl libpcre3-dev libssl-dev libjpeg-progs libcurl4-gnutls-dev munin munin-node libmime-lite-perl \
netpbm libbz2-dev subversion yasm libnet-ssleay-perl libauthen-pam-perl libio-pty-perl apt-show-versions git-core mysql-server mysql-client
Install libjpeg-turbo (AMD64)
wget http://downloads.sourceforge.net/project/libjpeg-turbo/1.1.1/libjpeg-turbo_1.1.1_amd64.deb?use_mirror=voxel
mv libjpeg-turbo_1.1.1_amd64.deb?use_mirror=voxel libjpeg-turbo_1.1.1_amd64.deb && \
dpkg -i libjpeg-turbo_1.1.1_amd64.deb
ln -s /opt/libjpeg-turbo/lib/libjpeg.a /usr/lib/libjpeg.a && \
ln -s /opt/libjpeg-turbo/include/jconfig.h /usr/include/jconfig.h && \
ln -s /opt/libjpeg-turbo/include/jerror.h /usr/include/jerror.h && \
ln -s /opt/libjpeg-turbo/include/jmorecfg.h /usr/include/jmorecfg.h && \
ln -s /opt/libjpeg-turbo/include/jpeglib.h /usr/include/jpeglib.h
Configure Lib Paths
echo "/opt/libjpeg-turbo/lib" > /etc/ld.so.conf.d/libjpeg-turbo.conf && \
echo "/usr/local/lib" > /etc/ld.so.conf.d/ffmpeg.conf && \
echo "LD_LIBRARY_PATH=/usr/local/lib:/opt/libjpeg-turbo/lib:$LD_LIBRARY_PATH" >> /etc/bash.bashrc && \
echo "export LD_LIBRARY_PATH" >> /etc/bash.bashrc
ldconfig
Install Perl Modules
perl -MCPAN -e shell
install CPAN
exit
perl -MCPAN -e shell
install YAML PHP::Serialization Module::Load X10::ActiveHome
exit
LC_ALL=C perl -MCPAN -e shell
install Sys::Mmap
install DBI
install DBD::mysql
exit
Install FFmpeg
# !!!!!! Use the following line for FFMpeg from GIT skipping the 0.8.5 release instructions, otherwise, use the 0.8.5 release instructions
cd /usr/src && git clone git://git.videolan.org/ffmpeg.git
cd /usr/src/ffmpeg/ && ./configure --enable-gpl --enable-shared --enable-pthreads
#FFMPEG 0.8.5 release#
cd /usr/src && wget http://ffmpeg.org/releases/ffmpeg-0.8.5.tar.gz
tar -xzvf ffmpeg-0.8.5.tar.gz
mv ffmpeg-0.8.5 ffmpeg
#Continue from here regardless of method of getting FFMPEG#
cd /usr/src/ffmpeg/ && ./configure --enable-gpl --enable-shared --enable-pthreads --enable-libx264 --enable-libfaac --enable-nonfree --enable-x11grab --enable-version3
make
make install
make install-libs
cd /lib && ln -s /usr/local/lib/libswscale.so.0 && \
ln -s /usr/local/lib/libavformat.so.52 && \
ln -s /usr/local/lib/libavcodec.so.52 && \
ln -s /usr/local/lib/libavutil.so.50 && \
ln -s /usr/local/lib/libavdevice.so.52
ldconfig
Install Zoneminder
cd /usr/src
wget http://www.zoneminder.com/downloads/ZoneMinder-1.25.0.tar.gz
tar -xzvf ZoneMinder-1.25.0.tar.gz
mv ZoneMinder-1.25.0 zm
cd /usr/src/zm && \
./configure --with-webdir=/var/www/zm --with-cgidir=/usr/lib/cgi-bin \
ZM_DB_HOST=localhost ZM_DB_NAME=zm ZM_DB_USER=zmuser \
ZM_DB_PASS=zmpass ZM_SSL_LIB=openssl --enable-debug=no \
--with-webgroup=www-data --with-webuser=www-data --enable-mmap=yes CPPFLAGS="-D__STDC_CONSTANT_MACROS ${CPPFLAGS}"
autoconf
aclocal
automake
make
mysql -u root -p < db/zm_create.sql
#Password that is requested is Mysql root password that was entered during install, not your ubuntu password.
mysql -u root -p
grant select,insert,update,delete on zm.* to 'zmuser'@localhost identified by 'zmpass';
quit
mysqladmin -p reload
make install
Create Zoneminder startup script
nano -w /etc/init.d/zm
#Paste everything BETWEEN the lines below:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#!/bin/sh
# description: Control ZoneMinder as a Service
# chkconfig: 2345 99 99
# Source function library.
#. /etc/rc.d/init.d/functions
prog=ZoneMinder
ZM_PATH_BIN="/usr/local/bin"
command="$ZM_PATH_BIN/zmpkg.pl"
start() {
echo -n "Starting $prog: "
$command start
RETVAL=$?
[ $RETVAL = 0 ] && echo success
[ $RETVAL != 0 ] && echo failure
return $RETVAL
}
stop() {
echo -n "Stopping $prog: "
$command stop
RETVAL=$?
[ $RETVAL = 0 ] && echo success
[ $RETVAL != 0 ] && echo failure
}
status() {
result=`$command status`
if [ "$result" = "running" ]; then
echo "ZoneMinder is running"
RETVAL=0
else
echo "ZoneMinder is stopped"
RETVAL=1
fi
}
case "$1" in
'start')
start
;;
'stop')
stop
;;
'restart')
stop
start
;;
'status')
status
;;
*)
echo "Usage: $0 { start | stop | restart | status }"
RETVAL=1
;;
esac
exit $RETVAL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
chmod 755 /etc/init.d/zm
Edit Zoneminder Script
#Because Ubuntu clears /tmp for some kind of reason at reboot,
#the /tmp/zm directory will also be deleted after a reboot. Creation in script/zmdc.in.pl
#will resolve this.
nano /usr/local/bin/zmdc.pl
#Add following lines (marked with a '+'-sign between the lines given (do not include '+').
foreach my $arg ( @ARGV )
}
}
+# Create the directory first if it doesn't exist
+mkdir(ZM_PATH_SOCKS);
+
socket( CLIENT, PF_UNIX, SOCK_STREAM, 0 ) or Fatal( "Can't open socket: $!" );
my $saddr = sockaddr_un( SOCK_FILE );
#To be sure, add rights to the /tmp/zm directory
chown www-data:www-data -R /tmp/zm
Setup additional hard drives
mkfs.ext4 for all drives
/dev/sda1 as /
/dev/sdb1 as /var/www/zm/event/
/dev/sdc1 as /network
/dev/sdd1 as /backups
For the ZoneMinder share I set the noatime attribute to reduce write cycles.
sudo chown www-data:www-data events/
Setup PCI capture card
I used a Kodicom 4400R and an Ad-Link RTV24 card, with the following settings the machine booted in 18 seconds!
sudo nano /etc/modprobe.d/bttv.conf
alias char-major-81 bttv
options i2c-algo-bit bit_test=1
options bttv gbuffers=16 card=133,132,133,133,134,134,134,134 tuner=4,4,4,4,4,4,4,4
sudo adduser www-data video
Edit Apache:
change /var/www to /var/www/zm in /etc/apache2/sites-available/default
Well, that's the first part. The machine is now basically running. I can configure the rest of the ZoneMinder setup and I can see video now. Next up is transferring the data to this new machine, setup Samba, MySQL, backups and Apache.
Saturday, 2 July 2011
Updated NFS backup script
After some searching and playing with SH (beware! different from BASH) scripts, I came up with this solution:
#!/bin/sh
/usr/bin/MountSaturnBackup
sleep 2
DT=`stat -f -L -c %T /mnt/backup`
if [ "$DT" = "nfs" ]; then
tar cpf /mnt/backup/FullBackup.tar --same-owner --exclude=/network/backups/Jupiter --exclude=/var/cache/zoneminder/events/* --exclude=/proc/* --exclude=/media/* --exclude=/d$
fi
The correct thing to do would be to add an else statement and have the script send me an email. I'll look at that later....
Monday, 17 August 2009
DaviCAL installation
Fed up with waiting for Snow Leopard I decided to set up a calender server on our Linux server. After some searching I found out that Davical would be the best solution, especially since we use thunderbird and would like to use lightning as a calerdar client.
I installed DaviCAL on our Ubuntu 7.10 server, this was not very easy! All in all I can't even rember how I exactly did it. My plan is do to the same on a second server and document this step by step.
Most important thing for me is that is all works! There are some disadvantages, but none to big.
Some things to watch out for:
- In the permissions file of PostgreSQL (pg_hba.conf), make sure to post the access control rule at the top of the configuration document!
- The username in the caldav access URL is casesensitive!
- Use this as URL: http://
/ /caldav.php/ /home/
Since DaviCAL uses PostgreSQL instead of MySQL a extra backup routine is important for me. Our MySQL database is backup, so I added an extra backup script:
#################################################################################################
# PostgresSQL Backup:
mv /network/backups/Database_Backup/Daily6_PostgreSQL_DaviCAL.sql /network/backups/Database_Backup/Daily7_PostgreSQL_DaviCAL.sql
mv /network/backups/Database_Backup/Daily5_PostgreSQL_DaviCAL.sql /network/backups/Database_Backup/Daily6_PostgreSQL_DaviCAL.sql
mv /network/backups/Database_Backup/Daily4_PostgreSQL_DaviCAL.sql /network/backups/Database_Backup/Daily5_PostgreSQL_DaviCAL.sql
mv /network/backups/Database_Backup/Daily3_PostgreSQL_DaviCAL.sql /network/backups/Database_Backup/Daily4_PostgreSQL_DaviCAL.sql
mv /network/backups/Database_Backup/Daily2_PostgreSQL_DaviCAL.sql /network/backups/Database_Backup/Daily3_PostgreSQL_DaviCAL.sql
mv /network/backups/Database_Backup/Daily1_PostgreSQL_DaviCAL.sql /network/backups/Database_Backup/Daily2_PostgreSQL_DaviCAL.sql
mv /network/backups/Database_Backup/Daily_PostgreSQL_DaviCAL.sql /network/backups/Database_Backup/Daily1_PostgreSQL_DaviCAL.sql
su postgres -c 'pg_dump davical' > /network/backups/Database_Backup/Daily_PostgreSQL_DaviCAL.sql
#################################################################################################
Friday, 7 August 2009
Adding an Axiz PTZ camera to ZoneMinder
I added a Axis 213 PTZ dome camera to our ZoneMinder server using the following settings (After turning on PTZ in the Options->System tab (ZM_OPT_CONTROL), and restarting ZoneMinder):
General Tab:
- Source type: remote
- FPS: 6.00 (keep this high or you will get a delay in the image)
- Remote host name: xxx.xxx.xxx.xxx
- Remote host port: 80
- Remote host path: /axis-cgi/mjpg/video.cgi?resolution=CIF
- 24 bit colour
- Capture width: 384
- Capture Height: 288
- Control Type: Axis API v2
- Control Device:
- Control Address: xxx.xxx.xxx.xxx:80 (mind the ':' and the port number !!)
- sudo apt-get install libmodule-load-perl
Friday, 24 April 2009
Weird Time Machine errors on Leopard Server
I looked through the logs and found this: (note that most backups finish ok, just some with the error below)
4/24/09 9:38:15 AM /System/Library/CoreServices/backupd[64124] Starting standard backup
4/24/09 9:38:15 AM /System/Library/CoreServices/backupd[64124] Backing up to: /Volumes/Time Machine Backups/Backups.backupdb
4/24/09 9:38:17 AM /System/Library/CoreServices/backupd[64124] No pre-backup thinning needed: 131.4 MB requested (including padding), 187.56 GB available
4/24/09 9:38:21 AM /System/Library/CoreServices/backupd[64124] Stopping backup.
4/24/09 9:38:21 AM /System/Library/CoreServices/backupd[64124] Error: (-43) SrcErr:NO Copying /private/var/imap/user/a to /Volumes/Time Machine Backups/Backups.backupdb/Saturn/2009-04-24-093816.inProgress/83D4516C-94ED-4201-9F20-9547F53209EF/Leopard Server/private/var/imap/user
4/24/09 9:38:21 AM /System/Library/CoreServices/backupd[64124] Copied 4 files (10.7 MB) from volume Leopard Server.
4/24/09 9:38:21 AM /System/Library/CoreServices/backupd[64124] Copy stage failed with error:11
4/24/09 9:38:27 AM /System/Library/CoreServices/backupd[64124] Backup failed with error: 11
I always thought that Time Machine does not backup mail stuff, but it apparently does backup parts (useless parts) of the IMAP mail store.
The fix I'm trying is to add the /var/imap/users/ dir to the exclude list in the Time Machine preferences screen. See if this helps....
Thursday, 16 April 2009
Rebuilding a Zoneminder system after a Crash
This time we installed a 160GB disk for the system itself and swap, and a seperate 500GB disk for events and backups of the system.
The start of the guide still applies, but we also did the following:
using fdisk created a 6GB for backups and a 482GB partition for events
Formated both as EXT3 and added the following to /etc/fstab:
/dev/sda1 /backups ext3 defaults,errors=remount-ro 0 1
/dev/sda2 /var/cache/zoneminder/events ext3 defaults,errors=remount-ro 0 1
Set correct permissions for the events dir:
sudo chown www-data:www-data /var/cache/zoneminder/events/
Added a time sync script:
sudo nano /etc/cron.daily/timesync
And added:
#!/bin/sh
ntpdate 0.nl.pool.ntp.org
Then:
sudo chmod 755 /etc/cron.daily/timesync
Fix Video permissions:
chmod 4755 /usr/bin/zmfix
Added a MySQL backup script I also use on our main Linux server:
sudo nano /etc/cron.daily/DB_backup
Added:
#!/bin/sh
mv /backups/MySQL_Backup/Daily6.sql /backups/MySQL_Backup/Daily7.sql
mv /backups/MySQL_Backup/Daily5.sql /backups/MySQL_Backup/Daily6.sql
mv /backups/MySQL_Backup/Daily4.sql /backups/MySQL_Backup/Daily5.sql
mv /backups/MySQL_Backup/Daily3.sql /backups/MySQL_Backup/Daily4.sql
mv /backups/MySQL_Backup/Daily2.sql /backups/MySQL_Backup/Daily3.sql
mv /backups/MySQL_Backup/Daily1.sql /backups/MySQL_Backup/Daily2.sql
mv /backups/MySQL_Backup/Daily.sql /backups/MySQL_Backup/Daily1.sql
mysqldump -r /backups/MySQL_Backup/Daily.sql --all-databases
Then ran:
sudo chmod 755 /etc/cron.daily/DB_backup
sudo mkdir /backups/MySQL_Backup
Set up main backup system using Rsnapshot:
sudo apt-get install rsnapshot
sudo nano /etc/rsnapshot.conf (Watch for tabs instead of spaces!!!)
Contents of config file: (used cat /etc/rsnapshot.conf | grep -v -e ^# to get this output btw)
(ALL SPACES MUST BE TABS!!)
config_version 1.2
snapshot_root /backups/
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
cmd_ssh /usr/bin/ssh
cmd_logger /usr/bin/logger
cmd_du /usr/bin/du
cmd_rsnapshot_diff /usr/bin/rsnapshot-diff
interval hourly 6
interval daily 7
interval weekly 4
interval monthly 6
verbose 2
loglevel 3
logfile /var/log/rsnapshot.log
lockfile /var/run/rsnapshot.pid
rsync_short_args -a
rsync_long_args --delete --numeric-ids --relative --delete-excluded
exclude /backups
exclude /dev
exclude /media
exclude /proc
exclude /tmp
exclude /mnt
exclude /sys
exclude /proc
exclude /var/cache/zoneminder/events
backup / localhost/
Then add this to /etc/crontab:
0 */4 * * * root /usr/bin/rsnapshot hourly
50 23 * * * root /usr/bin/rsnapshot daily
40 23 1,8,15,22 * * root /usr/bin/rsnapshot weekly
30 23 1 * * root /usr/bin/rsnapshot monthly
Installed X and setup autologin and auto script run:
sudo apt-get install gdm
sudo apt-get install xorg
On the machine itself (not SSH) ran:
sudo startx
sudo gdmsetup
In gdmsetup I set the user 'zoneminder' as autologin, and set the default session to run the Xclient script.
Create the Xscript:
nano /home/zoneminder/.xsession
And insert:
#!/bin/sh
while ( true )
do
/usr/sbin/xlib_shm-script
sleep 1
done
Then run:
chmod 755 /home/zoneminder/.xsession
Set the correct screen resolution:
sudo apt-get install displayconfig-gtk
sudo displayconfig-gtk
I set this to 800x600, this suits the cameras best
Stop screen blanking by adding this to the /etc/X11/xorg.conf file:
Section "ServerFlags"
Option "blank time" "0"
Option "standby time" "0"
Option "suspend time" "0"
Option "off time" "0"
EndSection
Create, setup & install the Xlib_shm Zoneminder raw viewing script:
Check the version numbers! not all Xlib_shm version is compatible with each version of Zonderminder!
sudo wget http://dig.hopto.org/xlib_shm/xlib_shm-0.6.3.tgz
sudo tar xfz xlib_shm-0.6.3.tgz
sudo apt-get install libxv-dev
sudo apt-get install build-essentials
sudo apt-get install libmysqlclient15-dev
sudo nano xlib_shm.c
And changed /usr/local/etc/zm.conf into /etc/zm/zm.conf
sudo make all
Place the binary xlib_shm in /usr/sbin/
sudo chown root.root /usr/sbin/xlib_shm
sudo chmod +s /usr/sbin/xlib_shm
For the actual auto starting of xlib_shm I first created a new shell script:
sudo nano /usr/sbin/xlib_shm-script
And added this:
#!/bin/sh
xlib_shm -m 1 -m 23 -m 11 -m 12 -k 0x07a6d0000 -d 40000
Then:
sudo chmod 755 /usr/sbin/xlib_shm-script