How do you install and use rsync to synchronize files and directories from one location (or one server) to another location? - A common question asked by new sys admin.
rsync is a free software computer program for Unix and Linux like systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction.
It can perform differential uploads and downloads (synchronization) of files across the network, transferring only data that has changed. The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection.
Use any one of the following commands to install rsync.
# apt-get install rsync
OR
$ sudo apt-get install rsync
# up2date rsync
# yum install rsync
Since rsync does not provide any security while transferring data it is recommended that you use rsync over ssh . This allows a secure remote connection. Now let us see some examples of rsync.
Copy file from /www/backup.tar.gz to a remote server called openbsd.nixcraft.in
$ rsync -v -e ssh /www/backup.tar.gz jerry@openbsd.nixcraft.in:~Output:
Password:
sent 19099 bytes received 36 bytes 1093.43 bytes/sec
total size is 19014 speedup is 0.99
Please note that symbol ~ indicate the users home directory (/home/jerry).
Copy file /home/jerry/webroot.txt from a remote server openbsd.nixcraft.in to a local computer /tmp directory:
$ rsync -v -e ssh jerry@openbsd.nixcraft.in:~/webroot.txt /tmp
Password
$ rsync -r -a -v -e "ssh -l jerry" --delete openbsd.nixcraft.in:/webroot/ /local/webroot
$ rsync -r -a -v -e "ssh -l jerry" --delete /local/webroot openbsd.nixcraft.in:/webroot
$ rsync -r -a -v --delete rsync://rsync.nixcraft.in/cvs /home/cvs
You can mirror a directory between my "old" (my.old.server.com) and "new" web server with the command (assuming that ssh keys are set for password less authentication)
$ rsync -zavrR --delete --links --rsh="ssh -l vivek" my.old.server.com:/home/lighttpd /home/lighttpd
There also exists a utility called rdiff, which uses the rsync algorithm to generate delta files Using rdiff. A utility called rdiff-backup has been created which is capable of maintaining a backup mirror of a file or directory over the network, on another server. rdiff-backup stores incremental rdiff deltas with the backup, with which it is possible to recreate any backup point. Next time I will write about these Utilities :)
Please note if you are using Windows, try any one of the program:
http://www.cyberciti.biz/tips/my-10-unix-command-line-mistakes.html
Anyone who has never made a mistake has never tried anything new. -- Albert Einstein.
Here are a few mistakes that I made while working at UNIX prompt. Some mistakes caused me a good amount of downtime. Most of these mistakes are from my early days as a UNIX admin.
The file /etc/deluser.conf was configured to remove the home directory (it was done by previous sys admin and it was my first day at work) and mail spool of the user to be removed. I just wanted to remove the user account and I end up deleting everything (note -r was activated via deluser.conf):
userdel foo
On Linux killall command kill processes by name (killall httpd). On Solaris it kill all active processes. As root I killed all process, this was our main Oracle db box:
killall process-name
I wanted to append a new zone to /var/named/chroot/etc/named.conf file., but end up running:
./mkzone example.com > /var/named/chroot/etc/named.conf
I had only one backup copy of my QT project and I just wanted to get a directory called functions. I end up deleting entire backup (note -c switch instead of -x):
cd /mnt/bacupusbharddisk
tar -zcvf project.tar.gz functions
I had no backup. Similarly I end up running rsync command and deleted all new files by overwriting files from backup set (now I've switched to rsnapshot)
rsync -av -delete /dest /src
Again, I had no backup.
I had sym links for my web server docroot (/home/httpd/http was symlinked to /www). I forgot about symlink issue. To save disk space, I ran rm -rf on http directory. Luckily, I had full working backup set.
Accidentally changed the current hostname (I wanted to see current hostname settings) for one of our cluster node. Within minutes I received an alert message on both mobile and email.
hostname foo.example.com
I wanted to shutdown VPN interface eth0, but ended up shutting down eth1 while I was logged in via SSH:
ifconfig eth1 down
I made changes to sshd_config and changed the ssh port number from 22 to 1022, but failed to update firewall rules. After a quick kernel upgrade, I had rebooted the box. I had to call remote data center tech to reset firewall settings. (now I use firewall reset script to avoid lockdowns).
I wanted to shutdown my local Fedora desktop system, but I issued halt on remote server (I was logged into remote box via SSH):
halt
service httpd stop
Created a wrong DNS CNAME entry in example.com zone file. The end result - a few visitors went to /dev/null:
echo 'foo 86400 IN CNAME lb0.example.com' >> example.com && rndc reload
In 2006 ORDB went out of operation. But, I failed to update my Postfix RBL settings. One day ORDB was re-activated and it was returning every IP address queried as being on its blacklist. The end result was a disaster.
All men make mistakes, but only wise men learn from their mistakes -- Winston Churchill.
From all those mistakes I've learnt that:
Mistakes are the inevitable, so did you made any mistakes that have caused some sort of downtime? Please add them into the comments below.
http://www.comptechdoc.org/os/linux/howlinuxworks/linux_hlprocess.html
Process control and the ability for inter process communication is handled by the Linux kernel.
Intro
Sometimes it happens that you want to laubnch some heavy operations in
the background without slowing down other software. For example:
Copying files
Backups
Indexing (updatedb).
In this situation nice and ionice can be used.
* nice allows you to reduce the priority of a process on the CPU
* ionice reduces the priority of a process on disk access.
Launching a program
ionice -c3 nice -n15 myprogram
The program launched will use CPU and disk resources only when no
other program is making use of it.
In a shell script
Place at the beginning of your shell script:
#!/bin/sh
ionice -c3 -p$$
renice +15 -p $$
The shell will have its priority reduced (and this will be applicable
for all programs launched within this shell).
This is useful in the scripts launched in /etc/cron.
You can add these lines to your script: /etc/cron.daily/slocate
http://www.cyberciti.biz/tips/linux-filesystem-benchmarking-with-iozone.html
User contributed tutorial written by Ramesh Natarajan.
Following are few situations where you may be interested in performing a filesystem benchmarking.
=> Deploying a new application that is very read and write intensive.
=> Purchased a new storage system and would like to measure the performance.
=> Changing the RAID level and would like to measure the performance of the new RAID.
=> Changing the storage parameters and would like to know the performance impact of this change
This article gives you a jumpstart on performing benchmark on filesystem using iozone a free Filesystem Benchmark utility.
Go to iozone and download the iozone for your appropriate platform. I downloaded the "Linux i386 RPM". Install the iozone from the RPM as shown below. By default this installs the iozone under /opt/iozone/bin
# cd /tmp
# wget http://www.iozone.org/src/current/iozone-3-303.i386.rpm
# rpm -ivh iozone-3-303.i386.rpm
Sample output:
Preparing... ########################################### [100%]
1:iozone ########################################### [100%]
Note: You can install iozone under any UNIX / Linux or Windows operating system.
Execute the following command in the background to begin the performance test.
# /opt/iozone/bin/iozone -R -l 5 -u 5 -r 4k -s 100m -F /home/f1 /home/f2 /home/f3 /home/f4 /home/f5 | tee -a /tmp/iozone_results.txt &
Let us review all the individual parameter passed to iozone command.
The first part of the output will contain the details about every individual filesystem performance metrics that was tested. for e.g. Initial write, rewrite etc as shown below.
Iozone: Performance Test of File I/O
Version $Revision: 3.303 $
Compiled for 32 bit mode.
Build: linux
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.
Run began: Thu Jun 22 00:08:51 2008
Excel chart generation enabled
Record Size 4 KB
File size set to 102400 KB
Command line used: /opt/iozone/bin/iozone -R -l 5 -u 5 -r 4k -s 100m -F /home/f1 /home/f2 /home/f3 /home/f4 /home/f5
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Min process = 5
Max process = 5
Throughput test with 5 processes
Each process writes a 102400 Kbyte file in 4 Kbyte records
Children see throughput for 2 initial writers = 60172.28 KB/sec
Parent sees throughput for 2 initial writers = 45902.89 KB/sec
Min throughput per process = 28564.52 KB/sec
Max throughput per process = 31607.76 KB/sec
Avg throughput per process = 30086.14 KB/sec
Min xfer = 92540.00 KB
Children see throughput for 2 rewriters = 78658.92 KB/sec
Parent sees throughput for 2 rewriters = 34277.52 KB/sec
Min throughput per process = 35743.92 KB/sec
Max throughput per process = 42915.00 KB/sec
Avg throughput per process = 39329.46 KB/sec
Min xfer = 85296.00 KB
Similar values like above will be generated for readers, re-readers, reverse readers, stride readers, random readers, mixed workload, random writers, pwrite writers, pread readers. The last part of the iozone output will contain the Throughput summary for different metrics as shown below.
Throughput report Y-axis is type of test X-axis is number of processes
Record size = 4 Kbytes
Output is in Kbytes/sec
Initial write 60172.28
Rewrite 78658.92
Read 2125613.88
Re-read 1754367.31
Reverse Read 1603521.50
Stride read 1633166.38
Random read 1583648.75
Mixed workload 1171437.78
Random write 5365.59
Pwrite 26847.44
Pread 2054149.00
(Fig.01: iozone in action)
Iozone does a benchmarking on different types of file system performance metrics. for e.g. Read, Write, Random read. Depending on the application that you are planning to deploy on that particular filesystem, pay attention to the appropriate items. for e.g. If the filesystem hosts an read intensive OLTP database, then pay attention to Random Read, Random write and Mixed workload. If the application does lot of streaming media content, pay attention to the Sequential Read. On a final note, you can generate graphs using the Generate_Graphs and gengnuplot.sh located under /opt/iozone/bin, based on the iozone output.
This article was written by Ramesh Natarajan. At the The Geek Stuff blog he shares his knowledge and experience on Linux and other Geeky stuff. He has more than 15 years of experience in IT industry and has performed very intensive work on Linux system administration, DBA, Hardware and Storage. nixCraft welcomes readers' tips / howtos.
The Debian packaging system is one of the most elegant methods of installing, upgrading, and removing software available. For all you fans of other packaging systems, before you send your flames, please note that I said "one of" and not simply "the most." Other packaging systems have their charms, but in this article I'm going to focus on the Debian packaging system. Specifically, I'm going to look at creating Debian packages so you can distribute your packages in Debian format -- or simply create packages for your own use.
I'll walk through the basic components of a package, what you need to create your own packages, and finally how to pull it all together to create a package.
Understanding Debian packages
In this section, we'll take a quick look at what Debian packages are. Debian packages are much like RPM (Red Hat Package Manager) packages, but they differ a little in the details. If you've already worked with creating RPMs, you'll probably find that it's not at all difficult to add Debian packaging to your repertoire.
What is a package, exactly?
To put it very simply, a package is a collection of files with instructions on what to do with them. A package usually contains a program or programs, but sometimes it has only documentation, window manager themes, or other files that are easier to distribute in an installable package.
The package contains instructions on where those files should reside in the filesystem, what libraries or other programs the contents of the package are dependent on (if any), setup instructions, and basic configuration scripts. Note that many packages cannot be used or should not be used with the default settings contained in their configuration files. With packages such as Apache, you'll still need to configure your installation after the package has been set up.
Packages usually contain precompiled software, but you can also package source code. Some admins may prefer to install from source, or your application may require customization prior to compilation, so if you're distributing software that is under a free or open source license, you may wish to create a source package as well as a "binary" package.
Note: Pre-compiled software packages are usually called "binaries," although this is something of a misnomer. Yes, executables are in binary format, but so are JPEGs, MP3s, and many other non-executable files. It would probably make more sense to call precompiled software "executable" packages.
All binary Debian packages consist of three basic things: a text file called debian-binary, a compressed tarball called control.tar.gz, and another compressed tarball called data.tar.gz.
The debian-binary text file contains the version number for the binary package, which should be 2.0. The control.tar.gz file contains the control file; the postinst file, which contains instructions on what to do after installing the package; and the prerm file, which contains removal instructions. control.tar.gz may also contain a file with information about configuration files for the package called conffiles and a file with the MD5 checksums for the package called md5sums.
The data.tar.gz contains the actual "payload" of the package. That is, it contains a filesystem with all the relevant files for your program that, when installed, will be placed in the appropriate spots in your system's filesystem.
If you want to see what a package looks like for yourself, download a few packages from the Debian site (see the link in the Resources later in this article) and run ar -x packagename.deb. Debian packages are simply archives of the files mentioned above.
Do I need to be a Debian Developer?
You don't need to be an official Debian Developer to create and distribute Debian-compatible packages. However, you do need to be a Debian Developer to actually upload packages and be a package maintainer for official Debian releases. If you're interested in becoming a Debian Developer, you can start by reading the "How You Can Help?" page on the Debian Web site (see Resources for a link).
There are a number of companies and projects that create and distribute Debian packages of their software. These are not "official" Debian packages, but they're still of great convenience to Debian users who wish to be able to install software using dpkg or APT (Advanced Packaging Tool).
What you need to get started
Obviously, you're going to need a current Debian system if you're going to create and test Debian packages. You'll also need to have something that you want to package. We'll start with the assumption that you already have those two things and go from there.
There are several packages that you'll need to have installed to build packages on a Debian system. Most of those packages will likely be installed by default, since you'll need dpkg, gcc, make, and so forth on most systems anyway. You will need the build-essential package and dpkg-dev package as well, which may or may not already be installed on your system.
Debian package naming conventions
If you've used Debian for any amount of time, you might have noticed that Debian package files all follow certain naming conventions. Every Debian binary package should have a filename that follows this format: packagename_version_arch.deb, where "packagename" is the name of the package, "version" is the package version with major, minor, and revision numbers, and "arch" is the architecture for the package.
The name of the package itself (not the filename, just the name of the package) can contain lowercase letters, numbers, and the "-" and "+" characters. (It might sound odd for a package to have a "+" in the package name, but it is used for several packages like "doc++" -- a documentation system for C/C++ code.) So, "mypackage-2++" is a perfectly valid package name, while "MY_Package" or "mypackage2.0" would not be.
Every package should also have a unique name that doesn't already exist in the Debian archive. Even though you technically do not need to adhere to this for your package to install, you should be sure that your package name does not conflict with an official Debian package. For example, if you wanted to distribute a specially patched version of Apache as an alternative to the official Debian version, you shouldn't use apache_X.X.X-i386.deb, because the Debian folks already have an apache package. Instead, you'd want to use something like acme-apache_X.X.X-i386.deb.
Creating a Debian package
Now that we've covered the basics, what packages are, and what you need, it's time to talk about the nuts and bolts of building packages.
Control files
You'll need to create a control file for your package. Don't worry; it's not very hard and all you need to create the file is your favorite text editor and a little information. The following is a sample control file. Note that I've included all the optional fields here for demonstration purposes -- your package is unlikely to need all of these fields.
|
Control file fields
The syntax for the control fields is very simple: the name of the field, followed by a colon, and then the body or value of the field. So, "Fieldname: value" is all you need. Some fields require a one-word value, while others allow a free-form description or set of values.
In this section we'll look at what each field in the control file means and what kind of information you need to provide. Most of these fields are optional, so your package will not have to include every field described here if they aren't relevant to your package. For example, a package will not require the "Source" field if source for the package is unavailable. If your package won't conflict with any other packages, then you need not include the "Conflicts" information, and so on.
The first field in any control file is the "Package" field. In this field you'll specify the name of the package. The next field, "Version" is the version number for the package. You may use whatever version numbering scheme the original author of the package uses, but it may not include a hyphen. Anything after a hyphen is considered the Debian revision number. For example, if a program's native version number is given as 1.01-1 for some reason, the version number would be considered "1.01" with a Debian revision number of "1" -- clearly, not the result you're looking for.
The "Maintainer" field should be your full name (or company name) and an e-mail address contained within angle brackets as shown above. The "Installed-Size" is pretty self-explanatory.
Dependencies
As you probably already know, one of the nicest things about Debian packages is the dependency resolution. When you use apt-get to retrieve packages, it automatically decides whether you have what you need on your system to use the requested package, and offers to resolve any conflicts you might have or retrieve any packages you're in need of. That feature depends on the package's dependencies being well thought out and properly set up in the package.
There are several optional fields that deal with dependencies. We'll cover each of them briefly here. Be sure you've sorted out exactly which packages your package may depend on before installation -- the dependency system is only as smart as the developers giving it information.
Dependencies make up five of the fields in a control file: Depends, Enhances, Pre-Depends, Recommends, and Suggests.
The Depends field should contain the name of any packages that your package absolutely needs to work. If there's a "nice to have" package related to yours, but not required for normal operation of your package, then you should use the "Recommends" or "Suggests" fields instead. You'll note in the example above that the "Recommends" field has two separate package separated by a pipe (|). This means that either package will satisfy the dependency; both are not required.
What's the difference between "Recommends" and "Suggests," anyway? Any packages that you should probably have, but are not absolutely required, belong in the "Recommends" field. According to the policy manual, "list packages that would be found together with this one in all but unusual installations." Packages that are likely to be useful but aren't in any way required should be listed in the "Suggests" field.
For example, the DocBook XML DTD package (docbook-xml) lists the DocBook SGML (docbook) package as a recommended package, and the DocBook documentation (docbook-doc) as a suggested package. In most circumstances, if you have the XML DTD, you'll probably have the SGML DTD too -- but not in every instance. It's usually (read: always) a good idea to install documentation about a package or related programs, like the DocBook docs, but it's not necessary for normal operation of your program.
The "Enhances" field is basically stating that your package may make another package more useful. So, the docbook-doc package might have an "Enhances" field with docbook-xml as one of the packages enhanced.
The "Pre-Depends" field is reserved for special cases. When a package is listed as a "Pre-Depends," it forces the system to make sure that the named packages are completely installed before attempting to install your package.
Finally, when working with version numbers, there are several symbols you'll need to understand. You'll note that one of the packages listed as a dependency has a version number given, acme-base (>= 1.2). This means, "this package requires the package acme-base version equal to or higher than 1.2."
The relations available are strictly earlier (<<), earlier or equal (<=), equal to only (=), equal to or later than (>=), and strictly later than (>>).
Specifying conflicts
The "Conflicts" and "Replaces" fields specify packages that are not dependencies, but rather packages that don't belong on the system at the same time as your package.
Packages that conflict with your package outright should be listed in the "Conflicts" fields. For example, if you're packaging a modified version of Apache, you won't want the normal version of Apache to be on the system as well as yours -- so "apache" would be listed as a conflict.
Finally, the control file should be located under a directory named DEBIAN, as should the scripts we discuss in the next section. This directory won't turn up in the final Deb archive.
Scripts
Your package will also need a few scripts to be run by the system when installing and removing the package. (Remember, no matter how wonderful your program is, someone may wish to uninstall it later on down the road.)
The postinst script should contain any necessary steps to be done after installing your program. The prerm script should contain instructions on what should be done to remove the program. Both of these scripts are required (see Resources for more on this).
The final step
After all the prep work that goes into making a Debian package, the actual package creation is somewhat anticlimactic.
You'll use the dpkg command with the -b or --build option (-b and --build are the same). The syntax for creating a package is dpkg -b directory packagename.deb where directory contains the filesystem tree with all the requisite files for your program. Note, you can build the package without specifying the name of the new package, but then it will simply place the package file under directory as ".deb" -- which might lead you to believe that the package wasn't created at all.
So, for example, if your program has a configuration file that belongs in etc/, the program itself, which will live in usr/bin/, and some documentation that lives under usr/share/doc/package/, you'll recreate that filesystem tree in a directory that will be used to create your package.
Beyond the basics
That's pretty much all there is to creating a Debian package. It might seem a bit complicated, but really it's just a number of simple steps. None are too complicated; you simply have to get them all right.
There are a few intricacies of creating packages that are beyond the scope of this article. For example, if you're packaging a mail user agent or mail transport agent, there are some specific considerations that apply. This article only covers the bare basics of creating a Debian package. Make sure you check the Debian Policy Manual (see Resources for a link) to make sure you're in full compliance with the guidelines for packages.
There are several categories of programs that require extra attention, including any Web servers, daemons, Perl programs or modules, X programs, and so forth. You'll find what you need to know in the Customized Programs section of the policy manual.
Posts
All Comments
