Linux Basics

= Linux Booting Process = Source: technochords.com

The following are the 6 high level stages of a typical Linux boot process:


 * 1) BIOS
 * MBR
 * 1) GRUB
 * 2) Kernel
 * 3) Init
 * 4) Runlevel programs

Detailed explanation of each stage:


 * BIOS
 * Performs some system integrity checks (POST-Power On Self Test)
 * Searches, loads, and executes the boot loader program.
 * It looks for boot loader in floppy, cd-rom, or hard drive.
 * You can press a key (typically F2 or F12, but it depends on your system) during the BIOS startup to change the boot sequence.
 * Once the boot loader program is detected and loaded into the memory, BIOS gives the control to it.
 * In simple terms BIOS loads and executes the MBR boot loader.


 * MBR
 * MBR stands for Master Boot Record.
 * It is located in the 1st sector of the bootable disk.
 * Typically /dev/hda, or /dev/sda
 * MBR is less than 512 bytes in size.
 * This has three components:
 * 1) primary boot loader info in 1st 446 bytes,
 * 2) partition table info in next 64 bytes(16,16,16,16) 4 partitions,
 * 3) magic numbers as mbr validation check in last 2 bytes.
 * It contains information about GRUB (or LILO in old systems).
 * In simple terms MBR loads and executes the GRUB boot loader.

default=0 timeout=5 splashimage=(hd0,0)/boot/grub/splash.xpm.gz hiddenmenu title CentOS (2.6.18-194.el5PAE) root (hd0,0) kernel /boot/vmlinuz-2.6.18-194.el5PAE ro root=LABEL=/ initrd /boot/initrd-2.6.18-194.el5PAE.img
 * GRUB
 * GRUB stands for Grand Unified Bootloader.
 * It is a Multiboot boot loader.
 * If you have multiple kernel images installed on your system, you can choose which one to be executed.
 * GRUB displays a splash screen, waits for few seconds, if you don’t enter anything, it loads the default kernel image as specified in the grub configuration file.
 * GRUB has the knowledge of the filesystem (the older Linux loader LILO didn’t understand filesystem).
 * Grub configuration file is /boot/grub/grub.conf (/etc/grub.conf is a link to this).
 * 1) boot=/dev/sda
 * As you notice from the above info, it contains kernel and initrd image.
 * Linux initial RAM disk (initrd) is a temporary root file system that is mounted during system boot to support the two-state boot process.
 * Initrd contains various executables and drivers that permit the real root file system to be mounted.
 * So, in simple terms GRUB just loads and executes Kernel and initrd images.


 * Kernel
 * Once the control is given to kernel which is the central part of all your OS and act as a mediator between hardware and software.
 * Kernel once loaded into to RAM it always resides on RAM until the machine is shutdown.
 * Once the Kernel starts its operations the first thing it do is executing INIT process.

0 – halt 1 – Single user mode 2 – Multiuser, without NFS 3 – Full multiuser mode 4 – unused 5 – X11 6 – reboot
 * Init (initialization)
 * Looks at the /etc/inittab file to decide the Linux run level.
 * Following are the available run levels


 * Init identifies the default initlevel from /etc/inittab and uses that to load all appropriate program.
 * Execute ‘grep initdefault /etc/inittab’ on your system to identify the default run level
 * Typically you would set the default run level to either 3 or 5.

Run level 0 – /etc/rc.d/rc0.d/ Run level 1 – /etc/rc.d/rc1.d/ Run level 2 – /etc/rc.d/rc2.d/ Run level 3 – /etc/rc.d/rc3.d/ Run level 4 – /etc/rc.d/rc4.d/ Run level 5 – /etc/rc.d/rc5.d/ Run level 6 – /etc/rc.d/rc6.d/
 * Runlevel programs
 * When the Linux system is booting up, you might see various services getting started.
 * For example, it might say “starting sendmail …. OK”.
 * Those are the runlevel programs, executed from the run level directory as defined by your run level.
 * Depending on your default init level setting, the system will execute the programs from one of the following directories.


 * Please note that there are also symbolic links available for these directory under /etc directly.
 * So, /etc/rc0.d is linked to /etc/rc.d/rc0.d.
 * Under the /etc/rc.d/rc*.d/ directories, you would see programs that start with S and K.
 * 1) Programs starts with S are used during startup. S for startup.
 * 2) Programs starts with K are used during shutdown. K for kill.
 * 3) There are numbers right next to S and K in the program names.
 * 4) Those are the sequence number in which the programs should be started or killed.
 * 5) For example, S12syslog is to start the syslog deamon, which has the sequence number of 12.
 * 6) S80sendmail is to start the sendmail daemon, which has the sequence number of 80.
 * So, syslog program will be started before sendmail.

= Linux file system layout =

This is a layout file system structure from a CentOS linux, depends on the system and linux distro, the structure may vary, and directories may be removed or added.


 * / – The Root Directory


 * Everything on Linux system is located under the / or root directory.
 * The meaning of / or root and root user are often confusing to new Linux users.
 * In Linux, the root directory “/” is a separator between a file and a directory contains all underlying directories and files, yet root user is a super user or administrator user with has administrative privileges on the system vs Linux privileged users who only have limited privileges to protect system security.


 * /bin – Essential command binaries


 * The /bin directory is a place contains most commonly essential executable terminal binaries programs or file required during booting, repairing like cat, ls, mount, rm, du, df, tar, rpm, wc, etc.


 * /boot – Boot loader files


 * All of the required files to boot the system contains in /boot directory, including GRUB boot loader’s files, Linux kernels, the Linux initial RAM disk (initrd),


 * /dev – Device Files


 * All of the hardware devices on the machine like cdrom, cpu, hard drives, etc will be stored as special device files that represent all the devices in /dev directory. Device files are created during your Linux system installation.


 * /etc – Configuration Files


 * Contain host-specific files and directories, e.g. information about system and application configuration files like startup, shutdown, start, stop script for every individual program.
 * /etc data is very similar to Control panel in Microsoft Windows.


 * /home – Home Directory


 * Home directory of the users. Every time you create a new user, a new directory with user’s name is created in /home directory for users to store their own files within their own home’s directory.
 * Most common automatic directories created within /home ‘s user are Desktop, Downloads, Documents, Music, Movie, etc.
 * Most programs configuration file for a specific user will be saved in it’s users /home directory like web browser settings, web browser bookmarks, desktop wallpaper, themes, and passwords.


 * /lib – Essential Libraries


 * Similar to Windows ‘dll’ files, all Linux shared libraries and kernel modules files stores in /lib directory.
 * These important Linux dynamic libraries are required to boot the system and run commands in the root file system.


 * /lost+found – Recovering Files


 * Every Linux file system and partitions has a lost+found in it’s directory.
 * In the event your system is crashed or unexpected shutdown, you can run fsck command to check and repair the filesystem, fsck will turn any corrupted or almost-deleted files back into files that you can recover them later in /lost+founddirectory.


 * /media – Removable Media Devices


 * Every time you insert a removable device such as external hard drive, floppy disk, zip drive, CDs, DVDs, flash drive to a Linux system, a new directory will automatically be created inside the /media directory.
 * It is a temporary mount directory for removable devices.


 * /mnt – Temporarily mounted filesystems


 * While /media is where the system automatically mounts removable media, /mnt is for you to mount things (partitions, file systems, devices) manually and temporarily.


 * /opt – Optional software packages


 * The /opt directory is reserved to store addition software or extra and third-party software for your system, those addition software usually don’t follow the standard file system hierarchy and not handled by the package manager.


 * /proc – Kernel & Process Information


 * Similar to /dev, /proc directory contains information about running process, system resources and information.
 * You can view information about any running process with a specific process-id (pid) or hardware’s information such as memory, cpu, io, etc.


 * /root – Root Home Directory


 * Don’t be confused with “/” or root directory, /root is a root account’s home directory determined by developer or local preference rather than /home/root to allow for booting the system even if /home/ is not available.
 * Sometimes /home is located on a different partition or even on another separate system and it’s inaccessible to “root”, that is why “root’s home directory” need to be in the same partition as “/” directory.


 * /sbin – System binaries


 * Similar to /bin, /sbin contains essential binaries that are generally intended to be run by the root user for system administration and maintenance purpose.
 * For example iptables, reboot, fdisk, ifconfig, swapon, init, ip, mount


 * /selinux – Security-Enhanced Linux


 * Selinux comes with RedHat based distro (fedora, centos), selinux is a security architecture integrated into the 2.6.x kernel using the Linux Security Modules (LSM).
 * For some reason Centos 6 created an emtpry selinux directory in root directory, the real selinux directory with its configuration files are stored in /etc/selinux/ directory.


 * /srv – Service Data


 * Server (srv) contains data of services such as HTTP, FTP, rsync, cvs


 * /sys – virtual filesystem


 * Some newer Linux distros have /sys directory with sysfs virtual filesystem to store information and statistics about (physical and virtual) device and device names.
 * It is newly added since Linux kernel 2.6 /sys contains similar information with /proc which display device information about the kernel’s view of the system.


 * /tmp – Temporary files


 * System’s Temporary Directory, all users and programs in your system can access/read/write in this directory.
 * Most files in this directory are required temporarily.
 * Many programs use /tmp to create lock file to save temporary data or files.
 * Normally don’t delete files from /tmp unless you know what you are doing because most files are required for current running programs.
 * You should not save or store any important files/directories under /tmp since all files will be removed after system is rebooted.


 * /usr – binaries, documentation, source code, libraries


 * Pronounced as ‘user’, /usr contains the majority of user utilities, programs, libraries, documentation etc for all user-related second level programs rather than applications and files used by the system.
 * Some user programs are stored here like telnet, ftp, etc.. /usr is shareable between various FHS-compliant hosts but can not be written to.


 * /var – Variable Files


 * Variable or /var contains data that is expected to change and grow as the system is running (system log files, mail, printer spool, temporary files).
 * Some sub directories under /var are not shareable between systems like /var/log, /var/lock, or /var/run, while other sub directories are shareable like /var/mail, var/cache/man, var/cache/fonts, and /var/spool/news

= File Details=

Passwd file

 * Password file /etc/passwd is human readable file.
 * By default /etc/passwd file permission is 644 i.e. -rw-r–r– and ownership root:root.
 * Means file is world readable and only root users can edit it.
 * However it is not recommended it manually.

root:x:0:0:ROOT account:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin myuser:x:513:520:Test User:/home/myuser:/bin/bash - output truncated -
 * 1) cat /etc/passwd


 * For every user (row) there are 7 fields defined separated by colon

Username Encrypted password   x means encrypted password is stored in /etc/shadow file UID GID Comment Home directory Shell

Shadow file

 * Located in /etc/shadow, it is not world readable and can be read by root user only.
 * Shadow file permissions are 400 i.e. -r——– and ownership is root:root.
 * Means it can be only read and by root users only.
 * Reason for such security is password related information which is being stored in this file.

root:$1$UFnkhP.mzcMyajdD9OEY1P80:17413:0:99999:7::: bin:*:15069:0:99999:7::: daemon:*:15069:0:99999:7::: adm:*:15069:0:99999:7::: testuser:$1$FrWa$ZCMQ5zpEG61e/wI45N8Zw.:17413:0:33:7:::
 * 1) cat /etc/shadow

There are total of 8 fields in shadow file separated by colon Username Encrypted password Last password change Min days Max days Warn days Inactive days Expiry

= Command Editing Shortcuts =

Ctrl + a – go to the start of the command line Ctrl + e – go to the end of the command line Ctrl + k – delete from cursor to the end of the command line Ctrl + u – delete from cursor to the start of the command line Ctrl + w – delete from cursor to start of word (i.e. delete backwards one word) Ctrl + y – paste word or text that was cut using one of the deletion shortcuts (such as the one above) after the cursor Ctrl + xx – move between start of command line and current cursor position (and back again) Alt + b – move backward one word (or go to start of word the cursor is currently on) Alt + f – move forward one word (or go to end of word the cursor is currently on) Alt + d – delete to end of word starting at cursor (whole word if cursor is at the beginning of word) Alt + c – capitalize to end of word starting at cursor (whole word if cursor is at the beginning of word) Alt + u – make uppercase from cursor to end of word Alt + l – make lowercase from cursor to end of word Alt + t – swap current word with previous Ctrl + f – move forward one character Ctrl + b – move backward one character Ctrl + d – delete character under the cursor Ctrl + h – delete character before the cursor Ctrl + t – swap character under cursor with the previous one

Command Recall Shortcuts

Ctrl + r – search the history backwards Ctrl + g – escape from history searching mode Ctrl + p – previous command in history (i.e. walk back through the command history) Ctrl + n – next command in history (i.e. walk forward through the command history) Alt +. – use the last word of the previous command

Command Control Shortcuts

Ctrl + l – clear the screen Ctrl + s – stops the output to the screen (for long running verbose command) Ctrl + q – allow output to the screen (if previously stopped using command above) Ctrl + c – terminate the command Ctrl + z – suspend/stop the command

Bash Bang (!) Commands

Bash also has some handy features that use the ! (bang) to allow you to do some funky stuff with bash commands.

!! – run last command !blah – run the most recent command that starts with ‘blah’ (e.g. !ls) !blah:p – print out the command that !blah would run (also adds it as the latest command in the command history) !$ – the last word of the previous command (same as Alt + .) !$:p – print out the word that !$ would substitute !* – the previous command except for the last word (e.g. if you type ‘find some_file.txt /‘, then !* would give you ‘find some_file.txt‘) !*:p – print out what !* would substitute

= Manually Boot using Grub =

Source: viktorpetersson.com

Locate where the vmlinuz and initrd.* files are located

grub> ls (hd0) (hd0,msdos5) (hd1) (hd1,msdos0)

Boot the system:

grub> linux (hd1,msdos1)/install/vmlinuz root=/dev/sdb1 grub> initrd (hd1,msdos1)/install/initrd.gz grub> boot

This should even work if your BIOS doesn’t support booting off of USB.

= ProcFS =


 * Procfs or /proc is a special FS under Linux used to present process information and kernel processes.
 * Much of the information for kernel level of 2.6 & above have been moved to "sysfs" generally mounted under /sys.
 * /proc is stored in memory.

"siblings" = (HT per CPU package) * (# of cores per CPU package) "cpu cores" = (# of cores per CPU package)
 * On multi-core CPUs, /proc/cpuinfo contains the fields for "siblings" and "cpu cores":


 * A CPU package means physical CPU which can have multiple cores (single core for one, dual core for two, quad core for four).
 * This allows a distinction between hyper-threading and dual-core, i.e. the number of hyper-threads per CPU package can be calculated by siblings / CPU cores.
 * If both values for a CPU package are the same, then hyper-threading is not supported.
 * For instance, a CPU package with siblings=2 and "cpu cores"=2 is a dual-core CPU but does not support hyper-threading.

/proc/cmdline      – Kernel command line information. /proc/consoles     – Information about current consoles including tty. /proc/crypto	   – list of available cryptographic modules /proc/devices      – Device drivers currently configured for the running kernel. /proc/diskstats    – /proc/dma          – Info about current DMA channels. /proc/fb           – Framebuffer devices. /proc/filesystems  – Current filesystems supported by the kernel. /proc/iomem        – Current system memory map for devices. /proc/ioports      – Registered port regions for input output communication with device. /proc/kmsg	    – holding messages output by the kernel /proc/loadavg      – System load average. /proc/locks        – Files currently locked by kernel. /proc/meminfo      – Summary of how the kernel is managing its memory. /proc/misc         – Miscellaneous drivers registered for miscellaneous major device. /proc/modules      – Currently loaded kernel modules. /proc/mounts       – List of all mounts in use by system. /proc/partitions   – Detailed info about partitions available to the system. /proc/pci          – Information about every PCI device. /proc/scsi	    – Information about any devices connected via a SCSI or RAID controller /proc/stat         – Record or various statistics kept from last reboot. /proc/swap         – Information about swap space. /proc/tty	    – Information about the current terminals /proc/uptime       – Uptime information (in seconds). /proc/version      – Kernel version, gcc version, and Linux distribution installed.

/proc/PID/cmdline  – Command line arguments. /proc/PID/cpu      – Current and last cpu in which it was executed. /proc/PID/cwd	    – Link to the current working directory. /proc/PID/environ  – Values of environment variables. /proc/PID/exe	    – Link to the executable of this process. /proc/PID/fd	    – Directory, which contains all file descriptors. /proc/PID/maps	    – Memory maps to executables and library files. /proc/PID/mem	    – Memory held by this process. /proc/PID/root	    – Link to the root directory of this process. /proc/PID/stat	    – Process status. /proc/PID/statm    – Process memory status information. /proc/PID/status   – Process status in human readable form (eg: GID, UID, etc) /proc/PID/limits   – Contains information about the limits of the process

Usage: ls -l /proc/$(pgrep -n python)/exe

= Inode Number =

Source: linoxide.com


 * Inode is entry in inode table containing metadata about a regular file and directory.
 * An inode is a data structure on a traditional Unix-style file system such as ext3 or ext4.
 * Stores all the information about a file except its name and its actual data.
 * Linux extended filesystems such as ext2 or ext3 maintain an array of these inodes: the inode table.
 * This table contains list of all files in that filesystem.
 * The individual inodes in inode table have a unique number (unique to that filesystem) - the inode number.
 * There are some data about files, such as their size, ownership, permissions, timestamp etc.
 * This meta-data about a file is managed with a data structure known as an inode (index node).


 * There is no entry for file name in the Inode, file name is kept as a separate entry parallel to Inode number.
 * This is for maintaining hard-links to files.


 * Copy file: cp allocates a free inode number and placing a new entry in inode table.
 * Move or Rename a file: if destination is same filesystem as the source, Has no impact on inode number, it only changes the time stamps in inode table.
 * Delete a file: Deleting a file in Linux decrements the link count and freeing the inode number to be reused.


 * A Directory cannot hold two files with same name because it cannot map one name with two different inode numbers.
 * The inode number of / directory is fixed, and is always 2.


 * There exists an algorithm which is used to create number of Inodes in a file system.
 * This algorithm takes into consideration the size of the file system and average file size.
 * The user can tweak the number of Inodes while creating the file system.


 * Inode number (or index number) consists following attributes:

File type:                Regular file, directory, pipe etc. Permissions:               Read, write, execute Link count:               The number of hard link relative to an inode User ID:                  Owner of file Group ID:                 Group owner Size of file:             or major/minor number in case of some special files Time stamp:               Access time, modification time and (inode) change time Attributes:               Immutable' for example Access control list:      Permissions for special users/groups Link to location of file Other metadata about the file

df -i                               ==> Inodes on Filesystem df -i /dev/vda1                     ==> Inodes on Filesystem ls -il myfile.txt                   ==> Show inode no of file find /home/rahul -inum 1150561      ==> Find file using inode no stat unetbootin.bin                  ==> Show all details of file stat --format=%i unetbootin.bin     ==> Shows only inode no
 * Check info:

List the contents of the filesystem superblock tune2fs -l /dev/sda6 | grep inode
 * Manipulate the filesystem meta data

Make sure files on the file system are not being accessed: mount -o remount /yourfilesystem

debugfs /dev/sda1                   ==> Manipulate FS here

You can use debugfs to undelete a file by using its inode and indicating a file

In the case of inodes are full, You need to remove unused files from the filesystem to make Inode free. There is no option to increase/decrease inodes on disk. Its only created during the creation of filesystem on any disk.
 * Free Inodes on Filesystem

= File Descriptor =


 * A FD or fildes is an abstract indicator (handle) used to access a file or other input/output resource, such as a pipe or network socket.
 * File descriptors form part of the POSIX application programming interface.

Int value 	Name 	 	 	symbolic constant	 file stream 0 	 	Standard input 	 	STDIN_FILENO 	 	 stdin 1 	 	Standard output 	STDOUT_FILENO 	 	 stdout 2 	 	Standard error 	 	STDERR_FILENO 	 	 stderr
 * Each Unix process (except perhaps a daemon) should have three standard POSIX file descriptors corresponding to the three standard streams:

root@myserver:/proc/28994/fd# ls -lh total 0 lr-x-- 1 root root 64 Nov 29 17:04 0 -> /dev/null l-wx-- 1 root root 64 Nov 29 17:04 1 -> /dev/null lrwx-- 1 root root 64 Nov 29 17:04 10 -> /tmp/.ZendSem.tjJF6q (deleted) lrwx-- 1 root root 64 Nov 29 17:04 11 -> anon_inode:[eventpoll] l-wx-- 1 root root 64 Nov 29 17:04 2 -> /var/log/apache2/error.log
 * Example


 * In Unix, File Descriptors index into a Per-Process File Descriptor Table maintained by the Kernel.
 * That in turn indexes into a system-wide table of files opened by all processes called the File Table.
 * This table records the mode with which the file has been opened: for Reading, Writing, Appending, and possibly other modes.
 * It also indexes into a third table called the Inode Table that describes the actual underlying files.
 * To perform input or output, the process passes the file descriptor to the kernel through a system call.
 * The Kernel will access the file on behalf of the process.
 * The process does not have direct access to the file or inode tables.
 * The set of file descriptors open in a process can be accessed under the path /proc/PID/fd/, where PID is the process identifier.
 * File Descriptors can refer to any Unix file type named in a file system.
 * As well as regular files, this includes directories, block and character devices (called "special files"), Unix domain sockets, and Named Pipes.
 * File Descriptors can also refer to other objects that do not normally exist in the file system - Anonymous Pipes and Network Sockets.


 * Using lsof to Recover Deleted Files

Source masterkenneth.com

lsof | grep myfile        ==> get pid, say 11230 ls -l /proc/11230/fd/4 cp /proc/11230/fd/4 myfile


 * Recover Disk space by deleted file; truncate the file



= Sort links vs Hard link =

drwxr-xr-x 6 aman aman    4096 Mar 30 11:50  Documents drwxr-xr-x 3 aman aman    4096 Sep 15 19:11  Downloads ^
 * Links and index number in Linux
 * In the output of ls -l, the column following the permissions and before owner is the link count.
 * Link count is the number of Hard Links to a file.
 * A link is a pointer to another file.
 * There are two types of links:

ln -s /home/bob/sync.sh filesync
 * Symbolic links (or Soft Links)
 * A separate file whose contents point to the linked-to file.
 * When creating a Sym link, first refer to the name of the original file and then to the name of the link:


 * Editing Sym link is like directly edit the original file.
 * If we delete or move the original file, the link will be broken and our filesync file will not be longer available.

ls -l filesync lrwxrwxrwx 1 root root 20 Apr 7 06:08 filesync -> /home/bobbin/sync.sh
 * The ls -l command shows that the resulting file is a symbolic link:


 * The contents of a symbolic link are the name of target file only.
 * The permissions on the symbolic link are completely open.
 * This is because the permissions are not managed
 * The original file is just a name that is connected directly to the inode, and the symbolic link refers to the name.
 * The size of the symbolic link is the number of bytes in the name of the file it refers to, because no other information is available in the symbolic link.

find. -type l -ls ls -la | grep "\->"
 * Find Sym Links


 * Hard links


 * The identity of a file is its inode number, not its name.
 * A hard link is a name that references an inode.
 * It means that if file1 has a hard link named file2, then both of these files refer to same inode.
 * So, when you create a hard link for a file, all you really do is add a new name to an inode.
 * there is no difference between the original file and the link: they are just two names connected to the same inode.

ln /home/bob/sync.sh synchro
 * Create a Hard link:

ls -il /home/bob/sync.sh synchro 517333 -rw-r- 2 root root 5 Apr 7 06:09 /home/bob/sync.sh 517333 -rw-r- 2 root root 5 Apr 7 06:09 synchro
 * Compare:


 * The directories cannot be hard linked as Linux does not permit this to maintain the acyclic tree structure of directories.
 * A hard link cannot be created across filesystems. Both the files must be on the same filesystems, because different filesystems have different independent inode tables (two files on different filesystems, but with same inode number will be different).

/home/bob/sync.sh /root/synchro
 * How to find hard link in Linux
 * 1) find / -inum 517333


 * Remove files
 * When rm command is issued, first it checks the link count of the file.
 * If the link count is greater than 1, then it removes that directory entry and decreases the link count.
 * Still, data is present, nor is the inode affected.
 * And when link count is 1, the inode is deleted from the inode table, inode number becomes free, and the data blocks that this file was occupying are added to the free data block list.

= Hosts file =


 * All operating systems with network support have a hosts file in order to translate hostnames to IP addresses.
 * The file /etc/hosts started in the old days of DARPA as the resolution file for all the hosts connected to the internet (before DNS existed).
 * It has the maximum priority ahead of any other name system

hosts:         files dns
 * Order of name resolution is actually defined in /etc/nsswitch.conf, which usually has this entry:


 * This means "try files (/etc/hosts); and if it fails, try DNS."
 * i.e. If the host name is not found there, then consult the remote DNS name servers identified by the /etc/resolv.conf file.
 * This order could be changed or expanded.


 * As a single file, it doesn't scale well: the size of the file becomes too big very soon.
 * That is why the DNS system was developed, a hierarchical distributed name system.
 * It allows any host to find the numerical address of some other host efficiently.


 * On Linux and Mac OS it is located here: /etc/hosts
 * On Windows it is under: Windows\System32\drivers\etc\


 * The hosts file contains lines of text consisting of an IP address field followed by One or More Host names.
 * Each field is separated by white space – tabs or spaces.
 * Comment lines are indicated by an octothorpe (#) in the first position.
 * Entirely blank lines in the file are ignored.
 * One name may resolve to several addresses (192.168.0.8 10.0.0.27).
 * However which one is used depends on the routes (and their priorities) set for the computer.

Block a website Handle an attack or resolve a prank Create an alias for locations on your local server Override addresses that your DNS server provides Control access to network traffic
 * By editing the hosts files, you can achieve:


 * IP-to-hostname conversion usually display only the first name found:

192.168.10.12 server.example.com myftp.example.com myhost myftp

$ ping myftp PING myhost.example.com (192.168.10.12) 56(84) bytes of data. 64 bytes from myhost.example.com (192.168.10.12): icmp_seq=1 ttl=64 time=0.023 ms 64 bytes from myhost.example.com (192.168.10.12): icmp_seq=2 ttl=64 time=0.028 ms

Note that we pinged myftp but results come from host myhost. This is a reliable hint that you are addressing an alias, not the actual host.

= Vlan Config =

= File permission =


 * Linux File Permission Basics


 * The first character represents the type of file.
 * The remaining nine bits in groups of three represent the permissions for the user, group, and global respectively.

File Type	       User	 Group	 Global d    Directory        rwx	 r-x	 r-x -    Regular file     rw-	 r--	 r-- l    Symbolic Link    rwx	 rwx	 rwx

Permission       On a file                     On a directory r (read)         read file content (cat)       read directory content (ls) w (write)        change file content (vi)      create file in directory (touch) x (execute)      execute the file              enter the directory (cd)
 * Permissions Meaning

Who (Letter)	Meaning u	       user g	       group o	       others a	       all
 * Targeted Users:

Binary	 Octal	Permission 000	 0	— 001	 1	–x 010	 2	-w- 011	 3	-wx 100	 4	r– 101	 5	r-x 110	 6	rw- 111	 7	rwx
 * Permissions Table:

chmod [who][+,-,=][permissions] filename
 * chmod Command Syntax and Options

chmod g+w ~/group-project.txt
 * Example:

chmod g=u ~/group-project.txt
 * The + operator grants permissions whereas the - operator takes away permissions.
 * Copying permissions is also possible:


 * The parameter g=u means grant group permissions to be same as the user’s.

chmod g+w,o-rw,a+x ~/group-project-files/
 * Multiple permissions can be specified by separating them with a comma, as in the following example:


 * Owner of the file is referred to as the user (e.g. u+x).

chmod -R +w,g=rw,o-rw, ~/group-project-files/
 * The -R option applies the modification to the permissions recursively to the directory specified:

chmod 600 .msmtprc chmod g-rwx,o-rwx .fetchmail
 * Restrict File Access: Remove all Group and World PermissionsPermalink


 * Octal Notation for File Permissions:

chmod u=rwx,g=rx,o= group-project.txt chmod 750 group-project.txt
 * The permissions to be set for file:

111 101 000 - rwx r-x ---
 * Disregarding the first bit, each bit that is occupied with a - can be replaced with a 0 while r, w, or x is represented by a 1:


 * This is called octal notation because the binary numbers are converted to base-8 by using the digits 0 to 7

Allows R,W,X permissions for the owner R permissions for the group and “world” users
 * Typical default permission: 744
 * Other default permissions are 600 or 644
 * For executable files, the equivalent settings would be 700 and 755


 * umask
 * Known as User Mask or User File creation MASK.
 * While creating a file or directory, by default a set of permissions are applied.
 * These default permissions are viewed by umask command.
 * For safety reasons all Unix systems doesn't provide execution permission to newly created files.
 * The 'mkdir -m' command can be used to set the mode.

mkdir -m 777 dir1 mkdir -m 000 dir2

cp -p list dupli.txt
 * Preserves the permissions and time stamps from source file:

= Commands =

CPU
lscpu lshw -C CPU hardinfo           ==>  sudo apt install hardinfo nproc sudo dmidecode -t 4 cpuid cat /proc/cpuinfo cat /proc/cpuinfo | grep processor | wc -l
 * CPU Info


 * The number of processors shown by /proc/cpuinfo might not be the actual number of cores on the processor.
 * For example a processor with 2 cores and hyperthreading would be reported as a processor with 4 cores.
 * If there are 4 different core ids, this indicates that there are 4 actual cores.

core id        : 0 core id        : 2 core id        : 1 core id        : 3
 * 1) cat /proc/cpuinfo | grep 'core id'

top -o %CPU htop vmstat sar 1 3       ==>  yum install sysstat iostat        ==>  yum install sysstat
 * CPU Usage

top - 01:07:37 up 2:40,  1 user,  load average: 0.37, 0.37, 0.39 Tasks: 286 total,  1 running, 285 sleeping,   0 stopped,   0 zombie %Cpu(s): 4.7 us,  1.6 sy,  0.0 ni, 93.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st MiB Mem :  15935.7 total,   9403.3 free,   3045.2 used,   3487.1 buff/cache MiB Swap:  4100.0 total,   4100.0 free,      0.0 used. 11720.3 avail Mem
 * Top Command

PID USER     PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND 6865 aman     20   0  982620  85280  53716 S   6.2   0.5   2:52.77 Xorg 10082 aman     20   0 3537624 285448 118848 S   6.2   1.7   5:45.24 gnome-shell

CPU Section

us     user cpu time           % CPU time spent in user space sy     system cpu time         % CPU time spent in kernel space ni     user nice cpu time      % CPU time spent on low priority processes id     idle cpu time           % CPU time spent idle wa     io wait cpu time        % CPU time spent in wait (on disk) hi     hardware irq            % CPU time spent servicing/handling hardware interrupts si     software irq            % CPU time spent servicing/handling software interrupts st     steal time              % CPU time stolen from a virtual machine

Main Section: %MEM   directly related to RES, percentage use of total physical memory by the process. VIRT   total memory that this process has access to shared memory, mapped pages, swapped out pages, etc. RES     total physical memory used shared or private that the process has access to. SHR    total physical shared memory that the process has access to.

RES is most close to the memory used by the process in memory, excluding what’s swapped out. This includes the SHR (shared physical memory) which mean it could have been used by some other process as well.

pgrep -n python pidof chrome              - return all PIDs pidof -s chrome           - return only 1 PID ps -C chrome -o pid=      - C = CMD
 * Obtain the PID:

Memory
dmidecode -t 17
 * Info

cat /proc/meminfo  ==> egrep --color 'Mem|Cache|Swap' /proc/meminfo top -o %MEM free -m total       used        free      shared  buff/cache   available Mem:         15935        3046        9470         767        3418       11787 Swap:         4099           0        4099
 * Usage

vmstat vmstat -s  ==> More detailed htop

ps -o pid,user,%mem,command ax | sort -b -k3 -r sudo pmap 917                                      ==> Libraries, other files, etc usage of memory sudo pmap 917 | tail -n 1                          ==> Total used by this process
 * Per Process usage check

HDD
du -h          ==> space by dir including all subdir in dir tree du -sh /etc/   ==> total disk space used by dir and suppress subdir du -ah /etc/ ==> see all files, not just directories:

df -h Filesystem    Type      Size  Used Avail Use% Mounted on    /dev/sda4      ext4       77G   51G   22G  71% / df -T -h       ==>  List Filesystem type as well df -t ext4     ==>  Only see ext4 file system df -a          ==>  List all filesystems that have a size of zero blocks as well df -i          ==>  Display File System Inodes

lsblk          ==>  Lists out all the storage blocks, which includes disk partitions and optical drives NAME  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda     8:0    0   1.8T  0 disk ├─sda1  8:1    0   500M  0 part /boot/efi ├─sda2  8:2    0   128M  0 part

sudo fdisk -l  ==> Partition & FS Type details parted         ==> List out partitions and modify them

Port
cat /sys/class/net/wlp2s0/operstate
 * Check Port state

Source: serverfault.com
 * Explanation of ip link command:

ip link ip link show INTERFACE as follows.

This is a port which is administratively up, but physically down: 2: eth0:  mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000

the UP indicates the system is configured to try and use the NIC for networking. The NO-CARRIER here tells what is the issue.

This is a port that is administratively down (its physical layer is 'up', technically - it is a VM): 3: eth1:  mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

Finally, this port is working normally: 2: eth0:  mtu 1500 qdisc fq state UP mode DEFAULT group default qlen 1000

It is administratively UP, the LOWER_UP indicates the physical layer is working (i.e. there is a carrier), and the second UP confirms (in effect) the IP layer is up.

IP
ip addr show  (ip a) ifconfig hostname -I ip route get 8.8.8.8 | head -1 | awk '{print $7}' ip route get 8.8.8.8 | head -1 | cut -d' ' -f7
 * Show info

nano /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=static IPADDR=192.168.1.2 NETMASK=255.255.255.0 ONBOOT=yes
 * Assign IP CentOS:

sudo nano /etc/network/interfaces
 * Ubuntu

auto ens31 iface ens31 inet dhcp

auto ens33 iface ens33 inet static address 192.168.0.2 netmask 255.255.255.0 network 192.168.0.0 broadcst 192.168.0.255 gateway 192.168.0.1 dns-nameservers 192.168.0.3

ifconfig eth0 10.10.10.45 netmask 255.255.255.0 up ip addr add 10.20.30.176/24 dev eth0
 * Other methods:

ifconfig eth0:100 192.168.66.1
 * IP Alias:

vconfig add eth0 700 ifconfig eth0.700 192.168.66.1
 * VLAN:

auto eth0.10 iface eth0.10 inet static address 192.168.1.61 netmask 255.255.255.0 gateway 192.168.1.11
 * VLAN & Alias:

auto eth0.10:1 iface eth0.10:1 inet static address 10.20.100.100 netmask 255.255.255.0

auto eth0.10:2 iface eth0.10:2 inet static address 10.20.100.200 netmask 255.255.255.0

Route
netstat -nr   (n => Numerical not hostname) ip route route route -n

U  route is up H   target is a host G  use gateway R  reinstate route for dynamic routing D  dynamically installed by daemon or redirect M  modified from routing daemon or redirect A  installed by addrconf C  cache entry !  reject route

Config Info
cat /etc/resolv.conf nmcli dev show | grep DNS systemd-resolve --status resolvectl status | grep -1 'DNS Server'

Host Command
host google.com host -t a google.com host -t mx google.com host -t soa google.com host -t cname files.google.com host -t txt google.com host google.com ns2.google.com     ==> Query a particular host host -t any google.com

DIG Command
dig google.com a dig google.com mx dig google.com ns dig google.com txt dig @ns1.google.com a dig @4.2.2.2 google.com soa         ==> SOA record dig +nssearch google.com           ==> SOA record dig +short google.com              ==> only IP address dig +noall +answer google.com      ==> Just answer line dig +noall +answer google.com any  ==> Just answers for all records

NSLOOKUP
nslookup yahoo.com                 ==> Find A Record nslookup 209.191.122.70            ==> Reverse Domain Lookup nslookup -query=mx www.yahoo.com   ==> Query MX (Mail Exchange) record nslookup -query=ns www.yahoo.com   ==> NS(Name Server) record nslookup -query=any yahoo.com      ==> query all Available DNS records nslookup -debug yahoo.com          ==> verbose information like TTL, etc

TCP Parameters
cat /proc/sys/net/ipv4/tcp_fin_timeout
 * MSL

To change it: echo 5 > /proc/sys/net/ipv4/tcp_fin_timeout

cat /proc/sys/net/ipv4/tcp_window_scaling
 * WSF

Kill Command
Signal Types: 1) SIGHUP    On POSIX-compliant platforms, SIGHUP ("signal hang up") is a signal sent to a process when its controlling terminal is closed. It was originally designed to notify the process of a serial line drop. 2) SIGINT 3) SIGQUIT   Same as SIGTERM but this can also dumps core 4) SIGILL     raised when an attempt is made to execute an invalid, privileged, or ill-formed instruction. 5) SIGTRAP 6) SIGABRT   Signal Abort, should generate a Core Dump 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV   Segmenation Fault - Accessing Memeory incorrectly; trying to modify read only memory; accessing memory outside of your process' mappings 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM   Used to terminate a process by Kill command. 16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR 31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 63) SIGRTMAX-1 64) SIGRTMAX


 * Default for Kill command is 15: TERM

= Process states in Linux =

R running or runnable    On run queue D uninterruptible sleep  Usually IO S  interruptible sleep    Waiting for an event to complete Z defunct/zombie         Terminated but not reaped by its parent T Stopped                Either by a job control signal or because it is being traced

= Character vs Block Devices =

This is one with which the Driver communicates by sending and receiving single characters (bytes, octets). Ex: serial ports, parallel ports, sounds cards.
 * Character ('c') Device:

This is one with which the Driver communicates by sending entire blocks of data. Ex: hard disks, USB cameras, Disk-On-Key.
 * Block ('b') Device

= System Calls = Source: usna.edu, github.io

System resources that OS is responsible for managing:

Hardware devices, such as keyboard, monitors, printers, hard drives, etc., are resources managed by the computer. When a programmer wishes to interact with these devices, a unified interface provided by the OS is used.
 * Device Management:

The invocation and execution of a program, a process, is managed by the OS, including managing its current state, running or stopped, as well as the loading of code.
 * Process Management:

The access to physical and virotual memory is controlled by the OS, and a programs memory layout and current allocations is carefully managed.
 * Memory Management:

The OS is also responsible for ensuring that programs can read and write from the filesystem, but also that programs don't corrupt the file sysystem or access files/directory that they do not have permission to.
 * File System Management:

List of System Calls
= Piping =


 * Pipe command allows sending the output of one command to another.
 * Piping can redirect the standard output, input, or error of one process to another for further processing.

Command-1 | Command-2 | …| Command-N


 * Syntax for unnamed pipe command is the | character.
 * Here, the pipe cannot be accessed via another session.
 * it is created temporarily to accommodate the execution of Command-1 and redirect the standard output.
 * It is deleted after successful execution.


 * Named Pipe

mkfifo  mknod  p
 * A named pipe can last until as long as the system is up and running or until it is deleted.
 * It is a special file that follows the FIFO (first in, first out) mechanism.
 * It can be used just like a normal file - you can write to it, read from it, and open or close it.
 * To create a named pipe, the commands are:
 * This creates a named pipe file that can be used even over multiple shell sessions.

= Init vs Systemd =

Source tecmint.com


 * Init (Initialization)


 * The init is a daemon process which starts as soon as the computer starts and continue running till, it is shutdown.
 * In-fact init is the first process that starts when a computer boots, making it the parent of all other running processes directly or indirectly and hence typically it is assigned “pid=1“.
 * If somehow init daemon could not start, no process will be started and the system will reach a stage called “Kernel Panic“.
 * Also called System V init as System V is first commercial UNIX Operating System designed and usages of init is identical to most Linuxes.


 * Systemd (System Management Daemon)
 * A init replacement daemon designed to start process in parallel.
 * Implemented in a number of standard distribution – Fedora, OpenSuSE, Arch, RHEL, CentOS, etc.
 * Similar to init, systemd is the parent of all other processes directly or indirectly.
 * It is the first process that starts at boot hence typically assigned a “pid=1“.
 * It was designed to overcome the shortcomings of init.
 * It itself is a background processes which is designed to start processes in parallel, thus reducing the boot time and computational overhead.
 * It has a lot other features as compared to init.


 * Need to replace Init
 * Init process starts serially i.e., one task starts only after the last task startup was successful and it was loaded in the memory.
 * This often resulted into delayed and long booting time.
 * Systemd was not designed for speed but for getting the things done neatly which in turns avoid all the UN-necessary delay.

Clean, stateforward and efficient design. Simpler boot process. Concurrent and parallel processing at boot. Better API. Simple Unit Syntax. Ability to remove optional components. Low memory footprints. Improved technique to express dependencies. Initialization instruction written in config file and not in shell script. Make use of Unix Domain Socket. Job Scheduling using systemd Calendar Timers. Event Logging with journald. Choice of logging System events with systemd as well as syslog. Logs are stored in binary file. systemd state can be preserved to be called later in future. Track process using kernel’s cgroup and not PID. Users login managed by systemd-logind. Better integration with Gnome for interoperability.
 * Features of systemd

Everything at one place. Not POSIX standard.
 * Bottlenecks systemd


 * Upstart – A init replacement daemon implemented in Ubuntu GNU/Linux and designed to start process asynchronously.

= Troubleshooting Slow Server =

Source: scoutapm.com

1: Check I/O wait and CPU Idletime Look for "wa" (I/O wait) and "id" (CPU idletime) I/O Wait represents the amount of time the CPU waiting for disk or network I/O. Anything above 10% I/O wait should be considered high. CPU idle time is a metric you WANT to be high If your idle time is consistently above 25%, consider it "high enough"

2: IO Wait is low and idle time is low: check CPU user time Look for the %us column (first column), then look for a process or processes that is doing the damage. If %usertime is high, see which program is monopolizing the CPU Be default, top sorts the process list by %CPU, so you can just look at the top process or processes. If situation seems anomalous: kill/restart the offending processes. If situation seems typical given history: upgrade server or add more servers.

3: IO wait is low and idle time is high Your slowness isn't due to CPU or IO problems, so it's likely an application-specific issue. Slowness might be caused by another server in your cluster or by an external services like DB If you suspect another server in your cluster use - Strace and Lsof Strace will show you which file descriptors are being read or written to. Lsof can give you a mapping of those file descriptors to network connections.

4: IO Wait is high: check your swap usage Use top or free -m Cache swaps will monopolize the disk Processes with legitimate IO needs will be starved for disk access. In other words, checking disk swap separates "real" IO wait problems from what are actually RAM problems that "look like" IO Wait problems.

5: Swap usage is high High swap usage means that you are actually out of RAM.

6: Swap usage is low Low swap means you have a "real" IO wait problem iotop is an awesome tool for identifying io offenders.

7: Check memory usage Once top is running, press the M key - this will sort applications by the memory used. Important: don't look at the "free" memory -- it's misleading. To get the actual memory available, subtract the "cached" memory from the "used" memory. This is because Linux caches things liberally, and often the memory can be freed up when it's needed. A memory leak can be satisfactorily addressed by a one-time or periodic restart of the process. If memory usage seems anomalous: kill the offending processes. If memory usage seems business-as-usual: add RAM to the server, or split high-memory using services to other servers.


 * References