LSOF (Listing of open files)

lsof stands for list open files. Which will list information about files opened by processes.

Command to show all opened Internet sockets
#lsof -i

Command to print all tcp connections
#lsof -i tcp

Command to print all udp connections
#lsof -i udp

Command to find who are the users using the folder right now
#lsof +D /tmp 

Command to print who and what are the files using in /dev/sda5 partitions
#lsof /dev/sda5 

Command to list all the files associated with process ID 2665
#lsof +p 2665 

Command to listing of files for processes executing the command that begins with vmstat
#lsof -c vmstat 

Listing of files for processes executing the command that begins with character a
#lsof -c a 

Command to print all the files opened by the user ashish
#lsof -u ashish

Consider that you are usnig nfs to transfer files between multiple systems in you local LAN. Assume that the folder which shares for NFS is /sharing, if you what to find out who all are the user’s using the folder means use the below command

 #lsof +D /sharing

 Lsof being asked for information about a file

bash-3.00# lsof /var/run/sendmail.pid COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sendmail 605 root 8wW VREG 281,3 32 8778600 /var/run/sendmail.pid

As the output shows, /var/run/sendmail.pid is owned by PID 605, which is sendmail, and has been opened for writing with an exclusive lock. If for some reason you needed to get rid of the file, the intelligent thing to do would be to stop the process, rather than just deleting the file. Otherwise, the daemon might fail to start properly next time, or another instance might start up later and cause contention.

Sometimes you know only that a file is open at some part of the file system. When unmounting a file system, the operation fails if any files are open on the file system. You can use lsof to show all open files on a file system by specifying the name of the mount point.

  Using lsof to find out who is using a file system

bash-3.00# umount /export/home umount: /export/home busy bash-3.00# lsof /export/home COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME bash 1943 root cwd VDIR 136,7 1024 4 /export/home/sean bash 2970 sean cwd VDIR 136,7 1024 4 /export/home/sean ct 3030 sean cwd VDIR 136,7 1024 4 /export/home/sean ct 3030 sean 1w VREG 136,7 0 25 /export/home/sean/output

In this example, a user (sean) is doing some work in his home directory. There are two instances of bash (a shell) running, with the current directory being sean's home directory. There is also an application named ct that is running out of the same directory and has its standard output (file descriptor 1) redirected to a file called output. To successfully unmount /export/home, these processes must be stopped, hopefully after giving the user a call to make sure it's all right.

This example shows how the current working directory of an application is important, because it still holds a file resource and can prevent a file system from being unmounted. This is why most daemons (background processes) change their directory to the root, or a service-specific directory, such as /var/spool/mqueue in the case of sendmail, to prevent the daemon from preventing an unrelated file system from being unmounted. Had sendmail been started from /export/home/sean and not changed its directory to /var/spool/mqueue, it would have had to be stopped before unmounting /export/home.

If you are interested in the open files under a directory that is not a mount point, you must specify the name of the directory with +dor +D, depending on if you need to recurse into subdirectories (+D), or not (+d). For instance, to see all open files under /export/home/sean, use lsof +D /export/home/sean. This is a subtle difference from the previous example, where the directory in question was a mount point, and is just a limitation of the way lsof and the kernel interact. It also brings up a potential problem in that lsof /export/home is different from lsof /export/home/ (note the trailing slash). The first works because it refers to the mount point. The second doesn't produce any output because it refers to the directory. You might run into this if you use tab completion in your shell, which helpfully adds the trailing slash. In this case, you either have to remove the slash or specify the directory with +D. The first is the preferred method, because it is faster than specifying an arbitrary directory.

fuser -mu /export/home

/export/home: 25781c(sean)

kill -9 25781
# umount /export/home

If you still cannot unmount the fs, or fuser hangs, try the "lazy" option to umount:
# umount -f -l /export/home

This will force the unmount, and to quote the man page:


# fuser -km /export/home
Where,

• -k : Kill processes accessing the file.
• -m : Name specifies a file on a mounted file system or a block device that is mounted. In above example you are using /export/home

Recovering a deleted file

A common thing that happens when a UNIX machine is broken into is that the log files are deleted to cover the attacker's tracks. Administrative error can also cause important files to be inadvertently deleted, such as accidentally removing the active transaction log of a database while cleaning up the old logs. Sometimes these files can be recovered, and lsof can help you.

When a process opens a file, it exists on disk even if deleted, as long as the process holds the file open. This means that the process doesn't know the file has been deleted; it can still read and write to the file descriptor it was granted when the file was opened. If you're not that process, the file is invisible because the directory entries have been removed.

Recall that in A diversion through the /proc directory section you had access file descriptors of a process by looking in the appropriate directory. You later discovered that lsof shows the file descriptor of a process and the associated file name. See where I'm going here?

If only it were that easy. When you pass lsof the name of a file, such as in lsof /file/I/deleted, it first uses the stat()system call to get information about the file, which is, unfortunately, gone. Depending on the operating system, lsof might be able to grab the name of the file from kernel memory. Below shows a Linux system where an Apache log has accidentally been deleted, and I am using the grep tool to find out if anyone has it open.

 Using lsof on Linux to look for a deleted file

# lsof | grep error_log httpd 2452 root 2w REG 33,2 499 3090660 /var/log/httpd/error_log (deleted) httpd 2452 root 7w REG 33,2 499 3090660 /var/log/httpd/error_log (deleted) ... more httpd processes ...

From this, you can see that PID 2452 has the file opened on file descriptors 2 (standard error) and 7. Thus, the data is available by looking at /proc/2452/fd/7

Looking at a deleted file through /proc

# cat /proc/2452/fd/7 [Sun Apr 30 04:02:48 2006] [notice] Digest: generating secret for digest authentication [Sun Apr 30 04:02:48 2006] [notice] Digest: done [Sun Apr 30 04:02:48 2006] [notice] LDAP: Built with OpenLDAP LDAP SDK

Linux was nice in that it saved the name of the file and even told us it was deleted. This is a handy thing to look for when investigating a compromised system because attackers often delete logs to hide their tracks. Solaris doesn't offer this information. However, knowing that error_log is used by the httpd daemon and that I can find the PID with the ps command, I can look at all the open files for the daemon.

 Looking for deleted files in Solaris

# lsof -a -p 8663 -d ^txt COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME httpd 8663 nobody cwd VDIR 136,8 1024 2 / httpd 8663 nobody 0r VCHR 13,2 6815752 /devices/pseudo/mm@0:null httpd 8663 nobody 1w VCHR 13,2 6815752 /devices/pseudo/mm@0:null httpd 8663 nobody 2w VREG 136,8 185 145465 / (/dev/dsk/c0t0d0s0) httpd 8663 nobody 4r DOOR 0t0 58 /var/run/name_service_door (door to nscd[81]) (FA:->0x30002b156c0) httpd 8663 nobody 15w VREG 136,8 185 145465 / (/dev/dsk/c0t0d0s0) httpd 8663 nobody 16u IPv4 0x300046d27c0 0t0 TCP *:80 (LISTEN) httpd 8663 nobody 17w VREG 136,8 0 145466 /var/apache/logs/access_log httpd 8663 nobody 18w VREG 281,3 0 9518013 /var/run (swap)

I use the -a and -d parameters to filter my output to exclude code segments, which I know aren't what I'm looking for. A look at theName column shows that two of the files (FDs 2 and 15) have the disk name instead of a file name and that they're of type VREG(regular file). In Solaris, a deleted file shows up as the name of the disk that the file resides on. This is your clue that the FD refers to a deleted file. Indeed, a look at /proc/8663/fd/15 gives me the data I'm looking for.

When you can view the data through the file descriptor, you can copy it into a file with I/O redirection, such as in cat /proc/8663/fd/15 > /tmp/error_log. At this point, you can stop the daemon (which eliminates the FD and, hence, the deleted file), copy the temporary file to where it is expected, and then restart the daemon.

This method of recovering deleted files is handy for many applications, especially log files and databases. As you can see, some operating systems (and versions of lsof) make it easier than others to find the data.

Finding network connections

Network connections are also files, which means that lsof can also get information about them. You saw an example of this above. It assumed you already knew the PID, which is not always the case. If you know only the port, use the -i parameter to search using socket information. Below shows a search for TCP port 25.

 Looking for the process listening on port 25

# lsof -i :25 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sendmail 605 root 5u IPv4 0x300010ea640 0t0 TCP *:smtp (LISTEN) sendmail 605 root 6u IPv6 0x3000431c180 0t0 TCP *:smtp (LISTEN)

The lsof utility expects that you will pass it something in the form of protocol:@ip:port, where the protocol is TCP or UDP (and optionally prefixed by 4 or 6 to refer to the version of IP), the IP is a resolvable name or IP address, and the port is a number or name (out of /etc/services) representing the service. One or more elements (port, IP, protocol) are required. Above, :25 refers to port 25. The output shows that process 605 is listening on port 25 using both IPv6 and IPv4. If you're not interested in IPv4, you can change the filter to 6:25 to refer to an IPv6 socket listening on port 25, or simply 6 to mean all IPv6 connections.

In addition to seeing what daemons are listening, lsof can also spy on what connections are happening, again using the -i parameter. Below shows a search for all connections to or from 192.168.1.10.

Searching for active connections

# lsof -i @192.168.1.10 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 1934 root 6u IPv6 0x300046d21c0 0t1303608 TCP sun:ssh->linux:40379 (ESTABLISHED) sshd 1937 root 4u IPv6 0x300046d21c0 0t1303608 TCP sun:ssh->linux:40379 (ESTABLISHED)

In this example, there are two IPv6 connections between sun and linux. A closer look shows that the connections are owned by two separate processes, but they're the same because both the hosts are the same and so are the ports (ssh and 40379). This is caused by the connection coming into a master process that forks off a handler, passing it the socket. You can also see that the machine called sun is using port 22 (ssh), while linux has a port of 40379. This indicates that sun is the recipient of the connection because it is the well-known port associated with the service. 40379 is the source, or ephemeral port, and is only significant for this connection.

Because, at least in UNIX, a socket is just another file, lsof can get detailed information about the connections and find out who is responsible for them.

How to check if a file is locked in Linux?

Answer:

Suppose a file test.txt is being locked by a program, e.g. using the flock system call, how can we know if this file is really being locked?

# lsof test.txt
COMMAND  PID USER   FD   TYPE DEVICE SIZE   NODE NAME
perl    5654 john 3uW  REG    8,1    1 983057 test.txt

The W means the file is currently held by an exclusive lock

The easiest way to get all the process IDs of a running program in Linux, see the following example:
pidof apache2 29747 29745 29700 29698 29696 29695 29687 29678 29677 29664 23931