Log files are one of the first things we look at when something fails. Some other times, we look at them because they are also useful to extract statistics. However, log files are not always easy to read, since too much information is displayed. This drawback can be overcome using vim syntax files, which enhance readability, making information more accessible and helping to detect problems faster.

Let’s take a look at the following example (CNR log with Intraway Extensions):

There is too much information and everything has the same level of “Importance”. So much text with this high contrast is difficult to read.

Vim syntax highlighting

Luckily, with vim we can change this very easily. There are 2 steps that are hard and they don’t have to do with vim:

• Decide what to highlight.

• Decide how to highlight (colors, contrast).

It’s not always easy to strike a good balance between what is highlighted and how much contrast there is. Too much highlighting or too much contrast could be even worse than no highlighting at all. As a rule of thumb, you should give a stark contrast to things that don’t appear often and are important for you. However, try not to highlight too many things.

In this example, we want the following:

• Better visibility of dates.

• Warnings and errors are VERY important!

• Tags are useful (in our logs, tags are between brackets).

• Mac addresses and IPs are important.

• Type of packets (DISCOVERs, REQUESTs, etc.) are rare and important.

• Selection tags.

Note: gvim has a much richer highlighting feature than vim. This syntax file was made to be used with gvim.

Look at the following syntax file:

" iwaylog.vim
" Vim syntax file
" Based on messages.vim - syntax file for highlighting kernel messages
if exists("b:current_syntax")
 finish
endif
“ we use solarized scheme as a base. It has a very good contrast 
“ for log files
colorscheme solarized
“ We want to match lines with keywords like FATAL or ERROR
syn match log_error 'c.*<(FATAL|ERROR|ERRORS|FAIL|FAILED|FAILURE).*'
“ The same with WARNING, but we want a different highlighting
“ for those lines.
syn match log_warning 'c.*<(WARNING).*'
“ Things between quotes are strings
syn region log_string start=/"/ end=/"/ skip=/\./
syn match log_number '0x[0-9a-fA-F]*|[<[0-9a-f]+>]|<d[0-9a-fA-F]*'
syn match log_number '{d{6,}}'
“ Match the date: dd/mm/yyyy hh:mm:ss
syn match log_date 'd{2}/d{2}/d{4}s*d{2}:d{2}:d{2}'
“ A component is something between brackets
syn match component '[[^]]*]'
“ Match IP addresses
syn match internet 'dd*.dd*.dd*.dd*'
“ Match IPv6 address, macaddresses and some other stuff
syn match internet '(x*:){5,}xx*'
syn keyword dhcp_keywords DISCOVER OFFER REQUEST ACK INFORM RENEW
syn keyword dhcp_keywords DHCPDISCOVER DHCPOFFER DHCPREQUEST DHCPACK DHCPINFORM DHCPRENEW
syn keyword dhcp_keywords SOLICIT ADVERTISE REPLY
syn keyword hard_keywords CMTS MTA CPE CM
“ Now we apply color
hi def link log_string String
hi def link log_error ErrorMsg
hi def link log_warning WarningMsg
hi def log_date guifg=#bbbbbb
hi def component guifg=#bbbbbb
hi def internet guifg=#cccc33
hi def internet guifg=#cccc33
hi def dhcp_keywords guifg=#bbbbdd
hi def hard_keywords guifg=#ddddff guibg=#444444

This file should be placed in ~/.vim/syntax/iwaylog.vim

On your .vimrc file:

au BufNewFile,BufRead *.iwaylog set filetype=iwaylog
au BufNewFile,BufRead *.log set filetype=iwaylog

This tells vim to use our syntax to log or iwaylog files (for example, files ending with .log or .iwaylog).

The syntax file is quite straightforward. Let’s explain the important parts:

• keywords: These are easy, for example:
  syn keyword hard_keywords CMTS MTA CPE CM

• matches: These are good for regex, for example:
  syn match internet ‘dd*.dd*.dd*.dd*’ matches things like 10.0.123.5
  syn match internet ‘(x*:){5,}xx*’ matches things like MAC addresses (00:11:22:33:44:55) or IPv6 (2001:470:5::1).

• links: We define how to treat keywords or matches (in our case, the groups we used for links are already defined and have the coloring defined in solarized theme). For example:
  hi def link log_warning WarningMsg treats log_warning matches as WarningMsg.

• Custom coloring: Here we tell vim how to highlight keywords and matches, setting the foreground and background color. Notice that in our case we are setting the colors for the gui (gvim).

For further information, type :help syntax in vim.

Result

This is what we get after we apply the syntax highlighting with gvim.

Notice that:

• IP addresses (v4 and v6) and MAC addresses are very easy to find.
• Warning lines stand out (though it is not shown in this example, error lines stand out even more).
• Tags are a bit different from normal text, but not so much.
• Important keywords like “CM”, “CPE”, “MTA” can be found immediately.

Conclusion

With this new highlighting:

• Log files are easier to read and understand.
• We can find errors quickly.
• Warning and error lines are almost impossible to overlook.
• All the important data stands out.

If you would like to learn more about us, please click the following link to visit our website: www.intraway.com.