Monday, November 17, 2014

Monitor Outgoing Internet Connections - #2 (Interim)

Originally posted at OpenWRT forum at 

Hi folks,
short update with option 3 (DNSMASQ logging) after 24 hrs with one device connected:
- browsing experience on iPad is NOT slower than normal - great!
- router is working fine (my log/grep process taking 1% CPU/Mem)
- logfile has grown to 800kB in a day with 7500 log entries (plenty of memory left...)
- download log to PC and quick analysis in Excel (pivot table & little string formatting)
- quick graphs to show #of DNS queries by minute, top-requested domain names

And voila, here you are: 
DNS Queries by Minute/by Domain

Summary: Option 3 (DNS Query logging)
- no noticable performance degredation
- logfile size is manageable
- detail level okay (domain name only, not the full URL)
Improvement ideas:
- switch more devices to my modded DNS server (performance? log size?...)
- check that really every connection shows up in DNS log (at least per minute)
- restart log process every day, and offload previous day via Email/FTP
- minimize logfile by writing only relevant text (grep/awk/sed magic)
- create log analysis files in HTML with charting via google chart api?
- start filtering all these advertising trackers
Comments to my commenters:
  • CPU power on router is fine (1%), analyis is done in Excel. I'm not so interested in bandwidth tracking, more in domains/URLs with traffic
  • yes, DNS-logging can be circumvented by changing DNS setting on client. And, my log does not show traffic to IP addresses (without hostnames). I want to log/observe - not control access.

Also, TCPDUMP does NOT run in background even when starting from scripts - google "tcpdump in background" for plenty of people having tried this route (and I did not find a success story).

Stay posted for updates.....

No comments:

Post a Comment