Audit all the things

Auditing all the things: The future of smarter monitoring and detection

Jen Andre

Founder @threatstack

@fun_cuddles

• Started with question on twitter:
• • Can you produce a list of all process running on your network?
  • But then expanded… wanted to know everything
• Why? Is there a reason to be this paranoid?
• • prevention fails. 
• should you care?
• • if you’re a startup about pets and you get hacked, you just change all passwords
  • but if you’re a pharmaceutical company, then you really do care. 
• “We found no evidence that any customer data was accessed, changed or lost”
• • Did you look for evidence?
  • Do you really know what happened?
  • If you log everything (the right things), then you don’t have to do forensic evidence.
• “We’re in the cloud!”
• Continuous security monitoring
• • auditing + analytics + automation
• Things to monitor:
• • Systems: authentications, process activity, network activity, kernel modules, file system
  • Apps: authentications, db requests, http requests
  • services: AWS api calls, SaaS api calls
• In order to do:
• • Intrusion detection
  • “active defense”
  • rapid incident response
• “Use the host, Luke”
• apt-get install audit
• • pros:
  • • super powerful
    • build into your kernel
    • relatively low overhead
  • you can audit logins, system calls.
• auditd
• • the workings:
  • • userland audit daemon and tools <- link="" net="" socket=""> kernel thread queue <- audit="" doing="" kernel="" li="" messages="" things="" threads="">
    • /var/log/audit
  • not so nice:
  • • obtuse logging
    • enable rate limiting or it could ‘crash’ your box
    • • auditctl -b 1000 -r 1500 # 100 buffers, 15000 eps max)
• alternative: connect directly to net link socket and write your own audit listening
• • wrote a JSON format exporter
  • luajit! for filtering, transformation & alerting
• authentications
• • who is logging in and from where?
  • Can use wtmp
  • • can turn into json
  • auditd also returns login information
  • pam_loginid will add a session id to all executed commands so you can link real user to sudo’d commands
• Detecting attacks
• • most often a long time goes by before people are hacked, sometimes years.
  • often they get a phone call from the government: hey, you’ve got servers sending data to china.
  • the hardest attack to detect is when the attacker is using valid credentials to access it.
  • things to think about:
  • • is that user running commands he should;’t be?
    • • ex: why is anyone except chef user running gcc on a production system?
    • why is a server that only accepts inbound connections suddenly making outbound ones?
    • • or why connecting to machines other than expected ones?
    • are accounts logging in from unexpected locations? (or at unexpected times)
    • are files being copied to /lib /bin, etc.
• Now go and audit!