Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts

Audit all the things

Auditing all the things: The future of smarter monitoring and detection
Jen Andre
Founder @threatstack
@fun_cuddles
  • Started with question on twitter:
    • Can you produce a list of all process running on your network?
    • But then expanded… wanted to know everything
  • Why? Is there a reason to be this paranoid?
    • prevention fails. 
  • should you care?
    • if you’re a startup about pets and you get hacked, you just change all passwords
    • but if you’re a pharmaceutical company, then you really do care. 
  • “We found no evidence that any customer data was accessed, changed or lost”
    • Did you look for evidence?
    • Do you really know what happened?
    • If you log everything (the right things), then you don’t have to do forensic evidence.
  • “We’re in the cloud!”
  • Continuous security monitoring
    • auditing + analytics + automation
  • Things to monitor:
    • Systems: authentications, process activity, network activity, kernel modules, file system
    • Apps: authentications, db requests, http requests
    • services: AWS api calls, SaaS api calls
  • In order to do:
    • Intrusion detection
    • “active defense”
    • rapid incident response
  • “Use the host, Luke”
  • apt-get install audit
    • pros:
      • super powerful
      • build into your kernel
      • relatively low overhead
    • you can audit logins, system calls.
  • auditd
    • the workings:
      • userland audit daemon and tools <- link="" net="" socket=""> kernel thread queue <- audit="" doing="" kernel="" li="" messages="" things="" threads="">
      • /var/log/audit
    • not so nice:
      • obtuse logging
      • enable rate limiting or it could ‘crash’ your box
        • auditctl -b 1000 -r 1500 # 100 buffers, 15000 eps max)
  • alternative: connect directly to net link socket and write your own audit listening
    • wrote a JSON format exporter
    • luajit! for filtering, transformation & alerting
  • authentications
    • who is logging in and from where?
    • Can use wtmp
      • can turn into json
    • auditd also returns login information
    • pam_loginid will add a session id to all executed commands so you can link real user to sudo’d commands
  • Detecting attacks
    • most often a long time goes by before people are hacked, sometimes years.
    • often they get a phone call from the government: hey, you’ve got servers sending data to china.
    • the hardest attack to detect is when the attacker is using valid credentials to access it.
    • things to think about:
      • is that user running commands he should;’t be?
        • ex: why is anyone except chef user running gcc on a production system?
      • why is a server that only accepts inbound connections suddenly making outbound ones?
        • or why connecting to machines other than expected ones?
      • are accounts logging in from unexpected locations? (or at unexpected times)
      • are files being copied to /lib /bin, etc.
  • Now go and audit!

Dan Slimmon on Monitoring

Car Alarms & Smoke Alarms & Monitoring
Dan Slimmon
@danslimmon
Senior Platform Engineer at Exosite
  • I work in Ops, so I wear a lot of hats
  • One of those is data scientist
    • Learn data analysis and visualization
    • You’ll be right more often and people will believe your right even more often than you are
  • A word problem
    • Plagiarism: 90% chance of positive
    • No Plagiarism: 20% chance of positive
    • Kids plagiarize 30% of the time
    • Given a random paper, what’s the probability that you’ll get a negative result?
      • 0.3*0.9 + 0.7*0.2 = 0.27+0.14=0.41
      • 59% likely to get negative result
    • If you get a positive result, how likely is it to really be plagiarized?
      • 65.8% likely
      • this is terrible.
      • Teachers will stop trusting the test.
  • Sensitivity & Specificity
    • Sensitivity: % of actual positives that are identified as such
    • Specificity: % of negative results that are indicated as such
    • Prevalence: percentage of people with problem
    • http://i.imgur.com/LkxcxLt.png
    • Positive Predictive Value: the probably that something is actually wrong.


  • Car Alarms
    • Go off all the time for reasons that aren’t someone stealing your car.
    • Most people ignore them.
  • Smoke Alarms
    • You get your ass outside and wait for the alarm to go off and the fire trucks.
  • We need monitoring tools that are both highly sensitive and highly specific.
  • Undetected outages are embarrassing, so we tend to focus on sensitivity.
    • That’s good.
    • But be careful with thresholds.
    • Too high, and you miss real problems. Too low, and too many false alarms.
    • There’s only one line with thresholds, so only one knob to adjust.
  • Get more degrees of freedom.
    • Hysteresis is a great way to add degrees of freedom. 
      • State machines
      • Time-series analysis
  • As your uptime increases, you must get more specific.
    • Going back to the chart…our positive predictive value goes down when there’s less actual problems.
  • A lot of nagios configs combine detecting problem with identifying what the problem is.
    • You need to separate those concerns.
    • Baron Schwartz says: Your alerting should tell you whether work is getting done.
    • Knowing that nginx is down doesn’t affect if your site is up. Check to see if you site is up (detecting problem), which is separate from source of problem (nginx isn’t running)
    • Alert on problems, bot on diagnosis.
Links:

Katherine Daniels on Monitoring

Katherine Daniels
@beerops
kd@gamechanger.io
  • The site is going down.
  • But everything seemed to be fine.
    • checked web servers, databases, mongo, more.
  • What was wrong? The monitoring tool wasn’t telling us.
  • One idea: monitor more. monitor everything.
    • But if you’re looking for a needle in a haystack, the solution is not to add more hay.
    • Monitoring everything just adds more stuff to weed through. Including thousands of things that might be not good (e.g. disk quote too high), but aren’t actually whats causing the problem.
  • Monitor some of the things. The right things. But which things? If we knew, we’d already be monitoring.
  • Talk to Amazon…
    • “try switching the load balancer”
    • “try switching the web server”
  • We had written a service called healthd that was supposed to monitor api1, and api2.
  • But we didn’t have logging for healthd, so we didn’t know what was wrong.
  • We needing more detail.
  • So adding logging, so we knew which API had a problem.
  • We also had some people who tried the monitor everything problem.
  • They uncovered a user who seemed to be scripting the site.
  • They added metrics for where the time was being spent with the API handlers
  • The site would go down for a minute each time things would blip.
  • We set the timeouts to be lower.
  • We found some database queries to be optimized.
  • We found some old APIs that we didn’t need and we removed them.
  • The end result was that things got better. The servers were mostly happy.
  • But the real question is: How did we get to a point where our monitoring didn’t tell us what we needed? We thought we were believers in monitoring. And yet we got stuck.
  • Black Boxes (of mysterious mysteries)
    • Using services in the cloud gives you less visibility
  • Why did we have two different API services…cohabiting…and not being well monitored?
    • No one had the goal of creating a bad solution.
    • But we’re stuck. So how do we fix it?
    • We stuck nginx in front and let it route between them.
  • What things should you be thinking about?
    • Services: 
      • Are the services that should be running actually running?
      • Use sensu or nagio
    • Responsiveness:
      • Is the service responding?
    • System metrics:
      • CPU utilization, disk space, etc.
      • What’s worth an alert depends: on a web server it shouldn’t use all the memory, on a mongo db it should, and if it isn’t, that’s a problem.
    • Application metrics?
      • Are we monitoring performance, errors?
      • Do we have the thresholds set right?
      • We don’t want to to look at a sea of red: “Oh, just ignore that. It’s supposed to be red.”
  • Work through what happens?
    • Had 20 servers running 50 queues each. 
    • Each one has its own sensu monitor. HipChat shows an alert for each one… a thousand outages.
  • You must load test your monitoring system: Will it behave correctly under outages and other problems?
  • “Why didn’t you tell me my service was down?” “Service, what service? You didn’t tell us you were running a service.”

Adrian Cockcroft on Monitoring Cloud Services

Adrian Cockcroft
@adrianco
Battery Ventures
Please, no More Minutes, Milliseconds, Monoliths… Or Monitoring Tools!
#Monitorama May 2014

  • Why at a Monitoring talk when I’m known as the Cloud guy?
  • 20 Years of free and open source tools for monitoring
  • “Virtual Adrian” rules
    • disk rule for all disks at once: look for slow and unbalanced usage
    • network rule” slow and unbalanced usage
    • …
  • No more monitoring tools
    • We have too many already
    • We need more analysis tools
  • Rule #1: Spend more time working on code that analyzes the meaning of metrics than the code that collects, moves, stores, and displays metrics.
  • What’s wrong with minutes?
    • Takes too long to see a problem
    • Something broke at 2m20s.
    • 40s of failure didn’t trigger (3m)
    • 1st high metrics seen at agent on instance
    • 1st high metric makes it to central server (3m30s)
    • 1 data collection isn’t enough, so it takes 3 data points (5m30s)
    • 5 minutes later, we take action that something is wrong.
  • Should be monitoring by the second
  • SaaS based products show what can be done
    • monitoring by the second
  • Netflix: Streaming metrics directly from front end services to a web browser
  • Rule #2: Metric to display latency needs to be less than human attention span (~10s)
  • What’s wrong with milliseconds?
    • Some JVM tools measure response times in ms
      • Network round trip within a datacenter is less than 1ms
      • SSD access latency is usually less than 1 ms
      • Cassandra response times can be less than 1ms
    • Rounding errors make 1ms insufficient to accurately measure and detect problems.
  • Rule #3: Validate that tour measurement system has enough accuracy and precision
  • Monolithic Monitoring Systems
    • Simple to build and install, but problematic
    • What is it goes down? gets deployed?
    • Should be a pool of analysis/display aggregators, a pool of distribution collection systems, all monitoring a large number of application. 
    • Scalability: 
      • problems scaling data collection, analysis, and reporting throughput
      • limitations on the number of metrics that can be monitored
  • In-Band, Out-of-band, or both?
    • In-band: can leave you blind during outage
    • SaaS: is out of band, but can also sometimes go down.
    • So the right answer is to have both: SaaS and internal. No one outage can take everything out.
  • Rule #4: Monitoring systems need to be more available and scalable than the systems being monitored.
  • Issues with Continouus Deliver and Microservices
    • High rate of change
      • Code pushes can cause floods of new instances and metrics
      • Short baseline for alert threshold analysis-everything looks unusual
    • Ephermeral configurations
      • short lifetimes make it hard to aggregate historical views
      • Hand tweaked monitoring tools take too much work to keep running
    • Microservices with complex calling patterns
      • end-to-end request flow measurements are very important
      • Request flow visualizations get very complicated
      • How many? Some companies go from zero to 450 in a year.
    • “Death Star” Architecture Diagrams
      • You have to spend time thinking about visualizations
      • You need hierarchy: ways to see micro services but also groups of services
  • Autoscaled ephermeral instances at Netflix (the old way)
    • Largest services use autoscaled red/block code pushes
    • average lifetime of an instance is 36 hours
    • Uses trailing load indicators
  • Scryer: Predictive Auto-scaling at Netflix
    • More morning load Sat/Sun high traffic
    • lower load on wednesday
    • 24 hours predicted traffic vs. ctually
    • Uses forward prediction to scale based on expected load. 
  • Monitoring Tools for Developers
    • Most monitoring tools are build to be used by operations people
      • Focus on individual systems rather than applications
      • Focus on utilization rather than throughput and response time.
      • Hard to integrate and extend
    • Developer oriented monitoring tools
      • Application Performance Measurement (APM) and Analysis
      • Business transactions, response time, JVM internal metrics
      • Logging business metrics directly
      • APIs for integration, data extraction, deep linking and embedding
        • deep linking: should be able to cut and paste link to show anyone exactly the data I’m seeing
        • embedding: to be able to put in wiki page or elsewhere.
  • Dynamic and Ephemeral Challenges
    • Datacenter Assets
      • Arrive infrequently, disappear infrequently
      • Stick around for three years or so before they get retired
      • Have unique IP and mac addresses
    • Cloud Assets
      • Arrive in bursts. A netflix code push creates over a hundred per minute
      • Stick around for a few hours before they get retired
      • Often reuse the IP and Mac address that was just vacated.
      • Use Netflix OSS Edda to record a full history of your configuration
  • Distributed Cloud Application Challenges
    • Cloud provider data stores don’t have the usual monitoring hooks: no way to install an agent on AWS mySQL.
    • Dependency on web services as well as code.
    • Cloud applications span zones and regions: monitoring tools also need to span and aggregate zones and regions. 
    • Monit
  • Links
    • http://techblog.netflix.com: Read about netflix tools and solutions to these problems
    • Adrian’s blog: http://perfcap.blogspots.com
    • Slideshare: http://slideshare.com/adriancockcroft
  • Q&A:
    • Post-commit, pre-deploy statistical tests. What do you test?
      • Error rate. Performance. Latency.
      • Using JMeter to drive.


Printable Obama for America AWS Architecture

The Obama for America AWS infrastructure diagram is a bundle of awesome. This mature cloud architecture diagram clearly shows the incredible web application architecture the Obama Campaign was able to create in a very short time.

A tiny portion of the AWS Obama for America 2012 infrastructure diagram
It's been available online at awsofa.info for a while, which is a nice, browsable Google Maps powered way to see the intricacies of the diagram.

However, if you've tried to print it, it's next to impossible. Using the browser File->Print just captures what is onscreen, so you're forced to choose between viewing a tiny portion of the diagram in high resolution or the entire diagram in a low, unusable resolution.

I really wanted to print the whole diagram on a plotter, and after playing with the HTML for an hour, I got a good 24"x200" print hanging on the wall at my office.

Now hanging on the wall at my work.

I'm sharing the high-resolution PDF that was an intermediate step in my workflow. It's 27"x240", but if you select scale to paper size on a plotter, this should print well on any large size. Note that it's considerably wider than it is tall, so it's helpful to set a custom size.

Download the PDF here. (60MB, PDF)

I hope this helps anyone trying to print it.

Update 2013-07-25: I just learned via JP that:

  1. Miles Ward is the author of the diagram. Thank you Miles for creating such an awesome, useful diagram.
  2. There's a hidden easter egg in the diagram. Look for a Take 5 candy bar.
  3. You can download a PNG of the diagram courtesy of JP. 

Scaling A Web App 1,000x in 3 Days

When I'm not writing science fiction novels, I work on web apps for a largish company*. This week was pretty darn exciting: we learned on Monday afternoon that we needed to scale up to a peak volume of 10,000 simultaneous visitors by Thursday morning at 5:30am.

This post will be about what we did, what we learned, and what worked.

A little background: our stack is nginx, Ruby on Rails, and mysql. The web app is a custom travel guide. The user comes into our app and either imports a TripIt itinerary, or selects a destination and enters their travel details. They can expressed preferences in dozens of different categories (think about rating cuisine types of restaurants, categories of attractions, etc.). They can preview their travel guide live, in the browser, and order a high quality paperback version.

Our site was in beta, and we've been getting a few hundred visitors a day. At 2pm on Monday we learned that we would be featured on a very popular TV show on Thursday morning (9:30am EST), and based on their previous experiences, we could expect anywhere from 20,000 to 150,000 unique visitors that day, with a peak load of 10,000 simultaneous visitors. This would be a 100x to 1,000x increase in the number of unique visitors that we would normally serve. We estimated that our peak volume would be 10,000 visitors at once, which would create a load of ~40,000 page requests per minute (rpm).

What we had going for us:
  • We were already running on Amazon Web Services (AWS), using EC2 behind a load balancer.
  • We already had auto-scale rules in place to detect when when CPU load exceeded a certain percentage on our app servers, and to double the number of EC2 instances we had.
  • We had reliable process for deploying (although this became a constraint later).
  • A month before we had spent a week optimizing performance of our app, and we had already cut  page load time by 30%, largely by reducing database calls.
On Monday we got JMeter installed on a few computers. With JMeter you can write a script that mimics a typical customer visit. In our case, we wrote a script that loaded the home page, selected a destination, went through the preferences, and went to check out. Although we didn't post to any pages (which we would have liked to, if we had more time), a number of database artifacts did get created as the simulated user went through the user interface. (This is important later.)

After some experimentation, we decided that we couldn't run more than 125 threads on a single JMeter instance. So if we wanted to simulate truly high loads, we needed to run JMeter on several computers. We also needed to make sure not to saturate any part of our local network. At a minimum, we made sure we didn't have all the machines going through the same router. We also VPNed into an entirely different site, and ran another client from a geographically distant location.

We quickly learned that:
  • we needed to change our initial auto-scale group from two servers to four.
  • we needed to reduce our CPU load threshold from 80% to 50%.
  • we needed to reduce the time we sustained that load from 5 minutes to 1 minute.
After making these changes, and letting the app servers scale to 16 servers, we saw that we were now DB bound. The DB server would quickly go to 100% load. We increased to the maximum AWS DB size (extra extra large, with extra CPUs and memory) but we still swamped the DB server. We watched the JMeter tests run, sending the average response times into the 30 second territory. We knew we were sunk if we didn't solve this problem. It was mid-day Tuesday.

I said "we knew we were sunk", but to be honest, this actually took some convincing. Some members of the team felt like any changes so close to the big day would be too risky to make. We could introduce a bug. But a 100% correct application experience that will definitely get buried under load will result in delivering no pages to anyone. That's 100% risk. After a few hours, we had everyone convinced that we had to solve the scalability issues. 

We briefly explored several different large scale changes we could have made, but ultimately judged to be too risky:
  • duplicating our entire stack, and using DynDNS or Amazon's Route 53 to route between them. Because we didn't have users accounts or other shared data, we could have done this in theory. If we had, we would later have needed to synchronize a bunch of databases through some scripts to collect all the customer data together. If we had a week or an Ops expert, we probably would have done this.
  • offloading part of the DB load by creating a read-only replica, and shunting some of the database calls to certain mostly-static data (e.g. list of destinations and attractions at those destinations) to the read-only database. We didn't do this mostly because we didn't know how we'd point some database queries at one db server, and others at a different db server.
  • using memcache or other mechanisms to hit only memory instead of database: this too would have been extensive code changes. Given the short timeframe, it felt too risky.
The action we did decide to take:
  • We had a few easily identified DB queries that ran with every page request, that loaded the same data each time. We rewrote the code to cache these in memory in global variables that stayed resident between calls. So they would get loaded once when the servers rebooted, and never again.
Around this time, which was Wednesday morning, we started to get smarter about how we used JMeter. Instead of slamming the servers with many hundreds of simultaneous threads from many clients and watching request times spiral around of control, we instead spun up threads slowly, let the servers stabilize, counted our total throughput in requests-per-second, and checked CPU load at that level, then increased the threads, and kept rechecking. We grabbed a flipchart, and started recording data where everyone could see it. Our goal was to determine the maximum RPM we could sustain without increasing response times.

As the day went on, we incrementally added code changes to cache some database requests, as well as other late-breaking requests we needed before we were live on Thursday. This is where our deployment process started to break down.

We have the following environments: local machines, dev, demo, staging, production. Normally, we have a test script that runs (takes about 40 minutes) before pushing to dev. The deployment script takes 15-30 minutes (80% of which is spent waiting for git clone. Why???) And we have to repeat that process to move closer and closer to production. 

The only environment that is a mirror of production is staging. If we had a code optimization we needed to performance test, it would take hours to deploy the change to staging. Conversely, we can (and did) log into the staging servers and apply the changes manually (ssh into server, vi to edit files, reboot mongrel), but we would need to repeat across four servers, bring down the auto-scale servers, rebuild the auto-scale AMI, then bring up the auto-scale servers. Even the short-cut process took 20 to 30 minutes.

We did this several times, and carefully keeping track of our data, we noticed that we were going in the wrong direction. We were able to sustain 15 requests per second (900 rpm), and the number kept get lower as the day went on. Now the astute database administrators who are reading this will chuckle, knowing exactly what is causing this. But we don't have a dedicated DBA on the team, and my DB skills are somewhat rusty.

We already had New Relic installed. I have always loved New Relic since I used it several years ago when I developed a Facebook app. It's a performance monitoring solution for Rails and other web app environments. I had really only used it as a development tool, as it lets you see where your app is spending it's time.

It's also intended for production monitoring, and what I didn't know up until that point is that if we paid for the professional plan, it would identify slow-running page requests and database queries, and provide quite a lot of detail about exactly what had executed.

So, desperate to get more data, we upgraded to the New Relic Pro plan. Instantly (ok, within 60 seconds), we had a wealth of data about exactly what DB requests were slow and where our app was spending it's time.

One DB query was taking 95% of all of our DB time. As soon as I saw that, and I remembered that our performance was getting worse during the day, it all snapped into place, and I realized that we had to have an unindexed field we were querying on. Because the JMeter tests were creating entries in the database tables, and because they had been running for hours, we had hundreds of thousands of rows in the database, so we were doing entire table scans.

I checked the DB schema.rb file, as saw that someone had incorrectly created a multi-column index. (In general, I would say to avoid multi-column indexes unless you know specifically that you need one.) In a multiple column index, if you specify columns A, B, and C in a query, you can use it to query against A, A and B, or A and B and C. But the columns are order dependent, and you cannot leave out column A.

That's exactly what had happened in our case. So I quickly created a migration to create the index we needed, we ran the migration on our server, let the server have five minutes to build the index, and then we reran the tests. We scaled to 120 requests per second (7,200 rpm), and the DB load was only 16%.

This was Wednesday afternoon, about 3pm. At this point we figured the DB server could handle the load from about 36,000 page requests per minute. We were bumping up against the app server CPU load again.

At some point during the day we had maxed out the number of EC2 instances we could get. We had another team doing scalability testing that day, and we were hitting some upper limit. We killed unnecessary running AMIs, we asked everyone else to stop testing, and we requested an increase in our upper limit via AWS, then went to our account rep and asked them to escalate the request.

By the time we left late Wednesday night, we felt confident we could handle the load on Thursday morning.

Thursday came, and we all had Google analytics, New Relic, and our backdoor interfaces covering all screens. Because it was all happening at 5:30am PST, we were spread across hotel rooms and homes. We were using Campfire for real-time chat.

The first load came at 5:38am, and the spike happened quickly: from 0 visitors we went to over a thousand visitors in just a minute.  Our page response times stayed rocky steady at 500ms, and we never experienced any delays in serving customer pages.

We had a pricing bug come up, and had to hot patch the servers and reboot them while under load (one at a time, of course), and still were able to keep serving up pages. Our peak load was well under the limits we had load tested, but was well over what we could have sustained on Monday afternoon prior to the changes we made. So the two and a half days of grueling effort paid off.

Lessons learned:
  • Had we paid for the Pro plan for New Relic earlier, we probably would have saved ourselves a lot of effort, but either way, I do believe New Relic saved our butts: if we hadn't gotten the data we needed on Wednesday afternoon, we would not have survived Thursday morning.
  • Load testing needs to be done regularly, not just when you are expecting a big event. At any point someone might make a database change, forget an index, or introduce some other weird performance anomaly that isn't spotted until we have the system under load. 
  • There needed to be a relatively safe, automated way to deliver expedited deployments to staging and production for times when you must shortcut the regular process. That shouldn't be dependent on one person using vi to edit files and not making any mistakes.
  • We couldn't have been scaling app instances and database servers so quickly if we weren't using a cloud service.
  • It really helped to have the team all in one place for this. Normally we're distributed, and that works fine when it's not crisis mode, but I wouldn't have wanted to do this and be 1,000 miles away.

*As usual, I'm speaking as and for myself, even when I'm writing about my day job, not my employer.

SQL Recipe: Calculating Ratios of Things

Let's say we have an sql table of jobs. They can be in any of several states. They have a starte time and a completed time. And they have have a type of A or B.

type started completed state
A9:01am 9:02am completed
A9:10am 9:14am aborted
B9:20am 9:21am cancelled
A9:20am 9:22am completed

Let's say we want to know for A and B, what percentage of jobs complete. We'd also like to know what percentage of jobs finish under a certain time threshold. Now this is drastically simplified, but you can imagine that job type could be any complex set of variables you want to analyze by: region, language, type of rendering, whatever.

Here's a query that will calculate the ratio of completed jobs by type:

select type,
avg(cast(
case
  when state='completed' then 1
  else 0
end
as float)
) as completed_ratio
from jobs
group by type


The way it works:

  1. The case statement sets of value of 1 for the success condition, and 0 for any other.
  2. These are integers, so we cast them as float, so we'll get the averaging behavior we want.
  3. We avg them, to get the average.
Similarly, if we wanted to calculate the ratio of jobs completed under a certain time threshold, we could do that in the case statement, such as this example which calculates the ratio of jobs whose end time is under 3 minutes:

select type,
avg(cast(
case
  when datediff(minute, completed, started) < 3 then 1
  else 0
end
as float)
) as completed_ratio
from jobs
group by type



Ruby Recursive Hash with SQL-like capabilities

I recently was working on a Ruby project where I had to query an SQL server, and the connection setup/query/response/close latency was about 5 seconds. I needed to generate many unique data slices for a reporting project, and the naive implementation generated some 6,000 database queries, which would take about eight hours to run.

I could fetch all the data in one query that had a large group by clause, which would return all the data I needed in 60 seconds instead of 8 hours.

But then I had a pile of data in rows. That's not very handy when you have to slice it by seven different characteristics.

So I wrote a recursive Ruby hash structure with semantics that allow me to retrieve by arbitrary attributes:

h=RecursiveHash.new([:business, :product_line, :format, :country, :state, :count], 0)

h.insert ['ConsumerBusiness', 'Product A', 'HTML', "UnitedStates", 1, 10]

h.retrieve { :product_line => 'ConsumerBusiness' }
h.retrieve { :product_line => 'ConsumerBusiness', :doc_format => 'HTML' }

The default mode is to sum the responses, but it can also return an array of matches.
It's really fast, and has a simple interface that, to me, is highly readable.

Here it is - RecursiveHash.rb:

class Array; def sum; inject( nil ) { |sum,x| sum ? sum+x : x }; end; end

class RecursiveHash
 def initialize(attributes, default_value, mode=:sum)
  @attribute_name = attributes[0]
  @child_attributes=attributes[1..-1]
  @default_value=default_value
  @master=Hash.new
  @mode=mode
 end
 
 def insert(values)
  if values.size > 2
   #puts "Inserting child hash at key #{values[0]}, child: #{values[1..-1].join(',')}"
   if @master[values[0]]==nil
    @master[values[0]]=RecursiveHash.new(@child_attributes, @default_value)
   end
   @master[values[0]].insert(values[1..-1])
  else
   puts "Inserting value at key #{values[0]}, value: #{values[1]}"
   @master[values[0]]=values[1]
  end
 end
 
 def return_val(obj, attributes)
  if obj.is_a? RecursiveHash
   return obj.retrieve(attributes)
  elsif obj==nil
   return @default_value
  else
   return obj
  end
 end
 
 def retrieve(attributes)
  if attributes[@attribute_name]==nil or attributes[@attribute_name]=='*' or attributes[@attribute_name].is_a? Array
   keys=nil
   if attributes[@attribute_name].is_a? Array
    keys=attributes[@attribute_name]
   else
    keys=@master.keys
   end
   
   v=keys.collect { |key| return_val(@master[key], attributes) }
   #puts "v: #{v.join(',')}"
   return @mode==:sum ? v.sum : v
  else
   return return_val(@master[attributes[@attribute_name]], attributes)
  end   
 end
 
 def pprint(n=0, parent_key="N/A")
  indent = "  " * n
  puts "#{indent}#{parent_key} (holds #{@attribute_name})"
  @master.each_key { |key| 
   if @master[key].is_a? RecursiveHash 
    @master[key].pprint(n+1, key)
   else
    puts "#{indent}    #{key}: #{@master[key]}"
   end
  }
 end
end

Square root of x divided by zero: The speed, size and dependability of programming languages

This may be the most interesting blog post I've read this year. It compares 33 programming languages in both speed of execution and expressiveness of the language. How does Python compare to Ruby? How fast is Java?

Square root of x divided by zero: The speed, size and dependability of programming languages


Database Anti-Patterns from PDX Ruby Tuesday

On Tuesday night, Xavier Shay (@xshay) gave a short talk on database anti-patterns.

Here are my rough notes:



  • STI for Shared Data
    • STI = single table inheritance. google “rails sti”.
    • this is sometimes called a "god table". It is easily found by having a single table that has many null columns, because different kinds of objects are being stored in a single table.
    • Example: a database for books with columns id, type, name, illustrator. for books, there is no illustrator, so you have a null field.
    • This gets complicated over the time. you have to loosen database constraints (you can’t enforce a value for illustrator), and logic is required to handle the null case, even for books. 
    • Using class table inheritence is one solution: books is one table, a second table is comics, and comics table has the illustrator field. In this case, instead of complicating the handling for books with a null illustrator field, we make comics, which needs the extra data, handling getting it from the table.
  • Not Deleting Data
    • People are afraid to delete data
      • in part, afraid of creating dead references, e.g. can’t use has_many
      • business needs to go back in past
    • It’s bad to try to use one database for both reporting and operational needs.
    • Example: a users database, in which records can marked as deleted.
      • then either user names can’t be unique, or then user names can’t be reused. either way, this then gets into extra coding and/or relaxing constraints.
      • plus, all your queries become “find users where state is not deleted”, so all queries become more complex and slower.
    • Solution: have two tables, and move the deleted users into the old_users database, which gives you your history.
  • Different data, same database
    • No notes here, sadly.
  • Not locking data
    • Example: it’s easy, if the user hits pay twice, that you could have a race condition here:
      • order = Order.find(id)
      • order.mark_as_paid!
    • Fix
      • Order.transaction do 
        • order = Order.find(id, :lock => true)
        • order.mark_as_paid!
      • end


Notes from CloudCamp PDX 2010

CloudCamp 2010
Portland, Oregon
#oscon #cloudcamp #pdx
CloudCamp is a free birds of a feather session at OSCON, the O’Reilly Open Source Conference. I came out of general interest, and because one of the promised tracks is deploying your own cloud using open source tools.
Promo: New user group: pdxdevops
Lightning Talks
open cloud
Sam Johnson
Google, Zurich
  • open source / open cloud: freedom. You can move from one cloud to another. avoids lock-in.
  • unfettered competition leads to commoditization leads to utility computing.
  • case study: free software 
    • open source is a happy medium between free software and proprietary software that leads to useful stuff, good for business.
    • open source is trademarked, giving it some instant recognizability and specific criteria for being open source
  • criteria for open cloud
    • open interfaces (atompub)
    • open formats (open document)
  • http://opencloudinitiative.org
Adrian Cole
Ops Code
@adrianfcole
5 APIs for Provisioning
  • Provisioning
    • Allows access to cheap resources
    • APIs -> automation
    • Tools exists
  • Manage Complexity
    • multi-cloud APIs
      • abstract what is commoditized
      • provide a consistent substrate
      • reduce complexity and lock-in
  • Dasein Cloud
    • Written by guy who did first JDBC
    • Focuses on services
  • Apache Deltacloud http://deltacloud.org
    • Ruby implementation
    • provides REST endpoint. Can use curl to manipulate the clouds.
  • fog
    • ruby cloud computing library
    • compute an storage across many providers (about 6)
  • jclouds
    • multi-cloud framework
    • zero lock-in to cloud apis
    • written in java
    • runs in google app engine
  • libcloud http://libcloud.org 
    • was a python library, with java coming soon
    • is about compute
    • works with 16 providers
The Simple Cloud API
Doug Tidwel
The Simple Cloud API brings cloud technologies to PHP and the PHPilosophy to the cloud, starting with common interfaces for three cloud application services: File Storage Services, Document Storage Services, Simple Queue Services.
  • Joint effort of Zend, GoGrid, IBM, MS, Nirvanix, and Rackspace.
    • But you can build libraries to support other clouds
  • Supports 3 areas:
    • File storage (s3, nirvanix, azure blob, rackspace)
    • Document storage (s3 doc, azure doc)
    • Simple queues (sqs, ...)
  • Uses Factory and Adapter design patterns
Eric
Principal consultant with Center Stance
Cloud Consultants: do implementations in the cloud
Not much of an open source person, more of a cloud person.
  • SalesForce.com, SAAS.
  • VisualForce is a templating language + Apex (java like) = to do addons for SalesForce.
  • App Exchange: app marketplace. 
  • managed and unmanaged packages.
    • managed packages are controlled, no code.
  • 940 packages in the app exchange.
  • less than 10% of those are open source: about 80 packages.
Cory 
Dyn, Inc.
DynDNS.org
  • Doing DynDNS for over 12 years. 3.5M people using it.
  • Dynect Platforms: hosts companies like twitter, 37signals, zappos.
  • Geotarget multiple clouds
    • Users in EU, go to Amazon EU, users in the Western USA go to GoGrid, users in the Eastern USA go to ...
    • Automatically redirect traffic to servers that are running (active failover)
  • DNS can give you a slider for your traffic: how much do you want to send to the cloud vs. your own servers? you can base it on latency, on location, on etc.
  • DNS resolution time is part of overall latency for users. DynDNS is faster (like 32 ms vs 120 ms in example.) that’s 90ms you’re getting back to be able to do more in your own server.
Unpanel
Hahahaha: They asked “who considers themselves an expert on the topic of open source and cloud computing?” Five people raised their hands. “OK, you’re the panel. Come on up.”
  • How is CC going to change the choice of the dev platform?
  • Is open source still relevant in cloud computing?
  • Will open source save us from a handful of monopolies?
  • What are the implications on hardware? What will change for hardware?
Stuart Smith, Rackspace: Is open source still relevant?
  • Only if you value freedom.
  • In fact, it is even more important.
  • When your proprietary software vendor goes out of business, you still have the software, you still have the license key.
  • When your proprietary cloud vendor goes out of business, your company is fucked.
Will open source save us from monopolies?
  • Just being free isn’t enough. There have been other free efforts that have been crushed by monopolies.
  • You have to have people adopt it.
  • The only way it is going to work is if everyone gets involved. otherwise cloud computing will be dominated by a few proprietary stacks.
How does this influence our choice of platforms?
  • With some platforms, like Google App Engine, you either drink the koolaid, or you don’t.
We’re going through this change between latency sensitive and bandwidth sensitive. Everything moving to data centers. highly multicore systems. now losing in the market place to classic out of order design. we’re going to see lots more cores, lots more latency sensitive. gpu assisted. more message passing hardware to avoid going through the OS.
Breakout Discussions
Why open? Open stack. Open cloud.
Open is:
  • creative commons license on the specs themselves. if the specs themselves are copyrighted, you can’t even tell your customers about them.
  • patents: you can’t have key technology locked up.
  • trademarks: when you start talking about “amazon compatibility”, you have problems. so the relevant names must be open for use.
  • implementations: you need to have multiple implementations.
  • open design / transparency / open process: so the community can participate, so i can understand the design, what is going on.
    • open process is hard: because standards bodies are in theory open, but they cost $12,000 to join, so it;’s not really open.
    • if it’s not open, then other people can’t innovate and move things forward. that’s limited to the standards setters.
  • then what are the options? a different standards body?
    • we had an unconference, and invited people to participate, and we were able to learn from each other and move things forward.
      • (this was on the format used for virtual machines)
  • Open cloud is:
    • open formats
    • open interfaces
    • open source
    • open data
  • “multiple, interoperable implementations, at least one of which is open source”
    • having an open source implementation does give you a real viable alternative. 
    • example: if there was an open format for microsoft office, and they said, well all you have to do is implement microsoft office yourself, then it isn’t really viable, unless there really is an existing open source implementation.
  • part of the core of open source is the right to fork.
    • if you don’t have the right to go, then you are married to the solution (e.g. whoever will buy MySQL)
    • this would include the right to fork a spec
      • let the best API float to the top.

Machine Learning and Data Mining in Ruby and R

My notes from the @pdxruby talk on 2010/04/06


Machine Learning and Data Mining
Randall Thomas
Engine Yard
  • Randall's Slides from Talk
  • netflix, amazon, google: recommending movies, books and music, links based on your personal experience
    • the future is about information...not data (how many gigabytes of data do you have sitting around?)
    • if it’s so cool, how come everyone isn’t doing it? it’s hard
  • world’s shortest stats course
    • two types of statistics
      • descriptive: the average height in this room is 5’ 6”
      • inferential: odds are, this horse is going to come in first. 
    • the two tasks
      • classification: you try to come up with a system for classification (cluster analysis, decision trees)
      • prediction: card counting, i predict that this deck is hot
      • or both: we want to both classify the data and draw inferences about new data
  • two types
    • supervised learning
    • unsupervised learning: the way a bayesian filter works... i have no idea what the inputs were, but i can look at the macro behavior, and then make predictions. this is also the way markov models work, the way spam filters work.
  • R
    • heavy-weight lifting tool for statistics
    • has shell for working in statistics
  • 5 numbers, one picture
    • pallas.telperion.info/ruby-stats
  • RSRuby
    • lets you eval R code
  • Computer friendly data descriptions
    • feature vector: simple 0 or 1 for each feature. beer, wine, whiskey, gin are the vectors. (1 if you like it, 0 if you don’t)
      • attempt bitwise and of vectors
  • Clustering...
    • Simple Geometric: just use the distance formula. If you have 2 dimensions, or 3 dimensions, there is a simple formula. that formula generalizes to N dimensions
    • R code: plot(sort(mydata$profits))
  • Not Simple Geometric Clustering
    • Support Vector Machines: create maximal separation of unseparatable data by projecting onto different planes.
    • You can seperate into two groups: one that is good, and one that is bad. one that are people attacking your IP ports, and one that isn’t. one that is spam, one that isn’t.
    • You can apply the SVM over and over again recursively... this turns into a decision tree.
  • Read: 
    • First: Introductory Statistics with R by Peter Dalgaard (2nd edition) - teaching you the basics in a tutorial fashion
    • Second: A Handbook of Statistical Analyses Using R by Brian S Everitt and Torsten Hothorn
      • load the free PDF in Rvignette(package = “HSAUR”)
    • The Elements of Statistical Learning by Hastie, Tibshirani, Friedman
      • www-stat.stanford.edu/~hastie/Papers/ESLII.pdf
  • Regression in R
  • Examples of companies doing this...
    • Collective Intellect: doing mining of memes

Kick Ass Mashups: Punk Rock APIs - notes from SXSWi presentation

Wow, I loved this presentation! Feel like a programming kung fu master... - Will

Revenge Of Kick-Ass Mash-Ups with Punk Rock APIs
Kent Brewster
@kentbrew
  • Notable Mash-Ups
    • Google Maps Mash-Up: first recorded AJAX mach-up, probably inspired most of the state of the modern art.
    • Flickr Blog badges
  • Punk Rock: DIY ethic
    • Other generative things
      • lego blocks, erectors sets, refrigerator boxes
      • original apple //e
    • is your site sterile?
      • users are cows, not customers
      • real customers are coke and GM
      • any unauthorized use is abuse
  • Your existing API
    • you already have an API: HTML
    • you’re already being screen-scraped
      • you know this
  • If you open up an API, you get pinpoint data about how it is being used.
    • Sterile APIs: HTML, RSS
    • Generative APIs: Free.
    • Punk Rock APIs: use generative APIs to turn sterile APIs into generative APIs.
  • Job interview at netflix: asked to review code. looking through real source code, he found that they had cribbed his own code. hired.
  • Netflix Bubble Widgets
    • single line javascript include
  • Pipes.yahoo.com
    • this is why yahoo is still relevant. they are doing amazing stuff like pipes.
  • Some very little javascript can do amazing things because it relies on Yahoo Pipes to do the heavy lifting.
  • YQL: yahoo query language. amazing tool.
    • select * from twitter.search where q=‘earthquake’;
    • This works because the community contributes tables (see community tables) that actually do the fetching/parsing of the data.
  • bit.ly/kb_twit 
  • bit.ly/kb_sxsw
    • used YQL, and a bit of xpath.
    • filtered results, nice presentation, runs fast.
  • Advice for Hackers
    • Go easy on the server. Since every request comes from a separate IP address, client-side mash-ups look like botnet attacks.
    • Respect robots.txt
      • Pipes and YQL respect robots.txt
    • Create and pass an application ID even if it’s not required. 
    • Let the site now what you’re doing. They might hire you. 
  • Advice for Site Owners
    • Build your API first. Build your site on your API, and then open it up to the community. Example: Flickr.
    • Whitelist Pipes and YQL: It’s the right thing to do.
      • They are giving you a free API caching mechanism
      • Twitter has done it. If you are running up against twitter API limits, try it.
  • How to open an API where you work
    • Build an interesting mash-up
    • Write the documentation for the API you wish you had.
    • Don’t write a spec. Write the actual docs.
    • Give it to the back-end guys.
  • To Be Useful for Client-Side Mash-Ups
    • Return Javascript
    • Wrap the requested JSON in the client’s preferred Javascript callback
  • To be useful for repeated calls... (some complicated stuff I didn’t get)
    • something having to do with square brackets
  • Every Javascript reply must have HTTP Status 200
    • If it comes back with anything else, the browser won’t see the response and the calling script will hang forever.
  • Demo the Last: Missing Kids CAPTCHA
  • Questions...
    • What if a call never returns?
      • You have to set a timeout. Probably requires a global variable. 
    • Examples of business mashups? Examples of doing it to correct a company’s bad UI?
      • People are more interesting to me... so not so aware.
      • Don’t surprise anyone in your IT group. If you should it to your boss, and they think it is awesome, you’ve really stuck the IT group in a corner.
    • If you’re a company, and you’ve never done this before, go talk to Mashery, or other companies like that.

Raw Notes from Coding For Pleasure at SXSWi 2010

Building Apps in Your Spare Time
#codingforpleasure
  • Gina Trapani
    • write stuff mainly to procastinate writing
    • Firefox scripts to improve gmail (better gmail 2 0.9.8.1)
    • ThinkTank - ask your friends
  • Matt Haughey
    • Side projects
      • Wrote fuelly: social miles per gallon.
      • MetaFilter (1999), written when blogs were still new
  • Adam Pash
    • MixTape: playlists shared with friends
    • Belvedere 
    • Texter: shorthand for your computer. Like textexpander for the mac.
  • Why should I develop an app in my spare time?
    • Just built a tool for ourselves (and 25,000 other users).
    • Just wanted something as clean as possible. Not an overbearing UI like slashdot.
    • Fill a need... Gmail
    • Want an archive of tweets.
    • Very important to scratch your own itch
    • Ego motivation... opportunity to get users right away, get feedback
    • You can build anything... that is really exciting.
      • Pash: I am not a programmer by trade, and I am not a great programmer, but I can still make anything.
      • Trapani: it’s amazing what you can do now between APIs and the languages available
    • Don’t expect to make money. Metafilter was a success, but it took 6 years before they made money. There can be a huge slog. If your motivation is only money, you’ll shutter the project. If you build an app you use every day, then at least you can still use it every day.
    • “The internet is so ready to give you an answer to any problem” -- Pash
    • You can work on stuff that will further your career
    • If you don’t have an idea you are excited about, then you aren’t going to make it happen.
  • All the beloved things... twitter, flickr... they didn’t start as a plan to make a lot of money.
  • How can I do it?
    • You have to dedicate time.
    • If you are really excited about it, you can find the time.
    • The first thing to go for most people is the television. Two hours of veg time at the end of the day is the easiest thing to go.
    • It can be a relaxing time... just enjoy it, watch TV, plan to put a year into it.
    • Use frameworks... don’t reinvent the wheel. Rapidly prototype. Google what you need to do, and copy and paste code. Use libraries and plugins that exist, there are plugins for everything.
    • Collaboration is a big deal
      • it’s so much more fun to work with someone
      • it’s so helpful to bounce ideas off something
    • You really don’t need to be a coder or to hire someone to start. You can go from zero to competent in just about any language about six months. 
    • Dan Bricklan, inventor of the spreadsheet (will: about a billion years ago), was like “iphone development, this sounds interesting“, and went out and bought an MacBook, an iPhone development book, and wrote an app, and put it in the store for $3
    • Did you ever pay anyone?
      • Yeah, I don’t really have the skills or competency anymore in design, so I hire some designers. Same for CSS... I don’t have the skills any more to make this work in dozens of browsers. I sent to it to some kids in (the middle of nowhere), and paid them $100.
      • I’ve never hired anyone because I’m cheap, but I barter with people. “I’ll build something for you if you design something for me.”
    • Open source
      • Trapani: everything I’ve done is open source. At lifehacker, we have this big community of people doing open source. Why not use those resources?
        • There is nothing more awesome than waking up to check your email and finding a code contribution.
      • But you can’t rely on that. It’s a big commitment for someone to get the code, work on it, and submit a change.
  • Pitching your idea to the company... to sponsor them
    • You’ve got to make the case for why to do
    • Google’s 20% time is a good example to cite
    • Or it may be synergistic: e.g. for lifehacker it raises their credibility for their employees to be doing open source
  • Questions...
    • Talk about ownership when you are working at a company
      • Check your company’s policy before hand. Some have weird policies like even what you do on your own time is owned by the company.
      • If you can convince your company to open source it, then it isn’t an issue at all.
    • I am a developer, and I like to build super-visualize things, but I am not a designer. How can i find someone to work with?
      • There are some sites to help. But that is kind of a crapshoot.
      • you network a lot. 
      • Go to an ignite in Portland. 
      • Look up the portfolio of designers you meet.
      • Don’t go to rubycon to find a designer.
      • Go to social events or design events.
    • Talk about programming where you might not want to open source the code. Talk about some successful examples of that.
      • I had security issues - a giant login system with crappy code. I wanted to keep that code secret.
      • One motivation to make your code good is to open source it.
      • But if you can’t do open source... then you have to hire programmers, or find one fan of your work to work with. and still keep it closed.
    • What about liability...worried about being sued.
      • I made a music sharing site that uses mp3s shared on servers around the country. So I made an LLC, and now MixTape belongs to that LLC. 
      • Having a terms of service can help. Lawyers can help you do terms of service and LLC for less than $1k. 
      • Or copy and paste from Google or someone else. Something is better than nothing.
    • Tradeoff with APIs... you are at the mercy of the service. You get a lot, but then the service could go away.
    • How do you get users? I’m the sole user of like a half a dozen apps.
      • It’s not easy. Integrate them into whatever you do. For fuelly, we made badges people could put on their blogs. Talk about it on twitter.
      • Talking to developers about things you made. No one want talks to a PR person. We want to talk to developers.
    • As a designer, I want to learn programming. Where should I go?
      • Google is great. 
    • I’m not hearing why the stuff you make is as awesome as it is. What are the decisions you can make, what are the freedoms you have, that you don’t have to make money off it
      • You are the user. You are the designer. You can make the application what you want it to be. It can be very satisfying.  
    • At what point do you reach break even on the server costs?
      • I’m spending $100/month for the server, and using AdSense will cover the costs. 
      • You can do “donate a dollar” via paypal, but that is sporadic.
      • It’s weird to do a project where covering the hosting cost is considered a success.
      • Amazon referrals, ads, are a passive way to do it.
    • Share a couple of websites that would be good resources
      • prototype
      • jquery
      • open languages have great documentation... documentation plus comments is amazing.
      • free git book online
      • stackoverflow
      • peepcode
      • just google your programming question