Google sometimes ignore it when it makes sense (ie big bank accidentally adds login page to ignore) or to check for spam activity (in which case google doesn't use their bot user agent)