Drop in CPU

    Newer Older

    Webserver CPU...anyone guess what the drop is from? (Flickr Staff, don't answer :) )

    Another hint here.

    ANSWER IS: php5. :)

    cdamian, kktsvetkov, lhl, dxjones pix, and 1 other people added this photo to their favorites.

    View 9 more comments

    1. precipice 73 months ago | reply

      Hmm. The spike is a little lower than the next one, though. Looks like the bars are five minute intervals. Maybe a restart that took less than about 5-6 seconds could happen there (assuming the first lowered bar took all of the downtime).

    2. • glub • 73 months ago | reply

      Reduced traffic led to a release of RAM to the OS, which led to an increase in disk caching, which led to lower CPU?

    3. jspaw 73 months ago | reply

      Again: not traffic related. :) Apache hits would have dropped proportionally. I'll give the answer later on today... :)

    4. dathan 73 months ago | reply

      php5 and or more servers on the front end and or more resources on the backend to reduce front-end wait-io

    5. ohesscue 73 months ago | reply

      My guess is that it's a new release of the Flickr codebase. From the graph, it looks like the drop-off at 8p is roughly proportional to an increase that begins around 3p. From the apache hits graph, the requests were roughly constant over the period shown. So, that sounds to me like code was pushed around 3 that was, perhaps, less efficient than it could have been -- and that a fix was pushed around 8p.

      Hedging my bets, there was also a new stable release of php5 about a month ago -- perhaps it contained some performance improvements...

      -sq

    6. getluky 73 months ago | reply

      Hard to tell without knowing whether this is from one specific box, an average of all boxes worldwide, an average of boxes in a colo, etc. It would be easier to tell if it was the result of a load distribution change, a misconfiguration, or a code change. :)

    7. rcrowley 73 months ago | reply

      This looks like the graph for a single host, if I remember correctly. They push code like every 9 seconds so I doubt a bad change would last that long in production.

      I like the PHP5 theory but I don't want to go slogging through changelogs to prove/disprove it.

      Did you clean up init_config? :)

    8. treebjen 73 months ago | reply

      Ugh -- I keep clicking "here" and it just goes HERE.

    9. rcrowley 73 months ago | reply

      9:15am is early for Flickr, isn't it?

    10. getluky 73 months ago | reply

      Hmm, this is a longshot, but it looks like there are a couple more pct points of cpu_wait directly following the drop. Perhaps developers added caching for something that was slightly cpu intensive, and the drop indicates the point at which the caching change was deployed. Then the webserver may have gone into cpu_wait while the i/o to store/fetch the new cached copy might have occurred.

      A deployment is probably also consistent with the steep one-pixel-width drop in the other hint - while the codebase is being deployed, i'm betting that you guys rotate the webserver off the vip temporarily during the code switch.

      If this was consistent across all webservers, and the pattern followed the type of peak cpu shown before the drop, then that would look like a more solid case. But it'd be easier just to ask the developers. :)

    11. jspaw 73 months ago | reply

      ANSWER IS: php5. :)

    12. getluky 73 months ago | reply

      Aww, I feel like a tool now. :p That was fun though, can you make it a regular game?

    13. precipice 73 months ago | reply

      Regular game! Regular tool! Uh, I mean...

      And yeah, @rcrowley++. 9:15a upgrades? Go to sleep! ;)

    14. dathan 73 months ago | reply

      You can significantly reduce your memory footprint if Flickr started to use Objects, but that would make Cal cry.

    15. jspaw 73 months ago | reply

      dathan: if memory was an issue, we'll consider it. :)

      thanks all for playing! :)

    16. smin 73 months ago | reply

      What do I win? :)

    17. shanan 73 months ago | reply

      When can we look forward to php6?

    keyboard shortcuts: previous photo next photo L view in light box F favorite < scroll film strip left > scroll film strip right ? show all shortcuts