Results 1 to 5 of 5

Thread: Windows Azure Linux Agent - High memory usage

  1. #1

    Default Windows Azure Linux Agent - High memory usage

    Hi,

    Is anyone else seeing high memory usage for the waagent service (provided by the python-azure-agent package) hosted on azure? On a machine with 4GB RAM, the process `python -u /usr/sbin/waagent -run-exthandlers` starts with around 20MB and in a couple of months goes up to 1.5GB. I have tried installing the package directly from Azure's github repo https://github.com/Azure/WALinuxAgent but that has the same issue as well. We also use Ubuntu instances on Azure and aren't seeing any issue on them.

    Thanks,
    Nishant.

  2. #2
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,278
    Blog Entries
    2

    Default Re: Windows Azure Linux Agent - High memory usage

    I don't deploy any Linux on Azure,

    But after skimming the waagent README.md,

    - I don't know if it's a typo in the documentation but one of the listed dependencies is something called "ip-route"
    On openSUSE, we have "ip route" as part of the ip tools package, but no "ip-route"

    - Have you tried to determine what is in your memory? Have you run top (or I'd recommend htop)?

    - Have you inspected and compared your logfiles?

    - What makes you think the memory issue (sounds like a memory leak when resources are not free) is related to the system and not the running application (generally far more likely)? In fact, what kind of app is running on your system and do you know enough about how that app was written to evaluate it for resource usage and possible memory leaks? For that matter, although not a definitive test, you could try deploying a copy in your own LAN, run some load testing and see if you can replicate the memory issue. It's not exactly the same as the Azure cloud, but if you know how to test, a lot can be learned running in a controlled environment.

    - Have you inspected your waagent logfile? Maybe you'll find something, maybe not... Or you might find something perhaps not definitive but possibly contributory... After all, waagent is supposed to collect and send performance metrics.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  3. #3

    Default Re: Windows Azure Linux Agent - High memory usage

    Quote Originally Posted by tsu2 View Post
    - Have you tried to determine what is in your memory? Have you run top (or I'd recommend htop)?
    top shows the process 'python' running with the high usage. ps shows the process to be 'python -u /usr/sbin/waagent -run-exthandlers'.
    - Have you inspected and compared your logfiles?
    There is not much in the waagent log files and no observable difference between Ubuntu and openSUSE(after removing the single preinstalled extension).
    - What makes you think the memory issue (sounds like a memory leak when resources are not free) is related to the system and not the running application (generally far more likely)? In fact, what kind of app is running on your system and do you know enough about how that app was written to evaluate it for resource usage and possible memory leaks? For that matter, although not a definitive test, you could try deploying a copy in your own LAN, run some load testing and see if you can replicate the memory issue. It's not exactly the same as the Azure cloud, but if you know how to test, a lot can be learned running in a controlled environment.
    I agree that the problem is likely to be with the application and not the OS. We have 12 instances running openSUSE, all having the same issue and yet our Microsoft Support Engineer claims that he can't reproduce the issue! Our instances are running different applications (mongodb/redis/tomcat/nodejs) and don't have any packages in common other than the base system.
    We don't have a local machine that's up all the time to test the memory leak.

  4. #4
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,278
    Blog Entries
    2

    Default Re: Windows Azure Linux Agent - High memory usage

    Recommend submitting a bug to the waagent project

    https://github.com/Azure/WALinuxAgent/issues

    I'd expect whoever takes an interest in your issue will want to be able to login to your running instance, perhaps instrument your code.

    Also,
    As you might expect, if you could replicate your issue on a local machine, you might have more options for troubleshooting.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  5. #5

    Default Re: Windows Azure Linux Agent - High memory usage

    Our support engineer finally accepted that the bug exists. For anyone else following this thread, here is the link to the issue on their github repo.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •