Why Do Go pprof and Docker Stats Show Different Memory Usage Metrics?

Why "Memory Used" Metric Differs Between Go pprof and Docker Stats

Context

In a Go application running on Docker containers, the docker stats command reports a linearly increasing memory usage. However, using HTTP pprof to monitor the running application, the runtime.MemStats.sys value remains constant. This discrepancy raises questions about potential memory leaks.

Cgroups and Memory Metrics

Docker stats retrieve memory usage statistics from cgroups, which optimize memory access to avoid cacheline false sharing. Consequently, the usage_in_bytes metric in cgroups includes both the Page Cache and RES, which account for File I/O within the container.

Memory Reclamation

When a container's memory usage reaches its maximum limit, it reclaims unused memory rather than terminating processes. This explains why pprof reports a constant runtime.MemStats.sys value, while docker stats shows a linearly increasing memory usage.

Verification

To confirm this behavior, add a memory limit to the container using either the docker run command or a docker-compose.yml file. Observe that when the limit is reached, the container will reclaim memory and maintain its usage below the limit, as indicated in cgroups at /sys/fs/cgroup/memory/docker//.

Conclusion

The difference in memory usage reported by pprof and docker stats arises from the inclusion of File I/O memory in docker stats and the memory reclamation机制 undertaken by cgroups when memory limits are imposed. By understanding these factors, developers can gain a more accurate picture of their container's memory consumption.

The above is the detailed content of Why Do Go pprof and Docker Stats Show Different Memory Usage Metrics?. For more information, please follow other related articles on the PHP Chinese website!