Moving garbage collection (GC) provides opportunities to rearrange object placement to reduce the working set and thereby enhance cache locality. Hot-Cold Objects Segregation GC (HCSGC) identifies objects whose reference are loaded through load reference barrier in ZGC as hot objects and segregates them from the other objects. If such hot objects are repeatedly accessed after segregation, the effective working set can shrink. However, we found that a substantial fraction of hot objects were not accessed again after experiencing a few, or even no, accesses immediately after the segregation. This paper presents load-site based filtering (LSBF) to further reduce the working set. Our key observation is that certain load sites tend to load references to persistently hot objects, while others tend to load references to transiently hot objects. We implemented LSBF in HCSGC, where we classified load sites using offline profiling. Our evaluation showed that LSBF reduced the working set for our synthetic benchmark where HCSGC failed to reduce the working set. For the DaCapo benchmarks, we did not observe improvements by LSBF mainly because these programs were small enough that the working set fit within the cache capacity regardless of object arrangements. Nevertheless, we confirmed that the overhead of LSBF was limited.