文法沒問題啊？牛津字典：

belong to somebody = be the property of somebody

belong with somebody/something = have a proper or usual place, as specified (=the things should be together)

用 belong with me 比較恰當沒錯。