Skyteam Elite Plus boarding group 5

Recently had an international flight from Amsterdam. My boarding pass correctly displayed my Skyteam Elite Plus and Sky Priority on the boarding pass but also showed boarding group 5. When I got on the boarding group 2 (Sky Priority) lane, the agent said I couldn't board until group 5. I showed him both Skyteam Elite Plus and Sky Priority designations but he would meet me through. I ended up not being able to get my carry on in a compartment near me due to being the last boarding group. Thoughts?