xfs_quota: Failed with Secondary Group

Unfortunately xfs_quota is only valid for Primary Group. It wont work for Secondary Group (in your case group: itshare).

You need Project Quota as workaround.

1. Mount your disk/partition with prjquota option:

    # mount -o rw,remount,prjquota /home
    # mount | grep home
   

   Make sure prjquota option is displayed.

2. Create itshare dir:

    # mkdir /home/itshare
    # chown -R testuser:itshare /home/itshare
    # chmod -R g+rwx /home/itshare
   

3. Create Project ID (can be any numerical value)

    # echo "11:/home/itshare" >> /etc/projects
    # echo "itshare:11" >> /etc/projid
   

4. Initialize project directory

    # xfs_quota -x -c 'project -s itshare' /home
   

5. Configure quotas for projects with initialized directories:

    # xfs_quota -x -c 'limit -p bsoft=3m bhard=4m itshare' /home
   

6. Test

Reference:

[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_file_systems/assembly_limiting-storage-space-usage-on-xfs-with-quotas_managing-file-systems
[2] https://stackoverflow.com/questions/40244862/xfs-quota-issue-for-group/76857835#76857835