I recall we passed by this issue in SCCM 2007 and we had to add the Valuelimit registry key and looks like its not fixed in SCCM 2012 and you need to do the same, however due to the change in the OS/SCCM version, the key location is changed, to fix this issue you need to apply the following:
- Open the Registry (using Regedit) on the SCCM server
- Explore HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\ConfigMgr10\AdminUI\QueryProcessors
- Add a new DWORD value named "Valuelimit" with Hexadecimal value of 10000 (The maximum value is 20000, its recommended to start up with 10000 and check if the problem is fixed instead of just entering the maximum number which can eat your memory). Check this article for more details http://support.microsoft.com/kb/269201
- Stop the SMS_EXECUTIVE & SMS_SITE_COMPONENT_MANAGER services in this order
- Start both services in the same order.
Note: If you managing the SCCM from a console on a terminal computer (most of the cases) and not directly from the SCCM server then you need to apply these settings on your console computer and reboot the computer after adding the registry key.